rahil-c commented on issue #19740: URL: https://github.com/apache/hudi/issues/19740#issuecomment-5419687692
Root-caused and fixed in #19575. This was not specific to Spark SQL. The executor metric registry is keyed by a digest of the base path, and one table is spelled more than one way: Spark SQL builds its write config from the catalog location, which is scheme-qualified, while the Spark DataSource passes the bare path. Digesting the raw string gives one logical table two different keys, so the executors register their counters under one and the commit-boundary drain looks under the other, finds nothing, and reports nothing. Instrumenting the drain shows both keys for the same table and temp directory in a single run: ``` basePath=/var/.../dataset key=hoodie_test::HoodieRecordIndexLookup.6b46c2c17e0a found=false basePath=file:///var/.../dataset key=hoodie_test::HoodieRecordIndexLookup.d7af07eb384d found=false ``` Both digests reproduce exactly as SHA-256 of the respective raw strings, truncated to 48 bits. The fix digests the authority and path rather than the raw string, so `file:///data/t` and `/data/t` agree while `s3://a/t` and `s3://b/t` stay distinct -- dropping the authority outright would collide two buckets sharing a key path. The test class attached above is restored in the PR and passes: 6/6 across the global and partitioned record index, covering the non-prepped UPDATE and MERGE INTO cases that previously reported nothing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
