sunchao opened a new pull request, #5503: URL: https://github.com/apache/datafusion-comet/pull/5503
## Which issue does this PR close? Closes #5502. ## Rationale for this change A configuration can route `s3` through Hadoop while leaving `s3a` on the native backend. Normalizing native `s3a` to `s3` currently gives those different backends the same process-cache key. Separately, DataFusion's registry identifies stores by scheme and authority, so registering a second configuration can overwrite the first mapping. This extracts the object-store correctness change from #5453. It contains no new scan counters or producer-lifecycle changes and does not depend on that metrics PR. ## What changes are included in this PR? Include backend identity in the cache key and register each non-local backend/configuration under a deterministic internal URL. The first registration receives the same identity it would receive after another store, so planning order cannot affect routing. Native `s3` and `s3a` share canonical `s3` identity; Hadoop-selected schemes retain their physical spelling. Native local files keep `file://` behavior. Reconstruct the physical URI before configuring encryption lookup, stripping only a complete internal identity suffix. This preserves existing `s3`/`s3a` key normalization and custom Hadoop schemes. The change isolates object-store lookup, not file-metadata caches. ## How are these changes tested? The focused tests use two distinct in-memory stores and verify actual returned bytes after registration in both orders. Cases cover a normalized S3 alias/backend collision, different native configurations, custom Hadoop routing, native S3 alias reuse, and native versus Hadoop-routed file URLs. Encryption-option tests compare ordinary and isolated physical URIs, including aliases, ports, custom schemes, and local files. The full native crate compiled with the two disjoint extracted changes combined at their shared base. All **90 Parquet tests passed**, including the six new isolation/encryption regressions, with a confirmed zero process exit status. The four companion producer-lifecycle tests also passed. `cargo fmt --all --check` and `git diff --check` pass. Default native features and JDK 21 were used; tests use deterministic in-memory stores rather than a live Hadoop/cloud deployment. Full Spark integration and this branch's hosted CI have not run locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
