andygrove opened a new issue, #4993:
URL: https://github.com/apache/datafusion-comet/issues/4993
## Describe the bug
`prepare_object_store_with_configs` in
`native/core/src/parquet/parquet_support.rs` caches object store instances
under `(url_key, config_hash)`, where `url_key` is built from:
```rust
let url_key = format!(
"{}://{}",
scheme,
&url[url::Position::BeforeHost..url::Position::AfterPort],
);
```
`Position::BeforeHost` starts *after* the URL userinfo. For ABFS, the
container is the userinfo, not the host:
```
abfss://[email protected]/path/to/file.parquet
```
So `abfss://[email protected]/...` and
`abfss://[email protected]/...` both produce the key
`abfss://acct.dfs.core.windows.net`. `config_hash` is computed over the
per-session Hadoop config map, which does not vary by container, so it does not
disambiguate them either.
Meanwhile `MicrosoftAzureBuilder::parse_url` bakes the container into the
store instance from the URL userinfo, and the `Path` handed to that store is
container-relative (the bucket/container is stripped by
`Path::from_url_path(url.path())` in `PhysicalPlanner::get_partitioned_files`).
The consequence: within one executor process, reading
`abfss://c2@acct/data/f.parquet` after `abfss://c1@acct/data/f.parquet` should
hit the cached `c1` store and read **c1's data**.
## Steps to reproduce
Not yet reproduced. This was found by code reading during review of an
unrelated change. Confirming it needs an Azure storage account with two
containers holding same-named paths, read from a single executor in one
application.
## Expected behavior
Each ABFS container resolves to its own object store instance, and a read
from one container never returns another container's data.
## Additional context
- S3, GCS, and HDFS are unaffected: their bucket/host lives in the URL host
component, which `url_key` does keep.
- The same key derivation is now also used to key a shared Parquet metadata
cache, where the impact is narrower (entries are additionally validated against
file size and modification time). That is documented as a known limitation
there; this issue is about the store cache itself, where no such validation
exists.
- The object store cache is process-wide and unbounded by design, so a wrong
entry persists for the executor's lifetime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
