andygrove opened a new issue, #5816:
URL: https://github.com/apache/datafusion-comet/issues/5816
## Describe the bug
`fs.comet.libhdfs.schemes` is compared against the scheme *after* the
s3-alias rewrite, so listing `s3` also routes `s3a://` reads through libhdfs.
Before #5314 the two were independent: the libhdfs decision was taken from the
URL the user typed, and `s3a` was normalized to `s3` only when it was not
libhdfs-routed.
`prepare_object_store_with_configs` in
`native/core/src/parquet/parquet_support.rs` now reads:
```rust
let url = normalize_object_store_url(url.as_str(), object_store_configs)?;
let is_hdfs_scheme = is_hdfs_scheme(&url, object_store_configs);
```
`normalize_object_store_url` guards its own early return with
`is_hdfs_scheme`, so an `s3a://` URL whose scheme is not in the list falls
through to `rewrite_alias_to_s3` and comes back as `s3://`. The classification
on the next line then sees `s3`, matches the list, and selects the Hadoop
backend.
Before #5314 the order was the other way round:
```rust
let is_hdfs_scheme = is_hdfs_scheme(&url, object_store_configs);
let mut scheme = url.scheme();
if !is_hdfs_scheme && scheme == "s3a" { scheme = "s3";
url.set_scheme("s3")?; }
```
## Steps to reproduce
With `fs.comet.libhdfs.schemes=s3` set and no `s3a` entry, read
`s3a://bucket/file.parquet`. The scan is served by `create_hdfs_object_store`
instead of the native S3 store.
Adding this to `native/core/src/parquet/objectstore/s3_blob_fs_support.rs`
on `424c31aa7` fails:
```rust
#[test]
fn s3a_is_not_libhdfs_routed_when_only_s3_is_listed() {
let configs = HashMap::from([(
"fs.comet.libhdfs.schemes".to_string(),
"s3".to_string(),
)]);
let typed = Url::parse("s3a://bucket/f.parquet").unwrap();
assert!(!is_hdfs_scheme(&typed, &configs));
let normalized = normalize_object_store_url("s3a://bucket/f.parquet",
&configs).unwrap();
assert!(!is_hdfs_scheme(&normalized, &configs));
}
```
Output:
```
PROBE typed scheme = s3a
PROBE typed is_hdfs = false
PROBE normalized scheme = s3
PROBE final is_hdfs = true
```
The converse case is fine: with `fs.comet.libhdfs.schemes=s3a`,
`normalize_object_store_url` returns early and the scheme keeps its spelling,
so `s3a` stays libhdfs-routed as asked.
## Expected behavior
The libhdfs decision should be taken once, from the scheme the user wrote,
and carried through the alias rewrite rather than being recomputed from the
rewritten URL.
## Additional context
Found while reviewing #5503, whose
`isolates_backends_even_when_s3_alias_and_configs_match` test covers exactly
this pair (`s3a` native versus `s3` libhdfs under one config) and fails in CI
on the merge with `main` while passing on the branch alone. That test is a good
regression guard for this fix.
Introduced by #5314. cc @comphead @sunchao
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
