dwsmith1983 opened a new issue, #6837:
URL: https://github.com/apache/datafusion-comet/issues/6837
### Describe the bug
`CometScanRule` and native Parquet scans read `fs.comet.libhdfs.schemes`
from different places, so the rule can decide a scheme is or is not served by
libhdfs differently from native.
- The rule reads `COMET_LIBHDFS_SCHEMES`, the SQLConf entry
`spark.hadoop.fs.comet.libhdfs.schemes`, defaulting to `hdfs`. It uses that
list to classify the scan's root paths, for the "Unsupported filesystem
schemes" gate, and to skip the object_store path probe for libhdfs paths.
- Native gets the list from the Hadoop conf: `NativeConfig` forwards
`hadoopConf.get("fs.comet.libhdfs.schemes")` to the scan's object store
options, and `is_hdfs_scheme` (`parquet_support.rs`) routes a scheme through
libhdfs only when it is in that option, or is `hdfs` when the option is absent.
- `fs.comet.s3Compliant.schemes` is already read from the Hadoop conf in the
same method, so `core-site.xml` is honored for it but not for the libhdfs list.
The two disagree in both directions:
1. Set only in `core-site.xml` (or another Hadoop-only source), e.g.
`fs.comet.libhdfs.schemes=hdfs,webhdfs`. Native would read `webhdfs://` through
libhdfs, but the rule does not see the list, so the scan falls back to Spark
with "Unsupported filesystem schemes: webhdfs".
2. Set at runtime with `SET
spark.hadoop.fs.comet.libhdfs.schemes=hdfs,webhdfs`. Spark's
`SessionState.newHadoopConf` copies SQLConf keys as they are, without dropping
the `spark.hadoop.` prefix, so the Hadoop conf has no
`fs.comet.libhdfs.schemes`. The rule admits the scan, native does not route
`webhdfs` through libhdfs, and the scan fails at execution when object_store
rejects the URL ("Unable to recognise URL").
### Steps to reproduce
1. Put `fs.comet.libhdfs.schemes=hdfs,webhdfs` in `core-site.xml`, enable
the native Parquet scan, and read Parquet files from a `webhdfs://` path. The
scan falls back with "Unsupported filesystem schemes: webhdfs".
2. Without it in `core-site.xml`, run `SET
spark.hadoop.fs.comet.libhdfs.schemes=hdfs,webhdfs` in the session and read the
same path. Comet claims the scan, and the query fails when native opens the
file.
### Expected behavior
The rule uses the same libhdfs list native will use: read it from the scan's
Hadoop conf (`newHadoopConfWithOptions(relation.options)`), as
`fs.comet.s3Compliant.schemes` already is. Then case 1 runs natively and case 2
falls back instead of failing.
### Additional context
A list set in the Spark conf at startup
(`spark.hadoop.fs.comet.libhdfs.schemes`) reaches both places, so the common
setup is not affected.
### Willingness to contribute
I can contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
