dwsmith1983 opened a new pull request, #6838:
URL: https://github.com/apache/datafusion-comet/pull/6838
## Which issue does this PR close?
Closes #6837.
## Rationale for this change
`CometScanRule` read `fs.comet.libhdfs.schemes` from the SQLConf entry
`spark.hadoop.fs.comet.libhdfs.schemes`, while native routes a scheme through
libhdfs from the Hadoop conf the scan forwards. Spark's
`SessionState.newHadoopConf` copies SQLConf keys as they are, without dropping
the `spark.hadoop.` prefix, so the two could disagree. A list set only in
`core-site.xml` made the rule fall back on a scheme native could read, and a
runtime `SET spark.hadoop.fs.comet.libhdfs.schemes` made it claim a scan native
then failed to open ("Unable to recognise URL").
## What changes are included in this PR?
- `CometScanRule` takes the libhdfs list from the scan's Hadoop conf
(`newHadoopConfWithOptions(relation.options)`, the conf the scan serializes for
native) through a new `NativeConfig.resolveLibhdfsSchemes`, which defaults to
`hdfs` when the key is unset or blank, as native `is_hdfs_scheme` does.
`fs.comet.s3Compliant.schemes` was already read this way.
- The `spark.hadoop.fs.comet.libhdfs.schemes` doc says it is read from the
Hadoop configuration and must be set before the SparkSession is created, as the
s3Compliant entry already says.
What users see, only where the two sources disagreed (query results do not
change, only native or fallback):
- A list set only in the Hadoop conf (`core-site.xml`, a reader or table
option, or the bare key `fs.comet.libhdfs.schemes`) now gets the scan run
natively instead of falling back.
- A runtime `SET spark.hadoop.fs.comet.libhdfs.schemes=...` no longer
changes what the planner claims. A scheme it added now falls back instead of
failing at execution.
- A runtime `SET` of that key that leaves out `hdfs` no longer makes
`hdfs://` scans fall back, since native never saw it. Set the list at startup
to change it.
The list set in the Spark conf at startup reaches both places and behaves as
before. `branch-1.1` has the same code, so this is a backport candidate.
`resolveLibhdfsSchemes` and its unit test are the same as in #6753, so
whichever of the two lands second resolves only a small conflict.
## How are these changes tested?
`CometScanSchemeFallbackSuite`, applying `CometScanRule` to the Spark plan
of a `fake://` scan:
- With the list as a reader option, and as the bare Hadoop key, the rule
claims the scan. Both fail on `main`, which declines it with "Unsupported
filesystem schemes: fake".
- With the list set at runtime under the `spark.hadoop.` key, the rule
declines the scan with that reason. On `main` it claims it.
`NativeConfigSuite` covers the resolver: unset, blank, and a list with
spaces and mixed case. Reverting the rule change fails all three rule tests.
CometScanSchemeFallbackSuite and NativeConfigSuite pass, and the build compiles
on the Spark 3.4 profile and with strict warnings on 3.5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
