dwsmith1983 commented on code in PR #6838:
URL: https://github.com/apache/datafusion-comet/pull/6838#discussion_r4261454278
##########
spark/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -1286,13 +1286,17 @@ object CometConf extends ShimCometConf {
val COMET_LIBHDFS_SCHEMES_KEY = "fs.comet.libhdfs.schemes"
+ // Declared so the value appears in the generated config reference and can
be set via
+ // `spark.hadoop.*`, but never read through this entry:
`NativeConfig.resolveLibhdfsSchemes`
Review Comment:
Moved it over in 335113f87. `declineReason` now reads the list with
`NativeConfig.resolveLibhdfsSchemes(hadoopConf)`, the same conf it keeps in the
memo and `CometDeltaNativeScan` hands to native, so `libhdfsSchemes` and its
import are gone. The comment here now names the scan gates as the readers.
The parse test in `DeltaScanContribSuite` is replaced by three that plan a
`fake://` read of a Delta table: the bare Hadoop key and a reader option each
get the native Delta scan, and a runtime `SET` of the `spark.hadoop.` key falls
back with the scheme reason. With the gate reading the SQLConf entry all three
fail, and the reader-option one also fails if the gate drops the relation
options. The Delta suites pass on 3.5 and 4.1.
Agreed on main only; I took the backport line out of the description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
