dwsmith1983 commented on code in PR #6753:
URL: https://github.com/apache/datafusion-comet/pull/6753#discussion_r4214914742
##########
spark/src/main/scala/org/apache/comet/serde/operator/CometNativeScan.scala:
##########
@@ -240,6 +240,22 @@ object CometNativeScan extends
CometOperatorSerde[CometScanExec] with CometTypeS
scan: CometScanExec,
builder: Operator.Builder,
childOp: OperatorOuterClass.Operator*):
Option[OperatorOuterClass.Operator] = {
+ val hadoopConf =
+
scan.relation.sparkSession.sessionState.newHadoopConfWithOptions(scan.relation.options)
+ // The root paths can miss a file, e.g. a catalog partition located
outside the table, so
+ // check the listed files. The static partitions are a superset of what
DPP keeps.
+ val multiStoreReason = CometScanUtils.multiStoreFallbackReason(
Review Comment:
> Could we forward the options of every scheme the scan reads, and keep the
fallback only for an alias next to `s3` or `s3a` in the same bucket, where the
settings really do collide?
Done in e909c077b. Both scans now forward the union of
`extractObjectStoreOptions` over one path per scheme, and per bucket for an
S3-compatible alias, with each alias bucket's translated settings applied last
so they win over a raw `fs.s3a.bucket.<b>.*` key. The fallback is now keyed on
the store: an alias next to `s3`, `s3a` or another alias on the same bucket, an
alias path with no bucket next to other S3 paths, and bucketed scans over
several stores. So a Hive table with partitions on HDFS and S3 stays native
again. Since alias settings are now translated per bucket, it also drops the
older fallback for alias paths over several buckets. Native hashes only the
settings a store reads (dcc61a56d), so a bucket keeps one cached store whatever
else the scan reads.
While checking this I found that `CometCsvNativeScanExec` printed its native
plan, forwarded options included, in `toString` and explain. The same commit
hides it as `CometNativeScanExec` already does, with a test over all three
explain modes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
