dwsmith1983 commented on code in PR #6753:
URL: https://github.com/apache/datafusion-comet/pull/6753#discussion_r4214858080
##########
spark/src/main/scala/org/apache/spark/sql/comet/CometCsvNativeScanExec.scala:
##########
@@ -92,7 +94,27 @@ object CometCsvNativeScanExec extends
CometOperatorSerde[CometBatchScanExec] {
val timeZone = sessionState.conf.sessionLocalTimeZone
new CSVOptions(csvScan.options.asScala.toMap, columnPruning, timeZone)
}
- val filePartitions = op.inputPartitions.map(_.asInstanceOf[FilePartition])
+ val hadoopConf =
+
sessionState.newHadoopConfWithOptions(op.session.sparkContext.conf.getAll.toMap)
+ val s3CompliantSchemes = NativeConfig.resolveS3CompliantSchemes(hadoopConf)
+ val libhdfsSchemes = NativeConfig.resolveLibhdfsSchemes(hadoopConf)
+ val inputPartitions = op.inputPartitions.map(_.asInstanceOf[FilePartition])
+ val multiStoreReason = CometScanUtils.multiStoreFallbackReason(
+ "Native CSV scan",
+ inputPartitions.view.flatMap(_.files.view.map(_.pathUri)),
+ s3CompliantSchemes,
+ libhdfsSchemes,
+ isBucketedScan = false)
+ if (multiStoreReason.nonEmpty) {
+ // CometExecRule falls back to the wrapped scan, so tag it too for the
explain output.
+ withFallbackReason(op, multiStoreReason.get)
+ withFallbackReason(op.wrapped, multiStoreReason.get)
+ return None
+ }
+ // Native planning reads each partition through one object store.
+ val filePartitions = CometScanUtils.splitPartitionsByStore(
Review Comment:
> Could the conversion fall back, or keep one partition, when the split
changes the partition count and `op.wrapped.outputPartitioning` is not an
`UnknownPartitioning`?
Done in e909c077b, as a fallback: keeping one partition is not an option,
since native refuses a partition that mixes stores.
`CometCsvNativeScanExec.convert` declines with a reason when the split changes
the partition count and Spark's scan reports anything but
`UnknownPartitioning`, which on 3.4 is the one-partition `SinglePartition`
case. Spark 3.5 and later report `UnknownPartitioning` there, so those scans
still split with an exchange above them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
