zhangfengcdt commented on code in PR #6754:
URL: https://github.com/apache/datafusion-comet/pull/6754#discussion_r4222381010
##########
spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala:
##########
@@ -394,6 +394,16 @@ case class CometScanRule(session: SparkSession)
return withFallbackReason(scanExec, "Iceberg Metadata tables are not
supported")
}
+ // As in transformV1Scan: these expressions read InputFileBlockHolder,
which the source's own
+ // reader sets per file. Comet's native V2 scans (Iceberg, CSV) do not, so
they would return
+ // empty/default values
(https://github.com/apache/datafusion-comet/issues/6707).
+ if (CometScanRule.readsInputFileBlock(plan)) {
Review Comment:
Sure! I have add both. `CometCsvNativeReadSuite` now has "input_file_name
falls back to Spark's reader". It fails with mismatched results if the V2
check is removed.
The CSV section of `datasources.md` now notes the fallback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
