sunchao commented on code in PR #6784:
URL: https://github.com/apache/datafusion-comet/pull/6784#discussion_r4320057319
##########
spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala:
##########
@@ -457,6 +458,9 @@ case class CometExecRule(session: SparkSession,
queryStagePrep: Boolean = false)
case scan: CometBatchScanExec if scan.wrapped.scan.isInstanceOf[CSVScan]
=>
convertToComet(scan, CometCsvNativeScanExec).getOrElse(scan.wrapped)
+ case scan: CometBatchScanExec if
scan.wrapped.scan.isInstanceOf[TextScan] =>
+ convertToComet(scan, CometTextNativeScanExec).getOrElse(scan)
Review Comment:
[P2] Restore the wrapped Spark scan when text conversion is declined. With
native V2 text enabled, reading small uncompressed files from two ABFS
containers passes the scan gate, but `extractScanObjectStoreOptions` rejects
the layout and the text serializer returns `None`. This branch then retains
`CometBatchScanExec`, which advertises columnar output although Spark’s text
reader produces `UnsafeRow`. The query fails with `ClassCastException` instead
of falling back and returning both files. Use `getOrElse(scan.wrapped)`, as the
adjacent CSV branch does, and cover a serializer-decline case.
Evidence: A bounded Spark 4.1.3 probe used the repository’s
FakeAbfsFileSystem to read local files through two abfss container authorities.
Spark returned alpha and beta. The unchanged exact-head
extractScanObjectStoreOptions method returned Left for the two containers.
Executing the exact-head CometBatchScanExec around that scan reproduced
`UnsafeRow cannot be cast to ColumnarBatch`; executing its wrapped
BatchScanExec returned alpha and beta. Source tracing connects that decline to
this getOrElse branch. This was a component reproduction without Azure access
or a full integrated Comet build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
