Abhishek-Mohan commented on issue #4354: URL: https://github.com/apache/datafusion-comet/issues/4354#issuecomment-5681445862
I have a local implementation and have verified the relevant failures against DataFusion main (cee7bae63ba1ba1213b2c4aaf3bb22d803f50bda). The investigation found two parts: - Spark 3.x rejects decimal scale widening that Comet currently accepts. A version-specific rejection fixes the SPARK-34212 failure. - With Parquet row-filter pushdown enabled, DataFusion 55.1.0 stringifies the predicate error. This is already fixed upstream in apache/datafusion#24638; Comet additionally needs to recover the structured Spark error through the preserved source chain. The original row-group test passes in both ParquetV1QuerySuite and ParquetV2QuerySuite on Spark 3.4.3 and 3.5.9 with the upstream fix and the Comet changes. The existing Spark 3.x shim wrapper is correct: the tagged Spark tests expect getCause.getCause. I plan to finish this against a published DataFusion release containing #24638. The next major release is currently tracked in apache/datafusion#24461 (56.0.0, planned for October); inclusion will need to be verified when it is available. No PR has been opened yet. AI assistance: OpenCode assisted with investigation, implementation, and test execution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
