andygrove commented on PR #7528: URL: https://github.com/apache/datafusion-comet/pull/7528#issuecomment-6949583853
I think the Spark 3.5 failure is Spark's own reader rather than Comet. The fallback works, which is why the error comes from Spark's vectorized reader. But Spark 3.5 can't read an INT32 column as `decimal(12,-2)` at all, because widening an integer to a decimal on read arrived with Parquet type widening in Spark 4.0. So there's no Spark answer to compare against on 3.5. Could the test check the fallback on every version and only compare answers where Spark can read it? For example, wrap the `checkSparkAnswerAndFallbackReason` call in `if (isSpark40Plus)`, and on 3.5 assert that both Spark and Comet raise and that the plan falls back with the same reason. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
