viirya commented on code in PR #7053:
URL: https://github.com/apache/datafusion-comet/pull/7053#discussion_r4457701581
##########
spark/src/test/scala/org/apache/comet/parquet/CometParquetWriterSuite.scala:
##########
@@ -651,6 +652,31 @@ class CometParquetWriterSuite extends
CometParquetWriterTestBase {
}
}
+ test("parquet write records the Spark version in the footer") {
Review Comment:
Thanks, this is the test I had in mind. One thing to make it robust: on
Spark 3.4 and 3.5 it relies on the native writer ignoring the default
`EXCEPTION` write mode, since 1500-01-01 is a value Spark refuses to write
there. Once #7047 makes the native write fail or fall back under `EXCEPTION`,
this test will fail on 3.x. Could the native write run under
`SQLConf.PARQUET_REBASE_MODE_IN_WRITE.key -> "CORRECTED"`? That is the mode
under which Spark writes the same footer, and it keeps the test about the read
side.
##########
spark/src/main/scala/org/apache/comet/serde/operator/CometDataWritingCommand.scala:
##########
@@ -126,6 +127,7 @@ object CometDataWritingCommand extends
CometOperatorSerde[DataWritingCommandExec
.newBuilder()
.setOutputPath(outputPath)
.setCompression(codec)
+ .setSparkVersion(SPARK_VERSION_SHORT)
Review Comment:
Thanks. The fallback test covers the `datetimeRebaseModeInWrite` branch with
a date column. The `int96RebaseModeInWrite` branch has its own condition
(timestamp columns only) and the docs describe it, but nothing exercises it.
Could you add a case with a timestamp column and
`int96RebaseModeInWrite=LEGACY` that falls back, and a date-only one with the
same setting that stays native? The second guards against the check widening to
every datetime column.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
