andygrove opened a new issue, #6751:
URL: https://github.com/apache/datafusion-comet/issues/6751
### Describe the bug
On Spark 3.4/3.5, a native Parquet write ignores `maxRecordsPerFile`, both
the write option and `spark.sql.files.maxRecordsPerFile`. Spark rolls to a new
file every N rows. `CometNativeWriteExec` writes one file per task, so the
output has fewer, larger files than Spark writes. The rows read back are the
same.
`CometWriteFiles` on Spark 4.0+ declines these writes, so they fall back to
Spark. `CometDataWritingCommand.getSupportLevel` (the 3.x serde) has no such
check. The test `write with maxRecordsPerFile falls back to Spark` in
`CometParquetWriterSuite` is gated to Spark 4.0+ for that reason.
### Steps to reproduce
On Spark 3.5, with `spark.comet.write.parquet.enabled=true` and
`spark.comet.operator.DataWritingCommandExec.allowIncompatible=true`, write ten
rows from a native scan with `.option("maxRecordsPerFile", 1)`. Spark writes
ten files. Comet writes one.
### Expected behavior
Decline the write on 3.x as the 4.0+ serde does, using the same effective
value Spark uses (`FileFormatWriter` reads the option first, then the session
conf), and drop the 4.0+ gate on the test.
### Additional context
Noted in the #4746 reviews (2026-09-28). It predates #4746.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
