andygrove commented on PR #5293: URL: https://github.com/apache/datafusion-comet/pull/5293#issuecomment-5219311507
## Relationship to #4658 @jordepic you may be interested in this one. It reaches the same shape as #4658 from the opposite direction: both separate per-task data-file writing from the commit so the write runs inside AQE. #4658 had to build that split because Spark's Iceberg V2 write is one operator; for V1 Parquet Spark already provides it (`DataWritingCommandExec` over `WriteFilesExec`), so this PR mostly deletes Comet's hand-rolled commit code. Relevant to writing Iceberg data files natively via iceberg-rust: that would inherit this writer's output, which is not yet Spark-comparable — INT96 (#3425), footer metadata keys (#3427), Catalyst nullability and field IDs (#5305), writer properties (#5304). Also borrowed from #4658: `IcebergWriteExec` has the SPARK-23271 zero-partition guard and `CometWriteFilesExec` does not — filed as #5303. No merge conflicts with `apache/main`. One coordination item: `spark.comet.write.iceberg.*` vs `spark.comet.parquet.write.*` are two namespaces for one feature family (#5306), cheap to settle while both are still experimental and off by default. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
