mbutrovich commented on code in PR #6664: URL: https://github.com/apache/datafusion-comet/pull/6664#discussion_r4260266960
########## docs/source/user-guide/latest/migration-guide.md: ########## @@ -111,6 +111,45 @@ it. `spark.comet.convert.oneRowRelation.enabled` is on by default, so such a que without either setting, and logs no warning. The list remains the way to convert other leaf operators, such as the scan of a Data Source V2 connector. +### Iceberg Writes + +`spark.comet.write.iceberg.enabled` now defaults to `true`. Comet plans an Iceberg `INSERT INTO`, +`INSERT OVERWRITE`, and copy-on-write `DELETE`, `UPDATE` or `MERGE` as two operators, `IcebergWrite` +under `IcebergCommit`, in place of Spark's single V2 write operator, and writes the data files of +each eligible write natively with iceberg-rust. A write that is not eligible still uses +iceberg-java's writer, and iceberg-java still commits every write. The table holds the same rows, +but natively written files differ from iceberg-java's in the ways listed under +[Accepted divergences](iceberg-writes.md#accepted-divergences), and explain output and the Spark UI +show the new operators. Set `spark.comet.write.iceberg.enabled=false` to plan Spark's own operator, +as in Comet 1.1.0. Comet 1.1.0 named this setting `spark.comet.iceberg.write.enabled`, in the +testing category, and Comet now ignores that name, so a deployment that set it to `false` gets the +new default unless it sets `spark.comet.write.iceberg.enabled=false`. +`spark.comet.write.iceberg.splitOperator.enabled` is now a testing setting, and setting it to Review Comment: Should this say the split flag remains a testing setting? "Is now" reads as if 1.1.0 treated it as a user setting, but it was already in the `testing` category in 1.1.0 ([`CometConf.scala` at 1.1.0](https://github.com/apache/datafusion-comet/blob/992c806a7e38c2e88bd018aa5774164b0850e1fa/spark/src/main/scala/org/apache/comet/CometConf.scala#L124-L126)). The rest of the paragraph covers the part that did change, that the 1.1.0 user guide told people to set it. ```suggestion `spark.comet.write.iceberg.splitOperator.enabled` remains a testing setting, and setting it to ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
