andygrove opened a new issue, #6147: URL: https://github.com/apache/datafusion-comet/issues/6147
### What is the problem the feature request solves? The Iceberg user guide has errors and gaps about native writes: - `docs/source/user-guide/latest/iceberg-writes.md:273-276` says float/double partition directories use Rust's shortest representation (`f=1` where iceberg-java writes `f=1.0`). Since #5840 they use Java's `Double.toString` rules and match iceberg-java. - `iceberg-writes.md:316-319` says the parity suite compares committed manifests byte-for-byte. `CometIcebergWriteActionSuite` compares aggregated `readable_metrics` (min/max, counts), not manifest bytes. - `iceberg-writes.md` does not say that writes whose input is a local relation, such as `INSERT ... VALUES` and `df.writeTo(...).append()` on a local DataFrame, need `spark.comet.exec.localTableScan.enabled` (default `false`). Without it they silently use iceberg-java even with both write flags on. - `iceberg-writes.md` does not list #6114 (dictionary page kept for high-cardinality columns) among the accepted divergences. - `docs/source/user-guide/latest/iceberg.md:184` lists "Iceberg writes (reads are accelerated, writes use Spark)" as unsupported. Native writes exist behind `spark.comet.iceberg.write.enabled`; that page should point to `iceberg-writes.md`. ### Describe the potential solution Fix the four points above. The "experimental, disabled by default" wording will need another pass when the defaults change (#5644). ### Additional context Found in an audit of the native Iceberg write path before enabling it by default. Part of #5649. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
