andygrove opened a new pull request, #6168: URL: https://github.com/apache/datafusion-comet/pull/6168
## Which issue does this PR close? Part of #5327. ## Rationale for this change The release process asks for a review of the user guide before the release branch is cut. I went through every page under `docs/source/user-guide/latest/` against the changes merged since 1.0.0 branched (af534e0fa), and checked each statement I touched against the code. ## What changes are included in this PR? Most of the diff corrects statements that stopped being true. The CI table in the installation guide still said every Spark version is tested on every change, but since #5963 only Spark 4.1 is gated before merge and the other profiles run nightly. The Spark 3.4/3.5 "known limitations" described Parquet reads that Comet has rejected, like Spark does, since before 1.0. A number of expression notes, and the cast legend, said "falls back" where Comet now routes the case through the codegen dispatcher by default. The S3 region and Iceberg `NOT IN` pushdown descriptions were wrong, as were the tokio thread defaults and several shuffle and scan metric names. The Kubernetes example pointed at a Java 11 / Spark 3.5 image and set no off-heap memory, which leaves Comet disabled. The compatibility guide listed divergences that 1.1.0 fixed: the boolean to decimal cast, explicit positive timestamp years, the ABFS object store cache key, and most of the ANSI error-message items. Those entries are removed. It also documents 1.1.0 behaviour that had no coverage. That includes the experimental in-memory cache, the native Iceberg writer (including the credential provider's `WRITE` access mode), nested hash shuffle keys, `BINARY` sort-merge join keys, and the aggregate spill and memory metrics. It also covers native TIME functions, scans of tables whose Variant columns aren't read, and missing expression and plan-node rows. The upgrade guide gets an "Upgrading to Comet 1.1.0" section. Diffing the generated config reference against 1.0.0 shows no removed keys, no changed defaults, and no `spark.comet.legacy.*` keys, so the section says no behavior change needs a legacy key, and notes that JDK 17 is now required. The release process also loses its two notes about tag pushes publishing a Docker image, since that workflow was removed in #4241. A few things are deliberately left alone: - The Iceberg transform-residual limitation still says those scans fall back. Some of them actually return wrong rows, and the right wording depends on whether #6154 lands before the cut. Likewise, #5783 isn't listed as a known issue here, pending #5654. - The links in `expressions.md` to `compatibility/expressions/<category>.md` don't resolve on the published site, because those pages only exist under `spark-<ver>/` at build time. 1.0.0 has the same problem. Should they point at the Spark 4.1 pages or at the index? - The `from_unixtime` example in `understanding-comet-plans.md` no longer falls back, and a replacement needs real plan output. - There is no Spark 4.2 expression-compatibility page yet. - Some generated text is stale at its source: the `scalaUDF.codegen.enabled` and `memoryOverhead` config descriptions, and the `TruncTimestamp` fallback reason. Those are code changes, so a follow-up. ## How are these changes tested? This is documentation only. Each corrected or added statement was checked against the code at dd68a531c: serde support levels, config definitions and defaults, operator node names, and metric names. Prettier passes. I haven't run the full docs build, but a script checked the relative links and anchors in the changed pages. The only targets that don't resolve are the pre-existing category links mentioned above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
