andygrove opened a new issue, #6169:
URL: https://github.com/apache/datafusion-comet/issues/6169
The 1.1.0 user guide review in #6168 left these items open. Some depend on
other PRs, some need a decision, and some need a code change or a test rather
than a doc edit.
### Before the 1.1.0 branch cut
- [ ] #5783: a Parquet struct with byte-identical child names silently
multiplies rows. This is a regression from #5602, which is new in 1.1.0. #5786
(approved, closes #5783) rejects the ambiguous reads with a clear error. If it
doesn't land before the cut, add #5783 to the incorrect-results list in
`compatibility/scans.md`.
- [ ] Iceberg transform residuals: a residual such as `bucket(4, id) = 2 AND
x` is pushed to iceberg-rust as `id = 2`, which drops rows. This is also true
in 1.0.0, and the fix is #6154. Once it lands, the transform-residual bullet
under "Current limitations" in `iceberg.md` should say that a single transform
predicate falls back, while a combined one stays native without pushing the
residual. If it misses 1.1.0, document it as a known wrong-results issue
instead.
- [ ] #6066 removes three keys that shipped in 1.0.0:
`spark.comet.memoryOverhead`, `spark.comet.exec.onHeap.memoryPool` and
`spark.comet.shuffle.jvm.memoryFactor`. They're in the Testing category, but
the versioning policy only allows removing a key in a major release and doesn't
exempt testing keys. Either keep them as deprecated no-ops or clarify the
policy.
- [ ] Check that the scheduled CI runs have been healthy over the release
window, the other release preparation step.
### Docs decisions
- [ ] Spark 4.2 has no expression-compatibility page. `SPARK_PROFILES` in
`docs/build.sh` and `dev/generate-release-docs.sh` stops at 4.1, while
`compatibility/expressions/index.md` tells readers to pick their version's
page. The registrations match 4.1, but `aggregates.scala` has `isSpark42Plus`
branches for `collect_list` / `collect_set` and `mode`, so a 4.2 page would
differ.
- [ ] The links in `expressions.md` to
`compatibility/expressions/<category>.md` (array, cast, datetime, map) don't
resolve on the published site, because those pages only exist under
`spark-<ver>/` at build time. 1.0.0 has the same problem. Should they point at
the Spark 4.1 pages, or at the index? The `reverse` row also links `array.md`,
while its generated notes land in `string.md`.
- [ ] In `understanding-comet-plans.md`, the `from_unixtime`
extended-explain example no longer falls back, since `CometFromUnixTime` routes
the incompatible case through the codegen dispatcher. Replace it with real
output from an expression that does fall back. The compatibility guide URL
quoted in it is also stale.
- [ ] `tuning.md` "Remote Shuffle with Celeborn" describes the native push
path in detail. However, `CelebornReflectionCompatibilitySuite` asserts that
released Celeborn clients are refused native push, so today users only get
ordinary Celeborn shuffle underneath Comet operators. Say so up front, or move
the detail to the contributor guide.
### Stale text generated from code
- [ ] The `spark.comet.exec.scalaUDF.codegen.enabled` description says the
regex family routes through the dispatcher by default. In-subset `rlike`
literals have run natively since #5415, and the description also understates
that the flag gates every dispatch path.
- [ ] The `spark.comet.memoryOverhead` description still talks about
additional memory allocated per executor. After #6054 it only sizes the native
pool in on-heap mode.
- [ ] The `TruncTimestamp` non-UTC incompatibility reason in
`serde/datetime.scala` cites #2649, which was closed on 2026-06-30. Check
whether the incompatibility still holds, then update or drop the reason. It
renders on the generated datetime compatibility page.
- [ ] The Iceberg write toggles,
`spark.comet.write.iceberg.splitOperator.enabled` and
`spark.comet.iceberg.write.enabled`, are in the Testing category. So
`configs.md` lists them as testing settings, even though `iceberg-writes.md`
documents them for users.
- [ ] GenerateDocs Implementation column:
- Builder-registered functions publish as "—". Examples are `hour`,
`minute`, `second`, `make_timestamp`, `mode`, `to_char` and `to_varchar`.
- `regexp_extract` and `regexp_extract_all` show Native, although they
dispatch by default.
- Each profile's pass rewrites `expressions.md`, so the last pass wins.
- [ ] Other GenerateDocs gaps:
- The cast tables use `Cast.canCast`, so casts that Spark rejects under
ANSI show U rather than N/A.
- There is no hash, predicate or json category, so some notes are never
published. For example, `hash`, `xxhash64` and `sha*` fall back for decimal
precision above 18 and for TIME.
### Needs a test or a check before changing the docs
- [ ] `compatibility/scans.md` says decimals encoded in binary format fall
back. There is no plan-time check for this, and arrow-rs reads BYTE_ARRAY
decimals as Decimal128, so the claim is likely stale.
- [ ] `iceberg.md` says REST-vended credentials aren't consumed
automatically. But the scan forwards `FileIO.properties()`, so static vended
`s3.*` keys probably do reach the reader. It may be only refresh, or
credentials delivered through `storage-credentials`, that are unsupported.
- [ ] `expressions.md` "Not currently planned": since #5692, several of
these should route through the codegen dispatcher rather than fall back. That
covers `is_valid_utf8`, `make_valid_utf8`, `validate_utf8`,
`try_validate_utf8`, `quote`, `sentences` and the scalar `bitmap_*` functions.
Add a fixture before changing the text.
- [ ] These are probably supported but untested:
- `current_time`, `time_diff` and `time_trunc`
- the two-argument TIME forms of `make_timestamp_ltz` /
`make_timestamp_ntz`
- the datetime form of `try_add`
- [ ] Reported by the review, not yet confirmed: `iceberg.md` "Current
limitations" may also be missing two fallbacks, equality deletes on struct or
variant columns and `fixed` partition columns.
- [ ] `datasources.md` doesn't cover native Parquet scans of `gs://`,
including which settings are honoured. It also doesn't explain
`fs.comet.libhdfs.schemes` routing.
### Other gaps that predate 1.1.0
- [ ] #5071 was auto-closed by #5162, but two of its items remain:
- Byte / Short `Add`, `Subtract` and `Multiply` overflow raises
`ARITHMETIC_OVERFLOW` instead of `BINARY_ARITHMETIC_OVERFLOW`.
- Integral overflow messages omit Spark's `try_` suggestion.
Reopen it or file a follow-up, so the compatibility guide entry can link
to it.
- [ ] `iceberg.md` "Current limitations" doesn't mention several fallbacks:
- Iceberg metadata tables
- metadata columns other than `_file`, `_pos`, `_spec_id` and `_partition`
- `FileIO` implementations outside the supported list
- [ ] `mode` is marked ✅, although every input falls back unless
`spark.comet.expression.Mode.allowIncompatible=true` is set. That stretches the
status legend.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
