jackylee-ch opened a new pull request, #10173:
URL: https://github.com/apache/paimon/pull/10173
## Purpose
`metadata.stats-mode` accepts `none` / `counts` / `truncate(N)` / `full`,
but pypaimon only recorded value stats for `full` — `counts` and
`truncate(N)` silently wrote none (not even null counts). A table that
makes Java's default explicit (`truncate(16)`) or sets `counts` therefore
produced files with no min/max, so Spark/Flink/Java queries could not
data-skip files pypaimon wrote.
## Change
- Add a `stats_mode` helper mirroring Java's `*SimpleColStatsCollector`:
`counts` keeps only the null count; `truncate(N)` truncates string/binary
min/max (min shortened, max bumped to stay `>=` the original, both dropped
when no sound upper bound exists) and leaves other types full; `none`
drops all; `full` unchanged.
- Writers record value stats for every mode except `none`, converting value
stats only. Key stats stay full (the LSM needs exact key bounds), and the
default stays `none`, so existing tables are untouched.
Per-field `fields.<f>.stats-mode` remains a follow-up; this covers the
table-level mode.
## Tests
- Unit parity with Java `SimpleColStatsCollectorTest` (astral-char string,
binary, all-`0xFF` drop, counts/none/full).
- End-to-end per mode, asserting the manifest value stats; verified
non-vacuous.
Written with Claude Code; verification is mine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
