sundapeng opened a new pull request, #9120:
URL: https://github.com/apache/paimon/pull/9120
### Purpose
`PartitionStatistics` says only that its fields "may be negative, indicating
that some data has been
removed". That covers one of the two planes the class is read on, and
consumers have been getting the
other one wrong.
- **Delta plane** — what a commit changed. A negative value is a decrement
the server adds to what it
holds. That is the existing meaning and **nothing here changes it**.
- **Observation plane** — what `listPartitions` returns for a partition as
it stands. A negative value
means nobody ever reported that field, and `0` means an exact zero.
Conflating them is not cosmetic. A consumer that reads unknown as zero plans
against an empty
partition that may hold a billion rows; one that does arithmetic on it gets
a number that is wrong
rather than missing. This has already produced a silent wrong answer in a
sibling project, where
`COUNT(*)` was answered from a placeholder row count and returned zero for a
partition full of data
(apache/paimon-rust#624).
So: the plane is named in the javadoc, unknown gets a name (`UNKNOWN`, with
`isKnown()` to test it
rather than each caller comparing against `-1`), and unknown is documented
as **per field** — a
reporter that only knows the file count leaves the record count unknown and
fills the rest.
### Why the fields stay primitive
Boxing them to express unknown as `null` would be a breaking change to a
`@Public` class, and the
encoding above needs no new type.
### Behaviour
Zero change. `UNKNOWN` is a name for a value already in use.
`FileSystemSplitEnumerator` now says
`PartitionStatistics.UNKNOWN` where it said `-1` — the same value under its
own name, because
discovering partitions by listing directories measures nothing about what is
inside them, which is
what unknown already meant there.
### Context
This is the first of a stack that lets a catalog-managed format table report
partition statistics; a
format table has no snapshot, so the channel a table snapshot uses does not
exist for it. This PR is
worth having on its own regardless of the rest of that stack: the ambiguity
it removes has already
cost one silent wrong answer.
### API and Format
`PartitionStatistics` gains `UNKNOWN`, `UNKNOWN_TOTAL_BUCKETS`,
`unknown(spec)` and
`isKnown(long)`. Additive only; no existing signature changes. No format
change.
### Documentation
The contract now lives in the class javadoc, which is where a consumer looks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
