sundapeng opened a new pull request, #9200: URL: https://github.com/apache/paimon/pull/9200
### Purpose A negative partition statistic means the field was never reported, not that it measured small. #9121 made that explicit — `PartitionStatistics#isKnown` is the predicate, and its javadoc says to use it rather than compare against `-1`. Two Spark display paths predate that and still take the sentinel literally. `DESCRIBE ... PARTITION` puts the raw `fileCount`, `fileSizeInBytes`, `lastFileCreationTime` and `recordCount` into the partition parameters, so an unreported field reads as `-1`, and an unreported creation time renders as a 1969 date. It also builds `CatalogStatistics` from the raw size and row count. `CatalogStatistics#toPlanStats` hands that size to Spark's cost estimation, so an unreported partition plans as a negative number of bytes. `SHOW TABLE EXTENDED ... PARTITION` prints the record count and byte size the partition metadata carries without looking at them, so an unreported statistic reads as `-1 rows, -1 bytes`. Separately on that path, a statistic the metadata does not carry at all defaults to an exact `0` — that one is reachable today for any `SupportsPartitionManagement` whose metadata map lacks these field names, and it is the worse failure of the two, because zero is a plausible answer. ### Approach An unreported statistic is left out of the partition parameters rather than rendered as a string. Spark already drops the whole `Partition Parameters` row when the map is empty, and `parameters` is a `Map[String, String]` with no null convention, so any placeholder we could write there would read as a measurement. An unreported creation time reads `UNKNOWN`, which is how Spark itself spells an unknown `Last Access` two rows below it. `CatalogStatistics` is built only when the size is known, and carries the row count only when that is known too. `CatalogStatistics(sizeInBytes, ...)` has no way to express "size unknown", so a partly-known partition yields no statistics object rather than a fabricated size. `SHOW TABLE EXTENDED` reads `UNKNOWN rows, UNKNOWN bytes`, and the unreported keys are dropped from the echoed parameter bag so the two adjacent lines cannot contradict each other. Non-statistic keys and non-numeric values pass through untouched. No new predicate was introduced; both sites call `PartitionStatistics#isKnown`. ### Tests `UnreportedPartitionStatisticsTest` in `paimon-spark-ut`, four cases, two per command: one that a statistic nobody reported is not displayed as a number, one that a statistic the catalog did report is still displayed. The two unknown cases fail on master and pass here; the two known cases pass on both, and they are what keeps the change from being a blanket suppression. The unknown is injected through a `DelegateCatalog` and a `BaseTable` subclass rather than produced by a stock table, because the in-repo `RESTCatalogServer` stores zeros. That injection is the shape a catalog which never received statistics returns, which is what the `isKnown` contract exists for. Existing coverage of both sites — `DescribeTableTest` and the four catalog-managed partition suites — passes unchanged. Built and tested with `-Pspark3`; also built with `-Pspark4` (Scala 2.13, JDK 17) since the change lives in `paimon-spark-common`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
