JingsongLi opened a new pull request, #10422:
URL: https://github.com/apache/paimon/pull/10422
### Purpose
Support unavailable file row counts for bucket-unaware append tables. A file
with `rowCount = -1` currently contaminates summed counts and can produce
incorrect metadata COUNT results, LIMIT planning, or filtering on newly added
nullable columns. This change propagates unknown counts and lets queries fall
back to reading rows.
Snapshot totals remain conservatively unknown after an unknown file
participates in a commit. Signed snapshot deltas retain their existing meaning:
a known deletion of one row is still `-1`; unavailable deltas are distinguished
with the snapshot property `delta-record-count.unknown` and
`Snapshot.deltaRecordCountKnown()`. Repair, rollback, and manifest compaction
preserve that distinction. Spark ANALYZE uses its computed count when the
snapshot count is unavailable; Flink split weights fall back to file size.
Unknown counts require a snapshot backend that explicitly supports them.
Filesystem/renaming snapshot commits support this; catalog-managed commits
(including REST) reject them before publishing a snapshot. The supported table
scope excludes row tracking, data evolution, and Iceberg metadata
synchronization. Known appends also reject Iceberg synchronization after an
unknown snapshot. Readers and writers must be upgraded before using these
files; the unchanged manifest encoding alone does not make older readers
correct.
This is an independent foundation change; it contains no file-import
procedure.
### Tests
Behavioral tests construct native files with unavailable count metadata
without relying on an importer. They cover serialization, mixed known/unknown
files, COUNT and grouped COUNT, nullable-column filtering, LIMIT, ANALYZE,
signed deletes, partition reporting, repair, rollback, manifest compaction,
backend rejection, and enabling Iceberg synchronization after an unknown
snapshot. Spark query tests cover Parquet, ORC, and Avro.
Validated with JDK 8, Spark 3, and Flink 1. Final Maven verification runs
without `fast-build`, including configured checkstyle, formatting, enforcer,
and license checks:
```sh
mvn -pl
paimon-core,paimon-flink/paimon-flink-common,paimon-spark/paimon-spark-ut -am \
-Pflink1,spark3 -DfailIfNoTests=false \
-DwildcardSuites=org.apache.paimon.spark.sql.UnknownRowCountTest,org.apache.paimon.spark.sql.PushDownAggregatesTest
\
'-Dtest=UnknownRowCountTest,DataSplitCompatibleTest,FileIndexEvaluatorTest,PartitionStatisticsReporterTest,RemoveUnexistingManifestsTest,SnapshotTest,CatalogSnapshotCommitTest,RenamingSnapshotCommitTest,SplitWeightUtilsTest,FlinkTableSourceStatisticsTest,AppendOnlySimpleTableTest#testLimitPushDown'
test
```
Result: **64 tests passed** (46 core, 5 Flink, 13 Spark).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
