sundapeng opened a new pull request, #9296: URL: https://github.com/apache/paimon/pull/9296
> **Stacked on #9295** — its commit shows up here too until it merges. Review only > `[core] Report what a format table commit wrote to the catalog`. ### Purpose The writer already counted the rows and the bytes, the commit already knows which partitions it wrote, and #9295 gave the catalog somewhere to put both. This connects them, behind `format-table.commit.report-partition-statistics`. Which mode a commit uses follows from what it did to the partition. An appending commit saw only its own files, so it reports an increment: a Flink sink commits once per writer subtask, and N increments over one partition add up to what the job wrote. An overwriting commit replaced everything the partitions held, so what it wrote is the total and it reports that. Static prefix overwrite is why a pure increment cannot express this. Clearing a prefix empties every partition beneath it, including ones this commit writes nothing to; their old data is gone and no increment says so. Those partitions report zero, an exact zero because they really are empty, and stay registered. The partition set is the catalog's authority and statistics never touch it; only ADD PARTITION and DROP PARTITION do. The directories emptied that way come out of the deletion listing, which already had to walk them, so the numbers cost no extra IO: the row count and byte size come from the writer, the file count from the commit messages, and the last file creation time from the commit's own clock. ### Why the option is off by default A non-empty `ADD` report makes the create-partitions request unsafe to replay, so a 429 or 503 the client used to retry by itself now surfaces to the caller instead of risking a double-counted increment. That is a change to how an existing write path fails, and it should be opted into rather than inherited. An increment can also drift for reasons no client can see: a writer that is not Paimon, a file deleted out of band. Convergence is a later full report over the same partition, which the next two PRs teach MSCK REPAIR TABLE and ANALYZE TABLE to produce. ### Tests `FormatTableCommitStatisticsTest`: append, dynamic overwrite, static prefix overwrite of a partition this commit does not write, the summation of the independent increments of concurrent writers of one partition, a listing that answers under another scheme, a nested directory in the value-only layout, and reporting turned off. ### API and Format One new option, off by default. No API or format change. ### Documentation The option is described in the generated core configuration table. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
