sundapeng opened a new pull request, #9297:
URL: https://github.com/apache/paimon/pull/9297
> **Stacked on #9295 and #9296** — their commits show up here too until they
merge. Review only
> `[core][spark] Measure format table partitions in MSCK REPAIR TABLE`.
### Purpose
A commit reports what it wrote (#9296). Nothing reports what is already
there, and for a format
table plenty is: partitions written by something that is not Paimon, files
deleted out of band, an
increment redelivered and counted twice. MSCK REPAIR TABLE is already the
command that reconciles
the partition set against the directories, so it is the natural place to
reconcile the numbers too.
Off by default, behind
`spark.paimon.format-table.repair.collect-statistics`, because measuring
changes what a repair costs: the plain diff lists partition directories, and
measuring lists the
files inside every one of them. That is a different order of magnitude on a
table with many
partitions, and a repair should not silently become that. When it is on,
`spark.paimon.format-table.statistics.parallelism` caps how many partitions
are measured at once, at
8: listing one is a round trip the driver spends waiting on, and the cap
keeps a table with many
partitions from turning that wait into a burst of requests.
When on it measures every partition that ends up registered with a directory
behind it, not only the
ones it just added. The stale numbers of partitions written outside Paimon
are exactly what a repair
exists to correct. Without ADD it stays inside the already-registered set,
so measuring never
registers a partition the command was not asked to.
### What the collector reports
What a reader would see. File count, byte size and last file creation time
come from the listing. It
stops there: the row count needs a file footer, which is what ANALYZE is
for. A listing failure
aborts the whole collection rather than reporting what it managed to see,
because a truncated
listing is indistinguishable from a partition that lost files. A partition
whose directory is gone
measures as an exact zero, with no last file to date.
### Tests
- `FormatTablePartitionStatsCollectorTest`: the staging trees a committer
leaves behind, exact
parquet row counts, an unreadable footer, a missing directory, the
one-for-one alignment of the
result with the given specs, a spec that omits a partition key, and a
listing failure aborting the
whole collection on both the serial and the parallel path.
- `FormatTablePartitionRepairTest`: measuring every partition on disk rather
than only the
additions, a repair without ADD registering nothing, and a listing failure
leaving the catalog
untouched.
- `CatalogManagedPartitionMsckRepairTest`: the command end to end with the
option off and on.
### API and Format
Two new Spark options, both off or bounded by default. No API or format
change.
### Documentation
Both options are described in the generated Spark connector configuration
table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
