sundapeng opened a new pull request, #9298: URL: https://github.com/apache/paimon/pull/9298
> **Stacked on #9295, #9296 and #9297** — their commits show up here too until they merge. Review > only `[spark] Support ANALYZE TABLE on catalog-managed format tables`. ### Purpose Spark rejects ANALYZE TABLE for every v2 table, so a user reaching for the standard command on a format table gets told it is unsupported. It is the command that exists to recompute statistics, and after this it does. What it means here is narrower than for a Paimon table, and the difference is worth stating. A format table has no snapshot to carry table-level statistics and no column statistics to compute; analyzing it measures its partitions. So both modifiers carry weight: NOSCAN stops at the listing, and a full ANALYZE also reads file footers for exact row counts, which no listing can give and no commit can give for data it did not write. `PARTITION(...)` must name a leading run of the partition keys, because that is the shape the catalog can select on. Anything else is rejected rather than quietly widened to more partitions than the user asked about; naming a partition that does not exist is rejected too, rather than reporting success for having measured nothing. Following Spark, a partition column named without a value means every value of it, and the column names are resolved case-insensitively like every other command's. A format table that discovers its partitions from the filesystem has no catalog to write to and keeps the upstream rejection. Measuring replaces rather than accumulates, so running it twice is running it once. Row counts stay unknown for formats without a footer, and a footer that cannot be read makes the whole partition's row count unknown rather than short. ### Tests `CatalogManagedPartitionAnalyzeTest`: a full spec measuring only its own partition and a leading prefix measuring every partition under it, a non-leading column and a column that is not a partition column both rejected rather than silently widened, a partition that is not registered failing instead of reporting success for having measured nothing, column names resolved the way the rest of Spark resolves them, NOSCAN keeping a row count that is already known while a full ANALYZE reads the one a NOSCAN cannot, a second run reporting the same measurement rather than accumulating, files a committer left staged not being counted, the registered set never changing, and a filesystem-discovering format table keeping the upstream rejection. ### API and Format No API or format change. A command Spark used to reject now runs on catalog-managed format tables. ### Documentation The command and what it measures are described in the Spark auxiliary statements page. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
