HaozeeWu opened a new pull request, #11798:
URL: https://github.com/apache/gravitino/pull/11798
### What changes were proposed in this pull request?
This PR adds automatic partition-level statistics aggregation for Hive
partitioned tables, enabling the REST API (`GET /tables/{table}`) to return a
table-level `totalSize` by summing up partition-level `totalSize` values at
query time.
**Changes include:**
1. New `listPartitionsByNames` API — Added to `HiveClient`, `HiveShim`,
`HiveShimV2`, `HiveShimV3` and `HiveClientImpl`. This method fetches partition
objects by a list of partition names, enabling batched metadata loading.
2. Batched partition statistics aggregation — Added
`aggregatePartitionTotalSize()` in `HiveCatalogOperations`. It uses
`listPartitionNames` (lightweight, strings only) followed by batched
`listPartitionsByNames` calls (batch size = 1000) to sum `totalSize` from all
partitions. The aggregated values are written into `HiveTable.properties()`
**in memory only** and are NOT persisted back to HMS.
3. Lazy trigger — Aggregation is skipped if the table-level `totalSize`
property already exists, avoiding redundant computation.
### Why are the changes needed?
Hive Metastore stores statistics at the partition level only. When a
partitioned table is described via Hive CLI, Hive internally iterates all
partitions and sums their stats in memory. Gravitino lacked this capability, so
the REST API returned no `totalSize` for partitioned Hive tables. This PR
replicates Hive's behavior with a memory-efficient batched approach.
### Does this PR introduce _any_ user-facing change?
- **Yes** — For Hive partitioned tables that have partition-level statistics
but no table-level statistics, the REST API and Java Client `loadTable()` will
now include `totalSize` and `numPartitions` in `table.properties()`.
- **No breaking changes** — The aggregation happens transparently at query
time and does not modify HMS data.
### How was this patch tested?
1. Created a Hive partitioned table with 3 partitions.
2. Verified via `curl GET /tables/{table}` that `totalSize` equals the sum
of partition-level values.
3. Verified non-partitioned tables and tables with existing table-level
stats are unaffected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
