neatHyperTxt-meesho opened a new pull request, #6822: URL: https://github.com/apache/hive/pull/6822
HIVE-30096: HiveAlterHandler forces a full file listing to recompute stats on every table-level alter Body: ### What changes were proposed in this pull request? HiveAlterHandler.alterTable no longer passes forceRecompute=true unconditionally to MetaStoreServerUtils.updateTableStatsSlow for table-level alters. It now forces the recompute only when the alter actually changed the table location (new package-private helper isTableLocationChanged). For metadata-only alters, the pre-existing "fast stats already present" shortcut in updateTableStatsSlow becomes reachable and no file listing happens. This also makes the alter path consistent with the create path (updateTableStatsForCreateTable), which has always passed forceRecompute=false. ### Why are the changes needed? For any non-rename table-level alter of an unpartitioned table, the metastore recursively lists the entire table location and materializes one FileStatus per file in memory inside a single RPC. Nothing gates this server-side (metastore.stats.autogather is only consulted on the create/add-partition paths; the DO_NOT_UPDATE_STATS table parameter is transient per the HIVE-10228 note). Table formats that keep partitioning in their own transaction log (Delta, Iceberg, Hudi) register as unpartitioned tables with millions of files under one directory: on our deployment, a property-only alter of a Delta table with ~4M files on GCS materialized ~7.9GB of FileStatus objects and OOMed the metastore twice with an 8GB heap. The recomputed values are also immediately marked not accurate by setBasicStatsState(FALSE) for any non-ANALYZE alter, so the listing cost bought numbers the planner is told not to trust. ### Does this PR introduce _any_ user-facing change? Metadata-only alters no longer refresh stale quick stats as a side effect; refreshing them without a location change requires ANALYZE TABLE (or happens via StatsTask on Hive-managed writes), which matches the create path's existing semantics. Location changes recompute exactly as before, and tables missing the fast stats entirely are still listed once. ### How was this patch tested? New unit test TestHiveAlterHandler#testTableLocationChangeDetection. Existing TestHiveAlterHandler and TestMetaStoreServerUtils suites pass (the !forceRecompute && containsAllFastStats shortcut this change routes into is already covered there). Verified the failure mode on a production deployment: property-only alter of a ~4M-file Delta table OOMs an 8GB-heap metastore before the change; with stats reuse the same alter applies instantly at flat heap. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
