wangyum opened a new pull request, #6886: URL: https://github.com/apache/hive/pull/6886
JIRA: [HIVE-30196](https://issues.apache.org/jira/browse/HIVE-30196) (umbrella epic: [HIVE-19715](https://issues.apache.org/jira/browse/HIVE-19715)) ### Problem `get_partitions_with_specs` (introduced by HIVE-20306) skipped the `hive.metastore.limit.partition.request` guard that every regular partition listing RPC enforces — the config's own description enumerates the 8 covered RPCs and this one was absent. The thrift request has no client-side max parameter, so a single call made the metastore materialize all matching partitions server-side before responding. HiveServer2 paths (`PartitionIterable` batched iteration, SHOW TABLE EXTENDED, EXCHANGE PARTITION, repl bootstrap, view rewrite analysis) and direct HMS clients (e.g. Spark) use this API, so an operator-configured guard was silently bypassed on it. ### What this change does 1. **Fail-fast pre-check** in `HMSHandler.get_partitions_with_specs`, before the expensive fetch, mirroring what the regular listing RPCs already do (`checkLimitNumberOfPartitionsByFilter` / `checkLimitNumberOfPartitionsByPs`), mapped per filter mode: - filter spec unset → `rs.getNumPartitionsByFilter(cat, db, tbl, "")` - `BY_NAMES` → the requested partition names count (same as `get_partitions_by_names`) - `BY_VALUES` → `rs.getNumPartitionsByPs` (identical predicate: same `MetaStoreUtils.makePartNameMatcher` on both the SQL and JDO paths) - `BY_EXPR` → `rs.getNumPartitionsByFilter` on the joined expressions (same `PartFilterExprUtil` grammar the projection path parses) 2. **Post-fetch backstop** on the fetched result size — covers the count/fetch TOCTOU window (concurrent `add_partitions`) and unknown filter modes. 3. Adds `get_partitions_with_specs` to the `LIMIT_PARTITION_REQUEST` config description. 4. Authorization ordering is preserved: `firePreEvent` (pre-event listeners) still runs before the limit checks. `exceedsPartitionFetchLimit` / `checkLimitNumberOfPartitions` were made `public static` (taking an explicit `Configuration`) so the handler flow and `get_partitions_with_specs` share one implementation. The check stays at the RPC layer on purpose: the same store method is also used internally by the column-stats path, so the guard must not become a store-level rule. ### Behavior changes (only when the limit is configured; default `-1` = disabled is unchanged) - Requests matching more partitions than the limit now throw the same `MetaException` (same message / config name) as the regular listing RPCs. - `BY_NAMES` now uses the strict requested-count semantics, mirroring `get_partitions_by_names`: requesting more names than the limit fails even if fewer of them exist. - no-filter / `BY_VALUES` / `BY_EXPR` requests pay one extra count query per request; `BY_NAMES` is free. - Policy decision: projected `PartitionSpec` responses count as partition objects — response size and DB work scale with the number of matched partitions regardless of projection. ### Tests `TestGetPartitionsUsingProjectionAndFilterSpecs` (server started with `LIMIT_PARTITION_REQUEST=100`), new tests against a 101-partition table: | test | covers | |---|---| | `...ExceedsPartitionLimit` | no-filter mode over limit → throws | | `...AtPartitionLimit` | boundary: exactly 100 → succeeds | | `...ByNamesExceeds...` | BY_NAMES, all 101 names exist → throws | | `...ByNamesRequestedCountExceeds...` | 150 requested / 50 exist → throws (pins the strict semantics) | | `...ByValuesExceeds...` | BY_VALUES partial values matching 101 → throws | | `...ByExprExceeds...` | BY_EXPR expression matching 101 → throws | | `...ExceedsPartitionLimitWithJDO` | same as the first, DirectSQL disabled → JDO count path | All 32 tests pass, plus `TestRemoteHiveMetaStore#testGetPartitionsWithSpec`, `TestEmbeddedHiveMetaStore#testGetPartitionsWithSpec`, and `TestHmsServerAuthorization` (authorization failure still precedes the limit check). ### Follow-ups - Push the limit down into `MetaStoreDirectSql.getPartitionIdsViaSqlFilter(max)` (the projection path currently passes `null`) to bound the ID query with `limit+1` in one round trip — removes the extra count query and the TOCTOU window. - Related: [HIVE-30195](https://issues.apache.org/jira/browse/HIVE-30195) — the same RPC also skips the table-level filter hook check (`checkDbAndTableFilters`); same class of gap, handled separately. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
