[
https://issues.apache.org/jira/browse/HIVE-30196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-30196:
----------------------------------
Labels: pull-request-available (was: )
> get_partitions_with_specs skips hive.metastore.limit.partition.request
> enforcement performed by all regular partition listing RPCs
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-30196
> URL: https://issues.apache.org/jira/browse/HIVE-30196
> Project: Hive
> Issue Type: Sub-task
> Reporter: Yuming Wang
> Priority: Major
> Labels: pull-request-available
>
> h3. Problem
> {\{get_partitions_with_specs}} (thrift RPC introduced by HIVE-20306) skipped
> the \{{hive.metastore.limit.partition.request}} guard that every regular
> partition listing RPC enforces (\{{get_partitions}},
> \{{get_partitions_by_names}}, \{{get_partitions_with_auth}},
> \{{get_partitions_by_filter}}, \{{get_partitions_by_expr}},
> \{{get_partitions_ps}}, \{{get_partitions_ps_with_auth}}). The request has no
> client-side max parameter, so one call made the metastore materialize all
> matching partitions server-side.
> h3. Fix
> * Fail fast *before* the partition fetch, using the same cheap count queries
> the regular listing RPCs use, mapped per filter mode:
> ** filter spec unset -> \{{rs.getNumPartitionsByFilter(cat, db, tbl, "")}}
> ** BY_NAMES -> the requested partition names count (as
> \{{get_partitions_by_names}} does)
> ** BY_VALUES -> \{{rs.getNumPartitionsByPs}} (same
> \{{MetaStoreUtils.makePartNameMatcher}} predicate as the fetch, both SQL and
> JDO paths)
> ** BY_EXPR -> \{{rs.getNumPartitionsByFilter}} on the joined expressions
> (same \{{PartFilterExprUtil}} grammar the projection path parses)
> * Keep a check on the fetched result size as the final backstop (covers the
> count/fetch TOCTOU window and unknown filter modes)
> * Add \{{get_partitions_with_specs}} to the \{{LIMIT_PARTITION_REQUEST}}
> config description
> Authorization ordering is preserved: \{{firePreEvent}} (pre-event listeners)
> still runs before the limit checks.
> h3. Behavior changes (only when the operator configures the limit; default -1
> is unchanged)
> * Requests matching more partitions than the limit now throw
> \{{MetaException}} (same message/config as the regular RPCs)
> * BY_NAMES uses the strict requested-count semantics, mirroring
> \{{get_partitions_by_names}} (requesting more names than the limit fails even
> if fewer exist)
> * no-filter / BY_VALUES / BY_EXPR requests pay one extra count query per
> request; BY_NAMES is free
> h3. Tests
> Added to \{{TestGetPartitionsUsingProjectionAndFilterSpecs}} (limit=100,
> table with 101 partitions): exceed (no filter), at-limit boundary (exactly
> 100 succeeds), BY_NAMES exceed, BY_NAMES requested-count exceed (150
> requested / 50 existing — pins the strict semantics), BY_VALUES exceed,
> BY_EXPR exceed, and a JDO-path variant (DirectSQL disabled). All 32 tests
> plus the related suites (\{{TestRemoteHiveMetaStore}},
> \{{TestEmbeddedHiveMetaStore}}, \{{TestHmsServerAuthorization}}) pass.
> h3. Follow-ups
> * Push the limit down into
> \{{MetaStoreDirectSql.getPartitionIdsViaSqlFilter(max)}} (the projection path
> currently passes \{{null}}) to bound the ID query with limit+1 in one round
> trip and remove the extra count query + TOCTOU window.
> * Related: HIVE-30195 — the same RPC also skips the table-level filter hook
> check (\{{checkDbAndTableFilters}}); same class of gap.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
