Hi Zsolt, Thanks for bringing this up! The high volume of database calls during token cleanup across multiple HMS instances is a significant pain point for large clusters.
Here is my feedback on the proposed options: - Option 1 & 2: Combining a renewal date column (Option 1) with batching (Option 2) is the most robust approach, efficiently handling this on the DB side without pounding the metastore with 100k individual calls. - Option 3: Fetching both token IDs and full tokens in a single bulk query raises security concerns about loading sensitive token material into memory at once and doubles transferred data. - Option 4: Passing large collections of IDs in WHERE token_ident IN (...) clauses will quickly hit SQL statement length limits or degrade query planner performance given the varchar(767) size. A few additional ideas to consider alongside Options 1 and 2: - Cursor-based pagination / Streaming: Avoids loading massive result sets into memory and keeps HMS memory usage flat. - Asynchronous / Rate-limited worker: Running cleanup via a background worker thread with rate limiting and random jitter across HMS instances would prevent thundering herd problems from concurrent metastores. My opinon: Combining Options 1 and 2 with batching seems like a solid direction forward. Thanks, Sai On Mon, Oct 5, 2026 at 6:16 AM Zsolt Miskolczi <[email protected]> wrote: > Hi community, > > I'm planning to pick up HIVE-27260 > <https://issues.apache.org/jira/browse/HIVE-27260> and please let me > bring up some possible ways to solve the problem. > > So, the issue that we faced at multiple customers is when the expired > token cleanup and renewal process ( > https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/security/TokenStoreDelegationTokenSecretManager.java#L263) > runs getToken for each token id previously collected in > tokenStore.getAllDelegationTokenIdentifiers(). > > In a really bad case scenario, assuming the customer has ~200k tokens and > only half of them pass to that point, it is ~100k calls to the metastore > database. And it is worse if they have multiple HMS instances and the clean > up fires in an overlapping time window. > > I see multiple possible ways to handle the issue, let me bring some of > them up. As I'm mostly familiar with the DbTokenStore I'm > especially curious if the possible solutions can potentially bring up > doubts at ZooKeeperTokenStore. > > So, the possible solutions came into my mind: > > 1.: Modify the data structure: Add renew date as a new column. So that, > getAllDelegationTokenIdentifier can be modified to contain the renew date > and we can make a decision about removing or renewing with a single > database call. Please note, with this, there will still be large reads (a > full table scan, no filtering or batching). But at least there will be only > one call, not 100k. > > 2.: Combine possible solution 1 with batching. Multiple small calls, like > 5k items in a batch. Honestly, I would do it in case solution 1 still has > performance problems due to the large amount of data transferred in a > single call. > > 3.: Instead of using getAllDelegationTokenIdentifiers() introduce a new > method that queries both the ids and the tokens together (basically the > whole table). The drawback is the traffic will be doubled (both of the > columns are varchar(767)). And also, I'm not completely sure if it can lead > to security issues (querying both the token ids and the tokens theirself > together). > > 4.: Instead of using a single getToken() call for each record, do it in > batches, like 1000 tokens in a batch. The problem with this solution is the > size of the token id (varchar(767)). Using this solution can lead to really > huge select texts (where token_ident in (....)). > > I personally think solution 1 can have the best performance but as it also > means schema change, it is the solution that will be hard to modify later. > Solution 4 can look like an easy solution but due to the large size of the > id, I believe we should choose a really small batch size and after all, > there would be no significant performance gain. > > Now I want to start an open discussion about the possible solutions. In > case there will be no second thoughts against that, I'm willing to start > developing solution 1. > > > So, my question is: what is your opinion? > > Thank you, > Zsolt >
