Hi community, I'm planning to pick up HIVE-27260 <https://issues.apache.org/jira/browse/HIVE-27260> and please let me bring up some possible ways to solve the problem.
So, the issue that we faced at multiple customers is when the expired token cleanup and renewal process ( https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/security/TokenStoreDelegationTokenSecretManager.java#L263) runs getToken for each token id previously collected in tokenStore.getAllDelegationTokenIdentifiers(). In a really bad case scenario, assuming the customer has ~200k tokens and only half of them pass to that point, it is ~100k calls to the metastore database. And it is worse if they have multiple HMS instances and the clean up fires in an overlapping time window. I see multiple possible ways to handle the issue, let me bring some of them up. As I'm mostly familiar with the DbTokenStore I'm especially curious if the possible solutions can potentially bring up doubts at ZooKeeperTokenStore. So, the possible solutions came into my mind: 1.: Modify the data structure: Add renew date as a new column. So that, getAllDelegationTokenIdentifier can be modified to contain the renew date and we can make a decision about removing or renewing with a single database call. Please note, with this, there will still be large reads (a full table scan, no filtering or batching). But at least there will be only one call, not 100k. 2.: Combine possible solution 1 with batching. Multiple small calls, like 5k items in a batch. Honestly, I would do it in case solution 1 still has performance problems due to the large amount of data transferred in a single call. 3.: Instead of using getAllDelegationTokenIdentifiers() introduce a new method that queries both the ids and the tokens together (basically the whole table). The drawback is the traffic will be doubled (both of the columns are varchar(767)). And also, I'm not completely sure if it can lead to security issues (querying both the token ids and the tokens theirself together). 4.: Instead of using a single getToken() call for each record, do it in batches, like 1000 tokens in a batch. The problem with this solution is the size of the token id (varchar(767)). Using this solution can lead to really huge select texts (where token_ident in (....)). I personally think solution 1 can have the best performance but as it also means schema change, it is the solution that will be hard to modify later. Solution 4 can look like an easy solution but due to the large size of the id, I believe we should choose a really small batch size and after all, there would be no significant performance gain. Now I want to start an open discussion about the possible solutions. In case there will be no second thoughts against that, I'm willing to start developing solution 1. So, my question is: what is your opinion? Thank you, Zsolt
