BsoBird commented on PR #4946: URL: https://github.com/apache/hive/pull/4946#issuecomment-1856015663
> > @deniskuzZ However, although this solves the problem, I have another doubt. If we use static cache, then the elements in SplitGroup.cache[] will never be deleted, when the user changes the partition, the elements here are not updated, so is there a problem? For PRE-JOB this is not too much of a problem, but for LLAP it is clearly very problematic. So. really. maybe set cache is non-static is a not bad idea.At least this way, the data in the cache won't be leaked. > > Yes, I think the static cache will never be cleaned for LLAP, so this maybe lead some issue. But it seems this cache has been in there for long long time ([HIVE-8409](https://issues.apache.org/jira/browse/HIVE-8409) -> [HIVE-9976](https://issues.apache.org/jira/browse/HIVE-9976)), and i don't have deep knowledge about this code at present. I'd like to hear others opinions. yea. Currently, the solution I'm using in our production environment is to use a non-static cache. So far it's working fine and I haven't observed any OOM issues due to the large number of partitions. Incidentally, our ICEBERG table has close to 100,000 partitions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
