Alexander Kolbasov commented on HIVE-18264:

I have some higher-level concerns with the pre-warn mechanism. HMS data can be 
pretty big and pre-warm keeps it all in memory whether it is needed or not. 
Essentially now we have in-memory HMS which uses RDBMS for persistency.

I can understand the use of pre-warm in the case where a table is accessed - it 
is likely that its partition data will be accessed as well. But doing this 
willy-nilly for all objects seems dangerous in terms of potential memory 
consumption impact.

[~pvary] [~alex.behm] What do you think?

> CachedStore: Store cached partitions/col stats within the table cache and 
> make prewarm non-blocking
> ---------------------------------------------------------------------------------------------------
>                 Key: HIVE-18264
>                 URL: https://issues.apache.org/jira/browse/HIVE-18264
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Vaibhav Gumashta
>            Assignee: Vaibhav Gumashta
>            Priority: Major
>         Attachments: HIVE-18264.1.patch, HIVE-18264.2.patch, 
> HIVE-18264.3.patch, HIVE-18264.4.patch, HIVE-18264.5.patch
> Currently we have a separate cache for partitions and partition col stats 
> which results in some calls iterating through each of these for 
> retrieving/updating. We can get better performance by organizing 
> hierarchically. We should also make prewarm non-blocking

This message was sent by Atlassian JIRA

Reply via email to