[
https://issues.apache.org/jira/browse/HIVE-18264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383961#comment-16383961
]
Alexander Kolbasov commented on HIVE-18264:
-------------------------------------------
I have some higher-level concerns with the pre-warn mechanism. HMS data can be
pretty big and pre-warm keeps it all in memory whether it is needed or not.
Essentially now we have in-memory HMS which uses RDBMS for persistency.
I can understand the use of pre-warm in the case where a table is accessed - it
is likely that its partition data will be accessed as well. But doing this
willy-nilly for all objects seems dangerous in terms of potential memory
consumption impact.
[~pvary] [~alex.behm] What do you think?
> CachedStore: Store cached partitions/col stats within the table cache and
> make prewarm non-blocking
> ---------------------------------------------------------------------------------------------------
>
> Key: HIVE-18264
> URL: https://issues.apache.org/jira/browse/HIVE-18264
> Project: Hive
> Issue Type: Bug
> Reporter: Vaibhav Gumashta
> Assignee: Vaibhav Gumashta
> Priority: Major
> Attachments: HIVE-18264.1.patch, HIVE-18264.2.patch,
> HIVE-18264.3.patch, HIVE-18264.4.patch, HIVE-18264.5.patch
>
>
> Currently we have a separate cache for partitions and partition col stats
> which results in some calls iterating through each of these for
> retrieving/updating. We can get better performance by organizing
> hierarchically. We should also make prewarm non-blocking
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
