Sergey Shelukhin created HIVE-18938:

             Summary: LLAP: cache priorities (soft pin, exclusions) 
                 Key: HIVE-18938
             Project: Hive
          Issue Type: Bug
            Reporter: Sergey Shelukhin

We want persistent configuration and not an ad-hoc pin API because the latter 
is both unreliable (e.g. node might go down and restart, so someone has to keep 
track of global state anyway - might as well have it done explicitly), and also 
dangerous because errant pin calls can fill cache with garbage and make it 

So, the idea is that we can allow to
1) Give some tables or partitions (tags, see HIVE-16992) a priority boost at 
cache time.
2) Exclude some tables or partitions from being cached.

Ideally we also want this changeable without LLAP restart (and managed by HS2). 
So the choice here is either make this a config setting, like many Hadoop 
settings that have user/group/etc lists, and implement ZK-based/HS2-based 
configuration changes without restart, that we want to do anyway at some point;
or make this a separate thing, probably managed and stored as policies in 
metastore (kind of like workload management) and update via ZK.

It's possible to go with config settings that do require restart for the first 

Separately, we can add a workload management policy that would disable cache 
for certain query pools (i.e. users/groups/apps).

This message was sent by Atlassian JIRA

Reply via email to