[jira] [Commented] (HIVE-12376) make hive.compactor.worker.threads use a thread pool, etc

2017-02-24 Thread Eugene Koifman (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-12376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883468#comment-15883468
 ] 

Eugene Koifman commented on HIVE-12376:
---

Together with HIVE-11685 it probably better to simply rely on 
_hive.compactor.job.queue_ for resource management and have the Worker just 
submit jobs to it.
Then we can have 1 polling thread to scan COMPACTION_QUEUE  for running jobs 
and check with Yarn if they are done.
Number of Worker thread could default to 5 or something since generating a 
compaction job may take time if there are very many deltas, it should not be 
used to throttle number of jobs.



> make hive.compactor.worker.threads use a thread pool, etc
> -
>
> Key: HIVE-12376
> URL: https://issues.apache.org/jira/browse/HIVE-12376
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> # use a thread pool with core/max capacities instead of creating all threads 
> upfront
> # make sure there is a limit (1000 threads?)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-12376) make hive.compactor.worker.threads use a thread pool, etc

2015-11-10 Thread Alan Gates (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-12376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999120#comment-14999120
 ] 

Alan Gates commented on HIVE-12376:
---

The default min/max for server threads in the metastore is 200/1000.  Given 
that we don't want compaction to eat too many metastore resources it would seem 
to make sense to set the limit as a percentage of the maximum (maybe 10%?).

> make hive.compactor.worker.threads use a thread pool, etc
> -
>
> Key: HIVE-12376
> URL: https://issues.apache.org/jira/browse/HIVE-12376
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> # use a thread pool with core/max capacities instead of creating all threads 
> upfront
> # make sure there is a limit (1000 threads?)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)