[
https://issues.apache.org/jira/browse/HIVE-15803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15855353#comment-15855353
]
Nandakumar commented on HIVE-15803:
-----------------------------------
We can split the {{hive.mv.files.thread}} count and create different pool for
each level (depth)
{code}
private List<ExecutorService> getExecutorServiceList(int
numberOfExecutorService, int totalThreads) {
int average = totalThreads / numberOfExecutorService;
int remainingThreads = totalThreads;
List<ExecutorService> serviceList = new
ArrayList<ExecutorService>(numberOfExecutorService);
while(numberOfExecutorService == 0) {
--numberOfExecutorService;
int numberOfThreads = numberOfExecutorService != 0 ? average :
remainingThreads;
serviceList.add(Executors.newFixedThreadPool(numberOfThreads, new
ThreadFactoryBuilder().setDaemon(true).setNameFormat("MSCK-GetPaths-%d").build()));
remainingThreads = remainingThreads - numberOfThreads;
}
return serviceList;
}
{code}
Now instead of passing {{pool}} to {{private void checkPartitionDirs(final
ExecutorService pool, final ConcurrentLinkedQueue<Path> basePaths, final
Set<Path> allDirs, final FileSystem fs, final int depth, final int maxDepth)}},
we can pass {{List<ExecutorService> poolList}} and for accessing the pool we
can use {{poolList\[depth\]#submit}}
> msck can hang when nested partitions are present
> ------------------------------------------------
>
> Key: HIVE-15803
> URL: https://issues.apache.org/jira/browse/HIVE-15803
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Priority: Minor
>
> Steps to reproduce.
> {noformat}
> CREATE TABLE `repairtable`( `col` string) PARTITIONED BY ( `p1` string,
> `p2` string)
> hive> dfs -mkdir -p /apps/hive/warehouse/test.db/repairtable/p1=c/p2=a/p3=b;
> hive> dfs -touchz
> /apps/hive/warehouse/test.db/repairtable/p1=c/p2=a/p3=b/datafile;
> hive> set hive.mv.files.thread;
> hive.mv.files.thread=15
> hive> set hive.mv.files.thread=1;
> hive> MSCK TABLE repairtable;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
