Gopal V created HIVE-4436:
-----------------------------
Summary: hive.exec.parallel=true doesn't work on hadoop-2
Key: HIVE-4436
URL: https://issues.apache.org/jira/browse/HIVE-4436
Project: Hive
Issue Type: Bug
Components: Query Processor
Environment: Ubuntu LXC (hive-trunk)
Reporter: Gopal V
While running a hive query with multiple independent stages, hive.exec.parallel
is a valid optimization to use.
The query tested has 3 MR jobs - the first job is the root dependency and the 2
further job depend on the first one.
When hive.exec.parallel is turned on, the job fails with the following exception
{code}
java.io.IOException: java.lang.InterruptedException
at org.apache.hadoop.ipc.Client.call(Client.java:1214)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
at $Proxy12.mkdirs(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
at $Proxy12.mkdirs(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:447)
at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2165)
at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2136)
at
org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:544)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1916)
at
org.apache.hadoop.hive.ql.exec.ExecDriver.createTmpDirs(ExecDriver.java:222)
at
org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:444)
at
org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:138)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:145)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:47)
Caused by: java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1279)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:921)
at org.apache.hadoop.ipc.Client.call(Client.java:1208)
{code}
The query plan is as follows
{code}
Stage-9 is a root stage
Stage-8 depends on stages: Stage-9
Stage-3 depends on stages: Stage-8
Stage-0 depends on stages: Stage-3
Stage-4 depends on stages: Stage-0
Stage-5 depends on stages: Stage-8
Stage-1 depends on stages: Stage-5
Stage-6 depends on stages: Stage-1
STAGE PLANS:
Stage: Stage-9
Map Reduce Local Work
Stage: Stage-8
Map Reduce
Map Join Operator
Stage: Stage-3
Map Reduce
Stage: Stage-0
Move Operator
Stage: Stage-4
Stats-Aggr Operator
Stage: Stage-5
Map Reduce
Stage: Stage-1
Move Operator
Stage: Stage-6
Stats-Aggr Operator
{code}
I cannot conclude that this is purely a hive issue, will file a bug on HDFS if
that does show up during triage.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
