[
https://issues.apache.org/jira/browse/HIVE-18227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16280956#comment-16280956
]
Daniel Dai commented on HIVE-18227:
-----------------------------------
In the above sql statement, Hive launches 3 DAG simultaneously to merge small
files, within the same session. Hive assume every session use the same AM and
thus we get issue. Ideally we shall detect idleness of the AM, and launch
additional AM as needed, like Pig does. But consider this is a niche case (when
overriding hive.exec.parallel=true), a simpler fix might be ok.
For the SessionState synchronization, I didn't fully get what could be the
problem, but I am totally fine to synchronize on other object.
> Tez parallel execution fail
> ---------------------------
>
> Key: HIVE-18227
> URL: https://issues.apache.org/jira/browse/HIVE-18227
> Project: Hive
> Issue Type: Bug
> Components: Tez
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Attachments: HIVE-18227.1.patch
>
>
> Running tez Dag in parallel within a session fail. Here is the test case:
> {code}
> set hive.exec.parallel=true;
> set hive.merge.tezfiles=true;
> set tez.grouping.max-size=100000;
> set tez.grouping.min-size=10000;
> from student
> insert overwrite table student4 select *
> insert overwrite table student5 select *
> insert overwrite table student6 select *;
> {code}
> The merge task run in parallel and result the exception:
> {code}
> org.apache.tez.dag.api.TezException: App master already running a DAG
> at
> org.apache.tez.dag.app.DAGAppMaster.submitDAGToAppMaster(DAGAppMaster.java:1255)
> at
> org.apache.tez.dag.api.client.DAGClientHandler.submitDAG(DAGClientHandler.java:118)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.submitDAG(DAGClientAMProtocolBlockingPBServerImpl.java:161)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7471)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2273)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2267)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
