[jira] [Updated] (HIVE-14244) bucketmap right outer join query throws ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-14244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhiyuan Yang updated HIVE-14244:
Status: Patch Available (was: Open)
> bucketmap right outer join query throws ArrayIndexOutOfBoundsException
> --
>
> Key: HIVE-14244
> URL: https://issues.apache.org/jira/browse/HIVE-14244
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 1.3.0
>Reporter: Jagruti Varia
>Assignee: Zhiyuan Yang
> Attachments: HIVE-14244.1.patch
>
>
> bucketmap right outer join on partitioned bucketed table throws this error:
> {noformat}
> Vertex failed, vertexName=Map 1, vertexId=vertex_1466710232033_0539_6_00,
> diagnostics=[Task failed, taskId=task_1466710232033_0539_6_00_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
> failure ) :
> attempt_1466710232033_0539_6_00_00_0:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:95)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:70)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:393)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> ... 14 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:850)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
> ... 17 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ArrayIndexOutOfBoundsException: -1
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:416)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorReduceSinkOperator.process(VectorReduceSinkOperator.java:104)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:762)
> ... 18 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
> at
> org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.write(UnorderedPartitionedKVWriter.java:314)
> at
> org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.write(UnorderedPartitionedKVWriter.java:257)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor$TezKVOutputCollector.collect(TezProcessor.java:253)
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.collect(ReduceSinkOperator.java:552)
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:398)
> ... 22 more
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
> attempt_1466710232033_0539_6_00_00_1:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
>
[jira] [Assigned] (HIVE-15570) LLAP: Exception in HostAffinitySplitLocationProvider when running in container mode
[
https://issues.apache.org/jira/browse/HIVE-15570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhiyuan Yang reassigned HIVE-15570:
---
Assignee: Zhiyuan Yang
> LLAP: Exception in HostAffinitySplitLocationProvider when running in
> container mode
> ---
>
> Key: HIVE-15570
> URL: https://issues.apache.org/jira/browse/HIVE-15570
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Rajesh Balamohan
>Assignee: Zhiyuan Yang
>Priority: Minor
> Attachments: HIVE-15570.1.patch
>
>
> Sometimes user might prefer to run with "hive.execution.mode=container" mode
> when LLAP is stopped. If hive config for LLAP had
> "hive.llap.client.consistent.splits=true" in client side, it would end up
> throwing the following exception in {{Utils.java}}.
> {noformat}
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:68)
> ... 25 more
> Caused by: java.lang.IllegalStateException:
> org.apache.hadoop.hive.ql.exec.tez.HostAffinitySplitLocationProvider needs at
> least 1 location to function
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:149)
> at
> org.apache.hadoop.hive.ql.exec.tez.HostAffinitySplitLocationProvider.(HostAffinitySplitLocationProvider.java:52)
> at
> org.apache.hadoop.hive.ql.exec.tez.Utils.getSplitLocationProvider(Utils.java:54)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.(HiveSplitGenerator.java:121)
> ... 30 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15570) LLAP: Exception in HostAffinitySplitLocationProvider when running in container mode
[
https://issues.apache.org/jira/browse/HIVE-15570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhiyuan Yang updated HIVE-15570:
Attachment: HIVE-15570.2.patch
Same exception may be thrown in following cases and is hard to debug.
1. llap mode is used but there is no llap daemon
2. llap mode is used but llap daemon is during recovery
3. container mode is used but hive.llap.client.consistent.splits is true
In the new patch, hive.llap.client.consistent.splits won't be effective if
container mode is used. If llap mode is used but there is no running daemon, we
fall back to locations provided by splits. Then If there is no llap daemon at
all, LlapTaskSchedulerService will detect this and report "No LLAP Daemons are
running"; or if llap daemon finish recovery, query can still succeed.
> LLAP: Exception in HostAffinitySplitLocationProvider when running in
> container mode
> ---
>
> Key: HIVE-15570
> URL: https://issues.apache.org/jira/browse/HIVE-15570
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Rajesh Balamohan
>Assignee: Zhiyuan Yang
>Priority: Minor
> Attachments: HIVE-15570.1.patch, HIVE-15570.2.patch
>
>
> Sometimes user might prefer to run with "hive.execution.mode=container" mode
> when LLAP is stopped. If hive config for LLAP had
> "hive.llap.client.consistent.splits=true" in client side, it would end up
> throwing the following exception in {{Utils.java}}.
> {noformat}
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:68)
> ... 25 more
> Caused by: java.lang.IllegalStateException:
> org.apache.hadoop.hive.ql.exec.tez.HostAffinitySplitLocationProvider needs at
> least 1 location to function
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:149)
> at
> org.apache.hadoop.hive.ql.exec.tez.HostAffinitySplitLocationProvider.(HostAffinitySplitLocationProvider.java:52)
> at
> org.apache.hadoop.hive.ql.exec.tez.Utils.getSplitLocationProvider(Utils.java:54)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.(HiveSplitGenerator.java:121)
> ... 30 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Attachment: HIVE-14731.14.patch Patch rebased. Now we're detecting both SIMPLE_EDGE and CUSTOM_SIMPLE_EDGE in CrossProductHandler. [~hagleitn] > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.10.patch, HIVE-14731.11.patch, > HIVE-14731.12.patch, HIVE-14731.13.patch, HIVE-14731.14.patch, > HIVE-14731.1.patch, HIVE-14731.2.patch, HIVE-14731.3.patch, > HIVE-14731.4.patch, HIVE-14731.5.patch, HIVE-14731.6.patch, > HIVE-14731.7.patch, HIVE-14731.8.patch, HIVE-14731.9.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15570) LLAP: Exception in HostAffinitySplitLocationProvider when running in container mode
[
https://issues.apache.org/jira/browse/HIVE-15570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhiyuan Yang updated HIVE-15570:
Attachment: HIVE-15570.3.patch
Thanks [~sseth] and [~sershe]! I've revised the patch according to your
comments. Now it simply ignores 'hive.llap.client.consistent.splits' in
container mode. If it's in llap mode but no daemon is available, an error will
be thrown.
> LLAP: Exception in HostAffinitySplitLocationProvider when running in
> container mode
> ---
>
> Key: HIVE-15570
> URL: https://issues.apache.org/jira/browse/HIVE-15570
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Rajesh Balamohan
>Assignee: Zhiyuan Yang
>Priority: Minor
> Attachments: HIVE-15570.1.patch, HIVE-15570.2.patch,
> HIVE-15570.3.patch
>
>
> Sometimes user might prefer to run with "hive.execution.mode=container" mode
> when LLAP is stopped. If hive config for LLAP had
> "hive.llap.client.consistent.splits=true" in client side, it would end up
> throwing the following exception in {{Utils.java}}.
> {noformat}
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:68)
> ... 25 more
> Caused by: java.lang.IllegalStateException:
> org.apache.hadoop.hive.ql.exec.tez.HostAffinitySplitLocationProvider needs at
> least 1 location to function
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:149)
> at
> org.apache.hadoop.hive.ql.exec.tez.HostAffinitySplitLocationProvider.(HostAffinitySplitLocationProvider.java:52)
> at
> org.apache.hadoop.hive.ql.exec.tez.Utils.getSplitLocationProvider(Utils.java:54)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.(HiveSplitGenerator.java:121)
> ... 30 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16082) Allow user to change number of listener thread in LlapTaskCommunicator
[ https://issues.apache.org/jira/browse/HIVE-16082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-16082: Attachment: HIVE-16082.1.patch Please help review @[~sseth], [~rajesh.balamohan]. > Allow user to change number of listener thread in LlapTaskCommunicator > -- > > Key: HIVE-16082 > URL: https://issues.apache.org/jira/browse/HIVE-16082 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-16082.1.patch > > > Now LlapTaskCommunicator always has same number of RPC listener thread with > TezTaskCommunicatorImpl. There are scenarios when we want them different: for > example, in Llap only mode, we want less TezTaskCommunicatorImpl's listener > thread to reduce off-heap memory usage. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16082) Allow user to change number of listener thread in LlapTaskCommunicator
[ https://issues.apache.org/jira/browse/HIVE-16082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang reassigned HIVE-16082: --- > Allow user to change number of listener thread in LlapTaskCommunicator > -- > > Key: HIVE-16082 > URL: https://issues.apache.org/jira/browse/HIVE-16082 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > > Now LlapTaskCommunicator always has same number of RPC listener thread with > TezTaskCommunicatorImpl. There are scenarios when we want them different: for > example, in Llap only mode, we want less TezTaskCommunicatorImpl's listener > thread to reduce off-heap memory usage. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16082) Allow user to change number of listener thread in LlapTaskCommunicator
[ https://issues.apache.org/jira/browse/HIVE-16082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-16082: Attachment: (was: HIVE-16082.1.patch) > Allow user to change number of listener thread in LlapTaskCommunicator > -- > > Key: HIVE-16082 > URL: https://issues.apache.org/jira/browse/HIVE-16082 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > > Now LlapTaskCommunicator always has same number of RPC listener thread with > TezTaskCommunicatorImpl. There are scenarios when we want them different: for > example, in Llap only mode, we want less TezTaskCommunicatorImpl's listener > thread to reduce off-heap memory usage. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16082) Allow user to change number of listener thread in LlapTaskCommunicator
[ https://issues.apache.org/jira/browse/HIVE-16082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-16082: Status: Patch Available (was: Open) > Allow user to change number of listener thread in LlapTaskCommunicator > -- > > Key: HIVE-16082 > URL: https://issues.apache.org/jira/browse/HIVE-16082 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-16082.1.patch > > > Now LlapTaskCommunicator always has same number of RPC listener thread with > TezTaskCommunicatorImpl. There are scenarios when we want them different: for > example, in Llap only mode, we want less TezTaskCommunicatorImpl's listener > thread to reduce off-heap memory usage. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16082) Allow user to change number of listener thread in LlapTaskCommunicator
[ https://issues.apache.org/jira/browse/HIVE-16082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-16082: Attachment: HIVE-16082.1.patch > Allow user to change number of listener thread in LlapTaskCommunicator > -- > > Key: HIVE-16082 > URL: https://issues.apache.org/jira/browse/HIVE-16082 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-16082.1.patch > > > Now LlapTaskCommunicator always has same number of RPC listener thread with > TezTaskCommunicatorImpl. There are scenarios when we want them different: for > example, in Llap only mode, we want less TezTaskCommunicatorImpl's listener > thread to reduce off-heap memory usage. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Attachment: HIVE-14731.2.patch Remove unnecessary log format change, which should fix most of test failures. > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Attachment: HIVE-14731.3.patch Modified jsonexplain to support XPROD_EDGE and overwrite some XPROD_EDGE related qtest output file. > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Attachment: HIVE-14731.1.patch Here is the patch based on initial patch provided by [~hagleitn]. It modifies root pom so that we can test it against tez 0.9.0-SNAPSHOT. A qtest has been added into mini llap qtest. > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Status: Patch Available (was: Open) > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Attachment: HIVE-14731.4.patch All failed tests beyond testCliDriver[dynamic_partition_pruning] are not age 1. Also there is no dynamic_partition_pruning test for testCliDriver, no sure why this test was ran. Reupload patch to trigger test run for debugging. > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15494520#comment-15494520 ] Zhiyuan Yang commented on HIVE-14731: - [~hagleitn] Any hint about the dynamic_partition_pruning test failure? I don't know much about this part. > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514904#comment-15514904 ] Zhiyuan Yang commented on HIVE-14731: - Test failures are irrelevant. > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch, > HIVE-14731.6.patch, HIVE-14731.7.patch, HIVE-14731.8.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517029#comment-15517029 ] Zhiyuan Yang commented on HIVE-14731: - Would you mind taking a review? [~hagleitn] > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch, > HIVE-14731.6.patch, HIVE-14731.7.patch, HIVE-14731.8.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Attachment: HIVE-14731.7.patch rebase patch to fix build failure > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch, > HIVE-14731.6.patch, HIVE-14731.7.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Attachment: HIVE-14731.5.patch > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Status: Open (was: Patch Available) > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Attachment: HIVE-14731.6.patch Fixed testconfiguration.properties > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch, HIVE-14731.6.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Status: Patch Available (was: Open) > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch, HIVE-14731.6.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Attachment: HIVE-14731.8.patch Upload patch to fix TestMiniLlapCliDriver[cross_join]. Other test failures are irrelevant. > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch, > HIVE-14731.6.patch, HIVE-14731.7.patch, HIVE-14731.8.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14951) ArrayIndexOutOfBoundsException in GroupByOperator
[
https://issues.apache.org/jira/browse/HIVE-14951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573083#comment-15573083
]
Zhiyuan Yang commented on HIVE-14951:
-
Here's the error message:
{code:java}
Error: Error while running task ( failure ) :
attempt_1475017598908_0282_2_02_00_3:java.lang.RuntimeException:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error while processing row (tag=0) {"key":{"_col0":2},"value":null}
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row (tag=0) {"key":{"_col0":2},"value":null}
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:284)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:252)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150)
... 14 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while processing row (tag=0) {"key":{"_col0":2},"value":null}
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:352)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:274)
... 16 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error while processing row (tag=1) {"key":{"_col0":1},"value":null}
at
org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:416)
at
org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:379)
at
org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.doFirstFetchIfNeeded(CommonMergeJoinOperator.java:485)
at
org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.process(CommonMergeJoinOperator.java:207)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1016)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.processAggr(GroupByOperator.java:821)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:695)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:761)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:343)
... 17 more
Caused by: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row (tag=1) {"key":{"_col0":1},"value":null}
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:284)
at
org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:404)
... 26 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while processing row (tag=1) {"key":{"_col0":1},"value":null}
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:352)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:274)
... 27 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:708)
at
[jira] [Updated] (HIVE-14951) ArrayIndexOutOfBoundsException in GroupByOperator (Hive on Tez)
[ https://issues.apache.org/jira/browse/HIVE-14951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14951: Summary: ArrayIndexOutOfBoundsException in GroupByOperator (Hive on Tez) (was: ArrayIndexOutOfBoundsException in GroupByOperator) > ArrayIndexOutOfBoundsException in GroupByOperator (Hive on Tez) > --- > > Key: HIVE-14951 > URL: https://issues.apache.org/jira/browse/HIVE-14951 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang > > Query: > select * from (select distinct a from f16) as f16, (select distinct a from > f1) as fprime where f16.a = fprime.a; > Table: > create table f1 (a int, b string); > create table f16 (a int, b string); > Config: > set hive.auto.convert.sortmerge.join=true; > set hive.auto.convert.join=false; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14951) ArrayIndexOutOfBoundsException in GroupByOperator (Hive on Tez)
[ https://issues.apache.org/jira/browse/HIVE-14951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14951: Description: Engine: Tez Query: select * from (select distinct a from f16) as f16, (select distinct a from f1) as fprime where f16.a = fprime.a; Table: create table f1 (a int, b string); create table f16 (a int, b string); Config: set hive.auto.convert.sortmerge.join=true; set hive.auto.convert.join=false; was: Query: select * from (select distinct a from f16) as f16, (select distinct a from f1) as fprime where f16.a = fprime.a; Table: create table f1 (a int, b string); create table f16 (a int, b string); Config: set hive.auto.convert.sortmerge.join=true; set hive.auto.convert.join=false; > ArrayIndexOutOfBoundsException in GroupByOperator (Hive on Tez) > --- > > Key: HIVE-14951 > URL: https://issues.apache.org/jira/browse/HIVE-14951 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang > > Engine: > Tez > Query: > select * from (select distinct a from f16) as f16, (select distinct a from > f1) as fprime where f16.a = fprime.a; > Table: > create table f1 (a int, b string); > create table f16 (a int, b string); > Config: > set hive.auto.convert.sortmerge.join=true; > set hive.auto.convert.join=false; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14951) ArrayIndexOutOfBoundsException in GroupByOperator
[ https://issues.apache.org/jira/browse/HIVE-14951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14951: Summary: ArrayIndexOutOfBoundsException in GroupByOperator (was: ArrayIndexOutOfBoundsException in GroupByOperator (Hive on Tez)) > ArrayIndexOutOfBoundsException in GroupByOperator > - > > Key: HIVE-14951 > URL: https://issues.apache.org/jira/browse/HIVE-14951 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang > > Engine: > Tez > Query: > select * from (select distinct a from f16) as f16, (select distinct a from > f1) as fprime where f16.a = fprime.a; > Table: > create table f1 (a int, b string); > create table f16 (a int, b string); > Config: > set hive.auto.convert.sortmerge.join=true; > set hive.auto.convert.join=false; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Attachment: HIVE-14731.10.patch > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.10.patch, > HIVE-14731.2.patch, HIVE-14731.3.patch, HIVE-14731.4.patch, > HIVE-14731.5.patch, HIVE-14731.6.patch, HIVE-14731.7.patch, > HIVE-14731.8.patch, HIVE-14731.9.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15727177#comment-15727177 ] Zhiyuan Yang commented on HIVE-14731: - Sorry I didn't know left outer join can also turn out a cross product. Will modify the patch to limit its scope to cross join. > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch, > HIVE-14731.6.patch, HIVE-14731.7.patch, HIVE-14731.8.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Status: Open (was: Patch Available) > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch, > HIVE-14731.6.patch, HIVE-14731.7.patch, HIVE-14731.8.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15742755#comment-15742755 ] Zhiyuan Yang commented on HIVE-14731: - [~jcamachorodriguez], I'm not sure whether it's a good idea to enable this new feature by default for user, but I'll use [~hagleitn]'s way to enable it for tests first. > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.10.patch, > HIVE-14731.2.patch, HIVE-14731.3.patch, HIVE-14731.4.patch, > HIVE-14731.5.patch, HIVE-14731.6.patch, HIVE-14731.7.patch, > HIVE-14731.8.patch, HIVE-14731.9.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Status: Patch Available (was: Open) > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.10.patch, > HIVE-14731.2.patch, HIVE-14731.3.patch, HIVE-14731.4.patch, > HIVE-14731.5.patch, HIVE-14731.6.patch, HIVE-14731.7.patch, > HIVE-14731.8.patch, HIVE-14731.9.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Status: Open (was: Patch Available) Resubmit patch for jenkins test. > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.10.patch, > HIVE-14731.2.patch, HIVE-14731.3.patch, HIVE-14731.4.patch, > HIVE-14731.5.patch, HIVE-14731.6.patch, HIVE-14731.7.patch, > HIVE-14731.8.patch, HIVE-14731.9.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Attachment: HIVE-14731.9.patch Upload new patch to address comments. A config has been added to enable cartesian product edge. By default it's disabled. > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch, > HIVE-14731.6.patch, HIVE-14731.7.patch, HIVE-14731.8.patch, HIVE-14731.9.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Status: Patch Available (was: Open) > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch, > HIVE-14731.6.patch, HIVE-14731.7.patch, HIVE-14731.8.patch, HIVE-14731.9.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Status: Open (was: Patch Available) > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch, > HIVE-14731.6.patch, HIVE-14731.7.patch, HIVE-14731.8.patch, HIVE-14731.9.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723819#comment-15723819 ] Zhiyuan Yang commented on HIVE-14731: - Test failures are irrelevant. > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch, > HIVE-14731.6.patch, HIVE-14731.7.patch, HIVE-14731.8.patch, HIVE-14731.9.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Attachment: HIVE-14731.12.patch Upload new patch to enable tez cartesian product edge for llap qtest. > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.10.patch, > HIVE-14731.11.patch, HIVE-14731.12.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch, > HIVE-14731.6.patch, HIVE-14731.7.patch, HIVE-14731.8.patch, HIVE-14731.9.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Attachment: HIVE-14731.13.patch > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.10.patch, > HIVE-14731.11.patch, HIVE-14731.12.patch, HIVE-14731.13.patch, > HIVE-14731.2.patch, HIVE-14731.3.patch, HIVE-14731.4.patch, > HIVE-14731.5.patch, HIVE-14731.6.patch, HIVE-14731.7.patch, > HIVE-14731.8.patch, HIVE-14731.9.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15768148#comment-15768148 ] Zhiyuan Yang commented on HIVE-14731: - Thanks [~hagleitn] for review! Final patch has been uploaded. > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.10.patch, > HIVE-14731.11.patch, HIVE-14731.12.patch, HIVE-14731.13.patch, > HIVE-14731.2.patch, HIVE-14731.3.patch, HIVE-14731.4.patch, > HIVE-14731.5.patch, HIVE-14731.6.patch, HIVE-14731.7.patch, > HIVE-14731.8.patch, HIVE-14731.9.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Attachment: HIVE-14731.11.patch Upload new patch to address comments: cross product edge is now enabled by default for unit test; more qtests related to cross product has been added. > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.10.patch, > HIVE-14731.11.patch, HIVE-14731.2.patch, HIVE-14731.3.patch, > HIVE-14731.4.patch, HIVE-14731.5.patch, HIVE-14731.6.patch, > HIVE-14731.7.patch, HIVE-14731.8.patch, HIVE-14731.9.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Attachment: (was: HIVE-14731.9.patch) > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch, > HIVE-14731.6.patch, HIVE-14731.7.patch, HIVE-14731.8.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726858#comment-15726858 ] Zhiyuan Yang commented on HIVE-14731: - Previous patch has severe issue. Will upload new one later. > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch, > HIVE-14731.6.patch, HIVE-14731.7.patch, HIVE-14731.8.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Attachment: HIVE-14731.9.patch Upload new patch to fix the outer join issue. XPROD_EDGE is no longer used for outer join. > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch, > HIVE-14731.6.patch, HIVE-14731.7.patch, HIVE-14731.8.patch, HIVE-14731.9.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Status: Patch Available (was: Open) > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch, > HIVE-14731.6.patch, HIVE-14731.7.patch, HIVE-14731.8.patch, HIVE-14731.9.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-16082) Allow user to change number of listener thread in LlapTaskCommunicator
[ https://issues.apache.org/jira/browse/HIVE-16082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-16082: Attachment: HIVE-16082.2.patch Re-submit patch for testing. > Allow user to change number of listener thread in LlapTaskCommunicator > -- > > Key: HIVE-16082 > URL: https://issues.apache.org/jira/browse/HIVE-16082 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-16082.1.patch, HIVE-16082.2.patch > > > Now LlapTaskCommunicator always has same number of RPC listener thread with > TezTaskCommunicatorImpl. There are scenarios when we want them different: for > example, in Llap only mode, we want less TezTaskCommunicatorImpl's listener > thread to reduce off-heap memory usage. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16082) Allow user to change number of listener thread in LlapTaskCommunicator
[ https://issues.apache.org/jira/browse/HIVE-16082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-16082: Status: Open (was: Patch Available) > Allow user to change number of listener thread in LlapTaskCommunicator > -- > > Key: HIVE-16082 > URL: https://issues.apache.org/jira/browse/HIVE-16082 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-16082.1.patch, HIVE-16082.2.patch > > > Now LlapTaskCommunicator always has same number of RPC listener thread with > TezTaskCommunicatorImpl. There are scenarios when we want them different: for > example, in Llap only mode, we want less TezTaskCommunicatorImpl's listener > thread to reduce off-heap memory usage. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16082) Allow user to change number of listener thread in LlapTaskCommunicator
[ https://issues.apache.org/jira/browse/HIVE-16082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-16082: Status: Patch Available (was: Open) > Allow user to change number of listener thread in LlapTaskCommunicator > -- > > Key: HIVE-16082 > URL: https://issues.apache.org/jira/browse/HIVE-16082 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-16082.1.patch, HIVE-16082.2.patch > > > Now LlapTaskCommunicator always has same number of RPC listener thread with > TezTaskCommunicatorImpl. There are scenarios when we want them different: for > example, in Llap only mode, we want less TezTaskCommunicatorImpl's listener > thread to reduce off-heap memory usage. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-17228) Bump tez version to 0.9.0
[ https://issues.apache.org/jira/browse/HIVE-17228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-17228: Attachment: HIVE-17228.1.patch > Bump tez version to 0.9.0 > - > > Key: HIVE-17228 > URL: https://issues.apache.org/jira/browse/HIVE-17228 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-17228.1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17228) Bump tez version to 0.9.0
[ https://issues.apache.org/jira/browse/HIVE-17228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang reassigned HIVE-17228: --- > Bump tez version to 0.9.0 > - > > Key: HIVE-17228 > URL: https://issues.apache.org/jira/browse/HIVE-17228 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17228) Bump tez version to 0.9.0
[ https://issues.apache.org/jira/browse/HIVE-17228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-17228: Status: Patch Available (was: Open) > Bump tez version to 0.9.0 > - > > Key: HIVE-17228 > URL: https://issues.apache.org/jira/browse/HIVE-17228 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-17228.1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124139#comment-16124139 ] Zhiyuan Yang commented on HIVE-14731: - [~hagleitn] Non deterministic behavior should come from multiple cross product reducers > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.10.patch, HIVE-14731.11.patch, > HIVE-14731.12.patch, HIVE-14731.13.patch, HIVE-14731.14.patch, > HIVE-14731.15.patch, HIVE-14731.16.patch, HIVE-14731.17.patch, > HIVE-14731.18.patch, HIVE-14731.19.patch, HIVE-14731.1.patch, > HIVE-14731.2.patch, HIVE-14731.3.patch, HIVE-14731.4.patch, > HIVE-14731.5.patch, HIVE-14731.6.patch, HIVE-14731.7.patch, > HIVE-14731.8.patch, HIVE-14731.9.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16690) Configure Tez cartesian product edge based on LLAP cluster size
[ https://issues.apache.org/jira/browse/HIVE-16690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-16690: Attachment: HIVE-16690.addendum.patch Upload addendum patch which avoids accessing uninitialized Llap cluster info (which cause NPE). > Configure Tez cartesian product edge based on LLAP cluster size > --- > > Key: HIVE-16690 > URL: https://issues.apache.org/jira/browse/HIVE-16690 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-16690.1.patch, HIVE-16690.addendum.patch > > > In HIVE-14731 we are using default value for target parallelism of fair > cartesian product edge. Ideally this should be set according to cluster size. > In case of LLAP it's pretty easy to get cluster size, i.e., number of > executors. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17047) Allow table property to be populated to jobConf to make FixedLengthInputFormat work
[
https://issues.apache.org/jira/browse/HIVE-17047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhiyuan Yang reassigned HIVE-17047:
---
> Allow table property to be populated to jobConf to make
> FixedLengthInputFormat work
> ---
>
> Key: HIVE-17047
> URL: https://issues.apache.org/jira/browse/HIVE-17047
> Project: Hive
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Fix For: 1.2.1
>
>
> To make FixedLengthInputFormat work in Hive, we need table specific value for
> the configuration "fixedlengthinputformat.record.length". Right now the best
> place would be table property. Unfortunately, table property is not alway
> populated to InputFormat configurations because of this in HiveInputFormat:
> {code}
> PartitionDesc part = pathToPartitionInfo.get(hsplit.getPath().toString());
> if ((part != null) && (part.getTableDesc() != null)) {
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17047) Allow table property to be populated to jobConf to make FixedLengthInputFormat work
[
https://issues.apache.org/jira/browse/HIVE-17047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhiyuan Yang updated HIVE-17047:
Target Version/s: 1.2.1
> Allow table property to be populated to jobConf to make
> FixedLengthInputFormat work
> ---
>
> Key: HIVE-17047
> URL: https://issues.apache.org/jira/browse/HIVE-17047
> Project: Hive
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: HIVE-17047.1.patch
>
>
> To make FixedLengthInputFormat work in Hive, we need table specific value for
> the configuration "fixedlengthinputformat.record.length". Right now the best
> place would be table property. Unfortunately, table property is not alway
> populated to InputFormat configurations because of this in HiveInputFormat:
> {code}
> PartitionDesc part = pathToPartitionInfo.get(hsplit.getPath().toString());
> if ((part != null) && (part.getTableDesc() != null)) {
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17047) Allow table property to be populated to jobConf to make FixedLengthInputFormat work
[
https://issues.apache.org/jira/browse/HIVE-17047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075556#comment-16075556
]
Zhiyuan Yang commented on HIVE-17047:
-
It turns out HIVE-15147 has fix for this accidentally. HIVE-15147 was for Hive
2.2.0 LLAP but not for earlier versions. Uploading a partial patch from
HIVE-15147 for earlier versions.
> Allow table property to be populated to jobConf to make
> FixedLengthInputFormat work
> ---
>
> Key: HIVE-17047
> URL: https://issues.apache.org/jira/browse/HIVE-17047
> Project: Hive
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Fix For: 1.2.1
>
>
> To make FixedLengthInputFormat work in Hive, we need table specific value for
> the configuration "fixedlengthinputformat.record.length". Right now the best
> place would be table property. Unfortunately, table property is not alway
> populated to InputFormat configurations because of this in HiveInputFormat:
> {code}
> PartitionDesc part = pathToPartitionInfo.get(hsplit.getPath().toString());
> if ((part != null) && (part.getTableDesc() != null)) {
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17047) Allow table property to be populated to jobConf to make FixedLengthInputFormat work
[
https://issues.apache.org/jira/browse/HIVE-17047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhiyuan Yang updated HIVE-17047:
Fix Version/s: (was: 1.2.1)
> Allow table property to be populated to jobConf to make
> FixedLengthInputFormat work
> ---
>
> Key: HIVE-17047
> URL: https://issues.apache.org/jira/browse/HIVE-17047
> Project: Hive
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: HIVE-17047.1.patch
>
>
> To make FixedLengthInputFormat work in Hive, we need table specific value for
> the configuration "fixedlengthinputformat.record.length". Right now the best
> place would be table property. Unfortunately, table property is not alway
> populated to InputFormat configurations because of this in HiveInputFormat:
> {code}
> PartitionDesc part = pathToPartitionInfo.get(hsplit.getPath().toString());
> if ((part != null) && (part.getTableDesc() != null)) {
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17047) Allow table property to be populated to jobConf to make FixedLengthInputFormat work
[
https://issues.apache.org/jira/browse/HIVE-17047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhiyuan Yang updated HIVE-17047:
Attachment: HIVE-17047.1.patch
> Allow table property to be populated to jobConf to make
> FixedLengthInputFormat work
> ---
>
> Key: HIVE-17047
> URL: https://issues.apache.org/jira/browse/HIVE-17047
> Project: Hive
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: HIVE-17047.1.patch
>
>
> To make FixedLengthInputFormat work in Hive, we need table specific value for
> the configuration "fixedlengthinputformat.record.length". Right now the best
> place would be table property. Unfortunately, table property is not alway
> populated to InputFormat configurations because of this in HiveInputFormat:
> {code}
> PartitionDesc part = pathToPartitionInfo.get(hsplit.getPath().toString());
> if ((part != null) && (part.getTableDesc() != null)) {
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17047) Allow table property to be populated to jobConf to make FixedLengthInputFormat work
[
https://issues.apache.org/jira/browse/HIVE-17047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhiyuan Yang updated HIVE-17047:
Status: Patch Available (was: Open)
> Allow table property to be populated to jobConf to make
> FixedLengthInputFormat work
> ---
>
> Key: HIVE-17047
> URL: https://issues.apache.org/jira/browse/HIVE-17047
> Project: Hive
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: HIVE-17047.1.patch
>
>
> To make FixedLengthInputFormat work in Hive, we need table specific value for
> the configuration "fixedlengthinputformat.record.length". Right now the best
> place would be table property. Unfortunately, table property is not alway
> populated to InputFormat configurations because of this in HiveInputFormat:
> {code}
> PartitionDesc part = pathToPartitionInfo.get(hsplit.getPath().toString());
> if ((part != null) && (part.getTableDesc() != null)) {
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17228) Bump tez version to 0.9.0
[ https://issues.apache.org/jira/browse/HIVE-17228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-17228: Attachment: HIVE-17228.1.patch > Bump tez version to 0.9.0 > - > > Key: HIVE-17228 > URL: https://issues.apache.org/jira/browse/HIVE-17228 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-17228.1.patch, HIVE-17228.1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17228) Bump tez version to 0.9.0
[ https://issues.apache.org/jira/browse/HIVE-17228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111428#comment-16111428 ] Zhiyuan Yang commented on HIVE-17228: - Test failures are unrelated. Please help review, [~hagleitn], [~sseth] > Bump tez version to 0.9.0 > - > > Key: HIVE-17228 > URL: https://issues.apache.org/jira/browse/HIVE-17228 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-17228.1.patch, HIVE-17228.1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17228) Bump tez version to 0.9.0
[ https://issues.apache.org/jira/browse/HIVE-17228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118818#comment-16118818 ] Zhiyuan Yang commented on HIVE-17228: - Thanks [~hagleitn]! > Bump tez version to 0.9.0 > - > > Key: HIVE-17228 > URL: https://issues.apache.org/jira/browse/HIVE-17228 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Fix For: 3.0.0 > > Attachments: HIVE-17228.1.patch, HIVE-17228.1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16007602#comment-16007602 ] Zhiyuan Yang commented on HIVE-14731: - Thanks [~hagleitn] for review! New patch was uploaded to address your comment. > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.10.patch, HIVE-14731.11.patch, > HIVE-14731.12.patch, HIVE-14731.13.patch, HIVE-14731.14.patch, > HIVE-14731.15.patch, HIVE-14731.16.patch, HIVE-14731.17.patch, > HIVE-14731.1.patch, HIVE-14731.2.patch, HIVE-14731.3.patch, > HIVE-14731.4.patch, HIVE-14731.5.patch, HIVE-14731.6.patch, > HIVE-14731.7.patch, HIVE-14731.8.patch, HIVE-14731.9.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Attachment: HIVE-14731.17.patch > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.10.patch, HIVE-14731.11.patch, > HIVE-14731.12.patch, HIVE-14731.13.patch, HIVE-14731.14.patch, > HIVE-14731.15.patch, HIVE-14731.16.patch, HIVE-14731.17.patch, > HIVE-14731.1.patch, HIVE-14731.2.patch, HIVE-14731.3.patch, > HIVE-14731.4.patch, HIVE-14731.5.patch, HIVE-14731.6.patch, > HIVE-14731.7.patch, HIVE-14731.8.patch, HIVE-14731.9.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Status: Patch Available (was: Open) > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.10.patch, HIVE-14731.11.patch, > HIVE-14731.12.patch, HIVE-14731.13.patch, HIVE-14731.14.patch, > HIVE-14731.15.patch, HIVE-14731.16.patch, HIVE-14731.17.patch, > HIVE-14731.1.patch, HIVE-14731.2.patch, HIVE-14731.3.patch, > HIVE-14731.4.patch, HIVE-14731.5.patch, HIVE-14731.6.patch, > HIVE-14731.7.patch, HIVE-14731.8.patch, HIVE-14731.9.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Attachment: HIVE-14731.18.patch > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.10.patch, HIVE-14731.11.patch, > HIVE-14731.12.patch, HIVE-14731.13.patch, HIVE-14731.14.patch, > HIVE-14731.15.patch, HIVE-14731.16.patch, HIVE-14731.17.patch, > HIVE-14731.18.patch, HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch, > HIVE-14731.6.patch, HIVE-14731.7.patch, HIVE-14731.8.patch, HIVE-14731.9.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Attachment: HIVE-14731.16.patch Upload new patch that won't use map join in case of cross product to get better parallelism. Please review [~hagleitn] > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.10.patch, HIVE-14731.11.patch, > HIVE-14731.12.patch, HIVE-14731.13.patch, HIVE-14731.14.patch, > HIVE-14731.15.patch, HIVE-14731.16.patch, HIVE-14731.1.patch, > HIVE-14731.2.patch, HIVE-14731.3.patch, HIVE-14731.4.patch, > HIVE-14731.5.patch, HIVE-14731.6.patch, HIVE-14731.7.patch, > HIVE-14731.8.patch, HIVE-14731.9.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16710) Make MAX_MS_TYPENAME_LENGTH configurable
[ https://issues.apache.org/jira/browse/HIVE-16710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang reassigned HIVE-16710: --- > Make MAX_MS_TYPENAME_LENGTH configurable > > > Key: HIVE-16710 > URL: https://issues.apache.org/jira/browse/HIVE-16710 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Fix For: 2.2.0 > > > HIVE-11985 introduced type name length check in 2.0.0. Before 2.3 > (HIVE-12274), users have no way to work around this check if they do get very > long type name. We should make max type name length configurable before 2.3. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16710) Make MAX_MS_TYPENAME_LENGTH configurable
[ https://issues.apache.org/jira/browse/HIVE-16710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-16710: Status: Patch Available (was: Open) > Make MAX_MS_TYPENAME_LENGTH configurable > > > Key: HIVE-16710 > URL: https://issues.apache.org/jira/browse/HIVE-16710 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Fix For: 2.2.0 > > Attachments: HIVE-16710.1.patch > > > HIVE-11985 introduced type name length check in 2.0.0. Before 2.3 > (HIVE-12274), users have no way to work around this check if they do get very > long type name. We should make max type name length configurable before 2.3. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16710) Make MAX_MS_TYPENAME_LENGTH configurable
[ https://issues.apache.org/jira/browse/HIVE-16710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-16710: Attachment: HIVE-16710.1.patch Please help review. [~sershe] > Make MAX_MS_TYPENAME_LENGTH configurable > > > Key: HIVE-16710 > URL: https://issues.apache.org/jira/browse/HIVE-16710 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Fix For: 2.2.0 > > Attachments: HIVE-16710.1.patch > > > HIVE-11985 introduced type name length check in 2.0.0. Before 2.3 > (HIVE-12274), users have no way to work around this check if they do get very > long type name. We should make max type name length configurable before 2.3. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16710) Make MAX_MS_TYPENAME_LENGTH configurable
[ https://issues.apache.org/jira/browse/HIVE-16710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16017794#comment-16017794 ] Zhiyuan Yang commented on HIVE-16710: - Do you know how can we trigger branch-2.2 build? [~sershe] > Make MAX_MS_TYPENAME_LENGTH configurable > > > Key: HIVE-16710 > URL: https://issues.apache.org/jira/browse/HIVE-16710 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Fix For: 2.2.0 > > Attachments: HIVE-16710.1.patch, HIVE-16710.2.patch > > > HIVE-11985 introduced type name length check in 2.0.0. Before 2.3 > (HIVE-12274), users have no way to work around this check if they do get very > long type name. We should make max type name length configurable before 2.3. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16710) Make MAX_MS_TYPENAME_LENGTH configurable
[ https://issues.apache.org/jira/browse/HIVE-16710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16017807#comment-16017807 ] Zhiyuan Yang commented on HIVE-16710: - Thanks [~sershe]! Agree that test is not useful. > Make MAX_MS_TYPENAME_LENGTH configurable > > > Key: HIVE-16710 > URL: https://issues.apache.org/jira/browse/HIVE-16710 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Fix For: 2.2.0 > > Attachments: HIVE-16710.1.patch, HIVE-16710.2.patch > > > HIVE-11985 introduced type name length check in 2.0.0. Before 2.3 > (HIVE-12274), users have no way to work around this check if they do get very > long type name. We should make max type name length configurable before 2.3. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16710) Make MAX_MS_TYPENAME_LENGTH configurable
[ https://issues.apache.org/jira/browse/HIVE-16710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-16710: Attachment: HIVE-16710.2.patch Thanks [~sershe] for review! I've upload new patch to address your comment. > Make MAX_MS_TYPENAME_LENGTH configurable > > > Key: HIVE-16710 > URL: https://issues.apache.org/jira/browse/HIVE-16710 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Fix For: 2.2.0 > > Attachments: HIVE-16710.1.patch, HIVE-16710.2.patch > > > HIVE-11985 introduced type name length check in 2.0.0. Before 2.3 > (HIVE-12274), users have no way to work around this check if they do get very > long type name. We should make max type name length configurable before 2.3. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16690) Configure Tez cartesian product edge based on LLAP cluster size
[ https://issues.apache.org/jira/browse/HIVE-16690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang reassigned HIVE-16690: --- > Configure Tez cartesian product edge based on LLAP cluster size > --- > > Key: HIVE-16690 > URL: https://issues.apache.org/jira/browse/HIVE-16690 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > > In HIVE-14731 we are using default value for target parallelism of fair > cartesian product edge. Ideally this should be set according to cluster size. > In case of LLAP it's pretty easy to get cluster size, i.e., number of > executors. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16690) Configure Tez cartesian product edge based on LLAP cluster size
[ https://issues.apache.org/jira/browse/HIVE-16690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-16690: Attachment: HIVE-16690.1.patch Please help review, [~hagleitn], [~sseth] > Configure Tez cartesian product edge based on LLAP cluster size > --- > > Key: HIVE-16690 > URL: https://issues.apache.org/jira/browse/HIVE-16690 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-16690.1.patch > > > In HIVE-14731 we are using default value for target parallelism of fair > cartesian product edge. Ideally this should be set according to cluster size. > In case of LLAP it's pretty easy to get cluster size, i.e., number of > executors. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16690) Configure Tez cartesian product edge based on LLAP cluster size
[ https://issues.apache.org/jira/browse/HIVE-16690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16013378#comment-16013378 ] Zhiyuan Yang commented on HIVE-16690: - Thanks [~sershe] for review! It works in a cluster. Later I'll submit additional patch combining HIVE-14731 and this one for jenkins. > Configure Tez cartesian product edge based on LLAP cluster size > --- > > Key: HIVE-16690 > URL: https://issues.apache.org/jira/browse/HIVE-16690 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-16690.1.patch > > > In HIVE-14731 we are using default value for target parallelism of fair > cartesian product edge. Ideally this should be set according to cluster size. > In case of LLAP it's pretty easy to get cluster size, i.e., number of > executors. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16596) CrossProductCheck failed to detect cross product between two unions
[
https://issues.apache.org/jira/browse/HIVE-16596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhiyuan Yang reassigned HIVE-16596:
---
> CrossProductCheck failed to detect cross product between two unions
> ---
>
> Key: HIVE-16596
> URL: https://issues.apache.org/jira/browse/HIVE-16596
> Project: Hive
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
>
> To reproduce:
> {code}
> create table f (a int, b string);
> set hive.auto.convert.join=false;
> explain select * from (select * from f union all select * from f) a join
> (select * from f union all select * from f) b;
> {code}
> No cross product warning is given.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Status: Open (was: Patch Available) Cancelling patch since TEZ-3708 isn't committed and Jenkins won't be able to compile. > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.10.patch, HIVE-14731.11.patch, > HIVE-14731.12.patch, HIVE-14731.13.patch, HIVE-14731.14.patch, > HIVE-14731.15.patch, HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch, > HIVE-14731.6.patch, HIVE-14731.7.patch, HIVE-14731.8.patch, HIVE-14731.9.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Attachment: HIVE-14731.15.patch Upload new patch that use new unpartitioned cross product edge implemented in TEZ-3708. Key features: 1. allow arbitrary parallelism by partitioning source output 2. estimate the workload based on #record 3. group by #cross-product-operation to evenly distribute the workload CC [~hagleitn] > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.10.patch, HIVE-14731.11.patch, > HIVE-14731.12.patch, HIVE-14731.13.patch, HIVE-14731.14.patch, > HIVE-14731.15.patch, HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch, > HIVE-14731.6.patch, HIVE-14731.7.patch, HIVE-14731.8.patch, HIVE-14731.9.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-17386) support LLAP workload management in HS2 (low level only)
[ https://issues.apache.org/jira/browse/HIVE-17386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165535#comment-16165535 ] Zhiyuan Yang commented on HIVE-17386: - Sure, I will review it soon. > support LLAP workload management in HS2 (low level only) > > > Key: HIVE-17386 > URL: https://issues.apache.org/jira/browse/HIVE-17386 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-17386.01.only.patch, HIVE-17386.01.patch, > HIVE-17386.01.patch, HIVE-17386.02.patch, HIVE-17386.03.patch, > HIVE-17386.04.patch, HIVE-17386.only.patch, HIVE-17386.patch > > > This makes use of HIVE-17297 and creates building blocks for workload > management policies, etc. > For now, there are no policies - a single yarn queue is designated for all > LLAP query AMs, and the capacity is distributed equally. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17641) Visibility issue of Task.done cause Driver skip stages in parallel execution
[
https://issues.apache.org/jira/browse/HIVE-17641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhiyuan Yang updated HIVE-17641:
Attachment: HIVE-17641.3-branch-0.14.patch
Removed check in Driver class since we don't have a way to tell how many stage
to run (due to conditional task)
> Visibility issue of Task.done cause Driver skip stages in parallel execution
>
>
> Key: HIVE-17641
> URL: https://issues.apache.org/jira/browse/HIVE-17641
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.14.1
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: HIVE-17641.1-branch-0.14.patch,
> HIVE-17641.2-branch-0.14.patch, HIVE-17641.3-branch-0.14.patch
>
>
> Task.done is not volatile. In case of parallel execution, TaskRunner thread
> set this value, and Driver thread read this value when it determines whether
> a child task is runnable
> DriverContext.java
> {code}
> public static boolean isLaunchable(Task tsk) {
> return !tsk.getQueued() && !tsk.getInitialized() && tsk.isRunnable();
> {code}
> Task.java
> {code}
> public boolean isRunnable() {
> boolean isrunnable = true;
> if (parentTasks != null) {
> for (Task parent : parentTasks) {
> if (!parent.done()) {
> {code}
> This happens without any synchronization, so a child can be not runnable even
> all parents finish.
> To make it worse, Driver think query is successful when there is no running
> task or runnable task, so query may finish without executing some stages.
> Driver.java
> {code}
> while (!destroyed && driverCxt.isRunning()) {
> {code}
> DriverContext.java
> {code}
> public synchronized boolean isRunning() {
> return !shutdown && (!running.isEmpty() || !runnable.isEmpty());
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17641) Visibility issue of Task.done cause Driver skip stages in parallel execution
[
https://issues.apache.org/jira/browse/HIVE-17641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhiyuan Yang updated HIVE-17641:
Attachment: HIVE-17641.2-branch-0.14.patch
Thanks [~ashutoshc] for review! I've attached new patch to address comment.
Actually that exception msg can be omit because other place already print how
many task in total, how many task has been executed. LOG.info won't be noise.
That function is not called a lot. Anyway, fixed all of them in new patch.
> Visibility issue of Task.done cause Driver skip stages in parallel execution
>
>
> Key: HIVE-17641
> URL: https://issues.apache.org/jira/browse/HIVE-17641
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.14.1
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: HIVE-17641.1-branch-0.14.patch,
> HIVE-17641.2-branch-0.14.patch
>
>
> Task.done is not volatile. In case of parallel execution, TaskRunner thread
> set this value, and Driver thread read this value when it determines whether
> a child task is runnable
> DriverContext.java
> {code}
> public static boolean isLaunchable(Task tsk) {
> return !tsk.getQueued() && !tsk.getInitialized() && tsk.isRunnable();
> {code}
> Task.java
> {code}
> public boolean isRunnable() {
> boolean isrunnable = true;
> if (parentTasks != null) {
> for (Task parent : parentTasks) {
> if (!parent.done()) {
> {code}
> This happens without any synchronization, so a child can be not runnable even
> all parents finish.
> To make it worse, Driver think query is successful when there is no running
> task or runnable task, so query may finish without executing some stages.
> Driver.java
> {code}
> while (!destroyed && driverCxt.isRunning()) {
> {code}
> DriverContext.java
> {code}
> public synchronized boolean isRunning() {
> return !shutdown && (!running.isEmpty() || !runnable.isEmpty());
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17641) Visibility issue of Task.done cause Driver skip stages in parallel execution
[
https://issues.apache.org/jira/browse/HIVE-17641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189069#comment-16189069
]
Zhiyuan Yang commented on HIVE-17641:
-
Thanks [~ashutoshc]!
> Visibility issue of Task.done cause Driver skip stages in parallel execution
>
>
> Key: HIVE-17641
> URL: https://issues.apache.org/jira/browse/HIVE-17641
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.14.1
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: HIVE-17641.1-branch-0.14.patch,
> HIVE-17641.2-branch-0.14.patch
>
>
> Task.done is not volatile. In case of parallel execution, TaskRunner thread
> set this value, and Driver thread read this value when it determines whether
> a child task is runnable
> DriverContext.java
> {code}
> public static boolean isLaunchable(Task tsk) {
> return !tsk.getQueued() && !tsk.getInitialized() && tsk.isRunnable();
> {code}
> Task.java
> {code}
> public boolean isRunnable() {
> boolean isrunnable = true;
> if (parentTasks != null) {
> for (Task parent : parentTasks) {
> if (!parent.done()) {
> {code}
> This happens without any synchronization, so a child can be not runnable even
> all parents finish.
> To make it worse, Driver think query is successful when there is no running
> task or runnable task, so query may finish without executing some stages.
> Driver.java
> {code}
> while (!destroyed && driverCxt.isRunning()) {
> {code}
> DriverContext.java
> {code}
> public synchronized boolean isRunning() {
> return !shutdown && (!running.isEmpty() || !runnable.isEmpty());
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16205373#comment-16205373 ] Zhiyuan Yang commented on HIVE-14731: - Rebased the patch again. Test was clean except for flaky or irrelevant test. Please review. CC [~hagleitn] > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.10.patch, > HIVE-14731.11.patch, HIVE-14731.12.patch, HIVE-14731.13.patch, > HIVE-14731.14.patch, HIVE-14731.15.patch, HIVE-14731.16.patch, > HIVE-14731.17.patch, HIVE-14731.18.patch, HIVE-14731.19.patch, > HIVE-14731.2.patch, HIVE-14731.20.patch, HIVE-14731.21.patch, > HIVE-14731.22.patch, HIVE-14731.3.patch, HIVE-14731.4.patch, > HIVE-14731.5.patch, HIVE-14731.6.patch, HIVE-14731.7.patch, > HIVE-14731.8.patch, HIVE-14731.9.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Attachment: HIVE-14731.22.patch > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.10.patch, > HIVE-14731.11.patch, HIVE-14731.12.patch, HIVE-14731.13.patch, > HIVE-14731.14.patch, HIVE-14731.15.patch, HIVE-14731.16.patch, > HIVE-14731.17.patch, HIVE-14731.18.patch, HIVE-14731.19.patch, > HIVE-14731.2.patch, HIVE-14731.20.patch, HIVE-14731.21.patch, > HIVE-14731.22.patch, HIVE-14731.3.patch, HIVE-14731.4.patch, > HIVE-14731.5.patch, HIVE-14731.6.patch, HIVE-14731.7.patch, > HIVE-14731.8.patch, HIVE-14731.9.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17393) AMReporter need hearbeat every external 'AM'
[ https://issues.apache.org/jira/browse/HIVE-17393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16154243#comment-16154243 ] Zhiyuan Yang commented on HIVE-17393: - Thanks [~sershe] for reivew! > AMReporter need hearbeat every external 'AM' > > > Key: HIVE-17393 > URL: https://issues.apache.org/jira/browse/HIVE-17393 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Fix For: 3.0.0 > > Attachments: HIVE-17393.1.patch, HIVE-17393.2.patch, > HIVE-17393.3.patch > > > AMReporter only remember first AM that submit the query and heartbeat to it. > In case of external client, there might be multiple 'AM's and every of them > need node heartbeat. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17386) support LLAP workload management in HS2 (low level only)
[ https://issues.apache.org/jira/browse/HIVE-17386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158061#comment-16158061 ] Zhiyuan Yang commented on HIVE-17386: - Yep. Sorry for the delay. I was working on other stuff and will start reviewing this tomorrow. > support LLAP workload management in HS2 (low level only) > > > Key: HIVE-17386 > URL: https://issues.apache.org/jira/browse/HIVE-17386 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-17386.01.only.patch, HIVE-17386.01.patch, > HIVE-17386.01.patch, HIVE-17386.02.patch, HIVE-17386.only.patch, > HIVE-17386.patch > > > This makes use of HIVE-17297 and creates building blocks for workload > management policies, etc. > For now, there are no policies - a single yarn queue is designated for all > LLAP query AMs, and the capacity is distributed equally. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17386) support LLAP workload management in HS2 (low level only)
[ https://issues.apache.org/jira/browse/HIVE-17386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163756#comment-16163756 ] Zhiyuan Yang commented on HIVE-17386: - Comments was posted on rb. Please take a look. > support LLAP workload management in HS2 (low level only) > > > Key: HIVE-17386 > URL: https://issues.apache.org/jira/browse/HIVE-17386 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-17386.01.only.patch, HIVE-17386.01.patch, > HIVE-17386.01.patch, HIVE-17386.02.patch, HIVE-17386.03.patch, > HIVE-17386.only.patch, HIVE-17386.patch > > > This makes use of HIVE-17297 and creates building blocks for workload > management policies, etc. > For now, there are no policies - a single yarn queue is designated for all > LLAP query AMs, and the capacity is distributed equally. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17386) support LLAP workload management in HS2 (low level only)
[ https://issues.apache.org/jira/browse/HIVE-17386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179966#comment-16179966 ] Zhiyuan Yang commented on HIVE-17386: - +1 (non-binding). CC [~hagleitn] > support LLAP workload management in HS2 (low level only) > > > Key: HIVE-17386 > URL: https://issues.apache.org/jira/browse/HIVE-17386 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-17386.01.only.patch, HIVE-17386.01.patch, > HIVE-17386.01.patch, HIVE-17386.02.patch, HIVE-17386.03.patch, > HIVE-17386.04.patch, HIVE-17386.only.patch, HIVE-17386.patch > > > This makes use of HIVE-17297 and creates building blocks for workload > management policies, etc. > For now, there are no policies - a single yarn queue is designated for all > LLAP query AMs, and the capacity is distributed equally. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17641) Visibility issue of Task.done cause Driver skip stages in parallel execution
[
https://issues.apache.org/jira/browse/HIVE-17641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhiyuan Yang updated HIVE-17641:
Attachment: HIVE-17641.1.patch
> Visibility issue of Task.done cause Driver skip stages in parallel execution
>
>
> Key: HIVE-17641
> URL: https://issues.apache.org/jira/browse/HIVE-17641
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Fix For: 1.2.1
>
> Attachments: HIVE-17641.1.patch
>
>
> Task.done is not volatile. In case of parallel execution, TaskRunner thread
> set this value, and Driver thread read this value when it determines whether
> a child task is runnable
> DriverContext.java
> {code}
> public static boolean isLaunchable(Task tsk) {
> return !tsk.getQueued() && !tsk.getInitialized() && tsk.isRunnable();
> {code}
> Task.java
> {code}
> public boolean isRunnable() {
> boolean isrunnable = true;
> if (parentTasks != null) {
> for (Task parent : parentTasks) {
> if (!parent.done()) {
> {code}
> This happens without any synchronization, so a child can be not runnable even
> all parents finish.
> To make it worse, Driver think query is successful when there is no running
> task or runnable task, so query may finish without executing some stages.
> Driver.java
> {code}
> while (!destroyed && driverCxt.isRunning()) {
> {code}
> DriverContext.java
> {code}
> public synchronized boolean isRunning() {
> return !shutdown && (!running.isEmpty() || !runnable.isEmpty());
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17641) Visibility issue of Task.done cause Driver skip stages in parallel execution
[
https://issues.apache.org/jira/browse/HIVE-17641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhiyuan Yang updated HIVE-17641:
Attachment: (was: HIVE-17641.1.patch)
> Visibility issue of Task.done cause Driver skip stages in parallel execution
>
>
> Key: HIVE-17641
> URL: https://issues.apache.org/jira/browse/HIVE-17641
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Fix For: 1.2.1
>
> Attachments: HIVE-17641.1.patch
>
>
> Task.done is not volatile. In case of parallel execution, TaskRunner thread
> set this value, and Driver thread read this value when it determines whether
> a child task is runnable
> DriverContext.java
> {code}
> public static boolean isLaunchable(Task tsk) {
> return !tsk.getQueued() && !tsk.getInitialized() && tsk.isRunnable();
> {code}
> Task.java
> {code}
> public boolean isRunnable() {
> boolean isrunnable = true;
> if (parentTasks != null) {
> for (Task parent : parentTasks) {
> if (!parent.done()) {
> {code}
> This happens without any synchronization, so a child can be not runnable even
> all parents finish.
> To make it worse, Driver think query is successful when there is no running
> task or runnable task, so query may finish without executing some stages.
> Driver.java
> {code}
> while (!destroyed && driverCxt.isRunning()) {
> {code}
> DriverContext.java
> {code}
> public synchronized boolean isRunning() {
> return !shutdown && (!running.isEmpty() || !runnable.isEmpty());
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17641) Visibility issue of Task.done cause Driver skip stages in parallel execution
[
https://issues.apache.org/jira/browse/HIVE-17641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhiyuan Yang updated HIVE-17641:
Affects Version/s: (was: 1.2.1)
0.14.1
> Visibility issue of Task.done cause Driver skip stages in parallel execution
>
>
> Key: HIVE-17641
> URL: https://issues.apache.org/jira/browse/HIVE-17641
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.14.1
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
>
> Task.done is not volatile. In case of parallel execution, TaskRunner thread
> set this value, and Driver thread read this value when it determines whether
> a child task is runnable
> DriverContext.java
> {code}
> public static boolean isLaunchable(Task tsk) {
> return !tsk.getQueued() && !tsk.getInitialized() && tsk.isRunnable();
> {code}
> Task.java
> {code}
> public boolean isRunnable() {
> boolean isrunnable = true;
> if (parentTasks != null) {
> for (Task parent : parentTasks) {
> if (!parent.done()) {
> {code}
> This happens without any synchronization, so a child can be not runnable even
> all parents finish.
> To make it worse, Driver think query is successful when there is no running
> task or runnable task, so query may finish without executing some stages.
> Driver.java
> {code}
> while (!destroyed && driverCxt.isRunning()) {
> {code}
> DriverContext.java
> {code}
> public synchronized boolean isRunning() {
> return !shutdown && (!running.isEmpty() || !runnable.isEmpty());
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17641) Visibility issue of Task.done cause Driver skip stages in parallel execution
[
https://issues.apache.org/jira/browse/HIVE-17641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhiyuan Yang updated HIVE-17641:
Status: Patch Available (was: In Progress)
> Visibility issue of Task.done cause Driver skip stages in parallel execution
>
>
> Key: HIVE-17641
> URL: https://issues.apache.org/jira/browse/HIVE-17641
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.14.1
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: HIVE-17641.1-branch-0.14.patch
>
>
> Task.done is not volatile. In case of parallel execution, TaskRunner thread
> set this value, and Driver thread read this value when it determines whether
> a child task is runnable
> DriverContext.java
> {code}
> public static boolean isLaunchable(Task tsk) {
> return !tsk.getQueued() && !tsk.getInitialized() && tsk.isRunnable();
> {code}
> Task.java
> {code}
> public boolean isRunnable() {
> boolean isrunnable = true;
> if (parentTasks != null) {
> for (Task parent : parentTasks) {
> if (!parent.done()) {
> {code}
> This happens without any synchronization, so a child can be not runnable even
> all parents finish.
> To make it worse, Driver think query is successful when there is no running
> task or runnable task, so query may finish without executing some stages.
> Driver.java
> {code}
> while (!destroyed && driverCxt.isRunning()) {
> {code}
> DriverContext.java
> {code}
> public synchronized boolean isRunning() {
> return !shutdown && (!running.isEmpty() || !runnable.isEmpty());
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17641) Visibility issue of Task.done cause Driver skip stages in parallel execution
[
https://issues.apache.org/jira/browse/HIVE-17641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhiyuan Yang updated HIVE-17641:
Attachment: HIVE-17641.1.patch
> Visibility issue of Task.done cause Driver skip stages in parallel execution
>
>
> Key: HIVE-17641
> URL: https://issues.apache.org/jira/browse/HIVE-17641
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: HIVE-17641.1.patch
>
>
> Task.done is not volatile. In case of parallel execution, TaskRunner thread
> set this value, and Driver thread read this value when it determines whether
> a child task is runnable
> DriverContext.java
> {code}
> public static boolean isLaunchable(Task tsk) {
> return !tsk.getQueued() && !tsk.getInitialized() && tsk.isRunnable();
> {code}
> Task.java
> {code}
> public boolean isRunnable() {
> boolean isrunnable = true;
> if (parentTasks != null) {
> for (Task parent : parentTasks) {
> if (!parent.done()) {
> {code}
> This happens without any synchronization, so a child can be not runnable even
> all parents finish.
> To make it worse, Driver think query is successful when there is no running
> task or runnable task, so query may finish without executing some stages.
> Driver.java
> {code}
> while (!destroyed && driverCxt.isRunning()) {
> {code}
> DriverContext.java
> {code}
> public synchronized boolean isRunning() {
> return !shutdown && (!running.isEmpty() || !runnable.isEmpty());
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17641) Visibility issue of Task.done cause Driver skip stages in parallel execution
[
https://issues.apache.org/jira/browse/HIVE-17641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhiyuan Yang updated HIVE-17641:
Fix Version/s: 1.2.1
> Visibility issue of Task.done cause Driver skip stages in parallel execution
>
>
> Key: HIVE-17641
> URL: https://issues.apache.org/jira/browse/HIVE-17641
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Fix For: 1.2.1
>
> Attachments: HIVE-17641.1.patch
>
>
> Task.done is not volatile. In case of parallel execution, TaskRunner thread
> set this value, and Driver thread read this value when it determines whether
> a child task is runnable
> DriverContext.java
> {code}
> public static boolean isLaunchable(Task tsk) {
> return !tsk.getQueued() && !tsk.getInitialized() && tsk.isRunnable();
> {code}
> Task.java
> {code}
> public boolean isRunnable() {
> boolean isrunnable = true;
> if (parentTasks != null) {
> for (Task parent : parentTasks) {
> if (!parent.done()) {
> {code}
> This happens without any synchronization, so a child can be not runnable even
> all parents finish.
> To make it worse, Driver think query is successful when there is no running
> task or runnable task, so query may finish without executing some stages.
> Driver.java
> {code}
> while (!destroyed && driverCxt.isRunning()) {
> {code}
> DriverContext.java
> {code}
> public synchronized boolean isRunning() {
> return !shutdown && (!running.isEmpty() || !runnable.isEmpty());
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17641) Visibility issue of Task.done cause Driver skip stages in parallel execution
[
https://issues.apache.org/jira/browse/HIVE-17641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhiyuan Yang updated HIVE-17641:
Attachment: (was: HIVE-17641.1.patch)
> Visibility issue of Task.done cause Driver skip stages in parallel execution
>
>
> Key: HIVE-17641
> URL: https://issues.apache.org/jira/browse/HIVE-17641
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Fix For: 1.2.1
>
>
> Task.done is not volatile. In case of parallel execution, TaskRunner thread
> set this value, and Driver thread read this value when it determines whether
> a child task is runnable
> DriverContext.java
> {code}
> public static boolean isLaunchable(Task tsk) {
> return !tsk.getQueued() && !tsk.getInitialized() && tsk.isRunnable();
> {code}
> Task.java
> {code}
> public boolean isRunnable() {
> boolean isrunnable = true;
> if (parentTasks != null) {
> for (Task parent : parentTasks) {
> if (!parent.done()) {
> {code}
> This happens without any synchronization, so a child can be not runnable even
> all parents finish.
> To make it worse, Driver think query is successful when there is no running
> task or runnable task, so query may finish without executing some stages.
> Driver.java
> {code}
> while (!destroyed && driverCxt.isRunning()) {
> {code}
> DriverContext.java
> {code}
> public synchronized boolean isRunning() {
> return !shutdown && (!running.isEmpty() || !runnable.isEmpty());
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17641) Visibility issue of Task.done cause Driver skip stages in parallel execution
[
https://issues.apache.org/jira/browse/HIVE-17641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhiyuan Yang updated HIVE-17641:
Fix Version/s: (was: 1.2.1)
> Visibility issue of Task.done cause Driver skip stages in parallel execution
>
>
> Key: HIVE-17641
> URL: https://issues.apache.org/jira/browse/HIVE-17641
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
>
> Task.done is not volatile. In case of parallel execution, TaskRunner thread
> set this value, and Driver thread read this value when it determines whether
> a child task is runnable
> DriverContext.java
> {code}
> public static boolean isLaunchable(Task tsk) {
> return !tsk.getQueued() && !tsk.getInitialized() && tsk.isRunnable();
> {code}
> Task.java
> {code}
> public boolean isRunnable() {
> boolean isrunnable = true;
> if (parentTasks != null) {
> for (Task parent : parentTasks) {
> if (!parent.done()) {
> {code}
> This happens without any synchronization, so a child can be not runnable even
> all parents finish.
> To make it worse, Driver think query is successful when there is no running
> task or runnable task, so query may finish without executing some stages.
> Driver.java
> {code}
> while (!destroyed && driverCxt.isRunning()) {
> {code}
> DriverContext.java
> {code}
> public synchronized boolean isRunning() {
> return !shutdown && (!running.isEmpty() || !runnable.isEmpty());
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Work started] (HIVE-17641) Visibility issue of Task.done cause Driver skip stages in parallel execution
[
https://issues.apache.org/jira/browse/HIVE-17641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-17641 started by Zhiyuan Yang.
---
> Visibility issue of Task.done cause Driver skip stages in parallel execution
>
>
> Key: HIVE-17641
> URL: https://issues.apache.org/jira/browse/HIVE-17641
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Fix For: 1.2.1
>
> Attachments: HIVE-17641.1.patch
>
>
> Task.done is not volatile. In case of parallel execution, TaskRunner thread
> set this value, and Driver thread read this value when it determines whether
> a child task is runnable
> DriverContext.java
> {code}
> public static boolean isLaunchable(Task tsk) {
> return !tsk.getQueued() && !tsk.getInitialized() && tsk.isRunnable();
> {code}
> Task.java
> {code}
> public boolean isRunnable() {
> boolean isrunnable = true;
> if (parentTasks != null) {
> for (Task parent : parentTasks) {
> if (!parent.done()) {
> {code}
> This happens without any synchronization, so a child can be not runnable even
> all parents finish.
> To make it worse, Driver think query is successful when there is no running
> task or runnable task, so query may finish without executing some stages.
> Driver.java
> {code}
> while (!destroyed && driverCxt.isRunning()) {
> {code}
> DriverContext.java
> {code}
> public synchronized boolean isRunning() {
> return !shutdown && (!running.isEmpty() || !runnable.isEmpty());
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17641) Visibility issue of Task.done cause Driver skip stages in parallel execution
[
https://issues.apache.org/jira/browse/HIVE-17641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhiyuan Yang updated HIVE-17641:
Attachment: HIVE-17641.1-branch-0.14.patch
> Visibility issue of Task.done cause Driver skip stages in parallel execution
>
>
> Key: HIVE-17641
> URL: https://issues.apache.org/jira/browse/HIVE-17641
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.14.1
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: HIVE-17641.1-branch-0.14.patch
>
>
> Task.done is not volatile. In case of parallel execution, TaskRunner thread
> set this value, and Driver thread read this value when it determines whether
> a child task is runnable
> DriverContext.java
> {code}
> public static boolean isLaunchable(Task tsk) {
> return !tsk.getQueued() && !tsk.getInitialized() && tsk.isRunnable();
> {code}
> Task.java
> {code}
> public boolean isRunnable() {
> boolean isrunnable = true;
> if (parentTasks != null) {
> for (Task parent : parentTasks) {
> if (!parent.done()) {
> {code}
> This happens without any synchronization, so a child can be not runnable even
> all parents finish.
> To make it worse, Driver think query is successful when there is no running
> task or runnable task, so query may finish without executing some stages.
> Driver.java
> {code}
> while (!destroyed && driverCxt.isRunning()) {
> {code}
> DriverContext.java
> {code}
> public synchronized boolean isRunning() {
> return !shutdown && (!running.isEmpty() || !runnable.isEmpty());
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
