ggangadharan commented on code in PR #6861:
URL: https://github.com/apache/hive/pull/6861#discussion_r4247431160
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFOperator.java:
##########
@@ -435,10 +435,11 @@ private void initBufferedColumns() {
* 1. order columns
* 2. the non-key input columns
* 3. any streaming columns that will already have their output values
- * 4. scratch columns (as they're used by input expressions of evaluators)
+ * 4. scratch columns holding the results of evaluator input expressions
*/
+ final int[] bufferedScratchIndices = getEvaluatorInputScratchIndices();
Review Comment:
Without this change, first commit failed CI on sketches_rewrite_ntile.q and
sketches_rewrite_ntile_partition_by.q with below ERROR -
``` org.apache.hadoop.hive.ql.exec.tez.TezRuntimeException: Vertex failed,
vertexName=Reducer 3, vertexId=vertex_1791496978238_0001_162_03,
diagnostics=[Task failed, taskId=task_1791496978238_0001_162_03_000000,
diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
failure ) :
attempt_1791496978238_0001_162_03_000000_0:java.lang.RuntimeException:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error while processing vector batch (tag=0) (vectorizedVertexNum 3)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:342)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:277)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:381)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:86)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:72)
at
java.base/java.security.AccessController.doPrivileged(AccessController.java:714)
at java.base/javax.security.auth.Subject.doAs(Subject.java:525)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:72)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:42)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at
org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:112)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing vector batch (tag=0) (vectorizedVertexNum 3)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:411)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:259)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:292)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:294)
... 15 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing vector batch (tag=0) (vectorizedVertexNum 3)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:519)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:402)
... 18 more
Caused by: java.lang.RuntimeException: No complex type support: LIST
at
org.apache.hadoop.hive.ql.exec.vector.VectorizedBatchUtil.copyNonSelectedColumnVector(VectorizedBatchUtil.java:802)
at
org.apache.hadoop.hive.ql.exec.vector.VectorizedBatchUtil.copyNonSelectedColumnVector(VectorizedBatchUtil.java:644)
at
org.apache.hadoop.hive.ql.exec.vector.ptf.VectorPTFGroupBatches.copyBufferedColumns(VectorPTFGroupBatches.java:1265)
at
org.apache.hadoop.hive.ql.exec.vector.ptf.VectorPTFGroupBatches.bufferGroupBatch(VectorPTFGroupBatches.java:545)
at
org.apache.hadoop.hive.ql.exec.vector.ptf.VectorPTFOperator.process(VectorPTFOperator.java:421)
at
org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:920)
at
org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:159)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:503)
... 19 more
```
This fixes sketches_rewrite_ntile.q and
sketches_rewrite_ntile_partition_by.q, which failed in CI on the first commit
with No complex type support: LIST.
initBufferedColumns buffered every scratch column of the reducer, including
the array<double> that the Select after the PTF computes for
ds_kll_cdf(...)[0], and VectorizedBatchUtil.copyNonSelectedColumnVector can't
copy lists. Since HIVE-29696, only evaluator input expressions are evaluated
before buffering, so those are the only scratch columns a buffered batch needs.
The rest belong to child operators, which recompute them after the batch is
forwarded (prepareBufferedBatchForForwarding resets the overflow batch first).
It isn't ntile-specific: the same query with cume_dist() instead of ntile fails
the same way. It only showed up now because vectorizing ntile sends those
queries through the buffered path
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
