ggangadharan commented on code in PR #6861:
URL: https://github.com/apache/hive/pull/6861#discussion_r4270401531
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFOperator.java:
##########
@@ -474,13 +476,31 @@ private void initBufferedColumns() {
bufferedTypeInfos[bufferedMapIndex] =
outputTypeInfos[streamingEvaluatorNum];
}
- for (int i = 0; i < scratchColumnPositions.length; i++) {
+ for (int i = 0; i < bufferedScratchIndices.length; i++) {
+ final int scratchIndex = bufferedScratchIndices[i];
final int bufferedMapIndex = orderColumnCount + nonKeyInputColumnCount +
streamingEvaluatorCount + i;
- bufferedColumnMap[bufferedMapIndex] = scratchColumnPositions[i];
- bufferedTypeInfos[bufferedMapIndex] =
TypeInfoUtils.getTypeInfoFromTypeString(vOutContext.getScratchColumnTypeNames()[i]);
+ bufferedColumnMap[bufferedMapIndex] =
scratchColumnPositions[scratchIndex];
+ bufferedTypeInfos[bufferedMapIndex] =
+
TypeInfoUtils.getTypeInfoFromTypeString(vOutContext.getScratchColumnTypeNames()[scratchIndex]);
}
}
+ /**
+ * Returns the indices (into scratchColumnPositions) of the scratch columns
written by evaluator
+ * input expressions. Those are evaluated before a batch is buffered, so
their results must be
+ * buffered too. The other scratch columns belong to child operators, which
recompute them after
+ * the batch is forwarded; buffering them is wasted work and fails for
complex types.
+ */
+ private int[] getEvaluatorInputScratchIndices() {
Review Comment:
Correct, it's not ntile-specific.
It's a pre-existing bug in buffering any vectorized PTF (cume_dist,
percent_rank, sum, …). Vectorizing ntile exposed it in
sketches_rewrite_ntile*.q. I've added a separate "Related fix" section to the
PR description
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
