ramitg254 commented on code in PR #6946:
URL: https://github.com/apache/hive/pull/6946#discussion_r4560997480
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFEvaluatorNTile.java:
##########
@@ -44,34 +44,41 @@
* evaluator keeps O(1) state regardless of partition size, and a batch that
falls entirely within
* one bucket is emitted as a repeating vector.
*
- * <p>The number of buckets must be a positive integer constant (e.g. {@code
ntile(4)}); the
- * Vectorizer leaves any other form to row mode.
+ * <p>The number of buckets is either a positive integer constant (e.g. {@code
ntile(4)}) or any
+ * other int expression (e.g. {@code ntile(col)}), which, as in {@code
GenericUDAFNTile}, is read
+ * from the first row of each partition. The Vectorizer leaves a non-positive
constant to row mode.
*/
public class VectorPTFEvaluatorNTile extends VectorPTFEvaluatorBase {
- private final long numBuckets;
+ // Number of buckets of a constant argument, or 0 when it is read from the
partition's first row.
+ private final long constantNumBuckets;
- // Per-partition bucket layout, set up by addStreamingGroupResults.
+ // Per-partition bucket layout, set up by setUpBuckets once the first batch
is replayed.
private int bucketSize;
private int largerBucketCount;
- // Replay position within the partition.
+ // Replay position within the partition; 0 until the bucket layout is set up.
private int currentBucket;
private int rowsLeftInBucket;
public VectorPTFEvaluatorNTile(WindowFrameDef windowFrameDef,
VectorExpression numBucketsExpression, int outputColumnNum) {
- super(windowFrameDef, outputColumnNum);
- if (!(numBucketsExpression instanceof ConstantVectorExpression
constantExpr)) {
- throw new RuntimeException(
- "NTILE expects a constant integer expression for the number of
buckets, got: "
- + (numBucketsExpression == null ? "null"
- : numBucketsExpression.getClass().getName()));
+ // A constant needs no input column; any other expression is evaluated
before batches are
+ // buffered, and its result column is read back when the partition's first
batch is replayed.
+ super(windowFrameDef,
+ numBucketsExpression instanceof ConstantVectorExpression ? null :
numBucketsExpression,
+ outputColumnNum);
+ if (numBucketsExpression == null) {
+ throw new RuntimeException("NTILE expects an expression for the number
of buckets");
Review Comment:
can we throw exception more earlier before calling super?
##########
ql/src/test/queries/clientnegative/vector_ptf_ntile_invalid_buckets.q:
##########
@@ -0,0 +1,31 @@
+set hive.vectorized.execution.enabled=true;
Review Comment:
these properties are true by defulat, we can avoid setting it explicitly
##########
ql/src/test/queries/clientnegative/vector_ptf_ntile_invalid_buckets.q:
##########
@@ -0,0 +1,31 @@
+set hive.vectorized.execution.enabled=true;
+set hive.vectorized.execution.ptf.enabled=true;
+
Review Comment:
can we add explain vectorization detail queries in this q file for
validation?
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFEvaluatorNTile.java:
##########
@@ -86,18 +93,56 @@ public void addStreamingGroupResults(List<Integer>
groupRowCounts) throws HiveEx
if (partitionSize <= 0) {
throw new HiveException("Partition size must be set before computing
ntile");
}
- // Peer groups are irrelevant: buckets depend only on row position and
partition size.
- // With more buckets than rows every row gets its own bucket, which is
ntile(partitionSize);
- // capping the bucket count keeps every bucket non-empty (and fits it in
an int).
- final int buckets = (int) Math.min(numBuckets, partitionSize);
+ // Peer groups are irrelevant: buckets depend only on row position and
partition size. The
+ // layout is set up on the first replayed batch, which holds the row a
non-constant number of
+ // buckets is read from.
+ currentBucket = 0;
+ }
+
+ /**
+ * Mirrors {@code GenericUDAFNTile}, which does not validate the number of
buckets: wherever row
+ * mode produces a result this produces the same one, and wherever row mode
fails this fails fast.
+ */
+ private void setUpBuckets(long numBuckets) throws HiveException {
+ final int buckets;
+ if (numBuckets > 0) {
+ // With more buckets than rows every row gets its own bucket, which is
ntile(partitionSize);
+ // capping the bucket count keeps every bucket non-empty (and fits it in
an int).
+ buckets = (int) Math.min(numBuckets, partitionSize);
+ } else if (numBuckets < -partitionSize) {
+ // Row mode's truncating division yields an empty base bucket size, so
every row gets its own
+ // bucket, as for a positive count larger than the partition.
+ buckets = partitionSize;
+ } else {
+ // Row mode divides by zero, or for any other negative count loops until
it runs out of
+ // memory.
+ throw new HiveException("NTILE requires a positive number of buckets,
got: " + numBuckets);
Review Comment:
this error message seems more or less redundant to the one in
`readNumBuckets` and `VectorPTFEvaluatorNTile constructor `, can it be moved to
some kind of single reference for consistent updation later on or some kind of
earlier throw in the code flow introduced rather than checking everywhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]