ggangadharan opened a new pull request, #6946:
URL: https://github.com/apache/hive/pull/6946
### What changes were proposed in this pull request?
Vectorize `ntile` when the number of buckets is not a constant: a column, an
expression, or a
partition key. Row mode (`GenericUDAFNTile`) reads the number of buckets
from the first row of each
partition, and the vectorized path now does the same.
- `VectorPTFEvaluatorNTile`: takes either a constant or a non-constant
expression. For a
non-constant one, the value is read from row 0 of the partition's first
batch when the buffered
batches are replayed for output. Partition key columns are restored at
that point, so a
partition key argument works even though it is not buffered. The bucket
sizes are set up once per
partition, so nothing is added per row. A constant is still neither
evaluated per batch nor
buffered.
- `Vectorizer`: non-constant arguments are now vectorized. A non-positive
constant still falls
back to row mode, as before. Lead/lag inside the argument now falls back
with an explicit reason.
ntile is in `COLUMN_AGNOSTIC_FUNCTIONS`, so that check was skipped for it
before.
- `VectorPTFDesc`: corrected the comment on `COLUMN_AGNOSTIC_FUNCTIONS`,
which said these functions
don't read input columns. ntile now reads its argument, but only from
forwarded batches.
Row mode does not validate the number of buckets. The vectorized evaluator
returns the same result
wherever row mode returns one, and fails right away wherever row mode fails:
| First-row value `n` (partition of `N` rows) | Row mode | Vectorized |
|---|---|---|
| `n > 0` | normal result | same result |
| `n < -N` | one row per bucket (truncating division) | same result |
| `0` | `ArithmeticException: / by zero` | `HiveException: NTILE requires a
positive number of buckets, got: 0` |
| `NULL` | `NullPointerException` | `HiveException: ... got: NULL` |
| `-N <= n < 0` | loops until it runs out of memory | `HiveException: ...
got: n` |
### Why are the changes needed?
HIVE-30018 vectorized `ntile` only for a positive int constant. Any other
argument made the whole
PTF fall back to row mode, with
`notVectorizedReason: PTF operator: ntile only supports a positive integer
constant number of buckets`.
This is part of the PTF vectorization coverage effort (HIVE-24872).
### Does this PR introduce _any_ user-facing change?
No, Released Hive versions do not vectorize `ntile` at all.
- Queries such as `ntile(col) over (...)` now run vectorized, with the same
results as row mode.
Example:
```sql
set hive.vectorized.execution.ptf.enabled=true;
create table t(name string, rowindex int) stored as orc;
insert into t values ('a', 3), ('a', 4), ('a', 5), ('a', 6), ('b', 2),
('b', 9);
explain vectorization detail
select name, rowindex, ntile(rowindex) over (partition by name order by
rowindex) as nt from t;
```
Before: the reducer is not vectorized, with the `notVectorizedReason`
above.
After: the reducer runs `VectorPTFOperator` with
`functionInputExpressions: [col 1:int]`. The
query returns `a: 1 2 3 3` (3 buckets) and `b: 1 1` (2 buckets) in both
modes.
- When the first row holds an invalid number of buckets (0, NULL, or a
negative count no larger in
magnitude than the partition), the query still fails, as in row mode. The
error is now a clear
`HiveException`, and the negative case fails immediately instead of
running out of memory.
No configuration changes.
### How was this patch tested?
New and updated qtests:
- `clientpositive/vector_ptf_ntile.q`: every non-constant case runs in row
mode and vectorized mode
with identical results. A new table holds each partition's number of
buckets in its first row,
with NULL, 0 and negative values in the later rows that must never be read.
- Bucket counts: 1 bucket; fewer buckets than rows (with and without a
remainder); as many as
rows; more than rows (the largest int); a negative count larger in
magnitude than the
partition; single-row partitions.
- Argument kinds: column, expression, partition key (not buffered), order
key. Also checks that
the first row is taken in ORDER BY order rather than insertion order.
- Execution paths: multi-row batches at batch size 2 and the default;
spilling every batch to
disk; constant and non-constant ntile in one PTF alongside `rank` and
`sum`.
- Fallbacks: `explain` shows the row-mode fallback for a non-positive
constant and for lead/lag in
the argument.
- `clientnegative/vector_ptf_ntile_invalid_buckets.q`: 0, NULL and `-N` in
the first row each fail
with the vectorized error.
- Regression run of 21 ntile and vector PTF qtests: `cbo_rp_windowing_2`,
`lineage_ptf`,
`ptf_decimal`, `sketches_materialized_view_ntile`,
`sketches_rewrite_ntile`,
`sketches_rewrite_ntile_partition_by`,
`vector_ptf_buffered_complex_scratch_col`,
`vector_ptf_ntile`, `vector_windowing`, `windowing`, `windowing_ntile`,
`vector_ptf_1`,
`vector_ptf_cume_dist`, `vector_ptf_percent_rank`,
`vector_ptf_unbuffered_partition_col`,
`vector_ptf_spill_partition_order_overlap`, `vector_ptf_part_simple`,
`vector_ptf_nested_input_expr`, `vector_ptf_lead_lag`,
`vector_windowing_rank`,
`vector_windowing_expressions`.
To reproduce:
```bash
mvn clean install -DskipTests -Pitests
cd itests/qtest
mvn test -Dtest=TestMiniLlapLocalCliDriver -Dqfile=vector_ptf_ntile.q
mvn test -Dtest=TestNegativeLlapLocalCliDriver
-Dqfile=vector_ptf_ntile_invalid_buckets.q
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
