ggangadharan opened a new pull request, #6898:
URL: https://github.com/apache/hive/pull/6898
### What changes were proposed in this pull request?
Vectorized PTF evaluators for the variance family (`variance`, `var_pop`,
`var_samp`, `std`,
`stddev`, `stddev_pop`, `stddev_samp`) over LONG / DOUBLE / DECIMAL inputs:
- `VectorPTFEvaluatorStreamingVariance` for `ROWS BETWEEN UNBOUNDED
PRECEDING AND CURRENT ROW`
(single pass, the result is written for each row while streaming).
- `VectorPTFEvaluatorVariance` for the other supported frames. When a range
starts where the
previous one started and does not end before it (UNBOUNDED PRECEDING
frames, including the
default `ORDER BY` frame), the previous aggregation state is reused and
only the new rows are
iterated, so these frames cost O(n) instead of O(n^2).
- `VectorPTFEvaluatorAbstractVariance` holds the shared state (count, sum,
variance) and the
result calculation, using the same `GenericUDAFVariance` helpers as row
mode. Values are
iterated in row order, so the results are identical to row mode.
- `VectorPTFDesc` / `Vectorizer`: register the functions and allow `ROWS ...
CURRENT ROW` frames,
like for avg / sum / min / max.
### Why are the changes needed?
These functions were not in the supported function list of the vectorized
PTF, so the whole
reducer fell back to row mode. Example from `vector_windowing.q`:
```sql
select p_mfgr, p_name, p_size,
rank() over(distribute by p_mfgr sort by p_name) as r,
...
avg(p_size) over(distribute by p_mfgr sort by p_name) as avg,
stddev(p_size) over(distribute by p_mfgr sort by p_name) as st,
...
from part;
```
Before:
```
Reducer 2
Execution mode: llap
Reduce Vectorization:
notVectorizedReason: PTF operator: stddev_pop not in
supported functions [avg, count, cume_dist, dense_rank, first_value, lag,
last_value, lead, max, min, ntile, percent_rank, rank, row_number, sum]
vectorized: false
```
After:
```
Reducer 2
Execution mode: vectorized, llap
Reduce Vectorization:
vectorized: true
...
PTF Vectorization:
className: VectorPTFOperator
evaluatorClasses: [VectorPTFEvaluatorRank,
VectorPTFEvaluatorDenseRank, VectorPTFEvaluatorCumeDist,
VectorPTFEvaluatorPercentRank, VectorPTFEvaluatorNTile,
VectorPTFEvaluatorCount, VectorPTFEvaluatorLongAvg, VectorPTFEvaluatorVariance,
VectorPTFEvaluatorLongFirstValue, VectorPTFEvaluatorLongLastValue]
functionNames: [rank, dense_rank, cume_dist,
percent_rank, ntile, count, avg, stddev_pop, first_value, last_value]
native: true
```
### Does this PR introduce _any_ user-facing change?
No. Such PTFs now run vectorized, so the `EXPLAIN VECTORIZATION` output
changes, but the query
results are unchanged.
### How was this patch tested?
New `vector_ptf_variance.q`, comparing row mode and vectorized output for:
- all frame types: whole partition, default RANGE frame with ties, streaming
ROWS frame, bounded
ROWS / RANGE frames, ROWS ... UNBOUNDED FOLLOWING
- all input types: tinyint, smallint, int, bigint, float, double,
decimal(10,2), decimal(38,18)
- NULL-only, single-row and repeated-value partitions, extreme values, NULL
partition key
- expression and constant (repeating) arguments
- streaming and non-streaming evaluators together with other functions in
one PTF
- CBO on and off (without CBO the original function names reach the
vectorizer)
- a 12288-row table with spilling forced, compared row by row: 0 mismatches
Updated `vector_windowing.q.out` (plan only, results unchanged).
Also ran all `vector_ptf*` and `vector_windowing*` q-files, `windowing.q`,
`cbo_rp_windowing_2.q`,
`udaf_binarysetfunctions.q`, `udaf_binarysetfunctions_no_cbo.q`
(TestMiniLlapLocalCliDriver) and
`TestVectorPTFGroupBatches`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
