beliefer opened a new pull request, #12709:
URL: https://github.com/apache/gluten/pull/12709
## What changes are proposed in this pull request?
Fixes #12708
For an **OUTER** generator, Velox's `Unnest` operator appends a trailing
`BOOLEAN`
marker column (output order: replicated columns → unnest value columns →
`[ordinality]` → `[marker]`). `GenerateExecTransformer.pullOutPostProject`
already
consumes this marker for `PosExplode`, `Inline`/`JsonTupleExplode`, and
`Explode`,
each by appending a boolean `isPresent` attribute to `generatorOutput` and
wrapping
every real output column in `CaseWhen(Seq((isPresent, col)), Literal(null,
col.dataType))`.
**`Stack` had no such branch** — an OUTER `Stack` fell through to `case _ =>
generate`,
leaving the marker unconsumed. Because Velox binds columns by position, the
extra
column shifts every upstream column by one index. When the exploded output
feeds a
columnar hash exchange, the boolean marker displaces the int32
`hash_partition_key`
that must sit at field 0, and `VeloxShuffleWriter::getFirstColumn` aborts:
```
Partition id (field 0) should be integer, but got BOOLEAN
```
The crash is deterministic and data-independent. Depending on the operator
downstream
of the shift, the same root cause can also surface as `values_->capacity()
>= byteSize`
(partial HashAggregate) or `Expected INT_ARRAY. Got BYTE_ARRAY`
(BroadcastHashJoin).
This PR adds a `case _: Stack if generate.outer =>` branch to
`pullOutPostProject`,
structurally identical to the existing `Explode(_) if generate.outer` branch
— `Stack`'s
native output layout (value columns, no ordinality, trailing marker) matches
`Explode`'s,
so no ordinality handling is needed. Inner (non-OUTER) `stack` produces no
marker column
and is gated out by the `if generate.outer` guard, so it is unchanged.
## How was this patch tested?
Added a regression test `test LATERAL VIEW OUTER stack followed by hash
shuffle` in
`MiscOperatorSuite` that runs `LATERAL VIEW OUTER stack(...)` whose exploded
key drives a
hash-partition columnar exchange into a SortMergeJoin (broadcast disabled via
`spark.sql.autoBroadcastJoinThreshold=-1`, which reproduces the exact
`field 0 should be integer, but got BOOLEAN` path), and compares results
against vanilla
Spark. Before this patch the query aborts; after, it returns the correct
rows including the
NULL-padded columns for OUTER padding rows.
Also verified manually:
- Inner `stack` (drop `OUTER`) is unchanged — same join result, no extra
column.
- OUTER `stack` with an argument count that pads a NULL value column
projects the NULL at
the correct position (no column shift) and still joins correctly.
- The original production query (OUTER `stack` → SortMergeJoin → CUBE +
`COUNT(DISTINCT)`),
with and without an outer `ORDER BY`, ran to completion with all tasks
succeeding.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude-opus-4-8.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
