schumiyi opened a new issue, #13655:
URL: https://github.com/apache/gluten/issues/13655
### Backend
VL (Velox)
### Bug description
## Backend and versions
- Velox backend.
- The affected rewrite is still present in Gluten `main` at
`3671995dfbd2badbce6c209aa9d7f1ec6b0ac4e6`.
- That revision selects `IBM/velox` branch `dft-2026_11_10`, which currently
resolves to `67fa1bbee62e9f8752b3c34cc92f179022cd2ac6`.
- The failure was originally observed in a downstream build. See the
validation note below for the distinction between that observation, the
current-main JVM regression, and a full native integration run.
## Problem
Velox `UnnestNode` emits its ordinality column as **BIGINT**, with positions
starting at **1**. Spark `posexplode` exposes an **INT** position starting at
**0**.
`PullOutGenerateProjectHelper.pullOutPostProject` accounts for the index
difference with `CAST(position AS INT) - 1`, but currently also uses Spark's
original **INT** attribute as the output attribute of the native Generate. The
declared input type therefore disagrees with the native column's physical type.
When that cast is evaluated on the JVM, the projection must first read the
cast's input. Binding the input as INT selects `getInt` on a BIGINT Arrow
vector, which throws `UnsupportedOperationException` before the conversion can
execute. A cast around the incorrectly declared input cannot repair its reader.
The original downstream stack included `ArrowWritableColumnVector.getInt`
and `ColumnarPartialProjectExec`. Current `main` uses Spark Arrow column
vectors instead of that removed Gluten vector implementation; the reader/type
mismatch remains in the rewrite. The expected failure mechanism is the same,
but those historical class names should not be mistaken for an observed
current-main native stack.
## Reproduction
In a Spark session configured with the Gluten Velox backend and its matching
native libraries, use a materialized Parquet input so constant folding cannot
bypass Generate:
```sql
SET spark.gluten.enabled=false;
CREATE TABLE posexplode_ordinality_repro (k INT, a ARRAY<INT>) USING PARQUET;
INSERT INTO posexplode_ordinality_repro VALUES
(0, array(10, 20)),
(1, array(CAST(NULL AS INT), 30)),
(2, array()),
(3, CAST(NULL AS ARRAY<INT>));
SET spark.gluten.enabled=true;
SET spark.sql.ansi.enabled=false;
SET spark.gluten.sql.columnar.generate=true;
SET spark.gluten.sql.columnar.partial.project=true;
SET spark.gluten.expression.blacklist=cast;
SELECT k, pos, item
FROM posexplode_ordinality_repro
LATERAL VIEW posexplode(a) e AS pos, item
ORDER BY k, pos;
```
The relevant executed plan must contain a native `GenerateExecTransformer`
whose ordinality is consumed by `ColumnarPartialProjectExec`; a plan that falls
back the entire Generate does not exercise this path. Depending on the version
and plan, the row conversion boundary can expose the same incorrect schema
through another JVM reader.
Also run the query with `LATERAL VIEW OUTER`, and the corresponding map
input cases. The regression coverage should include real null elements/values,
empty collections, and null collections.
## Expected behavior
- The query matches vanilla Spark.
- A real collection element retains its 0-based INT position, including an
element/value that is itself null.
- An OUTER padding row for an empty or null collection has a null position.
- The externally visible position keeps Spark's original output attribute
identity and schema.
## Root cause and proposed fix
Create a separate native ordinality attribute using:
```scala
val nativeOrdinal = originalOrdinal.withDataType(LongType).newInstance()
```
Use it in the rewritten native Generate's output and as the input of the
existing cast/subtraction. Keep the original Spark position's `exprId` and
qualifier on the final alias.
`LongType` makes JVM expression binding select `getLong`, matching the
native BIGINT column. A fresh `exprId` distinguishes that internal 1-based LONG
column from the externally visible 0-based INT result; the two attributes
should not share one identity with different types and meanings. The type
correction directly addresses the observed reader failure. Identity separation
preserves the plan contract between those two representations.
Apply this only to the rewritten native candidate. If native validation
rejects the candidate, fallback must retain the original Spark Generate and its
original INT output. The native ordinality layout and the OUTER marker behavior
remain as defined by Velox. This is a Gluten schema-adaptation fix; changing
Velox's BIGINT ordinality contract is unnecessary.
Removing `cast` from the blacklist may avoid the JVM partial-project path
for a particular query, but does not correct the declared native schema. Other
JVM consumers/fallback paths can still encounter that mismatch.
## Source evidence
- [Gluten's current position
rewrite](https://github.com/apache/gluten/blob/3671995dfbd2badbce6c209aa9d7f1ec6b0ac4e6/backends-velox/src/main/scala/org/apache/gluten/execution/GenerateExecTransformer.scala#L271):
uses `originalOrdinal` in both the native output and the cast.
- [Velox output
schema](https://github.com/IBM/velox/blob/67fa1bbee62e9f8752b3c34cc92f179022cd2ac6/velox/core/PlanNode.cpp#L1514):
ordinality is BIGINT.
- [Velox ordinality
values](https://github.com/IBM/velox/blob/67fa1bbee62e9f8752b3c34cc92f179022cd2ac6/velox/exec/Unnest.cpp#L356):
real rows use 1-based `int64_t`; OUTER padding uses zero before the
marker-controlled projection.
- [JVM partial
projection](https://github.com/apache/gluten/blob/3671995dfbd2badbce6c209aa9d7f1ec6b0ac4e6/backends-velox/src/main/scala/org/apache/gluten/execution/ColumnarPartialProjectExec.scala#L172):
binds expressions to input attributes and evaluates them against Arrow batch
rows.
Related: #9315 added partial-project support for blacklisted built-in
expressions. #12709 fixes a different OUTER `stack` marker/layout issue and
does not correct POSEXPLODE's ordinality type.
## Validation
A source-isolated JVM regression was run against the exact
`pullOutPostProject` method from the above main revision and the proposed
patch, using public Spark 3.5.5, Scala 2.12.18, Arrow 15.0.0, and JDK 17
dependencies. The harness compiles the production method with the actual
rewrite manager and runs the new `PullOutGenerateProjectSuite` without loading
a native backend.
- Unpatched main: 10 tests failed. The Arrow case reproduced
`UnsupportedOperationException` from `ArrowColumnVector.getInt`; the schema and
row-evaluation cases also exposed the incorrect INT binding.
- Patched method: all 10 tests passed, including ARRAY/MAP, inner/outer,
attribute identity, fallback restoration, and the Arrow/MutableProjection
reader check.
- This is not a full Gluten build or native SQL integration run. A
Parquet-backed native regression is included in the proposed patch for CI; its
plan assertions require both native Generate and the JVM partial project, so
complete fallback cannot produce a false pass.
AI assistance: Generated-by: OpenAI Codex (GPT-6.1-sol and GPT-6-astra).
### Gluten version
main branch
### Spark version
Spark-3.5.x
### Spark configurations
spark.gluten.enabled=true
spark.sql.ansi.enabled=false
spark.gluten.sql.columnar.generate=true
spark.gluten.sql.columnar.partial.project=true
spark.gluten.expression.blacklist=cast
### System information
_No response_
### Relevant logs
```bash
Source-isolated JVM regression against unpatched main (Spark 3.5.5):
java.lang.UnsupportedOperationException
at
org.apache.spark.sql.vectorized.ArrowColumnVector$ArrowVectorAccessor.getInt(ArrowColumnVector.java:234)
at
org.apache.spark.sql.vectorized.ArrowColumnVector.getInt(ArrowColumnVector.java:85)
at
org.apache.spark.sql.vectorized.ColumnarBatchRow.getInt(ColumnarBatchRow.java:106)
Patched method: Tests: succeeded 10, failed 0.
Full native SQL integration has not been run locally.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]