malinjawi opened a new pull request, #13393:
URL: https://github.com/apache/gluten/pull/13393
## What changes are proposed in this pull request?
Since #13096, the Spark 3.4/3.5 bundle ships Arrow under its original
package names, but it still relocates `com.google.flatbuffers`. So the bundled
`org.apache.arrow.flatbuf.*` classes extend
`org.apache.gluten.shaded.com.google.flatbuffers.Table`, while the classes with
the same names in Spark's `arrow-format` jar extend
`com.google.flatbuffers.Table`:
```
$ javap -cp gluten-velox-bundle-spark3.5_2.12-*.jar
org.apache.arrow.flatbuf.Null
public final class org.apache.arrow.flatbuf.Null extends
org.apache.gluten.shaded.com.google.flatbuffers.Table
$ javap -cp arrow-format-12.0.1.jar org.apache.arrow.flatbuf.Null
public final class org.apache.arrow.flatbuf.Null extends
com.google.flatbuffers.Table
```
When classpath order puts Spark's `arrow-vector` and `arrow-format` jars on
opposite sides of the bundle, `ArrowType` and the `flatbuf` classes come from
different copies, and class verification fails:
```
java.lang.VerifyError: Bad type on operand stack
Location:
org/apache/arrow/vector/types/pojo/ArrowType.getTypeForField(Lorg/apache/arrow/flatbuf/Field;)Lorg/apache/arrow/vector/types/pojo/ArrowType;
@112: invokevirtual
Reason:
Type 'org/apache/arrow/flatbuf/Null' (current frame, stack[1]) is not
assignable to 'com/google/flatbuffers/Table'
```
This is the failure currently hitting the ARM `tpc-test-centos8
(quay.io/centos/centos:stream9, spark-3.5, java-8)` job. gluten-it starts the
JVM with `-cp package/target/lib/*`, which picks up jars in directory order,
and the failures track the runner image exactly. From 2026-10-18 to 2026-10-20:
- `ubuntu-24.04-arm` 20261012.149.1: 0 of 48 jobs hit it.
- `ubuntu-24.04-arm` 20261018.154.1: 4 of 4 jobs hit it.
The same commit, 9ee941103, [failed on
.154.1](https://github.com/apache/gluten/actions/runs/39333463531/job/118338487147)
and [passed on
.149.1](https://github.com/apache/gluten/actions/runs/39347323279/job/118384087739),
with the same JDK and the same jars.
It also affects deployments that copy the bundle into `$SPARK_HOME/jars`.
wForget reported the mirror-image error on #13096, on Spark 3.5.4
([job](https://github.com/wForget/benchmarks-spark-native/actions/runs/38689658927/job/116313644110)).
Changes:
- **`package/pom.xml`:** drop the `com.google.flatbuffers` relocation. The
bundled Arrow then references the same flatbuffers names as upstream Arrow.
Mixing it with Spark's Arrow becomes ordinary version skew instead of a
guaranteed VerifyError.
- **`dev/check-arrow-c-shading.sh`:** add a check that no bundled
`org/apache/arrow/**` class references anything under
`${gluten.shade.packageName}`. It runs in the existing `verify-arrow-c-shading`
step. On the current main bundle it fails for 124 classes.
### Why removing the relocation is safe
The relocation came in with #1030, together with the `org.apache.arrow`
relocation, to keep Gluten's Arrow 15 apart from Spark's flatbuffers 1.12.0.
That isolation stopped working once `flatbuf` kept its real name in #13096. It
also isn't needed:
- **Only Arrow uses flatbuffers in the bundle.** Gluten has no direct
dependency on flatbuffers and doesn't reference it in its sources.
- **Arrow 15 and Arrow 12 both link against either flatbuffers version.**
Arrow 15 references 33 flatbuffers members, and all of them exist in
flatbuffers-java 1.12.0 except `Constants.FLATBUFFERS_23_5_26()`. Likewise,
every flatbuffers member Arrow 12 references exists in 23.5.26 except
`Constants.FLATBUFFERS_1_12_0()`. Those two constants are called only from the
generated `ValidateVersion()` methods, and neither Arrow version ever calls
`ValidateVersion()`.
- **Recommended setups behave as before.** With `extraClassPath` the bundle
comes first, so the bundled Arrow keeps using the bundled flatbuffers 23.5.26,
same as today.
- **Spark 4.x is unchanged.** Its bundle doesn't ship Arrow or flatbuffers
(`provided` since #12737).
## How was this patch tested?
- **`./build/mvn verify -pl package -am -DskipTests` on every profile set
that CI's install jobs use:** the new check reports `OK org/apache/arrow/**
references no gluten-shaded package` on all of them:
- Spark 3.4, 3.5, 4.0 and 4.1 with `-Pbackends-velox`;
- Spark 3.5 with `-Pceleborn`;
- Spark 3.5 with `-Pjava-17 -Pkafka -Pspark-ut`;
- Spark 3.5 and 4.0 with `-Piceberg -Pdelta -Ppaimon -Phudi -Pceleborn
-Puniffle`;
- Spark 4.2 with `-Pdelta -Pspark-ut`.
- **Against the bundle from main,** the check reports `FAIL
org/apache/arrow/** — 124 class(es) reference gluten-shaded packages:
org/apache/gluten/shaded/com/google/flatbuffers`.
- **JVM-only repro on JDK 8:** the Spark 3.5 runtime classpath plus the
bundle. I ran all 120 orderings of `arrow-vector-12.0.1`,
`arrow-format-12.0.1`, `flatbuffers-java-1.12.0`, `arrow-memory-core-12.0.1`
and the bundle. Each run round-trips a `Schema` and an IPC record batch.
- main: 80/120 pass and 40/120 fail with the CI's VerifyError.
- This PR: 120/120 pass.
- Spot checks on JDK 17 give the same result.
The new check only runs where the image has `unzip`, for example
`tpc-test-ubuntu-oom`, `tpc-test-ubuntu-randomkill` and
`tpc-test-centos8-uniffle`. In images without `unzip`, the script already skips
every check. That predates this PR and is left for a follow-up.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: IBM Bob
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
