LuciferYang opened a new pull request, #12543:
URL: https://github.com/apache/gluten/pull/12543
### What changes were proposed in this pull request?
Follow-up to #12525 (which cleaned up doc / velox / SparkJvmOptions
leftovers) and #12541 (CH-side workarounds). This picks up the remaining Spark
3.1 / 3.2-era leftovers in non-ClickHouse modules.
**Comment-only fixes**
- `gluten-substrait/.../WriteFilesExecTransformer.scala`: comment updated
from "Spark3.2/3.3/3.4" to "Spark 3.3/3.4", with an extra sentence clarifying
that Spark 3.5 strips these fields natively via SPARK-43123. Manual cleanup is
still needed for the 3.3/3.4 branches (SPARK-43123 was not backported), so the
code path stays.
- `gluten-arrow/.../ArrowColumnarBatch.scala`: rewrite the class docstring.
The old claim "Spark-3.2 declares ColumnarBatch as final" is factually wrong on
every currently supported branch (`public class ColumnarBatch implements
AutoCloseable` on 3.3 / 3.4 / 3.5 / 4.0 / 4.1). Replaced with a factual note
that this is a historical fork kept standalone to preserve the writable-column
contract.
- `gluten-celeborn/.../CelebornShuffleManager.java`: drop the "Added in
SPARK-32055, for Spark 3.1 and above" comment. That condition holds on every
currently supported Spark version, so the note has no signal.
**Small code refactors**
- `backends-velox/.../velox/VeloxFormatWriterInjects.scala`: drop the "Do
NOT add override keyword for compatibility on spark 3.1" comment and add the
`override` keyword to the anonymous `OutputWriter.path()`. Spark's
`OutputWriter.path(): String` is abstract on every supported branch, so
`override` is legal there and now matches the neighbouring `override def
close()`.
- `tools/gluten-it/common/.../SparkJvmOptions.java`: replace the
reflection-based lookup of `org.apache.spark.launcher.JavaModuleOptions` with a
direct `JavaModuleOptions.defaultModuleOptions()` call. The class is `public`
with a public static `defaultModuleOptions()` method, documented `@since
3.3.0`, and reachable via the `spark-launcher` transitive dep of `spark-core`
on every supported branch. The Spark 3.2 `ClassNotFoundException` fallback was
already removed by #12525; this eliminates the remaining reflection scaffolding.
No behavior change on any supported Spark version.
### How was this patch tested?
- `./build/mvn -Pbackends-velox -Pspark-3.5 -pl backends-velox -am
test-compile -DskipTests`: SUCCESS
- `./build/mvn -Pbackends-velox -Pspark-3.3 -pl backends-velox -am
test-compile -DskipTests`: SUCCESS
- `./build/mvn -Pbackends-velox -Pspark-3.5 -pl gluten-arrow -am
test-compile -DskipTests`: SUCCESS
- `./build/mvn -Pbackends-velox -Pspark-3.5 -pl gluten-substrait -am
test-compile -DskipTests`: SUCCESS
- `./build/mvn -Pbackends-velox -Pspark-3.5 -Pceleborn -pl gluten-celeborn
-am test-compile -DskipTests`: SUCCESS
- `javac -cp spark-launcher_2.12-3.3.1.jar
tools/gluten-it/common/.../SparkJvmOptions.java`: SUCCESS
- `javac -cp spark-launcher_2.12-3.5.7.jar
tools/gluten-it/common/.../SparkJvmOptions.java`: SUCCESS
- `javac -cp spark-launcher_2.13-4.1.0-SNAPSHOT.jar
tools/gluten-it/common/.../SparkJvmOptions.java`: SUCCESS
- Ran the compiled `SparkJvmOptions#main` against
`spark-launcher_2.12-3.5.7` and confirmed the output matches Spark's
`JavaModuleOptions.defaultModuleOptions()` string.
- Cross-checked upstream Spark on `branch-3.3` / `branch-3.4` / `branch-3.5`
/ `branch-4.0` / `branch-4.1`:
- `ColumnarBatch` is `public class ColumnarBatch implements AutoCloseable`
(non-final) on all branches.
- `OutputWriter.path(): String` is abstract on all branches.
- `JavaModuleOptions.defaultModuleOptions()` exists as `public static` on
all branches.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude claude-opus-4-7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
