dongjoon-hyun opened a new pull request, #60111:
URL: https://github.com/apache/spark/pull/60111
### What changes were proposed in this pull request?
Spark requires Java 21 or later since SPARK-60669 (`java.minimum.version` is
21), so code that branched on the running Java version always takes the same
side. This PR removes the branches that can no longer run, and the helpers that
only served them:
- `SparkSystemUtils`: remove `isJavaVersionAtMost17` (always `false`).
`isJavaVersionAtLeast21` (always `true`) is kept for now because
`SQLQueryTestSuite`, `ThriftServerQueryTestSuite`, `IsolatedClientLoader` and
`HiveClientVersions` still use it in other open PRs; it can be removed once
they are merged.
- `RestSubmissionServer`: call `Executors.newVirtualThreadPerTaskExecutor()`
directly instead of the `isJavaVersionAtLeast21` check plus the reflective
lookup. The config `spark.master.rest.virtualThread.enabled` keeps its name,
default and behavior.
- Tests keep only the Java 21 side:
- `StandaloneRestSubmitSuite`: the virtual threads executor is always set.
- `UtilsSuite`: `createDirectory` on a symbolic link always succeeds
(drops the SPARK-50946 `17.0.14` branch).
- `QueryExecutionErrorsSuite`: the lambda class name is
`Outer$$Lambda/...` and the `substring` error message is `Range [5, 6) out of
bounds for length 5`.
- `WriteAheadLogSuite`: always use `ByteBuffer.allocate(0)` instead of
`mock[ByteBuffer]` (SPARK-40731), keeping a short comment on why.
- `Platform`: always pick the `DirectByteBuffer(long, long)` constructor.
`majorVersion` stays because the `< 26` cleaner selection still uses it.
- `ClosureCleaner`: drop the always-true `javaVersion >= 18` term in
`cloneIndyLambda`, and rename `getFinalModifiersFieldForJava17` to
`getFinalModifiersField` with its comments fixed (the logic is unchanged; the
fallback that uses it is only reached on Java 21 when
`jdk.reflect.useDirectMethodHandle` is not `true`).
The code that depends on Java 22, 24, 25 and 26 (`NativeLibrary`,
`SerializationDebugger`, `SizeEstimator`, `Platform`'s `< 26`,
`ClosureCleaner`'s `>= 22`) and `HiveExternalCatalogVersionsSuite`'s version
map are intentionally left as they are.
### Why are the changes needed?
After SPARK-60669, one side of each of these branches is dead code. Removing
it makes the code and the tests easier to read, and it avoids a reflective call
for an API that is always available.
### Does this PR introduce _any_ user-facing change?
No. Every removed branch was unreachable because the minimum Java version is
21, so the remaining side is exactly what already ran.
### How was this patch tested?
Ran on JDK 21, and the suites below again on JDK 25:
```
build/sbt common-utils/Test/compile unsafe/Test/compile core/Test/compile
streaming/Test/compile sql/Test/compile
build/sbt "unsafe/test" \
"core/testOnly *StandaloneRestSubmitSuite org.apache.spark.util.UtilsSuite
*ClosureCleanerSuite*" \
"streaming/testOnly *WriteAheadLogSuite" "sql/testOnly
*QueryExecutionErrorsSuite"
```
- JDK 21: all pass (`unsafe` 161, `core` 125, `streaming` 31, `sql` 65
tests). JDK 25: all pass (`unsafe/testOnly *PlatformUtilSuite` 9, and the same
`core`, `streaming`, `sql` suites).
- `mimaReportBinaryIssues` for `common-utils` and `core` against 4.2.0
passes (`isJavaVersionAtMost17` is only on `private[spark]` `SparkSystemUtils`).
- `scalastyle` for `common-utils`, `core` (main and test), `streaming`
(test) and `sql` (test), and `checkstyle:check` for `common/unsafe` report no
violations.
- Also ran `Platform.allocateDirectBuffer` as a standalone smoke test on JDK
21 and 25 with the new constructor selection.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
