dongjoon-hyun opened a new pull request, #60160:
URL: https://github.com/apache/spark/pull/60160
### What changes were proposed in this pull request?
This PR removes the Mockito dependency from
`HighlyCompressedMapStatusBenchmark`.
- Instead of mocking `SparkEnv` for each case, the benchmark now starts a
single `local[1]` `SparkContext` and changes
`spark.shuffle.accurateBlockSkewedFactor` on `SparkEnv.get.conf` before each
case. `HighlyCompressedMapStatus` reads this value from `SparkEnv.get.conf` on
every call, so the behavior is the same as the mock.
- The `SparkContext` is stopped before the "Driver heap retained by
MapOutputTracker" report, which creates its own `SparkContext`.
### Why are the changes needed?
The benchmark job fails with:
```
java.lang.NoClassDefFoundError: org/mockito/Mockito
at
org.apache.spark.HighlyCompressedMapStatusBenchmark$.envWithSkewedFactor(HighlyCompressedMapStatusBenchmark.scala:52)
```
(https://github.com/dongjoon-hyun/spark/actions/runs/41684094207/job/127686502284)
The GitHub Actions benchmark workflow runs `Benchmarks` through
`spark-submit --jars <test jars>`. `mockito-core` is a test-scope dependency of
`core`, so it is not on that classpath. `build/sbt "core/Test/runMain ..."`
does include test dependencies, which is why the failure does not reproduce
locally.
The benchmark was added by SPARK-48290 (a1253377c4d, 2026-10-19). Other
benchmarks do not depend on Mockito.
### Does this PR introduce _any_ user-facing change?
No. This only changes a benchmark.
### How was this patch tested?
Ran the benchmark locally and confirmed that it completes successfully and
that the reported numbers (for example, the bytes retained per map task) are
the same as before the change:
```
build/sbt "core/Test/runMain
org.apache.spark.HighlyCompressedMapStatusBenchmark"
```
The benchmark has not been run through `spark-submit --jars` yet, so the
GitHub Actions benchmark job needs to confirm the fix.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
