This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new f610c5136809 [SPARK-60987][INFRA] Fix `NoClassDefFoundError` when
running Kafka benchmarks in the benchmark workflow
f610c5136809 is described below
commit f610c513680919d3f2995f8fbdaaed0defa21661
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Nov 5 22:22:03 2026 -0800
[SPARK-60987][INFRA] Fix `NoClassDefFoundError` when running Kafka
benchmarks in the benchmark workflow
### What changes were proposed in this pull request?
This PR adds the Kafka-related jars to the `--jars` option of
`spark-submit` in the `Run benchmarks` step of
`.github/workflows/benchmark.yml`:
- `spark-*kafka-0-10*-SNAPSHOT.jar` (the main jars of
`spark-sql-kafka-0-10` and `spark-token-provider-kafka-0-10`, built by
`Test/package`)
- From the coursier cache (the same way as `curator-test` is handled):
`kafka*.jar` (`kafka-clients`, `kafka_2.13`, `kafka-server`, `kafka-metadata`,
`kafka-raft`, `kafka-storage`, ...), `metrics-core-2.2.0.jar`, `jopt-simple-*`,
`scala-java8-compat_*`, `scala-logging_*`, `jackson-dataformat-csv-*`, and
`commons-pool2-*`. `-sources` and `-javadoc` jars are excluded.
The list was derived from `sql-kafka-0-10/Test/dependencyClasspath`,
keeping only the jars that are not part of the Spark distribution.
### Why are the changes needed?
`RTMKafkaKafkaBenchmark` and the other `RTMKafka*Benchmark` classes cannot
run in the benchmark workflow. Their classes are discovered through
`*-SNAPSHOT-tests.jar`, but `kafka-clients` and the other Kafka dependencies
are neither in `--jars` nor in the Spark distribution, so the JVM fails while
loading the class:
```
Running org.apache.spark.sql.kafka010.benchmark.RTMKafkaKafkaBenchmark:
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/kafka/clients/producer/Callback
at
org.apache.spark.sql.kafka010.benchmark.RTMKafkaKafkaBenchmark.main(RTMKafkaKafkaBenchmark.scala)
```
Because `SPARK_BENCHMARK_FAILFAST` defaults to `true`, this fails the whole
split. For example, the "Run benchmarks: * (JDK 21, Scala 2.13, 1 out of 64
splits)" job fails this way:
-
https://github.com/dongjoon-hyun/spark/actions/runs/41684094207/job/127686555631
-
https://github.com/dongjoon-hyun/spark/actions/runs/41684094207/job/127686511345
### Does this PR introduce _any_ user-facing change?
No. This only changes the GitHub Actions benchmark workflow.
### How was this patch tested?
Manually verified that each `find` pattern matches the required jars (e.g.,
`kafka-clients-3.9.2.jar`, `kafka_2.13-3.9.2.jar`,
`spark-sql-kafka-0-10_2.13-5.0.0-SNAPSHOT.jar`) against the local build output
and Maven repository.
The benchmark itself needs to be verified with the `Run benchmarks`
workflow in GitHub Actions on
`org.apache.spark.sql.kafka010.benchmark.RTMKafkaKafkaBenchmark`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 5.5
Closes #60161 from dongjoon-hyun/SPARK-60987.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.github/workflows/benchmark.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/benchmark.yml b/.github/workflows/benchmark.yml
index 17fbbbca5941..d7ee40bfb3cb 100644
--- a/.github/workflows/benchmark.yml
+++ b/.github/workflows/benchmark.yml
@@ -193,7 +193,7 @@ jobs:
# In benchmark, we use local as master so set driver memory only. Note
that GitHub Actions has 7 GB memory limit.
bin/spark-submit \
--driver-memory 6g --class org.apache.spark.benchmark.Benchmarks \
- --jars "`find . -name '*-SNAPSHOT-tests.jar' -o -name
'*avro*-SNAPSHOT.jar' | paste -sd ',' -`,`find ~/.cache/coursier -name
'curator-test-*.jar'`" \
+ --jars "`find . -name '*-SNAPSHOT-tests.jar' -o -name
'*avro*-SNAPSHOT.jar' -o -name 'spark-*kafka-0-10*-SNAPSHOT.jar' | paste -sd
',' -`,`find ~/.cache/coursier -name 'curator-test-*.jar'`,`find
~/.cache/coursier \( -name 'kafka*.jar' -o -name 'metrics-core-2.2.0.jar' -o
-name 'jopt-simple-*.jar' -o -name 'scala-java8-compat_*.jar' -o -name
'scala-logging_*.jar' -o -name 'jackson-dataformat-csv-*.jar' -o -name
'commons-pool2-*.jar' \) ! -name '*-sources.jar' ! -name '*-javadoc [...]
"`find . -name 'spark-core*-SNAPSHOT-tests.jar'`" \
"${{ inputs.class }}"
# To keep the directory structure and file permissions, tar them
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]