This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new f610c5136809 [SPARK-60987][INFRA] Fix `NoClassDefFoundError` when 
running Kafka benchmarks in the benchmark workflow
f610c5136809 is described below

commit f610c513680919d3f2995f8fbdaaed0defa21661
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Nov 5 22:22:03 2026 -0800

    [SPARK-60987][INFRA] Fix `NoClassDefFoundError` when running Kafka 
benchmarks in the benchmark workflow
    
    ### What changes were proposed in this pull request?
    
    This PR adds the Kafka-related jars to the `--jars` option of 
`spark-submit` in the `Run benchmarks` step of 
`.github/workflows/benchmark.yml`:
    
    - `spark-*kafka-0-10*-SNAPSHOT.jar` (the main jars of 
`spark-sql-kafka-0-10` and `spark-token-provider-kafka-0-10`, built by 
`Test/package`)
    - From the coursier cache (the same way as `curator-test` is handled): 
`kafka*.jar` (`kafka-clients`, `kafka_2.13`, `kafka-server`, `kafka-metadata`, 
`kafka-raft`, `kafka-storage`, ...), `metrics-core-2.2.0.jar`, `jopt-simple-*`, 
`scala-java8-compat_*`, `scala-logging_*`, `jackson-dataformat-csv-*`, and 
`commons-pool2-*`. `-sources` and `-javadoc` jars are excluded.
    
    The list was derived from `sql-kafka-0-10/Test/dependencyClasspath`, 
keeping only the jars that are not part of the Spark distribution.
    
    ### Why are the changes needed?
    
    `RTMKafkaKafkaBenchmark` and the other `RTMKafka*Benchmark` classes cannot 
run in the benchmark workflow. Their classes are discovered through 
`*-SNAPSHOT-tests.jar`, but `kafka-clients` and the other Kafka dependencies 
are neither in `--jars` nor in the Spark distribution, so the JVM fails while 
loading the class:
    
    ```
    Running org.apache.spark.sql.kafka010.benchmark.RTMKafkaKafkaBenchmark:
    Exception in thread "main" java.lang.NoClassDefFoundError: 
org/apache/kafka/clients/producer/Callback
            at 
org.apache.spark.sql.kafka010.benchmark.RTMKafkaKafkaBenchmark.main(RTMKafkaKafkaBenchmark.scala)
    ```
    
    Because `SPARK_BENCHMARK_FAILFAST` defaults to `true`, this fails the whole 
split. For example, the "Run benchmarks: * (JDK 21, Scala 2.13, 1 out of 64 
splits)" job fails this way:
    - 
https://github.com/dongjoon-hyun/spark/actions/runs/41684094207/job/127686555631
    - 
https://github.com/dongjoon-hyun/spark/actions/runs/41684094207/job/127686511345
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. This only changes the GitHub Actions benchmark workflow.
    
    ### How was this patch tested?
    
    Manually verified that each `find` pattern matches the required jars (e.g., 
`kafka-clients-3.9.2.jar`, `kafka_2.13-3.9.2.jar`, 
`spark-sql-kafka-0-10_2.13-5.0.0-SNAPSHOT.jar`) against the local build output 
and Maven repository.
    
    The benchmark itself needs to be verified with the `Run benchmarks` 
workflow in GitHub Actions on 
`org.apache.spark.sql.kafka010.benchmark.RTMKafkaKafkaBenchmark`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Sonnet 5.5
    
    Closes #60161 from dongjoon-hyun/SPARK-60987.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .github/workflows/benchmark.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.github/workflows/benchmark.yml b/.github/workflows/benchmark.yml
index 17fbbbca5941..d7ee40bfb3cb 100644
--- a/.github/workflows/benchmark.yml
+++ b/.github/workflows/benchmark.yml
@@ -193,7 +193,7 @@ jobs:
         # In benchmark, we use local as master so set driver memory only. Note 
that GitHub Actions has 7 GB memory limit.
         bin/spark-submit \
           --driver-memory 6g --class org.apache.spark.benchmark.Benchmarks \
-          --jars "`find . -name '*-SNAPSHOT-tests.jar' -o -name 
'*avro*-SNAPSHOT.jar' | paste -sd ',' -`,`find ~/.cache/coursier -name 
'curator-test-*.jar'`" \
+          --jars "`find . -name '*-SNAPSHOT-tests.jar' -o -name 
'*avro*-SNAPSHOT.jar' -o -name 'spark-*kafka-0-10*-SNAPSHOT.jar' | paste -sd 
',' -`,`find ~/.cache/coursier -name 'curator-test-*.jar'`,`find 
~/.cache/coursier \( -name 'kafka*.jar' -o -name 'metrics-core-2.2.0.jar' -o 
-name 'jopt-simple-*.jar' -o -name 'scala-java8-compat_*.jar' -o -name 
'scala-logging_*.jar' -o -name 'jackson-dataformat-csv-*.jar' -o -name 
'commons-pool2-*.jar' \) ! -name '*-sources.jar' ! -name '*-javadoc [...]
           "`find . -name 'spark-core*-SNAPSHOT-tests.jar'`" \
           "${{ inputs.class }}"
         # To keep the directory structure and file permissions, tar them


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to