obs0811 opened a new pull request #32394: URL: https://github.com/apache/spark/pull/32394
### What changes were proposed in this pull request? This PR fixes an error in `BenchmarkBase.scala` that occurs when creating a benchmark file in a non-existent directory. ### Why are the changes needed? When submitting a benchmark job using `org.apache.spark.benchmark.Benchmarks` class with `SPARK_GENERATE_BENCHMARK_FILES=1` option, an exception is raised if the directory where the benchmark file will be generated does not exist. For more information, please refer to [SPARK-35266](https://issues.apache.org/jira/browse/SPARK-35266). ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? After building Spark, manually tested with the following command: ``` SPARK_GENERATE_BENCHMARK_FILES=1 bin/spark-submit --class \ org.apache.spark.benchmark.Benchmarks --jars \ "`find . -name '*-SNAPSHOT-tests.jar' -o -name '*avro*-SNAPSHOT.jar' | paste -sd ',' -`" \ "`find . -name 'spark-core*-SNAPSHOT-tests.jar'`" \ "org.apache.spark.ml.linalg.BLASBenchmark" ``` It successfully generated the benchmark result files. **Why it is sufficient:** As illustrated in the comments in `Benchmarks.scala`, the command below runs all benchmarks and generates the results: ``` SPARK_GENERATE_BENCHMARK_FILES=1 bin/spark-submit --class \ org.apache.spark.benchmark.Benchmarks --jars \ "`find . -name '*-SNAPSHOT-tests.jar' -o -name '*avro*-SNAPSHOT.jar' | paste -sd ',' -`" \ "`find . -name 'spark-core*-SNAPSHOT-tests.jar'`" \ "*" ``` Of all the benchmarks (55 benchmarks in total), only `BLASBenchmark` fails due to the proposed issue for the current code in the master branch. Thus, it is currently sufficient to test `BLASBenchmark` to validate this change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
