Byungsoo Oh created SPARK-35266:
-----------------------------------
Summary: Fix an error in BenchmarkBase.scala that occurs when
creating a benchmark file in a non-existent directory
Key: SPARK-35266
URL: https://issues.apache.org/jira/browse/SPARK-35266
Project: Spark
Issue Type: Bug
Components: Tests
Affects Versions: 3.2.0
Reporter: Byungsoo Oh
When submitting a benchmark job using _org.apache.spark.benchmark.Benchmarks_
class with _SPARK_GENERATE_BENCHMARK_FILES=1_ option, an exception is raised if
the directory where the benchmark file will be generated does not exist.
For example, if you execute _BLASBenchmark_ like the command below, you get an
error unless you manually create _benchmarks/_ directory under
_spark/mllib-local/_.
{code:java}
SPARK_GENERATE_BENCHMARK_FILES=1 bin/spark-submit \
--driver-memory 6g --class org.apache.spark.benchmark.Benchmarks \
--jars "`find . -name '*-SNAPSHOT-tests.jar' -o -name '*avro*-SNAPSHOT.jar' |
paste -sd ',' -`" \
"`find . -name 'spark-core*-SNAPSHOT-tests.jar'`" \
"org.apache.spark.ml.linalg.BLASBenchmark"
{code}
This is caused by the code in _BenchmarkBase.scala_ where an attempt is made to
create the benchmark file without validating the path.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
