Yang Jie created SPARK-34900:
--------------------------------
Summary: Some `spark-submit` commands used to run benchmarks in
the user's guide is wrong
Key: SPARK-34900
URL: https://issues.apache.org/jira/browse/SPARK-34900
Project: Spark
Issue Type: Bug
Components: Tests
Affects Versions: 3.2.0
Reporter: Yang Jie
For example, the guide for running JoinBenchmark as follows:
{code:java}
/**
* Benchmark to measure performance for joins.
* To run this benchmark:
* {{{
* 1. without sbt:
* bin/spark-submit --class <this class> --jars <spark core test jar>
<spark sql test jar>
* 2. build/sbt "sql/test:runMain <this class>"
* 3. generate result:
* SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain <this
class>"
* Results will be written to "benchmarks/JoinBenchmark-results.txt".
* }}}
*/
object JoinBenchmark extends SqlBasedBenchmark {
{code}
but if we run JoinBenchmark with commnad
{code:java}
bin/spark-submit --class org.apache.spark.sql.execution.benchmark.JoinBenchmark
--jars spark-core_2.12-3.2.0-SNAPSHOT-tests.jar
spark-sql_2.12-3.2.0-SNAPSHOT-tests.jar
{code}
The following exception will be thrown：
{code:java}
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/spark/sql/catalyst/plans/SQLHelper
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369){code}
because SqlBasedBenchmark trait extends BenchmarkBase and SQLHelper, SQLHelper
def in spark-catalyst-tests.jar.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
