Josh Rosen created SPARK-36774:
----------------------------------
Summary: Use SparkSubmitTestUtils to core and use it in
SparkSubmitSuite
Key: SPARK-36774
URL: https://issues.apache.org/jira/browse/SPARK-36774
Project: Spark
Issue Type: Improvement
Components: Tests
Affects Versions: 3.0.0
Reporter: Josh Rosen
Assignee: Josh Rosen
{{SparkSubmitSuite}} test failures can be painful to debug: currently, failed
tests result in messages like
{code:java}
org.scalatest.exceptions.TestFailedException: Process returned with exit code
1. See the log4j logs for more detail.
{code}
which require the developer to hunt in test logs in order to identify the root
cause of the test failure.
In contrast, {{HiveSparkSubmitSuite}}'s output packs lots more useful
information into its test failure messages:
{code:java}
[info] - temporary Hive UDF: define a UDF and use it *** FAILED *** (4 seconds,
135 milliseconds)
[info] spark-submit returned with exit code 101.
[info] Command line: '/Users/joshrosen/oss-spark/bin/spark-submit' '--class'
'wrongClassName' '--name' 'TemporaryHiveUDFTest' '--master'
'local-cluster[2,1,1024]' '--conf' 'spark.ui.enabled=false' '--conf'
'spark.master.rest.enabled=false' '--driver-java-options'
'-Dderby.system.durability=test' '--jars'
'file:/Users/joshrosen/oss-spark/target/tmp/spark-32d0a47c-33eb-4488-866a-9994b3727b5b/testJar-1631766233448.jar,file:/Users/joshrosen/oss-spark/target/tmp/spark-e9e32588-83fa-43bd-a60f-9538fd30ab9a/testJar-1631766233601.jar'
'file:/Users/joshrosen/oss-spark/target/tmp/spark-daef4628-d953-41e3-b935-a726a7796418/testJar-1631766232701.jar'
'SparkSubmitClassA' 'SparkSubmitClassB'
[info]
[info] 2021-09-15 21:23:54.752 - stderr> SLF4J: Class path contains multiple
SLF4J bindings.
[info] 2021-09-15 21:23:54.752 - stderr> SLF4J: Found binding in
[jar:file:/Users/joshrosen/oss-spark/assembly/target/scala-2.12/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
[info] 2021-09-15 21:23:54.752 - stderr> SLF4J: Found binding in
[jar:file:/Users/joshrosen/.m2/repository/org/slf4j/slf4j-log4j12/1.7.30/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
[info] 2021-09-15 21:23:54.752 - stderr> SLF4J: See
http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
[info] 2021-09-15 21:23:54.753 - stderr> SLF4J: Actual binding is of type
[org.slf4j.impl.Log4jLoggerFactory]
[info] 2021-09-15 21:23:55.623 - stderr> Error: Failed to load class
wrongClassName. (SparkSubmitTestUtils.scala:97)
[info] org.scalatest.exceptions.TestFailedException:
[info] at
org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
[...]
{code}
This is done via code in {{SparkSubmitTestUtils}}, which is currently located
in the {{sql/hive}} module. I propose a refactoring to move this into the
{{core}} module so that it can be used by both {{SparkSubmitSuite}} and
{{HiveSparkSubmitSuite}}, enriching the former with nicer error outputs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
