Josh Rosen created SPARK-36774: ---------------------------------- Summary: Use SparkSubmitTestUtils to core and use it in SparkSubmitSuite Key: SPARK-36774 URL: https://issues.apache.org/jira/browse/SPARK-36774 Project: Spark Issue Type: Improvement Components: Tests Affects Versions: 3.0.0 Reporter: Josh Rosen Assignee: Josh Rosen
{{SparkSubmitSuite}} test failures can be painful to debug: currently, failed tests result in messages like {code:java} org.scalatest.exceptions.TestFailedException: Process returned with exit code 1. See the log4j logs for more detail. {code} which require the developer to hunt in test logs in order to identify the root cause of the test failure. In contrast, {{HiveSparkSubmitSuite}}'s output packs lots more useful information into its test failure messages: {code:java} [info] - temporary Hive UDF: define a UDF and use it *** FAILED *** (4 seconds, 135 milliseconds) [info] spark-submit returned with exit code 101. [info] Command line: '/Users/joshrosen/oss-spark/bin/spark-submit' '--class' 'wrongClassName' '--name' 'TemporaryHiveUDFTest' '--master' 'local-cluster[2,1,1024]' '--conf' 'spark.ui.enabled=false' '--conf' 'spark.master.rest.enabled=false' '--driver-java-options' '-Dderby.system.durability=test' '--jars' 'file:/Users/joshrosen/oss-spark/target/tmp/spark-32d0a47c-33eb-4488-866a-9994b3727b5b/testJar-1631766233448.jar,file:/Users/joshrosen/oss-spark/target/tmp/spark-e9e32588-83fa-43bd-a60f-9538fd30ab9a/testJar-1631766233601.jar' 'file:/Users/joshrosen/oss-spark/target/tmp/spark-daef4628-d953-41e3-b935-a726a7796418/testJar-1631766232701.jar' 'SparkSubmitClassA' 'SparkSubmitClassB' [info] [info] 2021-09-15 21:23:54.752 - stderr> SLF4J: Class path contains multiple SLF4J bindings. [info] 2021-09-15 21:23:54.752 - stderr> SLF4J: Found binding in [jar:file:/Users/joshrosen/oss-spark/assembly/target/scala-2.12/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] [info] 2021-09-15 21:23:54.752 - stderr> SLF4J: Found binding in [jar:file:/Users/joshrosen/.m2/repository/org/slf4j/slf4j-log4j12/1.7.30/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] [info] 2021-09-15 21:23:54.752 - stderr> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. [info] 2021-09-15 21:23:54.753 - stderr> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] [info] 2021-09-15 21:23:55.623 - stderr> Error: Failed to load class wrongClassName. (SparkSubmitTestUtils.scala:97) [info] org.scalatest.exceptions.TestFailedException: [info] at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472) [...] {code} This is done via code in {{SparkSubmitTestUtils}}, which is currently located in the {{sql/hive}} module. I propose a refactoring to move this into the {{core}} module so that it can be used by both {{SparkSubmitSuite}} and {{HiveSparkSubmitSuite}}, enriching the former with nicer error outputs. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org