Repository: spark Updated Branches: refs/heads/master 92024797a -> 431a3d04b
[SPARK-12653][SQL] Re-enable test "SPARK-8489: MissingRequirementError during reflection" ## What changes were proposed in this pull request? The purpose of [SPARK-12653](https://issues.apache.org/jira/browse/SPARK-12653) is re-enabling a regression test. Historically, the target regression test is added by [SPARK-8498](https://github.com/apache/spark/commit/093c34838d1db7a9375f36a9a2ab5d96a23ae683), but is temporarily disabled by [SPARK-12615](https://github.com/apache/spark/commit/8ce645d4eeda203cf5e100c4bdba2d71edd44e6a) due to binary compatibility error. The following is the current error message at the submitting spark job with the pre-built `test.jar` file in the target regression test. ``` Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.SparkContext$.$lessinit$greater$default$6()Lscala/collection/Map; ``` Simple rebuilding `test.jar` can not recover the purpose of testcase since we need to support both Scala 2.10 and 2.11 for a while. For example, we will face the following Scala 2.11 error if we use `test.jar` built by Scala 2.10. ``` Exception in thread "main" java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaMirrors$JavaMirror; ``` This PR replace the existing `test.jar` with `test-2.10.jar` and `test-2.11.jar` and improve the regression test to use the suitable jar file. ## How was this patch tested? Pass the existing Jenkins test. Author: Dongjoon Hyun <dongj...@apache.org> Closes #11744 from dongjoon-hyun/SPARK-12653. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/431a3d04 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/431a3d04 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/431a3d04 Branch: refs/heads/master Commit: 431a3d04b437df4ab323fd925f57873aa6b8a0c8 Parents: 9202479 Author: Dongjoon Hyun <dongj...@apache.org> Authored: Wed Mar 16 09:05:53 2016 +0000 Committer: Sean Owen <so...@cloudera.com> Committed: Wed Mar 16 09:05:53 2016 +0000 ---------------------------------------------------------------------- .../regression-test-SPARK-8489/test-2.10.jar | Bin 0 -> 6873 bytes .../regression-test-SPARK-8489/test-2.11.jar | Bin 0 -> 7039 bytes .../resources/regression-test-SPARK-8489/test.jar | Bin 6828 -> 0 bytes .../spark/sql/hive/HiveSparkSubmitSuite.scala | 9 +++++++-- 4 files changed, 7 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/431a3d04/sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.10.jar ---------------------------------------------------------------------- diff --git a/sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.10.jar b/sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.10.jar new file mode 100644 index 0000000..26d410f Binary files /dev/null and b/sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.10.jar differ http://git-wip-us.apache.org/repos/asf/spark/blob/431a3d04/sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.11.jar ---------------------------------------------------------------------- diff --git a/sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.11.jar b/sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.11.jar new file mode 100644 index 0000000..f347847 Binary files /dev/null and b/sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.11.jar differ http://git-wip-us.apache.org/repos/asf/spark/blob/431a3d04/sql/hive/src/test/resources/regression-test-SPARK-8489/test.jar ---------------------------------------------------------------------- diff --git a/sql/hive/src/test/resources/regression-test-SPARK-8489/test.jar b/sql/hive/src/test/resources/regression-test-SPARK-8489/test.jar deleted file mode 100644 index 5944aa6..0000000 Binary files a/sql/hive/src/test/resources/regression-test-SPARK-8489/test.jar and /dev/null differ http://git-wip-us.apache.org/repos/asf/spark/blob/431a3d04/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSparkSubmitSuite.scala ---------------------------------------------------------------------- diff --git a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSparkSubmitSuite.scala b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSparkSubmitSuite.scala index 12a5542..c14c94f 100644 --- a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSparkSubmitSuite.scala +++ b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSparkSubmitSuite.scala @@ -22,6 +22,7 @@ import java.sql.Timestamp import java.util.Date import scala.collection.mutable.ArrayBuffer +import scala.tools.nsc.Properties import org.scalatest.{BeforeAndAfterEach, Matchers} import org.scalatest.concurrent.Timeouts @@ -87,13 +88,17 @@ class HiveSparkSubmitSuite runSparkSubmit(args) } - ignore("SPARK-8489: MissingRequirementError during reflection") { + test("SPARK-8489: MissingRequirementError during reflection") { // This test uses a pre-built jar to test SPARK-8489. In a nutshell, this test creates // a HiveContext and uses it to create a data frame from an RDD using reflection. // Before the fix in SPARK-8470, this results in a MissingRequirementError because // the HiveContext code mistakenly overrides the class loader that contains user classes. // For more detail, see sql/hive/src/test/resources/regression-test-SPARK-8489/*scala. - val testJar = "sql/hive/src/test/resources/regression-test-SPARK-8489/test.jar" + val version = Properties.versionNumberString match { + case v if v.startsWith("2.10") || v.startsWith("2.11") => v.substring(0, 4) + case x => throw new Exception(s"Unsupported Scala Version: $x") + } + val testJar = s"sql/hive/src/test/resources/regression-test-SPARK-8489/test-$version.jar" val args = Seq( "--conf", "spark.ui.enabled=false", "--conf", "spark.master.rest.enabled=false", --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
