Kousuke Saruta created SPARK-37159: -------------------------------------- Summary: Change HiveExternalCatalogVersionsSuite to be able to test with Java 17 Key: SPARK-37159 URL: https://issues.apache.org/jira/browse/SPARK-37159 Project: Spark Issue Type: Bug Components: SQL, Tests Affects Versions: 3.3.0 Reporter: Kousuke Saruta Assignee: Kousuke Saruta
SPARK-37105 seems to have fixed most of tests in `sql/hive` for Java 17 but `HiveExternalCatalogVersionsSuite`. {code} [info] org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite *** ABORTED *** (42 seconds, 526 milliseconds) [info] spark-submit returned with exit code 1. [info] Command line: '/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/bin/spark-submit' '--name' 'prepare testing tables' '--master' 'local[2]' '--conf' 'spark.ui.enabled=false' '--conf' 'spark.master.rest.enabled=false' '--conf' 'spark.sql.hive.metastore.version=2.3' '--conf' 'spark.sql.hive.metastore.jars=maven' '--conf' 'spark.sql.warehouse.dir=/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/warehouse-69d9bdbc-54ce-443b-8677-a413663ddb62' '--conf' 'spark.sql.test.version.index=0' '--driver-java-options' '-Dderby.system.home=/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/warehouse-69d9bdbc-54ce-443b-8677-a413663ddb62' '/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test15166225869206697603.py' [info] [info] 2021-10-28 06:07:18.486 - stderr> Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties [info] 2021-10-28 06:07:18.49 - stderr> 21/10/28 22:07:18 INFO SparkContext: Running Spark version 3.2.0 [info] 2021-10-28 06:07:18.537 - stderr> 21/10/28 22:07:18 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [info] 2021-10-28 06:07:18.616 - stderr> 21/10/28 22:07:18 INFO ResourceUtils: ============================================================== [info] 2021-10-28 06:07:18.616 - stderr> 21/10/28 22:07:18 INFO ResourceUtils: No custom resources configured for spark.driver. [info] 2021-10-28 06:07:18.616 - stderr> 21/10/28 22:07:18 INFO ResourceUtils: ============================================================== [info] 2021-10-28 06:07:18.617 - stderr> 21/10/28 22:07:18 INFO SparkContext: Submitted application: prepare testing tables [info] 2021-10-28 06:07:18.632 - stderr> 21/10/28 22:07:18 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0) [info] 2021-10-28 06:07:18.641 - stderr> 21/10/28 22:07:18 INFO ResourceProfile: Limiting resource is cpu [info] 2021-10-28 06:07:18.641 - stderr> 21/10/28 22:07:18 INFO ResourceProfileManager: Added ResourceProfile id: 0 [info] 2021-10-28 06:07:18.679 - stderr> 21/10/28 22:07:18 INFO SecurityManager: Changing view acls to: kou [info] 2021-10-28 06:07:18.679 - stderr> 21/10/28 22:07:18 INFO SecurityManager: Changing modify acls to: kou [info] 2021-10-28 06:07:18.68 - stderr> 21/10/28 22:07:18 INFO SecurityManager: Changing view acls groups to: [info] 2021-10-28 06:07:18.68 - stderr> 21/10/28 22:07:18 INFO SecurityManager: Changing modify acls groups to: [info] 2021-10-28 06:07:18.68 - stderr> 21/10/28 22:07:18 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(kou); groups with view permissions: Set(); users with modify permissions: Set(kou); groups with modify permissions: Set() [info] 2021-10-28 06:07:18.886 - stderr> 21/10/28 22:07:18 INFO Utils: Successfully started service 'sparkDriver' on port 35867. [info] 2021-10-28 06:07:18.906 - stderr> 21/10/28 22:07:18 INFO SparkEnv: Registering MapOutputTracker [info] 2021-10-28 06:07:18.93 - stderr> 21/10/28 22:07:18 INFO SparkEnv: Registering BlockManagerMaster [info] 2021-10-28 06:07:18.943 - stderr> 21/10/28 22:07:18 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information [info] 2021-10-28 06:07:18.944 - stderr> 21/10/28 22:07:18 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up [info] 2021-10-28 06:07:18.945 - stdout> Traceback (most recent call last): [info] 2021-10-28 06:07:18.946 - stdout> File "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test15166225869206697603.py", line 5, in <module> [info] 2021-10-28 06:07:18.946 - stdout> spark = SparkSession.builder.enableHiveSupport().getOrCreate() [info] 2021-10-28 06:07:18.946 - stdout> File "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/pyspark.zip/pyspark/sql/session.py", line 228, in getOrCreate [info] 2021-10-28 06:07:18.946 - stdout> File "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/pyspark.zip/pyspark/context.py", line 392, in getOrCreate [info] 2021-10-28 06:07:18.946 - stdout> File "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/pyspark.zip/pyspark/context.py", line 146, in __init__ [info] 2021-10-28 06:07:18.946 - stdout> File "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/pyspark.zip/pyspark/context.py", line 209, in _do_init [info] 2021-10-28 06:07:18.946 - stdout> File "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/pyspark.zip/pyspark/context.py", line 329, in _initialize_context [info] 2021-10-28 06:07:18.946 - stdout> File "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/py4j-0.10.9.2-src.zip/py4j/java_gateway.py", line 1573, in __call__ [info] 2021-10-28 06:07:18.946 - stdout> File "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/py4j-0.10.9.2-src.zip/py4j/protocol.py", line 326, in get_return_value [info] 2021-10-28 06:07:18.947 - stdout> py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext. [info] 2021-10-28 06:07:18.948 - stdout> : java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x724b93a8) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x724b93a8 [info] 2021-10-28 06:07:18.948 - stdout> at org.apache.spark.storage.StorageUtils$.<init>(StorageUtils.scala:213) [info] 2021-10-28 06:07:18.948 - stdout> at org.apache.spark.storage.StorageUtils$.<clinit>(StorageUtils.scala) [info] 2021-10-28 06:07:18.948 - stdout> at org.apache.spark.storage.BlockManagerMasterEndpoint.<init>(BlockManagerMasterEndpoint.scala:110) [info] 2021-10-28 06:07:18.948 - stdout> at org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:348) [info] 2021-10-28 06:07:18.948 - stdout> at org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:287) [info] 2021-10-28 06:07:18.948 - stdout> at org.apache.spark.SparkEnv$.create(SparkEnv.scala:336) [info] 2021-10-28 06:07:18.948 - stdout> at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:191) [info] 2021-10-28 06:07:18.948 - stdout> at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:277) [info] 2021-10-28 06:07:18.948 - stdout> at org.apache.spark.SparkContext.<init>(SparkContext.scala:460) [info] 2021-10-28 06:07:18.948 - stdout> at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58) [info] 2021-10-28 06:07:18.948 - stdout> at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [info] 2021-10-28 06:07:18.948 - stdout> at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) [info] 2021-10-28 06:07:18.948 - stdout> at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [info] 2021-10-28 06:07:18.948 - stdout> at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) [info] 2021-10-28 06:07:18.948 - stdout> at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480) [info] 2021-10-28 06:07:18.948 - stdout> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247) [info] 2021-10-28 06:07:18.948 - stdout> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) [info] 2021-10-28 06:07:18.948 - stdout> at py4j.Gateway.invoke(Gateway.java:238) [info] 2021-10-28 06:07:18.948 - stdout> at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80) [info] 2021-10-28 06:07:18.948 - stdout> at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69) [info] 2021-10-28 06:07:18.948 - stdout> at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182) [info] 2021-10-28 06:07:18.948 - stdout> at py4j.ClientServerConnection.run(ClientServerConnection.java:106) [info] 2021-10-28 06:07:18.948 - stdout> at java.base/java.lang.Thread.run(Thread.java:833) [info] 2021-10-28 06:07:18.948 - stdout> [info] 2021-10-28 06:07:18.975 - stderr> 21/10/28 22:07:18 INFO ShutdownHookManager: Shutdown hook called [info] 2021-10-28 06:07:18.976 - stderr> 21/10/28 22:07:18 INFO ShutdownHookManager: Deleting directory /tmp/spark-5bd38557-2b71-43f1-80ee-3648a456b3fc (SparkSubmitTestUtils.scala:96) [info] org.scalatest.exceptions.TestFailedException: [info] at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472) [info] at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471) [info] at org.scalatest.funsuite.AnyFunSuite.newAssertionFailedException(AnyFunSuite.scala:1563) [info] at org.scalatest.Assertions.fail(Assertions.scala:933) [info] at org.scalatest.Assertions.fail$(Assertions.scala:929) [info] at org.scalatest.funsuite.AnyFunSuite.fail(AnyFunSuite.scala:1563) [info] at org.apache.spark.deploy.SparkSubmitTestUtils.runSparkSubmit(SparkSubmitTestUtils.scala:96) [info] at org.apache.spark.deploy.SparkSubmitTestUtils.runSparkSubmit$(SparkSubmitTestUtils.scala:43) [info] at org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.runSparkSubmit(HiveExternalCatalogVersionsSuite.scala:56) [info] at org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.$anonfun$beforeAll$2(HiveExternalCatalogVersionsSuite.scala:223) [info] at org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.$anonfun$beforeAll$2$adapted(HiveExternalCatalogVersionsSuite.scala:204) [info] at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) [info] at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) [info] at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) [info] at org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.beforeAll(HiveExternalCatalogVersionsSuite.scala:204) [info] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212) [info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210) [info] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208) [info] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:62) [info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:318) [info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:513) [info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:413) [info] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [info] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [info] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [info] at java.base/java.lang.Thread.run(Thread.java:833) {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org