[jira] [Commented] (SPARK-37159) Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
[
https://issues.apache.org/jira/browse/SPARK-37159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17472434#comment-17472434
]
Kousuke Saruta commented on SPARK-37159:
All right. Thank you [~dongjoon]!
> Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
> ---
>
> Key: SPARK-37159
> URL: https://issues.apache.org/jira/browse/SPARK-37159
> Project: Spark
> Issue Type: Sub-task
> Components: SQL, Tests
>Affects Versions: 3.3.0
>Reporter: Kousuke Saruta
>Assignee: Kousuke Saruta
>Priority: Minor
> Fix For: 3.3.0
>
>
> SPARK-37105 seems to have fixed most of tests in `sql/hive` for Java 17 but
> `HiveExternalCatalogVersionsSuite`.
> {code}
> [info] org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite *** ABORTED
> *** (42 seconds, 526 milliseconds)
> [info] spark-submit returned with exit code 1.
> [info] Command line:
> '/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/bin/spark-submit'
> '--name' 'prepare testing tables' '--master' 'local[2]' '--conf'
> 'spark.ui.enabled=false' '--conf' 'spark.master.rest.enabled=false' '--conf'
> 'spark.sql.hive.metastore.version=2.3' '--conf'
> 'spark.sql.hive.metastore.jars=maven' '--conf'
> 'spark.sql.warehouse.dir=/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/warehouse-69d9bdbc-54ce-443b-8677-a413663ddb62'
> '--conf' 'spark.sql.test.version.index=0' '--driver-java-options'
> '-Dderby.system.home=/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/warehouse-69d9bdbc-54ce-443b-8677-a413663ddb62'
>
> '/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test15166225869206697603.py'
> [info]
> [info] 2021-10-28 06:07:18.486 - stderr> Using Spark's default log4j
> profile: org/apache/spark/log4j-defaults.properties
> [info] 2021-10-28 06:07:18.49 - stderr> 21/10/28 22:07:18 INFO
> SparkContext: Running Spark version 3.2.0
> [info] 2021-10-28 06:07:18.537 - stderr> 21/10/28 22:07:18 WARN
> NativeCodeLoader: Unable to load native-hadoop library for your platform...
> using builtin-java classes where applicable
> [info] 2021-10-28 06:07:18.616 - stderr> 21/10/28 22:07:18 INFO
> ResourceUtils: ==
> [info] 2021-10-28 06:07:18.616 - stderr> 21/10/28 22:07:18 INFO
> ResourceUtils: No custom resources configured for spark.driver.
> [info] 2021-10-28 06:07:18.616 - stderr> 21/10/28 22:07:18 INFO
> ResourceUtils: ==
> [info] 2021-10-28 06:07:18.617 - stderr> 21/10/28 22:07:18 INFO
> SparkContext: Submitted application: prepare testing tables
> [info] 2021-10-28 06:07:18.632 - stderr> 21/10/28 22:07:18 INFO
> ResourceProfile: Default ResourceProfile created, executor resources:
> Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name:
> memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount:
> 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
> [info] 2021-10-28 06:07:18.641 - stderr> 21/10/28 22:07:18 INFO
> ResourceProfile: Limiting resource is cpu
> [info] 2021-10-28 06:07:18.641 - stderr> 21/10/28 22:07:18 INFO
> ResourceProfileManager: Added ResourceProfile id: 0
> [info] 2021-10-28 06:07:18.679 - stderr> 21/10/28 22:07:18 INFO
> SecurityManager: Changing view acls to: kou
> [info] 2021-10-28 06:07:18.679 - stderr> 21/10/28 22:07:18 INFO
> SecurityManager: Changing modify acls to: kou
> [info] 2021-10-28 06:07:18.68 - stderr> 21/10/28 22:07:18 INFO
> SecurityManager: Changing view acls groups to:
> [info] 2021-10-28 06:07:18.68 - stderr> 21/10/28 22:07:18 INFO
> SecurityManager: Changing modify acls groups to:
> [info] 2021-10-28 06:07:18.68 - stderr> 21/10/28 22:07:18 INFO
> SecurityManager: SecurityManager: authentication disabled; ui acls disabled;
> users with view permissions: Set(kou); groups with view permissions: Set();
> users with modify permissions: Set(kou); groups with modify permissions:
> Set()
> [info] 2021-10-28 06:07:18.886 - stderr> 21/10/28 22:07:18 INFO Utils:
> Successfully started service 'sparkDriver' on port 35867.
> [info] 2021-10-28 06:07:18.906 - stderr> 21/10/28 22:07:18 INFO SparkEnv:
> Registering MapOutputTracker
> [info] 2021-10-28 06:07:18.93 - stderr> 21/10/28 22:07:18 INFO SparkEnv:
> Registering BlockManagerMaster
> [info] 2021-10-28 06:07:18.943 - stderr> 21/10/28 22:07:18 INFO
>
[jira] [Commented] (SPARK-37159) Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
[
https://issues.apache.org/jira/browse/SPARK-37159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17472409#comment-17472409
]
Dongjoon Hyun commented on SPARK-37159:
---
I'll collect this to a subtask of SPARK-33772, [~sarutak].
> Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
> ---
>
> Key: SPARK-37159
> URL: https://issues.apache.org/jira/browse/SPARK-37159
> Project: Spark
> Issue Type: Bug
> Components: SQL, Tests
>Affects Versions: 3.3.0
>Reporter: Kousuke Saruta
>Assignee: Kousuke Saruta
>Priority: Minor
> Fix For: 3.3.0
>
>
> SPARK-37105 seems to have fixed most of tests in `sql/hive` for Java 17 but
> `HiveExternalCatalogVersionsSuite`.
> {code}
> [info] org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite *** ABORTED
> *** (42 seconds, 526 milliseconds)
> [info] spark-submit returned with exit code 1.
> [info] Command line:
> '/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/bin/spark-submit'
> '--name' 'prepare testing tables' '--master' 'local[2]' '--conf'
> 'spark.ui.enabled=false' '--conf' 'spark.master.rest.enabled=false' '--conf'
> 'spark.sql.hive.metastore.version=2.3' '--conf'
> 'spark.sql.hive.metastore.jars=maven' '--conf'
> 'spark.sql.warehouse.dir=/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/warehouse-69d9bdbc-54ce-443b-8677-a413663ddb62'
> '--conf' 'spark.sql.test.version.index=0' '--driver-java-options'
> '-Dderby.system.home=/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/warehouse-69d9bdbc-54ce-443b-8677-a413663ddb62'
>
> '/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test15166225869206697603.py'
> [info]
> [info] 2021-10-28 06:07:18.486 - stderr> Using Spark's default log4j
> profile: org/apache/spark/log4j-defaults.properties
> [info] 2021-10-28 06:07:18.49 - stderr> 21/10/28 22:07:18 INFO
> SparkContext: Running Spark version 3.2.0
> [info] 2021-10-28 06:07:18.537 - stderr> 21/10/28 22:07:18 WARN
> NativeCodeLoader: Unable to load native-hadoop library for your platform...
> using builtin-java classes where applicable
> [info] 2021-10-28 06:07:18.616 - stderr> 21/10/28 22:07:18 INFO
> ResourceUtils: ==
> [info] 2021-10-28 06:07:18.616 - stderr> 21/10/28 22:07:18 INFO
> ResourceUtils: No custom resources configured for spark.driver.
> [info] 2021-10-28 06:07:18.616 - stderr> 21/10/28 22:07:18 INFO
> ResourceUtils: ==
> [info] 2021-10-28 06:07:18.617 - stderr> 21/10/28 22:07:18 INFO
> SparkContext: Submitted application: prepare testing tables
> [info] 2021-10-28 06:07:18.632 - stderr> 21/10/28 22:07:18 INFO
> ResourceProfile: Default ResourceProfile created, executor resources:
> Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name:
> memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount:
> 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
> [info] 2021-10-28 06:07:18.641 - stderr> 21/10/28 22:07:18 INFO
> ResourceProfile: Limiting resource is cpu
> [info] 2021-10-28 06:07:18.641 - stderr> 21/10/28 22:07:18 INFO
> ResourceProfileManager: Added ResourceProfile id: 0
> [info] 2021-10-28 06:07:18.679 - stderr> 21/10/28 22:07:18 INFO
> SecurityManager: Changing view acls to: kou
> [info] 2021-10-28 06:07:18.679 - stderr> 21/10/28 22:07:18 INFO
> SecurityManager: Changing modify acls to: kou
> [info] 2021-10-28 06:07:18.68 - stderr> 21/10/28 22:07:18 INFO
> SecurityManager: Changing view acls groups to:
> [info] 2021-10-28 06:07:18.68 - stderr> 21/10/28 22:07:18 INFO
> SecurityManager: Changing modify acls groups to:
> [info] 2021-10-28 06:07:18.68 - stderr> 21/10/28 22:07:18 INFO
> SecurityManager: SecurityManager: authentication disabled; ui acls disabled;
> users with view permissions: Set(kou); groups with view permissions: Set();
> users with modify permissions: Set(kou); groups with modify permissions:
> Set()
> [info] 2021-10-28 06:07:18.886 - stderr> 21/10/28 22:07:18 INFO Utils:
> Successfully started service 'sparkDriver' on port 35867.
> [info] 2021-10-28 06:07:18.906 - stderr> 21/10/28 22:07:18 INFO SparkEnv:
> Registering MapOutputTracker
> [info] 2021-10-28 06:07:18.93 - stderr> 21/10/28 22:07:18 INFO SparkEnv:
> Registering BlockManagerMaster
> [info] 2021-10-28 06:07:18.943 - stderr> 21/10/28 22:07:18 INFO
>
[jira] [Commented] (SPARK-37159) Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
[
https://issues.apache.org/jira/browse/SPARK-37159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17435819#comment-17435819
]
Apache Spark commented on SPARK-37159:
--
User 'sarutak' has created a pull request for this issue:
https://github.com/apache/spark/pull/34425
> Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
> ---
>
> Key: SPARK-37159
> URL: https://issues.apache.org/jira/browse/SPARK-37159
> Project: Spark
> Issue Type: Bug
> Components: SQL, Tests
>Affects Versions: 3.3.0
>Reporter: Kousuke Saruta
>Assignee: Kousuke Saruta
>Priority: Minor
>
> SPARK-37105 seems to have fixed most of tests in `sql/hive` for Java 17 but
> `HiveExternalCatalogVersionsSuite`.
> {code}
> [info] org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite *** ABORTED
> *** (42 seconds, 526 milliseconds)
> [info] spark-submit returned with exit code 1.
> [info] Command line:
> '/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/bin/spark-submit'
> '--name' 'prepare testing tables' '--master' 'local[2]' '--conf'
> 'spark.ui.enabled=false' '--conf' 'spark.master.rest.enabled=false' '--conf'
> 'spark.sql.hive.metastore.version=2.3' '--conf'
> 'spark.sql.hive.metastore.jars=maven' '--conf'
> 'spark.sql.warehouse.dir=/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/warehouse-69d9bdbc-54ce-443b-8677-a413663ddb62'
> '--conf' 'spark.sql.test.version.index=0' '--driver-java-options'
> '-Dderby.system.home=/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/warehouse-69d9bdbc-54ce-443b-8677-a413663ddb62'
>
> '/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test15166225869206697603.py'
> [info]
> [info] 2021-10-28 06:07:18.486 - stderr> Using Spark's default log4j
> profile: org/apache/spark/log4j-defaults.properties
> [info] 2021-10-28 06:07:18.49 - stderr> 21/10/28 22:07:18 INFO
> SparkContext: Running Spark version 3.2.0
> [info] 2021-10-28 06:07:18.537 - stderr> 21/10/28 22:07:18 WARN
> NativeCodeLoader: Unable to load native-hadoop library for your platform...
> using builtin-java classes where applicable
> [info] 2021-10-28 06:07:18.616 - stderr> 21/10/28 22:07:18 INFO
> ResourceUtils: ==
> [info] 2021-10-28 06:07:18.616 - stderr> 21/10/28 22:07:18 INFO
> ResourceUtils: No custom resources configured for spark.driver.
> [info] 2021-10-28 06:07:18.616 - stderr> 21/10/28 22:07:18 INFO
> ResourceUtils: ==
> [info] 2021-10-28 06:07:18.617 - stderr> 21/10/28 22:07:18 INFO
> SparkContext: Submitted application: prepare testing tables
> [info] 2021-10-28 06:07:18.632 - stderr> 21/10/28 22:07:18 INFO
> ResourceProfile: Default ResourceProfile created, executor resources:
> Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name:
> memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount:
> 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
> [info] 2021-10-28 06:07:18.641 - stderr> 21/10/28 22:07:18 INFO
> ResourceProfile: Limiting resource is cpu
> [info] 2021-10-28 06:07:18.641 - stderr> 21/10/28 22:07:18 INFO
> ResourceProfileManager: Added ResourceProfile id: 0
> [info] 2021-10-28 06:07:18.679 - stderr> 21/10/28 22:07:18 INFO
> SecurityManager: Changing view acls to: kou
> [info] 2021-10-28 06:07:18.679 - stderr> 21/10/28 22:07:18 INFO
> SecurityManager: Changing modify acls to: kou
> [info] 2021-10-28 06:07:18.68 - stderr> 21/10/28 22:07:18 INFO
> SecurityManager: Changing view acls groups to:
> [info] 2021-10-28 06:07:18.68 - stderr> 21/10/28 22:07:18 INFO
> SecurityManager: Changing modify acls groups to:
> [info] 2021-10-28 06:07:18.68 - stderr> 21/10/28 22:07:18 INFO
> SecurityManager: SecurityManager: authentication disabled; ui acls disabled;
> users with view permissions: Set(kou); groups with view permissions: Set();
> users with modify permissions: Set(kou); groups with modify permissions:
> Set()
> [info] 2021-10-28 06:07:18.886 - stderr> 21/10/28 22:07:18 INFO Utils:
> Successfully started service 'sparkDriver' on port 35867.
> [info] 2021-10-28 06:07:18.906 - stderr> 21/10/28 22:07:18 INFO SparkEnv:
> Registering MapOutputTracker
> [info] 2021-10-28 06:07:18.93 - stderr> 21/10/28 22:07:18 INFO SparkEnv:
> Registering BlockManagerMaster
> [info] 2021-10-28 06:07:18.943 - stderr> 21/10/28 22:07:18
