[jira] [Updated] (SPARK-23872) You can not connect to another metastore uri using two Spark sessions
[ https://issues.apache.org/jira/browse/SPARK-23872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Park Chan Min updated SPARK-23872: -- Description: In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session metasore information is used when the second session is run --- Code -- val spark_1 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_A:9083") .getOrCreate() spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() SparkSession.clearActiveSession() SparkSession.clearDefaultSession() val spark_2 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_B:9083") .getOrCreate() spark_2.sql("SELECT B_FIELD FROM TABLE_B").show() --- run info result in spark 2.1.0 -- INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s + + / A_FIELD / ++ / A / ++ .. INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_B{color}*:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s + + / B_FIELD / ++ / B / ++ .. --- run info result in spark 2.3.0 -- INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s + + / A_FIELD / ++ / A / ++ .. INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 .. Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or view not found: `default`.`TABLE_B`; line 1 pos 19; was: In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session metasore information is used when the second session is run --- Code -- val spark_1 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_A:9083") .getOrCreate() spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() SparkSession.clearActiveSession() SparkSession.clearDefaultSession() val spark_2 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_B:9083") .getOrCreate() spark_2.sql("SELECT B_FIELD FROM TABLE_B").show() --- spark 2.1.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s + + / A_FIELD / ++ / A / ++ .. INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_B{color}*:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s + + / B_FIELD / ++ / B / ++ .. --- spark 2.3.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s + + / A_FIELD / ++ / A / ++ .. INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 .. Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or view not found: `default`.`TABLE_B`; line 1 pos 19; > You can not connect to another metastore uri using two Spark sessions > - > > Key: SPARK-23872 > URL: https://issues.apache.org/jira/browse/SPARK-23872 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.0 > Environment: OS :CentOS release 6.8 (Final) > JAVA : build 1.8.0_101-b13 > SPARK : 2.3.0 > >Reporter: Park Chan Min >Priority: Major > > In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session > metasore information is used when the second session is run > --- Code -- > val spark_1 = SparkSession.builder() > .enableHiveSupport() > .config("hive.metastore.uris", "thrift://HOST_A:9083") > .getOrCreate() > spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() > SparkSession.clearActiveSession() > SparkSession.clearDefaultSession() > val spark_2 = SparkSession.builder() > .enableHiveSupport() > .config("hive.metastore.uris", "thrift://HOST_B:9083") > .getOrCreate() > spark_2.sql("SELECT B_FIELD FROM
[jira] [Updated] (SPARK-23872) You can not connect to another metastore uri using two Spark sessions
[ https://issues.apache.org/jira/browse/SPARK-23872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Park Chan Min updated SPARK-23872: -- Description: In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session metasore information is used when the second session is run --- Code -- val spark_1 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_A:9083") .getOrCreate() spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() SparkSession.clearActiveSession() SparkSession.clearDefaultSession() val spark_2 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_B:9083") .getOrCreate() spark_2.sql("SELECT B_FIELD FROM TABLE_B").show() --- spark 2.1.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s + + / A_FIELD / ++ / A / ++ .. INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_B{color}*:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s + + / B_FIELD / ++ / B / ++ .. --- spark 2.3.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s + + / A_FIELD / ++ / A / ++ .. INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 .. Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or view not found: `default`.`TABLE_B`; line 1 pos 19; was: In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session metasore information is used when the second session is run --- Code -- val spark_1 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_A:9083") .getOrCreate() spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() SparkSession.clearActiveSession() SparkSession.clearDefaultSession() val spark_2 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_B:9083") .getOrCreate() spark_2.sql("SELECT B_FIELD FROM TABLE_B").show() --- spark 2.1.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s + + / A_FIELD / ++ / A / ++ .. INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_B{color}*:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s + + / B_FIELD / ++ / B / ++ .. --- spark 2.3.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s + + / A_FIELD / ++ / A / ++ .. INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s .. Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or view not found: `default`.`TABLE_B`; line 1 pos 19; > You can not connect to another metastore uri using two Spark sessions > - > > Key: SPARK-23872 > URL: https://issues.apache.org/jira/browse/SPARK-23872 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.0 > Environment: OS :CentOS release 6.8 (Final) > JAVA : build 1.8.0_101-b13 > SPARK : 2.3.0 > >Reporter: Park Chan Min >Priority: Major > > In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session > metasore information is used when the second session is run > --- Code -- > val spark_1 = SparkSession.builder() > .enableHiveSupport() > .config("hive.metastore.uris", "thrift://HOST_A:9083") > .getOrCreate() > spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() > SparkSession.clearActiveSession() > SparkSession.clearDefaultSession() > val spark_2 = SparkSession.builder() > .enableHiveSupport() > .config("hive.metastore.uris", "thrift://HOST_B:9
[jira] [Updated] (SPARK-23872) You can not connect to another metastore uri using two Spark sessions
[ https://issues.apache.org/jira/browse/SPARK-23872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Park Chan Min updated SPARK-23872: -- Description: In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session metasore information is used when the second session is run --- Code -- val spark_1 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_A:9083") .getOrCreate() spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() SparkSession.clearActiveSession() SparkSession.clearDefaultSession() val spark_2 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_B:9083") .getOrCreate() spark_2.sql("SELECT B_FIELD FROM TABLE_B").show() --- spark 2.1.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s + + / A_FIELD / ++ / A / ++ .. INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_B{color}*:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s + + / B_FIELD / ++ / B / ++ .. --- spark 2.3.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s + + / A_FIELD / ++ / A / ++ .. INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s .. Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or view not found: `default`.`TABLE_B`; line 1 pos 19; was: In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session metasore information is used when the second session is run *--- Code --* val spark_1 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_A:9083") .getOrCreate() spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() SparkSession.clearActiveSession() SparkSession.clearDefaultSession() val spark_2 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_B:9083") .getOrCreate() spark_2.sql("SELECT B_FIELD FROM TABLE_B").show() *--- spark 2.1.0 result --* INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s + + / A_FIELD / ++ / A / ++ .. INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_B{color}*:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s + + / B_FIELD / ++ / B / ++ .. *--- spark 2.3.0 result --* INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s + + / A_FIELD / ++ / A / ++ .. INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s .. Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or view not found: `default`.`TABLE_B`; line 1 pos 19; > You can not connect to another metastore uri using two Spark sessions > - > > Key: SPARK-23872 > URL: https://issues.apache.org/jira/browse/SPARK-23872 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.0 > Environment: OS :CentOS release 6.8 (Final) > JAVA : build 1.8.0_101-b13 > SPARK : 2.3.0 > >Reporter: Park Chan Min >Priority: Major > > In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session > metasore information is used when the second session is run > --- Code -- > val spark_1 = SparkSession.builder() > .enableHiveSupport() > .config("hive.metastore.uris", "thrift://HOST_A:9083") > .getOrCreate() > spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() > SparkSession.clearActiveSession() > SparkSession.clearDefaultSession() > val spark_2 = SparkSes
[jira] [Updated] (SPARK-23872) You can not connect to another metastore uri using two Spark sessions
[ https://issues.apache.org/jira/browse/SPARK-23872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Park Chan Min updated SPARK-23872: -- Description: In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session metasore information is used when the second session is run *--- Code --* val spark_1 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_A:9083") .getOrCreate() spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() SparkSession.clearActiveSession() SparkSession.clearDefaultSession() val spark_2 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_B:9083") .getOrCreate() spark_2.sql("SELECT B_FIELD FROM TABLE_B").show() *--- spark 2.1.0 result --* INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s + + / A_FIELD / ++ / A / ++ .. INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_B{color}*:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s + + / B_FIELD / ++ / B / ++ .. *--- spark 2.3.0 result --* INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s + + / A_FIELD / ++ / A / ++ .. INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s .. Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or view not found: `default`.`TABLE_B`; line 1 pos 19; was: In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session metasore information is used when the second session is run --- Code-- val spark_1 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_A:9083") .getOrCreate() spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() SparkSession.clearActiveSession() SparkSession.clearDefaultSession() val spark_2 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_B:9083") .getOrCreate() spark_2.sql("SELECT B_FIELD FROM TABLE_B").show() --- spark 2.1.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s + + / A_FIELD / ++ / A / ++ .. INFO metastore: Trying to connect to metastore with URI thrift://HOST_B:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s + + / B_FIELD / ++ / B / ++ .. --- spark 2.3.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s + + / A_FIELD / ++ / A / ++ .. INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s .. Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or view not found: `default`.`TABLE_B`; line 1 pos 19; > You can not connect to another metastore uri using two Spark sessions > - > > Key: SPARK-23872 > URL: https://issues.apache.org/jira/browse/SPARK-23872 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.0 > Environment: OS :CentOS release 6.8 (Final) > JAVA : build 1.8.0_101-b13 > SPARK : 2.3.0 > >Reporter: Park Chan Min >Priority: Major > > In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session > metasore information is used when the second session is run > *--- Code --* > val spark_1 = SparkSession.builder() > .enableHiveSupport() > .config("hive.metastore.uris", "thrift://HOST_A:9083") > .getOrCreate() > spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() > SparkSession.clearActiveSession() > SparkSession.clearDefaultSession() > val spark_2 = SparkSession.bui
[jira] [Updated] (SPARK-23872) You can not connect to another metastore uri using two Spark sessions
[ https://issues.apache.org/jira/browse/SPARK-23872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Park Chan Min updated SPARK-23872: -- Description: In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session metasore information is used when the second session is run --- Code-- val spark_1 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_A:9083") .getOrCreate() spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() SparkSession.clearActiveSession() SparkSession.clearDefaultSession() val spark_2 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_B:9083") .getOrCreate() spark_2.sql("SELECT B_FIELD FROM TABLE_B").show() --- spark 2.1.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ ! A_FIELD ! +---+ !| A! +---+ .. INFO metastore: Trying to connect to metastore with URI thrift://HOST_B:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |B_FIELD| +---+ | B | +---+ .. --- spark 2.3.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |A_FIELD| +---+ | A | +---+ .. INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s .. Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or view not found: `default`.`TABLE_B`; line 1 pos 19; was: In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session metasore information is used when the second session is run --- Code-- val spark_1 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_A:9083") .getOrCreate() spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() SparkSession.clearActiveSession() SparkSession.clearDefaultSession() val spark_2 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_B:9083") .getOrCreate() spark_2.sql("SELECT B_FIELD FROM TABLE_B").show() --- spark 2.1.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |A_FIELD| +---+ '| A ' | +---+ .. INFO metastore: Trying to connect to metastore with URI thrift://HOST_B:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |B_FIELD| +---+ | B | +---+ .. --- spark 2.3.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |A_FIELD| +---+ | A | +---+ .. INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s .. Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or view not found: `default`.`TABLE_B`; line 1 pos 19; > You can not connect to another metastore uri using two Spark sessions > - > > Key: SPARK-23872 > URL: https://issues.apache.org/jira/browse/SPARK-23872 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.0 > Environment: OS :CentOS release 6.8 (Final) > JAVA : build 1.8.0_101-b13 > SPARK : 2.3.0 > >Reporter: Park Chan Min >Priority: Major > > In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session > metasore information is used when the second session is run > --- Code-- > val spark_1 = SparkSession.builder() > .enableHiveSupport() > .config("hive.metastore.uris", "thrift://HOST_A:9083") > .getOrCreate() > spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() > SparkSession.clearActiveSession() > SparkSession.clearDefaultSession() > val spark_2 = SparkSession.builder() > .enableHiveSupport() > .config("hive.metastore.uris", "thrift://HOST_B:9083") >
[jira] [Updated] (SPARK-23872) You can not connect to another metastore uri using two Spark sessions
[ https://issues.apache.org/jira/browse/SPARK-23872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Park Chan Min updated SPARK-23872: -- Description: In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session metasore information is used when the second session is run --- Code-- val spark_1 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_A:9083") .getOrCreate() spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() SparkSession.clearActiveSession() SparkSession.clearDefaultSession() val spark_2 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_B:9083") .getOrCreate() spark_2.sql("SELECT B_FIELD FROM TABLE_B").show() --- spark 2.1.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s + + / A_FIELD / ++ / A / ++ .. INFO metastore: Trying to connect to metastore with URI thrift://HOST_B:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s + + / B_FIELD / ++ / B / ++ .. --- spark 2.3.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s + + / A_FIELD / ++ / A / ++ .. INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s .. Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or view not found: `default`.`TABLE_B`; line 1 pos 19; was: In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session metasore information is used when the second session is run --- Code-- val spark_1 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_A:9083") .getOrCreate() spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() SparkSession.clearActiveSession() SparkSession.clearDefaultSession() val spark_2 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_B:9083") .getOrCreate() spark_2.sql("SELECT B_FIELD FROM TABLE_B").show() --- spark 2.1.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ ! A_FIELD ! +---+ !| A! +---+ .. INFO metastore: Trying to connect to metastore with URI thrift://HOST_B:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |B_FIELD| +---+ | B | +---+ .. --- spark 2.3.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |A_FIELD| +---+ | A | +---+ .. INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s .. Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or view not found: `default`.`TABLE_B`; line 1 pos 19; > You can not connect to another metastore uri using two Spark sessions > - > > Key: SPARK-23872 > URL: https://issues.apache.org/jira/browse/SPARK-23872 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.0 > Environment: OS :CentOS release 6.8 (Final) > JAVA : build 1.8.0_101-b13 > SPARK : 2.3.0 > >Reporter: Park Chan Min >Priority: Major > > In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session > metasore information is used when the second session is run > --- Code-- > val spark_1 = SparkSession.builder() > .enableHiveSupport() > .config("hive.metastore.uris", "thrift://HOST_A:9083") > .getOrCreate() > spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() > SparkSession.clearActiveSession() > SparkSession.clearDefaultSession() > val spark_2 = SparkSession.builder() > .enableHiveSupport() >
[jira] [Updated] (SPARK-23872) You can not connect to another metastore uri using two Spark sessions
[ https://issues.apache.org/jira/browse/SPARK-23872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Park Chan Min updated SPARK-23872: -- Description: In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session metasore information is used when the second session is run --- Code-- val spark_1 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_A:9083") .getOrCreate() spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() SparkSession.clearActiveSession() SparkSession.clearDefaultSession() val spark_2 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_B:9083") .getOrCreate() spark_2.sql("SELECT B_FIELD FROM TABLE_B").show() --- spark 2.1.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |A_FIELD| +---+ '| A ' | +---+ .. INFO metastore: Trying to connect to metastore with URI thrift://HOST_B:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |B_FIELD| +---+ | B | +---+ .. --- spark 2.3.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |A_FIELD| +---+ | A | +---+ .. INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s .. Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or view not found: `default`.`TABLE_B`; line 1 pos 19; was: In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session metasore information is used when the second session is run --- Code-- val spark_1 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_A:9083") .getOrCreate() spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() SparkSession.clearActiveSession() SparkSession.clearDefaultSession() val spark_2 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_B:9083") .getOrCreate() spark_2.sql("SELECT B_FIELD FROM TABLE_B").show() --- spark 2.1.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |A_FIELD| +---+ | A | +---+ .. INFO metastore: Trying to connect to metastore with URI thrift://HOST_B:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |B_FIELD| +---+ | B | +---+ .. --- spark 2.3.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |A_FIELD| +---+ | A | +---+ .. INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s .. Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or view not found: `default`.`TABLE_B`; line 1 pos 19; > You can not connect to another metastore uri using two Spark sessions > - > > Key: SPARK-23872 > URL: https://issues.apache.org/jira/browse/SPARK-23872 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.0 > Environment: OS :CentOS release 6.8 (Final) > JAVA : build 1.8.0_101-b13 > SPARK : 2.3.0 > >Reporter: Park Chan Min >Priority: Major > > In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session > metasore information is used when the second session is run > --- Code-- > val spark_1 = SparkSession.builder() > .enableHiveSupport() > .config("hive.metastore.uris", "thrift://HOST_A:9083") > .getOrCreate() > spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() > SparkSession.clearActiveSession() > SparkSession.clearDefaultSession() > val spark_2 = SparkSession.builder() > .enableHiveSupport() > .config("hive.metastore.uris", "thrift://HOST_B:9083") >
[jira] [Updated] (SPARK-23872) You can not connect to another metastore uri using two Spark sessions
[ https://issues.apache.org/jira/browse/SPARK-23872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Park Chan Min updated SPARK-23872: -- Description: In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session metasore information is used when the second session is run --- Code-- val spark_1 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_A:9083") .getOrCreate() spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() SparkSession.clearActiveSession() SparkSession.clearDefaultSession() val spark_2 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_B:9083") .getOrCreate() spark_2.sql("SELECT B_FIELD FROM TABLE_B").show() --- spark 2.1.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |A_FIELD| +---+ | A | +---+ .. INFO metastore: Trying to connect to metastore with URI thrift://HOST_B:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |B_FIELD| +---+ | B | +---+ .. --- spark 2.3.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |A_FIELD| +---+ | A | +---+ .. INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s .. Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or view not found: `default`.`TABLE_B`; line 1 pos 19; was: In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session metasore information is used when the second session is run --- Code-- val spark_1 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_A:9083") .getOrCreate() spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() SparkSession.clearActiveSession() SparkSession.clearDefaultSession() val spark_2 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_B:9083") .getOrCreate() spark_2.sql("SELECT B_FIELD FROM TABLE_B").show() --- spark 2.1.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |A_FIELD| +---+ | A | +---+ .. INFO metastore: Trying to connect to metastore with URI thrift://HOST_B:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |B_FIELD| +---+ | B | +---+ .. --- spark 2.3.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |A_FIELD| +---+ | A | +---+ .. INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s .. Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or view not found: `default`.`TABLE_B`; line 1 pos 19; .. > You can not connect to another metastore uri using two Spark sessions > - > > Key: SPARK-23872 > URL: https://issues.apache.org/jira/browse/SPARK-23872 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.0 > Environment: OS :CentOS release 6.8 (Final) > JAVA : build 1.8.0_101-b13 > SPARK : 2.3.0 > >Reporter: Park Chan Min >Priority: Major > > In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session > metasore information is used when the second session is run > --- Code-- > val spark_1 = SparkSession.builder() > .enableHiveSupport() > .config("hive.metastore.uris", "thrift://HOST_A:9083") > .getOrCreate() > spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() > SparkSession.clearActiveSession() > SparkSession.clearDefaultSession() > val spark_2 = SparkSession.builder() > .enableHiveSupport() > .config("hive.metastore.uris", "thrift://HOST_B:9083") >
[jira] [Updated] (SPARK-23872) You can not connect to another metastore uri using two Spark sessions
[ https://issues.apache.org/jira/browse/SPARK-23872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Park Chan Min updated SPARK-23872: -- Description: In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session metasore information is used when the second session is run --- Code-- val spark_1 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_A:9083") .getOrCreate() spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() SparkSession.clearActiveSession() SparkSession.clearDefaultSession() val spark_2 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_B:9083") .getOrCreate() spark_2.sql("SELECT B_FIELD FROM TABLE_B").show() --- spark 2.1.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |A_FIELD| +---+ | A | +---+ .. INFO metastore: Trying to connect to metastore with URI thrift://HOST_B:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |B_FIELD| +---+ | B | +---+ .. --- spark 2.3.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |A_FIELD| +---+ | A | +---+ .. INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s .. Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or view not found: `default`.`TABLE_B`; line 1 pos 19; .. was: In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session metasore information is used when the second session is run --- Code-- val spark_1 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_A:9083") .getOrCreate() spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() SparkSession.clearActiveSession() SparkSession.clearDefaultSession() val spark_2 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_B:9083") .getOrCreate() spark_2.sql("SELECT B_FIELD FROM TABLE_B").show() --- spark 2.1.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |A_FIELD| +---+ | A | +---+ .. INFO metastore: Trying to connect to metastore with URI thrift://HOST_B:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |B_FIELD| +---+ | B | +---+ .. --- spark 2.3.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |A_FIELD| +---+ | A | +---+ .. INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s .. Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or view not found: `default`.`TABLE_B`; line 1 pos 19; .. > You can not connect to another metastore uri using two Spark sessions > - > > Key: SPARK-23872 > URL: https://issues.apache.org/jira/browse/SPARK-23872 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.0 > Environment: OS :CentOS release 6.8 (Final) > JAVA : build 1.8.0_101-b13 > SPARK : 2.3.0 > >Reporter: Park Chan Min >Priority: Major > > In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session > metasore information is used when the second session is run > --- Code-- > val spark_1 = SparkSession.builder() > .enableHiveSupport() > .config("hive.metastore.uris", "thrift://HOST_A:9083") > .getOrCreate() > spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() > SparkSession.clearActiveSession() > SparkSession.clearDefaultSession() > val spark_2 = SparkSession.builder() > .enableHiveSupport() > .config("hive.metastore.uris", "thrift://HOST_B:9083"
[jira] [Updated] (SPARK-23872) You can not connect to another metastore uri using two Spark sessions
[ https://issues.apache.org/jira/browse/SPARK-23872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Park Chan Min updated SPARK-23872: -- Description: In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session metasore information is used when the second session is run Code -- val spark_1 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_A:9083") .getOrCreate() spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() SparkSession.clearActiveSession() SparkSession.clearDefaultSession() val spark_2 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_B:9083") .getOrCreate() spark_2.sql("SELECT B_FIELD FROM TABLE_B").show() --- spark 2.1.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://{color:#d04437}*HOST_A*{color}:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |A_FIELD| +---+ |A| +---+ .. INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_B{color}*:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |B_FIELD| +---+ |B| +---+ .. * -- --- - -- --- - -- --- - -- --- - -- --- spark 2.3.0 result --* INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |A_FIELD| +---+ |A| +---+ .. INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s .. Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or view not found: `default`.`TABLE_B`; line 1 pos 19; was: In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session metasore information is used when the second session is run. *--- Code --* val spark_1 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_A:9083") .getOrCreate() spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() SparkSession.clearActiveSession() SparkSession.clearDefaultSession() val spark_2 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_B:9083") .getOrCreate() spark_2.sql("SELECT B_FIELD FROM TABLE_B").show() *--- spark 2.1.0 result --* INFO metastore: Trying to connect to metastore with URI thrift://{color:#d04437}*HOST_A*{color}:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |A_FIELD| +---+ | A | +---+ .. INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_B{color}*:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |B_FIELD| +---+ | B | +---+ .. *--- spark 2.3.0 result --* INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |A_FIELD| +---+ | A | +---+ .. INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s .. Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or view not found: `default`.`TABLE_B`; line 1 pos 19; > You can not connect to another metastore uri using two Spark sessions > - > > Key: SPARK-23872 > URL: https://issues.apache.org/jira/browse/SPARK-23872 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.0 > Environment: OS :CentOS release 6.8 (Final) > JAVA : build 1.8.0_101-b13 > SPARK : 2.3.0 > >Reporter: Park Chan Min >Priority: Major > > In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session > metasore information is used when the second session is run > Code -- > val spark_1 = SparkSession.builder() > .enableHiveSupport() > .config("hive.metastore.uris", "thrift://HOST_A:9083") > .getOrCreate() > spark_1.sql("SELECT A_FIELD FROM TAB
[jira] [Updated] (SPARK-23872) You can not connect to another metastore uri using two Spark sessions
[ https://issues.apache.org/jira/browse/SPARK-23872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Park Chan Min updated SPARK-23872: -- Description: In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session metasore information is used when the second session is run --- Code-- val spark_1 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_A:9083") .getOrCreate() spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() SparkSession.clearActiveSession() SparkSession.clearDefaultSession() val spark_2 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_B:9083") .getOrCreate() spark_2.sql("SELECT B_FIELD FROM TABLE_B").show() --- spark 2.1.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |A_FIELD| +---+ | A | +---+ .. INFO metastore: Trying to connect to metastore with URI thrift://HOST_B:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |B_FIELD| +---+ | B | +---+ .. --- spark 2.3.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |A_FIELD| +---+ | A | +---+ .. INFO metastore: Trying to connect to metastore with URI thrift://HOST_A:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s .. Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or view not found: `default`.`TABLE_B`; line 1 pos 19; .. was: In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session metasore information is used when the second session is run Code -- val spark_1 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_A:9083") .getOrCreate() spark_1.sql("SELECT A_FIELD FROM TABLE_A").show() SparkSession.clearActiveSession() SparkSession.clearDefaultSession() val spark_2 = SparkSession.builder() .enableHiveSupport() .config("hive.metastore.uris", "thrift://HOST_B:9083") .getOrCreate() spark_2.sql("SELECT B_FIELD FROM TABLE_B").show() --- spark 2.1.0 result -- INFO metastore: Trying to connect to metastore with URI thrift://{color:#d04437}*HOST_A*{color}:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |A_FIELD| +---+ |A| +---+ .. INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_B{color}*:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |B_FIELD| +---+ |B| +---+ .. * -- --- - -- --- - -- --- - -- --- - -- --- spark 2.3.0 result --* INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 .. INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s +---+ |A_FIELD| +---+ |A| +---+ .. INFO metastore: Trying to connect to metastore with URI thrift://*{color:#d04437}HOST_A{color}*:9083 INFO DAGScheduler: Job 3 finished: show at SparkHDFSTest.scala:20, took 0.807905 s .. Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or view not found: `default`.`TABLE_B`; line 1 pos 19; > You can not connect to another metastore uri using two Spark sessions > - > > Key: SPARK-23872 > URL: https://issues.apache.org/jira/browse/SPARK-23872 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.0 > Environment: OS :CentOS release 6.8 (Final) > JAVA : build 1.8.0_101-b13 > SPARK : 2.3.0 > >Reporter: Park Chan Min >Priority: Major > > In Spark 2.1.0, two sessions worked normally In 2.3.0, the first session > metasore information is used when the second session is run > --- Code-- > val spark_1 = SparkSession.builder() > .enableHiveSupport() > .config("hive.metastore.uris", "thrift://HOST_A:9083") > .getOrCreate()