[jira] [Updated] (HIVE-12538) After set spark related config, SparkSession never get reused
[ https://issues.apache.org/jira/browse/HIVE-12538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nemon Lou updated HIVE-12538: - Attachment: HIVE-12538.5.patch Uploading a patch addressing the following issues : 1, Property isSparkConfigUpdated of HiveConf has not been maintained properly , as shown in the unit test added by this patch . 2, In case of a asynchronous query, isSparkConfigUpdated should be set from session level instead of operation level . The second issue is hard to add a test case . > After set spark related config, SparkSession never get reused > - > > Key: HIVE-12538 > URL: https://issues.apache.org/jira/browse/HIVE-12538 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 1.3.0 >Reporter: Nemon Lou >Assignee: Nemon Lou > Attachments: HIVE-12538.1.patch, HIVE-12538.2.patch, > HIVE-12538.3.patch, HIVE-12538.4.patch, HIVE-12538.5.patch, HIVE-12538.patch > > > Hive on Spark yarn-cluster mode. > After setting "set spark.yarn.queue=QueueA;" , > run the query "select count(*) from test" 3 times and you will find 3 > different yarn applications. > Two of the yarn applications in FINISHED & SUCCEEDED state,and one in RUNNING > & UNDEFINED state waiting for next work. > And if you submit one more "select count(*) from test" ,the third one will be > in FINISHED & SUCCEEDED state and a new yarn application will start up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12538) After set spark related config, SparkSession never get reused
[ https://issues.apache.org/jira/browse/HIVE-12538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nemon Lou updated HIVE-12538: - Attachment: HIVE-12538.4.patch Changing to session level spark session lock. Changing back to HashMap. > After set spark related config, SparkSession never get reused > - > > Key: HIVE-12538 > URL: https://issues.apache.org/jira/browse/HIVE-12538 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 1.3.0 >Reporter: Nemon Lou >Assignee: Nemon Lou > Attachments: HIVE-12538.1.patch, HIVE-12538.2.patch, > HIVE-12538.3.patch, HIVE-12538.4.patch, HIVE-12538.patch > > > Hive on Spark yarn-cluster mode. > After setting "set spark.yarn.queue=QueueA;" , > run the query "select count(*) from test" 3 times and you will find 3 > different yarn applications. > Two of the yarn applications in FINISHED & SUCCEEDED state,and one in RUNNING > & UNDEFINED state waiting for next work. > And if you submit one more "select count(*) from test" ,the third one will be > in FINISHED & SUCCEEDED state and a new yarn application will start up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12538) After set spark related config, SparkSession never get reused
[ https://issues.apache.org/jira/browse/HIVE-12538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nemon Lou updated HIVE-12538: - Attachment: HIVE-12538.3.patch Using ConcurrentHashMap instead of HashMap. > After set spark related config, SparkSession never get reused > - > > Key: HIVE-12538 > URL: https://issues.apache.org/jira/browse/HIVE-12538 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 1.3.0 >Reporter: Nemon Lou >Assignee: Nemon Lou > Attachments: HIVE-12538.1.patch, HIVE-12538.2.patch, > HIVE-12538.3.patch, HIVE-12538.patch > > > Hive on Spark yarn-cluster mode. > After setting "set spark.yarn.queue=QueueA;" , > run the query "select count(*) from test" 3 times and you will find 3 > different yarn applications. > Two of the yarn applications in FINISHED & SUCCEEDED state,and one in RUNNING > & UNDEFINED state waiting for next work. > And if you submit one more "select count(*) from test" ,the third one will be > in FINISHED & SUCCEEDED state and a new yarn application will start up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12538) After set spark related config, SparkSession never get reused
[ https://issues.apache.org/jira/browse/HIVE-12538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nemon Lou updated HIVE-12538: - Attachment: HIVE-12538.2.patch Addressing issues mentioned in comments. > After set spark related config, SparkSession never get reused > - > > Key: HIVE-12538 > URL: https://issues.apache.org/jira/browse/HIVE-12538 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 1.3.0 >Reporter: Nemon Lou >Assignee: Nemon Lou > Attachments: HIVE-12538.1.patch, HIVE-12538.2.patch, HIVE-12538.patch > > > Hive on Spark yarn-cluster mode. > After setting "set spark.yarn.queue=QueueA;" , > run the query "select count(*) from test" 3 times and you will find 3 > different yarn applications. > Two of the yarn applications in FINISHED & SUCCEEDED state,and one in RUNNING > & UNDEFINED state waiting for next work. > And if you submit one more "select count(*) from test" ,the third one will be > in FINISHED & SUCCEEDED state and a new yarn application will start up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12538) After set spark related config, SparkSession never get reused
[ https://issues.apache.org/jira/browse/HIVE-12538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nemon Lou updated HIVE-12538: - Attachment: HIVE-12538.1.patch > After set spark related config, SparkSession never get reused > - > > Key: HIVE-12538 > URL: https://issues.apache.org/jira/browse/HIVE-12538 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 1.3.0 >Reporter: Nemon Lou >Assignee: Nemon Lou > Attachments: HIVE-12538.1.patch, HIVE-12538.patch > > > Hive on Spark yarn-cluster mode. > After setting "set spark.yarn.queue=QueueA;" , > run the query "select count(*) from test" 3 times and you will find 3 > different yarn applications. > Two of the yarn applications in FINISHED & SUCCEEDED state,and one in RUNNING > & UNDEFINED state waiting for next work. > And if you submit one more "select count(*) from test" ,the third one will be > in FINISHED & SUCCEEDED state and a new yarn application will start up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12538) After set spark related config, SparkSession never get reused
[ https://issues.apache.org/jira/browse/HIVE-12538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nemon Lou updated HIVE-12538: - Attachment: HIVE-12538.patch 1,Fixing the bug : " isSparkConfigUpdated = isSparkRelatedConfig(name)" 2,Setting "isSparkConfigUpdated =false " at session level 3,Taking concurrency into consideration by adding a session level conf lock.As multiple queries can run in a single session. > After set spark related config, SparkSession never get reused > - > > Key: HIVE-12538 > URL: https://issues.apache.org/jira/browse/HIVE-12538 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 1.3.0 >Reporter: Nemon Lou >Assignee: Nemon Lou > Attachments: HIVE-12538.patch > > > Hive on Spark yarn-cluster mode. > After setting "set spark.yarn.queue=QueueA;" , > run the query "select count(*) from test" 3 times and you will find 3 > different yarn applications. > Two of the yarn applications in FINISHED & SUCCEEDED state,and one in RUNNING > & UNDEFINED state waiting for next work. > And if you submit one more "select count(*) from test" ,the third one will be > in FINISHED & SUCCEEDED state and a new yarn application will start up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12538) After set spark related config, SparkSession never get reused
[ https://issues.apache.org/jira/browse/HIVE-12538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nemon Lou updated HIVE-12538: - Description: Hive on Spark yarn-cluster mode. After setting "set spark.yarn.queue=QueueA;" , run the query "select count(*) from test" 3 times and you will find 3 different yarn applications. Two of the yarn applications in FINISHED & SUCCEEDED state,and one in RUNNING & UNDEFINED state waiting for next work. And if you submit one more "select count(*) from test" ,the third one will be in FINISHED & SUCCEEDED state and a new yarn application will start up. was: Hive on Spark yarn-cluster mode. After setting "set spark.yarn.queue=QueueA;" , run the query "select count(*) from test" 3 times and you will find 3 different yarn applications. > After set spark related config, SparkSession never get reused > - > > Key: HIVE-12538 > URL: https://issues.apache.org/jira/browse/HIVE-12538 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 1.3.0 >Reporter: Nemon Lou > > Hive on Spark yarn-cluster mode. > After setting "set spark.yarn.queue=QueueA;" , > run the query "select count(*) from test" 3 times and you will find 3 > different yarn applications. > Two of the yarn applications in FINISHED & SUCCEEDED state,and one in RUNNING > & UNDEFINED state waiting for next work. > And if you submit one more "select count(*) from test" ,the third one will be > in FINISHED & SUCCEEDED state and a new yarn application will start up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
