[jira] [Commented] (HIVE-17964) HoS: some spark configs doesn't require re-creating a session
[ https://issues.apache.org/jira/browse/HIVE-17964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258760#comment-16258760 ] Lefty Leverenz commented on HIVE-17964: --- Doc note: This adds *hive.spark.rsc.conf.list* to HiveConf.java, so it needs to be documented in the wiki. * [Configuration Properties -- Spark | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Spark] Added a TODOC3.0 label. > HoS: some spark configs doesn't require re-creating a session > - > > Key: HIVE-17964 > URL: https://issues.apache.org/jira/browse/HIVE-17964 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Labels: TODOC3.0 > Fix For: 3.0.0 > > Attachments: HIVE-17964.1.patch, HIVE-17964.2.patch, > HIVE-17964.3.patch > > > I guess the {{hive.spark.}} configs were initially intended for the RSC. > Therefore when they're changed, we'll re-create the session for them to take > effect. There're some configs not related to RSC that also start with > {{hive.spark.}}. We'd better rename them so that we don't unnecessarily > re-create sessions, which is usually time consuming. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17964) HoS: some spark configs doesn't require re-creating a session
[ https://issues.apache.org/jira/browse/HIVE-17964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16257222#comment-16257222 ] Xuefu Zhang commented on HIVE-17964: +1 > HoS: some spark configs doesn't require re-creating a session > - > > Key: HIVE-17964 > URL: https://issues.apache.org/jira/browse/HIVE-17964 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Attachments: HIVE-17964.1.patch, HIVE-17964.2.patch > > > I guess the {{hive.spark.}} configs were initially intended for the RSC. > Therefore when they're changed, we'll re-create the session for them to take > effect. There're some configs not related to RSC that also start with > {{hive.spark.}}. We'd better rename them so that we don't unnecessarily > re-create sessions, which is usually time consuming. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17964) HoS: some spark configs doesn't require re-creating a session
[ https://issues.apache.org/jira/browse/HIVE-17964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16256959#comment-16256959 ] Rui Li commented on HIVE-17964: --- Failures are not related. [~xuefuz] could you take another look? Thanks. > HoS: some spark configs doesn't require re-creating a session > - > > Key: HIVE-17964 > URL: https://issues.apache.org/jira/browse/HIVE-17964 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Attachments: HIVE-17964.1.patch, HIVE-17964.2.patch > > > I guess the {{hive.spark.}} configs were initially intended for the RSC. > Therefore when they're changed, we'll re-create the session for them to take > effect. There're some configs not related to RSC that also start with > {{hive.spark.}}. We'd better rename them so that we don't unnecessarily > re-create sessions, which is usually time consuming. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17964) HoS: some spark configs doesn't require re-creating a session
[ https://issues.apache.org/jira/browse/HIVE-17964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16256807#comment-16256807 ] Hive QA commented on HIVE-17964: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12898109/HIVE-17964.2.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 11383 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks] (batchId=77) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] (batchId=146) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=162) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=102) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_public_create] (batchId=92) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorize_grant_public] (batchId=92) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints (batchId=223) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7880/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7880/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7880/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12898109 - PreCommit-HIVE-Build > HoS: some spark configs doesn't require re-creating a session > - > > Key: HIVE-17964 > URL: https://issues.apache.org/jira/browse/HIVE-17964 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Attachments: HIVE-17964.1.patch, HIVE-17964.2.patch > > > I guess the {{hive.spark.}} configs were initially intended for the RSC. > Therefore when they're changed, we'll re-create the session for them to take > effect. There're some configs not related to RSC that also start with > {{hive.spark.}}. We'd better rename them so that we don't unnecessarily > re-create sessions, which is usually time consuming. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17964) HoS: some spark configs doesn't require re-creating a session
[ https://issues.apache.org/jira/browse/HIVE-17964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252858#comment-16252858 ] Xuefu Zhang commented on HIVE-17964: That's good then. > HoS: some spark configs doesn't require re-creating a session > - > > Key: HIVE-17964 > URL: https://issues.apache.org/jira/browse/HIVE-17964 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Attachments: HIVE-17964.1.patch, HIVE-17964.2.patch > > > I guess the {{hive.spark.}} configs were initially intended for the RSC. > Therefore when they're changed, we'll re-create the session for them to take > effect. There're some configs not related to RSC that also start with > {{hive.spark.}}. We'd better rename them so that we don't unnecessarily > re-create sessions, which is usually time consuming. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17964) HoS: some spark configs doesn't require re-creating a session
[ https://issues.apache.org/jira/browse/HIVE-17964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252855#comment-16252855 ] Rui Li commented on HIVE-17964: --- [~xuefuz], changing configs that start with {{spark.}} will still result in a new session. This patch only changes how we handle {{hive.spark.}} configs. > HoS: some spark configs doesn't require re-creating a session > - > > Key: HIVE-17964 > URL: https://issues.apache.org/jira/browse/HIVE-17964 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Attachments: HIVE-17964.1.patch, HIVE-17964.2.patch > > > I guess the {{hive.spark.}} configs were initially intended for the RSC. > Therefore when they're changed, we'll re-create the session for them to take > effect. There're some configs not related to RSC that also start with > {{hive.spark.}}. We'd better rename them so that we don't unnecessarily > re-create sessions, which is usually time consuming. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17964) HoS: some spark configs doesn't require re-creating a session
[ https://issues.apache.org/jira/browse/HIVE-17964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252653#comment-16252653 ] Xuefu Zhang commented on HIVE-17964: [~lirui] What's your thought on configurations such as {{spark.driver.cores}}? > HoS: some spark configs doesn't require re-creating a session > - > > Key: HIVE-17964 > URL: https://issues.apache.org/jira/browse/HIVE-17964 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Attachments: HIVE-17964.1.patch, HIVE-17964.2.patch > > > I guess the {{hive.spark.}} configs were initially intended for the RSC. > Therefore when they're changed, we'll re-create the session for them to take > effect. There're some configs not related to RSC that also start with > {{hive.spark.}}. We'd better rename them so that we don't unnecessarily > re-create sessions, which is usually time consuming. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17964) HoS: some spark configs doesn't require re-creating a session
[ https://issues.apache.org/jira/browse/HIVE-17964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250420#comment-16250420 ] Xuefu Zhang commented on HIVE-17964: +1 > HoS: some spark configs doesn't require re-creating a session > - > > Key: HIVE-17964 > URL: https://issues.apache.org/jira/browse/HIVE-17964 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Attachments: HIVE-17964.1.patch > > > I guess the {{hive.spark.}} configs were initially intended for the RSC. > Therefore when they're changed, we'll re-create the session for them to take > effect. There're some configs not related to RSC that also start with > {{hive.spark.}}. We'd better rename them so that we don't unnecessarily > re-create sessions, which is usually time consuming. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17964) HoS: some spark configs doesn't require re-creating a session
[ https://issues.apache.org/jira/browse/HIVE-17964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249405#comment-16249405 ] Hive QA commented on HIVE-17964: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12897299/HIVE-17964.1.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 11374 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks] (batchId=77) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] (batchId=146) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=156) org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc] (batchId=94) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi] (batchId=111) org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.testCliDriver[spark_job_max_tasks] (batchId=245) org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.testCliDriver[spark_stage_max_tasks] (batchId=245) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=206) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints (batchId=223) org.apache.hive.jdbc.TestSSL.testMetastoreWithSSL (batchId=229) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7788/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7788/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7788/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 11 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12897299 - PreCommit-HIVE-Build > HoS: some spark configs doesn't require re-creating a session > - > > Key: HIVE-17964 > URL: https://issues.apache.org/jira/browse/HIVE-17964 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Attachments: HIVE-17964.1.patch > > > I guess the {{hive.spark.}} configs were initially intended for the RSC. > Therefore when they're changed, we'll re-create the session for them to take > effect. There're some configs not related to RSC that also start with > {{hive.spark.}}. We'd better rename them so that we don't unnecessarily > re-create sessions, which is usually time consuming. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17964) HoS: some spark configs doesn't require re-creating a session
[ https://issues.apache.org/jira/browse/HIVE-17964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249247#comment-16249247 ] Rui Li commented on HIVE-17964: --- At the moment, only two configures need re-creating session: {noformat} hive.spark.optimize.shuffle.serde hive.spark.client.future.timeout {noformat} Other {{hive.spark.}} configs either don't require a new session, or are already immutable at runtime. BTW, I missed {{hive.spark.client.rpc.sasl.mechanisms}} in HIVE-16876, so I made it immutable here. > HoS: some spark configs doesn't require re-creating a session > - > > Key: HIVE-17964 > URL: https://issues.apache.org/jira/browse/HIVE-17964 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Attachments: HIVE-17964.1.patch > > > I guess the {{hive.spark.}} configs were initially intended for the RSC. > Therefore when they're changed, we'll re-create the session for them to take > effect. There're some configs not related to RSC that also start with > {{hive.spark.}}. We'd better rename them so that we don't unnecessarily > re-create sessions, which is usually time consuming. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17964) HoS: some spark configs doesn't require re-creating a session
[ https://issues.apache.org/jira/browse/HIVE-17964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246136#comment-16246136 ] Xuefu Zhang commented on HIVE-17964: {quote} I think renaming a bunch of configs is not very user friendly. Maybe we should differentiate these configs in our code. {quote} Probably we can add a new param that enlists the params that require a session refresh. > HoS: some spark configs doesn't require re-creating a session > - > > Key: HIVE-17964 > URL: https://issues.apache.org/jira/browse/HIVE-17964 > Project: Hive > Issue Type: Improvement >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > > I guess the {{hive.spark.}} configs were initially intended for the RSC. > Therefore when they're changed, we'll re-create the session for them to take > effect. There're some configs not related to RSC that also start with > {{hive.spark.}}. We'd better rename them so that we don't unnecessarily > re-create sessions, which is usually time consuming. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17964) HoS: some spark configs doesn't require re-creating a session
[ https://issues.apache.org/jira/browse/HIVE-17964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16239980#comment-16239980 ] Rui Li commented on HIVE-17964: --- I think renaming a bunch of configs is not very user friendly. Maybe we should differentiate these configs in our code. > HoS: some spark configs doesn't require re-creating a session > - > > Key: HIVE-17964 > URL: https://issues.apache.org/jira/browse/HIVE-17964 > Project: Hive > Issue Type: Improvement >Reporter: Rui Li >Priority: Minor > > I guess the {{hive.spark.}} configs were initially intended for the RSC. > Therefore when they're changed, we'll re-create the session for them to take > effect. There're some configs not related to RSC that also start with > {{hive.spark.}}. We'd better rename them so that we don't unnecessarily > re-create sessions, which is usually time consuming. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17964) HoS: some spark configs doesn't require re-creating a session
[ https://issues.apache.org/jira/browse/HIVE-17964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235612#comment-16235612 ] Rui Li commented on HIVE-17964: --- Some examples are: {noformat} hive.spark.explain.user hive.spark.exec.inplace.progress hive.spark.use.op.stats hive.spark.dynamic.partition.pruning hive.spark.use.ts.stats.for.mapjoin hive.spark.dynamic.partition.pruning.max.data.size hive.spark.dynamic.partition.pruning.map.join.only hive.spark.use.groupby.shuffle {noformat} > HoS: some spark configs doesn't require re-creating a session > - > > Key: HIVE-17964 > URL: https://issues.apache.org/jira/browse/HIVE-17964 > Project: Hive > Issue Type: Improvement >Reporter: Rui Li >Priority: Minor > > I guess the {{hive.spark.}} configs were initially intended for the RSC. > Therefore when they're changed, we'll re-create the session for them to take > effect. There're some configs not related to RSC that also start with > {{hive.spark.}}. We'd better rename them so that we don't unnecessarily > re-create sessions, which is usually time consuming. -- This message was sent by Atlassian JIRA (v6.4.14#64029)