[jira] [Commented] (HIVE-17942) HiveAlterHandler not using conf from HMS Handler
[ https://issues.apache.org/jira/browse/HIVE-17942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16256434#comment-16256434 ] Vihang Karajgaonkar commented on HIVE-17942: patch merged to branch-2 as well. Thanks for your contribution [~janulatha] > HiveAlterHandler not using conf from HMS Handler > > > Key: HIVE-17942 > URL: https://issues.apache.org/jira/browse/HIVE-17942 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani > Fix For: 3.0.0, 2.4.0 > > Attachments: HIVE-17942-branch-2.01.patch, HIVE-17942.1.patch, > HIVE-17942.2.patch, HIVE-17942.3.patch, HIVE-17942.4.patch, HIVE-17942.5.patch > > > When HiveAlterHandler looks for conf, it is not getting the one from thread > local. So, local changes are not visible. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17942) HiveAlterHandler not using conf from HMS Handler
[ https://issues.apache.org/jira/browse/HIVE-17942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250536#comment-16250536 ] Vihang Karajgaonkar commented on HIVE-17942: patch merged to master branch but doesn't apply cleanly on branch-2 Hi [~janulatha] can you please attach the branch-2 patch as well? > HiveAlterHandler not using conf from HMS Handler > > > Key: HIVE-17942 > URL: https://issues.apache.org/jira/browse/HIVE-17942 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani > Fix For: 3.0.0 > > Attachments: HIVE-17942.1.patch, HIVE-17942.2.patch, > HIVE-17942.3.patch, HIVE-17942.4.patch, HIVE-17942.5.patch > > > When HiveAlterHandler looks for conf, it is not getting the one from thread > local. So, local changes are not visible. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17942) HiveAlterHandler not using conf from HMS Handler
[ https://issues.apache.org/jira/browse/HIVE-17942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246104#comment-16246104 ] Janaki Lahorani commented on HIVE-17942: @akolb Changing AlterHandler to be not configurable is a significant change - potentially backward incompatible. There can be other implementations that can be used other than HiveAlterHandler. https://github.com/apache/hive/blob/master/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L485 Also, though this can be considered as a good code cleanup, it is in theory cosmetic and not needed for this bug fix. And, the story about the way HMS Handler works with DDL handlers is a design detail that is common to all DDL handlers, hence doesn't necessarily belong in any DDL handler, but rather in a design document or possibly in HMS Handler implementation, which again IMHO outside the scope of this bug fix. There is an example in the test. Configuration changes within a connection is to be made using set =value If the configuration change is to be effective in HMS it is to be prepended with metaconf: set metaconf:=value Example: set metaconf:hive.metastore.disallow.incompatible.col.type.changes=false > HiveAlterHandler not using conf from HMS Handler > > > Key: HIVE-17942 > URL: https://issues.apache.org/jira/browse/HIVE-17942 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani > Fix For: 3.0.0 > > Attachments: HIVE-17942.1.patch, HIVE-17942.2.patch, > HIVE-17942.3.patch, HIVE-17942.4.patch, HIVE-17942.5.patch > > > When HiveAlterHandler looks for conf, it is not getting the one from thread > local. So, local changes are not visible. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17942) HiveAlterHandler not using conf from HMS Handler
[ https://issues.apache.org/jira/browse/HIVE-17942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246000#comment-16246000 ] Alexander Kolbasov commented on HIVE-17942: --- [~janulatha] Thanks for the explanation. Can you provide some example showing how HMSHandler configuration may be customized by a user? Looking at the fix and at your comment - the most important part is that HMSHandler's configuration may be different from general server configuration and that's the one we should use during alterTable. The fact that it is thread-local is interesting, but this is really an implementation detail of HMSHandler. It would be good to explain this in the story in the code comments. Another question that I asked - since now the server configuration isn't used by HiveALterHandle, should it still be Configurable? What is the value of setConf()/getConf() calls in the new code? > HiveAlterHandler not using conf from HMS Handler > > > Key: HIVE-17942 > URL: https://issues.apache.org/jira/browse/HIVE-17942 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani > Fix For: 3.0.0 > > Attachments: HIVE-17942.1.patch, HIVE-17942.2.patch, > HIVE-17942.3.patch, HIVE-17942.4.patch > > > When HiveAlterHandler looks for conf, it is not getting the one from thread > local. So, local changes are not visible. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17942) HiveAlterHandler not using conf from HMS Handler
[ https://issues.apache.org/jira/browse/HIVE-17942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245977#comment-16245977 ] Janaki Lahorani commented on HIVE-17942: Thanks [~akolb] for reviewing this patch. Based on what I understand about the interactions between HS2 and HMS: for every end user connection there is a thread in HS2 and an associated thread in HMS. The configuration changes that are done by the end user in that connection are supposed to be applicable only to that connection. The configuration changes in HMS are handled through a thread local variable (threadLocalConf in HMSHandler). https://github.com/apache/hive/blob/master/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L273 The first attempt to get configuration from HMS Handler will result in the initialization of the thread local. https://github.com/apache/hive/blob/master/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L576 HiveAlterHandler when instantiated in HMS will be instantiated as a single object. IOW, only one object of HiveAlterHandler will be in the JVM and will be shared amongst all the HMS threads. We need the end user configuration change to be visible only for that end user connection - the threadLocalConf exists in HMSHandler for this purpose. So, the bug here is that HiveAlterHandler is using not using the conf from HMSHandler, and the result is that the configuration changes are not visible to the end user after a set command. > HiveAlterHandler not using conf from HMS Handler > > > Key: HIVE-17942 > URL: https://issues.apache.org/jira/browse/HIVE-17942 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani > Fix For: 3.0.0 > > Attachments: HIVE-17942.1.patch, HIVE-17942.2.patch, > HIVE-17942.3.patch, HIVE-17942.4.patch > > > When HiveAlterHandler looks for conf, it is not getting the one from thread > local. So, local changes are not visible. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17942) HiveAlterHandler not using conf from HMS Handler
[
https://issues.apache.org/jira/browse/HIVE-17942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245190#comment-16245190
]
Hive QA commented on HIVE-17942:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12896711/HIVE-17942.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 11373 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks]
(batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=62)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=156)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi]
(batchId=111)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=206)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=223)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7721/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7721/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7721/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12896711 - PreCommit-HIVE-Build
> HiveAlterHandler not using conf from HMS Handler
>
>
> Key: HIVE-17942
> URL: https://issues.apache.org/jira/browse/HIVE-17942
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.1
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
> Fix For: 3.0.0
>
> Attachments: HIVE-17942.1.patch, HIVE-17942.2.patch,
> HIVE-17942.3.patch, HIVE-17942.4.patch
>
>
> When HiveAlterHandler looks for conf, it is not getting the one from thread
> local. So, local changes are not visible.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17942) HiveAlterHandler not using conf from HMS Handler
[ https://issues.apache.org/jira/browse/HIVE-17942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245036#comment-16245036 ] Alexander Kolbasov commented on HIVE-17942: --- Looking at the patch, it isn't about using thread-local config but about using config provided by the passed in handler. > HiveAlterHandler not using conf from HMS Handler > > > Key: HIVE-17942 > URL: https://issues.apache.org/jira/browse/HIVE-17942 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani > Fix For: 3.0.0 > > Attachments: HIVE-17942.1.patch, HIVE-17942.2.patch, > HIVE-17942.3.patch, HIVE-17942.4.patch > > > When HiveAlterHandler looks for conf, it is not getting the one from thread > local. So, local changes are not visible. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17942) HiveAlterHandler not using conf from HMS Handler
[ https://issues.apache.org/jira/browse/HIVE-17942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244802#comment-16244802 ] Vihang Karajgaonkar commented on HIVE-17942: +1 > HiveAlterHandler not using conf from HMS Handler > > > Key: HIVE-17942 > URL: https://issues.apache.org/jira/browse/HIVE-17942 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani > Fix For: 3.0.0 > > Attachments: HIVE-17942.1.patch, HIVE-17942.2.patch, > HIVE-17942.3.patch, HIVE-17942.4.patch > > > When HiveAlterHandler looks for conf, it is not getting the one from thread > local. So, local changes are not visible. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17942) HiveAlterHandler not using conf from HMS Handler
[
https://issues.apache.org/jira/browse/HIVE-17942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243526#comment-16243526
]
Hive QA commented on HIVE-17942:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12896502/HIVE-17942.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 11373 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_complex_types_multi_single_reducer]
(batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=62)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketmapjoin7]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=102)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketmapjoin7]
(batchId=116)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi]
(batchId=111)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14]
(batchId=243)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query23]
(batchId=243)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=206)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=223)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7699/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7699/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7699/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12896502 - PreCommit-HIVE-Build
> HiveAlterHandler not using conf from HMS Handler
>
>
> Key: HIVE-17942
> URL: https://issues.apache.org/jira/browse/HIVE-17942
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.1
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
> Fix For: 3.0.0
>
> Attachments: HIVE-17942.1.patch, HIVE-17942.2.patch,
> HIVE-17942.3.patch
>
>
> When HiveAlterHandler looks for conf, it is not getting the one from thread
> local. So, local changes are not visible.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
