[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313453#comment-15313453 ] Lefty Leverenz commented on HIVE-4239: -- HIVE-13882 changes the description of *hive.driver.parallel.compilation* in release 2.1.0. > Remove lock on compilation stage > > > Key: HIVE-4239 > URL: https://issues.apache.org/jira/browse/HIVE-4239 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Query Processor >Reporter: Carl Steinbach >Assignee: Sergey Shelukhin > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch, > HIVE-4239.03.patch, HIVE-4239.04.patch, HIVE-4239.05.patch, > HIVE-4239.06.patch, HIVE-4239.07.patch, HIVE-4239.08.patch, HIVE-4239.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14647266#comment-14647266 ] Carl Steinbach commented on HIVE-4239: -- It should probably go in both the hs2 and compiler sections. Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin Labels: TODOC2.0 Fix For: 2.0.0 Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch, HIVE-4239.03.patch, HIVE-4239.04.patch, HIVE-4239.05.patch, HIVE-4239.06.patch, HIVE-4239.07.patch, HIVE-4239.08.patch, HIVE-4239.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14647283#comment-14647283 ] Lefty Leverenz commented on HIVE-4239: -- Hmm ... a compiler section would be nice to have. Maybe we could add one. Thanks Carl. Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin Labels: TODOC2.0 Fix For: 2.0.0 Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch, HIVE-4239.03.patch, HIVE-4239.04.patch, HIVE-4239.05.patch, HIVE-4239.06.patch, HIVE-4239.07.patch, HIVE-4239.08.patch, HIVE-4239.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14646690#comment-14646690 ] Thejas M Nair commented on HIVE-4239: - Created HIVE-11402 - HS2 - disallow parallel query execution within a single Session Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin Labels: TODOC2.0 Fix For: 2.0.0 Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch, HIVE-4239.03.patch, HIVE-4239.04.patch, HIVE-4239.05.patch, HIVE-4239.06.patch, HIVE-4239.07.patch, HIVE-4239.08.patch, HIVE-4239.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14646892#comment-14646892 ] Carl Steinbach commented on HIVE-4239: -- bq. shouldn't it be named hive.server2.driver.parallel.compilation to match the other HS2 parameters? [~leftylev], please see my comment here: https://issues.apache.org/jira/browse/HIVE-4239?focusedCommentId=14564517page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14564517 Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin Labels: TODOC2.0 Fix For: 2.0.0 Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch, HIVE-4239.03.patch, HIVE-4239.04.patch, HIVE-4239.05.patch, HIVE-4239.06.patch, HIVE-4239.07.patch, HIVE-4239.08.patch, HIVE-4239.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14620970#comment-14620970 ] Sergey Shelukhin commented on HIVE-4239: That looks unrelated Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch, HIVE-4239.03.patch, HIVE-4239.04.patch, HIVE-4239.05.patch, HIVE-4239.06.patch, HIVE-4239.07.patch, HIVE-4239.08.patch, HIVE-4239.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14621327#comment-14621327 ] Lefty Leverenz commented on HIVE-4239: -- Doc note: This adds configuration parameter *hive.driver.parallel.compilation* to HiveConf.java, so it will need to be documented in the wiki for release 2.0. It belongs in the HiveServer2 section of Configuration Properties: * [Configuration Properties -- HiveServer2 | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-HiveServer2] But shouldn't it be named *hive.server2.driver.parallel.compilation* to match the other HS2 parameters? (Sorry I didn't notice that earlier.) And a nit: if you change the parameter name in a new jira, please start the parameter description on a new line so it will look better in the generated template file. Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin Labels: TODOC2.0 Fix For: 2.0.0 Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch, HIVE-4239.03.patch, HIVE-4239.04.patch, HIVE-4239.05.patch, HIVE-4239.06.patch, HIVE-4239.07.patch, HIVE-4239.08.patch, HIVE-4239.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[
https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14619882#comment-14619882
]
Hive QA commented on HIVE-4239:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12744314/HIVE-4239.08.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 9137 tests executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4546/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4546/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4546/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12744314 - PreCommit-HIVE-TRUNK-Build
Remove lock on compilation stage
Key: HIVE-4239
URL: https://issues.apache.org/jira/browse/HIVE-4239
Project: Hive
Issue Type: Bug
Components: HiveServer2, Query Processor
Reporter: Carl Steinbach
Assignee: Sergey Shelukhin
Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch,
HIVE-4239.03.patch, HIVE-4239.04.patch, HIVE-4239.05.patch,
HIVE-4239.06.patch, HIVE-4239.07.patch, HIVE-4239.08.patch, HIVE-4239.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14615692#comment-14615692 ] Thejas M Nair commented on HIVE-4239: - +1 Sorry about the delay! Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch, HIVE-4239.03.patch, HIVE-4239.04.patch, HIVE-4239.05.patch, HIVE-4239.06.patch, HIVE-4239.07.patch, HIVE-4239.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14615552#comment-14615552 ] Sergey Shelukhin commented on HIVE-4239: [~thejas] I just realized this actually still needs review :) Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch, HIVE-4239.03.patch, HIVE-4239.04.patch, HIVE-4239.05.patch, HIVE-4239.06.patch, HIVE-4239.07.patch, HIVE-4239.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14584031#comment-14584031 ] Sergey Shelukhin commented on HIVE-4239: I don't think we should remove the session code Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch, HIVE-4239.03.patch, HIVE-4239.04.patch, HIVE-4239.05.patch, HIVE-4239.06.patch, HIVE-4239.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14571402#comment-14571402 ] Sergey Shelukhin commented on HIVE-4239: [~thejas] ping? Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch, HIVE-4239.03.patch, HIVE-4239.04.patch, HIVE-4239.05.patch, HIVE-4239.06.patch, HIVE-4239.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[
https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569979#comment-14569979
]
Hive QA commented on HIVE-4239:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12736986/HIVE-4239.06.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 8991 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4146/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4146/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4146/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12736986 - PreCommit-HIVE-TRUNK-Build
Remove lock on compilation stage
Key: HIVE-4239
URL: https://issues.apache.org/jira/browse/HIVE-4239
Project: Hive
Issue Type: Bug
Components: HiveServer2, Query Processor
Reporter: Carl Steinbach
Assignee: Sergey Shelukhin
Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch,
HIVE-4239.03.patch, HIVE-4239.04.patch, HIVE-4239.05.patch,
HIVE-4239.06.patch, HIVE-4239.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569995#comment-14569995 ] Sergey Shelukhin commented on HIVE-4239: the test failure is unrelated Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch, HIVE-4239.03.patch, HIVE-4239.04.patch, HIVE-4239.05.patch, HIVE-4239.06.patch, HIVE-4239.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[
https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568617#comment-14568617
]
Hive QA commented on HIVE-4239:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12736698/HIVE-4239.05.patch
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 8997 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementParallel
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testGetFunctions
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testOpenSession
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4139/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4139/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4139/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12736698 - PreCommit-HIVE-TRUNK-Build
Remove lock on compilation stage
Key: HIVE-4239
URL: https://issues.apache.org/jira/browse/HIVE-4239
Project: Hive
Issue Type: Bug
Components: HiveServer2, Query Processor
Reporter: Carl Steinbach
Assignee: Sergey Shelukhin
Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch,
HIVE-4239.03.patch, HIVE-4239.04.patch, HIVE-4239.05.patch, HIVE-4239.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568442#comment-14568442 ] Thejas M Nair commented on HIVE-4239: - [~sershe] Thanks for all your hard work! This is much needed and long due improvement ! Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch, HIVE-4239.03.patch, HIVE-4239.04.patch, HIVE-4239.05.patch, HIVE-4239.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568435#comment-14568435 ] Thejas M Nair commented on HIVE-4239: - I am sorry about the delay in adding comments (as the patch has been around for couple of days). I will do some more review of associated code related to the HIVE-6245 issue by tomorrow. [~sershe] Did you do any survey of all static variables used in compilation ? I am just trying to to understand how you determined the changes that are needed. Maybe we should also check if there are potentially any libraries that we use that might be thread unsafe (say the parser libraries etc). For identifying potential race conditions, some testing alone would not suffice, a systematic review of code is very important. Any reports of race conditions are extremely hard to debug. Anything we can do to reduce the chances of that would be very valuable. Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch, HIVE-4239.03.patch, HIVE-4239.04.patch, HIVE-4239.05.patch, HIVE-4239.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[
https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568358#comment-14568358
]
Hive QA commented on HIVE-4239:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12736677/HIVE-4239.04.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 8997 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4135/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4135/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4135/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12736677 - PreCommit-HIVE-TRUNK-Build
Remove lock on compilation stage
Key: HIVE-4239
URL: https://issues.apache.org/jira/browse/HIVE-4239
Project: Hive
Issue Type: Bug
Components: HiveServer2, Query Processor
Reporter: Carl Steinbach
Assignee: Sergey Shelukhin
Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch,
HIVE-4239.03.patch, HIVE-4239.04.patch, HIVE-4239.05.patch, HIVE-4239.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14566747#comment-14566747 ] Vaibhav Gumashta commented on HIVE-4239: [~cwsteinbach] Just looked at your comment on getting rid of Driver and SessionState from HS2 codepath. [~thejas] and I were having a similar discussion sometime back. I've created HIVE-10876 to track and work on it. Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch, HIVE-4239.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[
https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14566100#comment-14566100
]
Hive QA commented on HIVE-4239:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12736232/HIVE-4239.02.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4106/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4106/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4106/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-4106/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
From https://github.com/apache/hive
83cc691..ae4790e master - origin/master
+ git reset --hard HEAD
HEAD is now at 83cc691 HIVE-10722 : external table creation with msck in Hive
can create unusable partition (Sergey Shelukhin, reviewed by Sushanth Sowmyan)
+ git clean -f -d
Removing common/src/java/org/apache/hadoop/hive/conf/HiveConf.java.orig
Removing ql/src/java/org/apache/hadoop/hive/ql/exec/spark/CacheTran.java
Removing ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SmallTableCache.java
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
+ git reset --hard origin/master
HEAD is now at ae4790e HIVE-10788 - Change sort_array to support non-primitive
types (Chao Sun, reviewed by Alexander Pivovarov)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12736232 - PreCommit-HIVE-TRUNK-Build
Remove lock on compilation stage
Key: HIVE-4239
URL: https://issues.apache.org/jira/browse/HIVE-4239
Project: Hive
Issue Type: Bug
Components: HiveServer2, Query Processor
Reporter: Carl Steinbach
Assignee: Sergey Shelukhin
Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch, HIVE-4239.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14564517#comment-14564517 ] Carl Steinbach commented on HIVE-4239: -- A couple comments on the patch: * Would you mind changing the name of the new configuration property to 'hive.driver.parallel.compilation'? The only reference to this config property is located in the Driver class, and both Driver and SessionState predate HiveServer2 by several years. Driver and SessionState were used by HiveServer1 and continue to be used by the HiveCli, and I know that there are at least a couple third-party libraries out there that attempt to support concurrency by scheduling queries across a pool of Driver objects. In other words, this property changes the behavior of a class which is not part of HiveServer2, and also has the potential to change the behavior of other user-facing interfaces built on top of Driver which are also not part of HiveServer2. I also hope that at some point the Driver and SessionState classes will go away completely, at which point we can deprecate and remove this property, but only if it references driver. * I think the docstring for the new property could use some wordsmithing: Whether to enable parallel compilation on HiveServer2. _Disable as a workaround for future bugs._ The last sentence isn't going to inspire much confidence in users about either the quality of the product or the development team that produced it. * In order to enable this feature by default I think we need to be pretty confident that parallel compilation works. The additional parallel test coverage included in this patch is a great start, but I think it falls well short of being comprehensive. I mentioned in an earlier comment that code already exists for running qfile tests in parallel on top of HiveServer2. Why not re-enable this? Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin Attachments: HIVE-4239.01.patch, HIVE-4239.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14565774#comment-14565774 ] Carl Steinbach commented on HIVE-4239: -- [~sershe]: Yup, that's cool with me. Thanks for making the changes I requested. +1 Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch, HIVE-4239.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14563854#comment-14563854 ] Sergey Shelukhin commented on HIVE-4239: Tested it on the cluster with doAs, but w/o multiple users. 5-10 threads (in separate sessions) compiling the same complex TPCDS query works fine. Throughput increases accordingly. Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin Attachments: HIVE-4239.01.patch, HIVE-4239.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14563617#comment-14563617 ] Vaibhav Gumashta commented on HIVE-4239: Yet to review the patch properly, but I'd like to point out that concurrent multiuser testing (with doAs = true) might be a good idea here. A while back, I played around with removing the compile lock and ran into issues with that use case. Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin Attachments: HIVE-4239.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14563594#comment-14563594 ] Sergey Shelukhin commented on HIVE-4239: Sounds like a plan... after looking at SessionState I don't want to get into that can of worms :) I'll file a separate JIRA. Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin Attachments: HIVE-4239.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14563588#comment-14563588 ] Sergey Shelukhin commented on HIVE-4239: Actually upon further examination SessionState is not meant to be thread-safe in this way. I am not sure why we don't have more bugs. E.g. mapRedStats is a non-thread-safe map accessed during execution to put values. It's not protected by compile lock or any other lock, so if 2 queries execute within the same session it could have issues. Two set commands running in parallel within the same session should have the same problem on hiveVariables (see SetProcessor::setVariable calling get for that map and modifying it Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin Attachments: HIVE-4239.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[
https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14563606#comment-14563606
]
Hive QA commented on HIVE-4239:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12735728/HIVE-4239.patch
{color:red}ERROR:{color} -1 due to 627 failed/errored test(s), 5328 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestJdbcWithMiniKdcSQLAuthHttp - did not produce a TEST-*.xml file
TestJdbcWithSQLAuthUDFBlacklist - did not produce a TEST-*.xml file
TestJdbcWithSQLAuthorization - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.initializationError
org.apache.hadoop.hive.cli.TestCliDriverMethods.testProcessSelectDatabase
org.apache.hadoop.hive.cli.TestCliDriverMethods.testprocessInitFiles
org.apache.hadoop.hive.cli.TestCompareCliDriver.initializationError
org.apache.hadoop.hive.cli.TestContribCliDriver.initializationError
org.apache.hadoop.hive.cli.TestContribNegativeCliDriver.initializationError
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.initializationError
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_external_table_ppd
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_external_table_queries
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_map_queries
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_map_queries_prefix
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_storage_queries
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key2
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_handler_bulk
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_handler_snapshot
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_joins
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_ppd_join
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_ppd_key_range
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_pushdown
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_scan_params
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_single_sourced_multi_insert
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_stats
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_stats2
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_stats3
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_stats_empty_partition
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_timestamp
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_timestamp_format
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_ppd_key_ranges
org.apache.hadoop.hive.cli.TestHBaseMinimrCliDriver.testCliDriver_hbase_bulk
org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.testCliDriver_cascade_dbdrop
org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.testCliDriver_cascade_dbdrop_hadoop20
org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.testCliDriver_generatehfiles_require_family_path
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.initializationError
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.initializationError
org.apache.hadoop.hive.cli.TestMinimrCliDriver.initializationError
org.apache.hadoop.hive.cli.TestNegativeCliDriver.initializationError
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.initializationError
org.apache.hadoop.hive.cli.TestSparkCliDriver.initializationError
org.apache.hadoop.hive.hooks.TestHs2Hooks.testHookContexts
org.apache.hadoop.hive.metastore.TestFilterHooks.testDefaultFilter
org.apache.hadoop.hive.metastore.TestFilterHooks.testDummyFilterForDb
org.apache.hadoop.hive.metastore.TestFilterHooks.testDummyFilterForIndex
org.apache.hadoop.hive.metastore.TestFilterHooks.testDummyFilterForPartition
org.apache.hadoop.hive.metastore.TestFilterHooks.testDummyFilterForTables
org.apache.hadoop.hive.metastore.TestMarkPartition.testMarkingPartitionSet
org.apache.hadoop.hive.metastore.TestMarkPartitionRemote.testMarkingPartitionSet
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListener.testListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetastoreVersion.testMetastoreVersion
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14563340#comment-14563340 ] Sergey Shelukhin commented on HIVE-4239: [~thejas] I wonder what the mainline scenario is. If most clients do that (which I suspect may be the case) it won't solve the problem for them, only alleviate it. How does Hive JDBC driver do that? [~cwsteinbach] RB at https://reviews.apache.org/r/34776/ Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin Attachments: HIVE-4239.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14563440#comment-14563440 ] Thejas M Nair commented on HIVE-4239: - The jdbc driver does not allow for concurrent compile to happen within a jdbc session because it has locks around the thrift client calls (actually, its partly a work in progress). However, if you consider Hue to be a mainline scenario, then it happens in mainline scenario. I am not sure about the case with third party odbc drivers. This change has potential to cause even more issues when that happens. So I would reccomend having serial compile within a single session for now. I was wrong when I said the change to make driver.compile syncrhonized would help. We would actually need to synchronize on current SesionState object. Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin Attachments: HIVE-4239.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14563418#comment-14563418 ] Sergey Shelukhin commented on HIVE-4239: Yeah, some more changes will be needed for that... meanwhile the rest of the patch should be reviewable :) Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin Attachments: HIVE-4239.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14563448#comment-14563448 ] Thejas M Nair commented on HIVE-4239: - The better and more involved fix for the parallel compile issues within a single session would be to move any 'operation' scope information out from session to some operation specific objects. That way parallel compiles within a session can also be allowed. Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin Attachments: HIVE-4239.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14563396#comment-14563396 ] Sergey Shelukhin commented on HIVE-4239: I'll look at sessionstate stuff Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin Attachments: HIVE-4239.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14564086#comment-14564086 ] Nemon Lou commented on HIVE-4239: - Pasting from email : I have tested it on a 7 nodes cluster, with 100 concurrent clients running for about 8 hours and haven't seen any issue. The workload is TPC-H style with both DDLs and queries. The 100 clients use different tables from each other and use different sessions. My Hive version is 0.13.1.And the changing part is the removal of the big lock. Hope this test helps. Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin Attachments: HIVE-4239.01.patch, HIVE-4239.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14562062#comment-14562062 ] Thejas M Nair commented on HIVE-4239: - There is another use case to consider- Hue (or other Thrift clients) may attempt to execute multiple concurrent queries using the same SessionID. This could be problematic for the Hive SessionState since there are several fields that are associated with the currently executing query, such as commandType, lastCommand. Also SessionState has a Conf object and this also gets set with query-specific information. (from an earlier offline discussion I had with [~jdere] and [~vgumashta]) Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin Attachments: HIVE-4239.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14562215#comment-14562215 ] Carl Steinbach commented on HIVE-4239: -- Hi [~sershe], I have some feedback. Would you mind posting an RB for this patch? Thanks. Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin Attachments: HIVE-4239.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14562045#comment-14562045 ] Sergey Shelukhin commented on HIVE-4239: [~hagleitn] would you also like to take a look? Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin Attachments: HIVE-4239.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14562064#comment-14562064 ] Thejas M Nair commented on HIVE-4239: - I haven't looked at this in depth, but a potential way to make things not worse in above use case, would be to have a synchronize keyword for Driver.compile(). That way the compile lock is not global, but local to a hiveserver2 session. Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin Attachments: HIVE-4239.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14561950#comment-14561950 ] Carl Steinbach commented on HIVE-4239: -- bq. Problem is that we don't have any massive parallel test case in hive. Actually, we used to have massively parallel test cases... The original HiveServer2 patch included a rewritten version of the CliDriver harness for Beeline (TestBeeLineDriver.vm) that supported a configurable parallelism setting (test.concurrency.num.threads). These tests were disabled in HIVE-4162 and HIVE-4176, but most of the code appears to still be there. Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14561959#comment-14561959 ] Carl Steinbach commented on HIVE-4239: -- I just wanted to add that even in single-threaded mode TestBeeLineDriver is able to find bugs that TestCliDriver misses due to over-aggressive diff masking. See HIVE-3583 for more details. Remove lock on compilation stage Key: HIVE-4239 URL: https://issues.apache.org/jira/browse/HIVE-4239 Project: Hive Issue Type: Bug Components: HiveServer2, Query Processor Reporter: Carl Steinbach Assignee: Sergey Shelukhin -- This message was sent by Atlassian JIRA (v6.3.4#6332)
