[jira] [Commented] (HIVE-13567) Auto-gather column stats - phase 2
[
https://issues.apache.org/jira/browse/HIVE-13567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969774#comment-15969774
]
Hive QA commented on HIVE-13567:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863530/HIVE-13567.07.patch
{color:green}SUCCESS:{color} +1 due to 11 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10580 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join43] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_7]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_order_null]
(batchId=27)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=212)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4702/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4702/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4702/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863530 - PreCommit-HIVE-Build
> Auto-gather column stats - phase 2
> --
>
> Key: HIVE-13567
> URL: https://issues.apache.org/jira/browse/HIVE-13567
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-13567.01.patch, HIVE-13567.02.patch,
> HIVE-13567.03.patch, HIVE-13567.04.patch, HIVE-13567.05.patch,
> HIVE-13567.06.patch, HIVE-13567.07.patch
>
>
> in phase 2, we are going to set auto-gather column on as default. This needs
> to update golden files.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16197) Incremental insert into a partitioned table doesn't get replicated.
[ https://issues.apache.org/jira/browse/HIVE-16197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-16197: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to master. Thanks, [~sankarh]! > Incremental insert into a partitioned table doesn't get replicated. > --- > > Key: HIVE-16197 > URL: https://issues.apache.org/jira/browse/HIVE-16197 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR > Fix For: 3.0.0 > > Attachments: HIVE-16197.01.patch, HIVE-16197.02.patch, > HIVE-16197.03.patch, HIVE-16197.04.patch > > > Insert to a partitioned table doesn't replicate properly in case of > incremental dump/load. Few key points to be noted. > 1. If insert command itself created the new partition, then the inserted row > is replicated. But the subsequent inserts into the same table doesn't get > replicated. > 2. If the partition is created using ALTER TABLE command, then none of the > inserted rows to this partition is getting replicated. However, the partition > metadata is getting replicated. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16197) Incremental insert into a partitioned table doesn't get replicated.
[ https://issues.apache.org/jira/browse/HIVE-16197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969725#comment-15969725 ] Sushanth Sowmyan commented on HIVE-16197: - Thanks for the update with the changes, [~sankarh], I'm +1 on .04.patch, and will go ahead and commit it. The 2 unit tests that failed above are unrelated so I'm going to ignore those failures. That said, I do have 2 comments/questions: a) Is there a reason you changed the "replace" field from Boolean to String in JSONInsertMessage.java ? The change is effectively equivalent and I don't mind, but is there an issue with the json parsing that you noticed that prompted you to make that change from .03.patch to .04.patch? b) In general, I think that when changing thrift structures, it has been standard practice to always add new fields at the "end", rather than the beginning - i.e., if a struct already has numbered elements for 1&2, then a new field should always be added as 3, rather than adding it as 1, and bumping the prior 1&2 to 2&3. I had to check to see whether doing so breaks backward compatibility (i.e. is ordering of field numbers important?) and from reading docs, I can't find anything that indicates that that is a problem, and so the patch is fine as-is. If there were reason to regenerate this patch, I would suggest changing it, but at this point, I think it's fine to leave it as-is. > Incremental insert into a partitioned table doesn't get replicated. > --- > > Key: HIVE-16197 > URL: https://issues.apache.org/jira/browse/HIVE-16197 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR > Attachments: HIVE-16197.01.patch, HIVE-16197.02.patch, > HIVE-16197.03.patch, HIVE-16197.04.patch > > > Insert to a partitioned table doesn't replicate properly in case of > incremental dump/load. Few key points to be noted. > 1. If insert command itself created the new partition, then the inserted row > is replicated. But the subsequent inserts into the same table doesn't get > replicated. > 2. If the partition is created using ALTER TABLE command, then none of the > inserted rows to this partition is getting replicated. However, the partition > metadata is getting replicated. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-13567) Auto-gather column stats - phase 2
[ https://issues.apache.org/jira/browse/HIVE-13567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13567: --- Status: Open (was: Patch Available) > Auto-gather column stats - phase 2 > -- > > Key: HIVE-13567 > URL: https://issues.apache.org/jira/browse/HIVE-13567 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13567.01.patch, HIVE-13567.02.patch, > HIVE-13567.03.patch, HIVE-13567.04.patch, HIVE-13567.05.patch, > HIVE-13567.06.patch, HIVE-13567.07.patch > > > in phase 2, we are going to set auto-gather column on as default. This needs > to update golden files. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-13567) Auto-gather column stats - phase 2
[ https://issues.apache.org/jira/browse/HIVE-13567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13567: --- Attachment: HIVE-13567.07.patch > Auto-gather column stats - phase 2 > -- > > Key: HIVE-13567 > URL: https://issues.apache.org/jira/browse/HIVE-13567 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13567.01.patch, HIVE-13567.02.patch, > HIVE-13567.03.patch, HIVE-13567.04.patch, HIVE-13567.05.patch, > HIVE-13567.06.patch, HIVE-13567.07.patch > > > in phase 2, we are going to set auto-gather column on as default. This needs > to update golden files. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-13567) Auto-gather column stats - phase 2
[ https://issues.apache.org/jira/browse/HIVE-13567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13567: --- Status: Patch Available (was: Open) > Auto-gather column stats - phase 2 > -- > > Key: HIVE-13567 > URL: https://issues.apache.org/jira/browse/HIVE-13567 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13567.01.patch, HIVE-13567.02.patch, > HIVE-13567.03.patch, HIVE-13567.04.patch, HIVE-13567.05.patch, > HIVE-13567.06.patch, HIVE-13567.07.patch > > > in phase 2, we are going to set auto-gather column on as default. This needs > to update golden files. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-12156) expanding view doesn't quote reserved keyword
[ https://issues.apache.org/jira/browse/HIVE-12156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong reassigned HIVE-12156: -- Assignee: Pengcheng Xiong > expanding view doesn't quote reserved keyword > - > > Key: HIVE-12156 > URL: https://issues.apache.org/jira/browse/HIVE-12156 > Project: Hive > Issue Type: Bug > Components: Parser >Affects Versions: 1.2.1 > Environment: hadoop 2.7 > hive 1.2.1 >Reporter: Jay Lee >Assignee: Pengcheng Xiong > Fix For: 2.0.0 > > Attachments: HIVE-12156.01.patch, HIVE-12156.02.patch, > HIVE-12156.03.patch > > > hive> create table testreserved (data struct<`end`:string, id: string>); > OK > Time taken: 0.274 seconds > hive> create view testreservedview as select data.`end` as data_end, data.id > as data_id from testreserved; > OK > Time taken: 0.769 seconds > hive> select data.`end` from testreserved; > OK > Time taken: 1.852 seconds > hive> select data_id from testreservedview; > NoViableAltException(98@[]) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.identifier(HiveParser_IdentifiersParser.java:10858) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6438) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:6768) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:6828) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7012) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7172) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7332) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAmpersandExpression(HiveParser_IdentifiersParser.java:7483) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseOrExpression(HiveParser_IdentifiersParser.java:7634) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceEqualExpression(HiveParser_IdentifiersParser.java:8164) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceNotExpression(HiveParser_IdentifiersParser.java:9177) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAndExpression(HiveParser_IdentifiersParser.java:9296) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceOrExpression(HiveParser_IdentifiersParser.java:9455) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.expression(HiveParser_IdentifiersParser.java:6105) > at > org.apache.hadoop.hive.ql.parse.HiveParser.expression(HiveParser.java:45840) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectItem(HiveParser_SelectClauseParser.java:2907) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectList(HiveParser_SelectClauseParser.java:1373) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1128) > at > org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:45827) > at > org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41495) > at > org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402) > at > org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413) > at > org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283) > at > org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1590) > at > org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1109) > at > org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202) > ... > FAILED: SemanticException line 1:29 cannot recognize input near 'end' 'as' > 'data_end' in expression specification in definition of VIEW testreservedview > [ > select `testreserved`.`data`.end as `data_end`, `testreserved`.`data`.id as > `data_id` from `test`.`testreserved` > ] used as testreservedview at Line 1:20 > When view is expanded, field should be quote with backquote. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16456) Kill spark job when InterruptedException happens or driverContext.isShutdown is true.
[
https://issues.apache.org/jira/browse/HIVE-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969698#comment-15969698
]
Hive QA commented on HIVE-16456:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863512/HIVE-16456.000.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10578 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_order_null]
(batchId=27)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4701/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4701/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4701/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863512 - PreCommit-HIVE-Build
> Kill spark job when InterruptedException happens or driverContext.isShutdown
> is true.
> -
>
> Key: HIVE-16456
> URL: https://issues.apache.org/jira/browse/HIVE-16456
> Project: Hive
> Issue Type: Improvement
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Attachments: HIVE-16456.000.patch
>
>
> Kill spark job when InterruptedException happens or driverContext.isShutdown
> is true. If the InterruptedException happened in RemoteSparkJobMonitor and
> LocalSparkJobMonitor, it will be better to kill the job. Also there is a race
> condition between submit the spark job and query/operation cancellation, it
> will be better to check driverContext.isShutdown right after submit the spark
> job. This will guarantee the job being killed no matter when shutdown is
> called. It is similar as HIVE-15997.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16451) Race condition between HiveStatement.getQueryLog and HiveStatement.runAsyncOnServer
[
https://issues.apache.org/jira/browse/HIVE-16451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969656#comment-15969656
]
Hive QA commented on HIVE-16451:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863506/HIVE-16451.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10578 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_order_null]
(batchId=27)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
org.apache.hive.jdbc.TestJdbcDriver2.testSelectExecAsync2 (batchId=221)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4700/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4700/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4700/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863506 - PreCommit-HIVE-Build
> Race condition between HiveStatement.getQueryLog and
> HiveStatement.runAsyncOnServer
> ---
>
> Key: HIVE-16451
> URL: https://issues.apache.org/jira/browse/HIVE-16451
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16451.02.patch, HIVE-16451.patch
>
>
> During the BeeLineDriver testing I have met the following race condition:
> - Run the query asynchronously through BeeLine
> - Querying the logs in the BeeLine
> In the following code:
> {code:title=HiveStatement.runAsyncOnServer}
> private void runAsyncOnServer(String sql) throws SQLException {
> checkConnection("execute");
> closeClientOperation();
> initFlags();
> [..]
> }
> {code}
> {code:title=HiveStatement.getQueryLog}
> public List getQueryLog(boolean incremental, int fetchSize)
> throws SQLException, ClosedOrCancelledStatementException {
> [..]
> try {
> if (stmtHandle != null) {
> [..]
> } else {
> if (isQueryClosed) {
> throw new ClosedOrCancelledStatementException("Method getQueryLog()
> failed. The " +
> "statement has been closed or cancelled.");
> } else {
> return logs;
> }
> }
> } catch (SQLException e) {
> [..]
> }
> [..]
> }
> {code}
> The runAsyncOnServer {{closeClientOperation}} sets {{isQueryClosed}} flag to
> true:
> {code:title=HiveStatement.closeClientOperation}
> void closeClientOperation() throws SQLException {
> [..]
> isQueryClosed = true;
> isExecuteStatementFailed = false;
> stmtHandle = null;
> }
> {code}
> The {{initFlags}} sets it to false:
> {code}
> private void initFlags() {
> isCancelled = false;
> isQueryClosed = false;
> isLogBeingGenerated = true;
> isExecuteStatementFailed = false;
> isOperationComplete = false;
> }
> {code}
> If the {{getQueryLog}} is called after the {{closeClientOperation}}, but
> before the {{initFlags}}, then we will have a following warning if verbose
> mode is set to true in BeeLine:
> {code}
> Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method
> getQueryLog() failed. The statement has been closed or cancelled.
> (state=,code=0)
> {code}
> This caused this fail:
> https://builds.apache.org/job/PreCommit-HIVE-Build/4691/testReport/org.apache.hadoop.hive.cli/TestBeeLineDriver/testCliDriver_smb_mapjoin_11_/
> {code}
> Error Message
> Client result comparison failed with error code = 1 while executing
> fname=smb_mapjoin_11
> 16a17
> > Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method
> > getQueryLog() failed. The statement has been closed or cancelled.
> > (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16455) ADD JAR command leaks JAR Files
[
https://issues.apache.org/jira/browse/HIVE-16455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969606#comment-15969606
]
Hive QA commented on HIVE-16455:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863502/HIVE-16455.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10577 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
org.apache.hive.jdbc.TestXSRFFilter.testFilterDisabledWithInjection
(batchId=219)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4699/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4699/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4699/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863502 - PreCommit-HIVE-Build
> ADD JAR command leaks JAR Files
> ---
>
> Key: HIVE-16455
> URL: https://issues.apache.org/jira/browse/HIVE-16455
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-16455.1.patch
>
>
> HiveServer2 is leaking file handles when using ADD JAR statement and the JAR
> file added is not used in the query itself.
> {noformat}
> beeline> !connect jdbc:hive2://localhost:1 admin
> 0: jdbc:hive2://localhost:1> create table test_leak (a int);
> 0: jdbc:hive2://localhost:1> insert into test_leak Values (1);
> -- Exit beeline terminal; Find PID of HiveServer2
> [root@host-10-17-80-111 ~]# lsof -p 29588 | grep "(deleted)" | wc -l
> 0
> [root@host-10-17-80-111 ~]# beeline -u jdbc:hive2://localhost:1/default
> -n admin
> And run the command "ADD JAR hdfs:///tmp/hive-contrib.jar; select * from
> test_leak"
> [root@host-10-17-80-111 ~]# lsof -p 29588 | grep "(deleted)" | wc -l
> 1
> java29588 hive 391u REG 252,3125987 2099944
> /tmp/57d98f5b-1e53-44e2-876b-6b4323ac24db_resources/hive-contrib.jar (deleted)
> java29588 hive 392u REG 252,3125987 2099946
> /tmp/eb3184ad-7f15-4a77-a10d-87717ae634d1_resources/hive-contrib.jar (deleted)
> java29588 hive 393r REG 252,3125987 2099825
> /tmp/e29dccfc-5708-4254-addb-7a8988fc0500_resources/hive-contrib.jar (deleted)
> java29588 hive 394r REG 252,3125987 2099833
> /tmp/5153dd4a-a606-4f53-b02c-d606e7e56985_resources/hive-contrib.jar (deleted)
> java29588 hive 395r REG 252,3125987 2099827
> /tmp/ff3cdb05-917f-43c0-830a-b293bf397a23_resources/hive-contrib.jar (deleted)
> java29588 hive 396r REG 252,3125987 2099822
> /tmp/60531b66-5985-421e-8eb5-eeac31fdf964_resources/hive-contrib.jar (deleted)
> java29588 hive 397r REG 252,3125987 2099831
> /tmp/78878921-455c-438c-9735-447566ed8381_resources/hive-contrib.jar (deleted)
> java29588 hive 399r REG 252,3125987 2099835
> /tmp/0e5d7990-30cc-4248-9058-587f7f1ff211_resources/hive-contrib.jar (deleted)
> {noformat}
> You can see the the session directory (and therefore anything in it) is set
> to delete only on exit.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16456) Kill spark job when InterruptedException happens or driverContext.isShutdown is true.
[ https://issues.apache.org/jira/browse/HIVE-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated HIVE-16456: - Attachment: HIVE-16456.000.patch > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. > - > > Key: HIVE-16456 > URL: https://issues.apache.org/jira/browse/HIVE-16456 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: HIVE-16456.000.patch > > > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. If the InterruptedException happened in RemoteSparkJobMonitor and > LocalSparkJobMonitor, it will be better to kill the job. Also there is a race > condition between submit the spark job and query/operation cancellation, it > will be better to check driverContext.isShutdown right after submit the spark > job. This will guarantee the job being killed no matter when shutdown is > called. It is similar as HIVE-15997. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16456) Kill spark job when InterruptedException happens or driverContext.isShutdown is true.
[ https://issues.apache.org/jira/browse/HIVE-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated HIVE-16456: - Status: Patch Available (was: Open) > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. > - > > Key: HIVE-16456 > URL: https://issues.apache.org/jira/browse/HIVE-16456 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: HIVE-16456.000.patch > > > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. If the InterruptedException happened in RemoteSparkJobMonitor and > LocalSparkJobMonitor, it will be better to kill the job. Also there is a race > condition between submit the spark job and query/operation cancellation, it > will be better to check driverContext.isShutdown right after submit the spark > job. This will guarantee the job being killed no matter when shutdown is > called. It is similar as HIVE-15997. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16456) Kill spark job when InterruptedException happens or driverContext.isShutdown is true.
[ https://issues.apache.org/jira/browse/HIVE-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu reassigned HIVE-16456: > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. > - > > Key: HIVE-16456 > URL: https://issues.apache.org/jira/browse/HIVE-16456 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. If the InterruptedException happened in RemoteSparkJobMonitor and > LocalSparkJobMonitor, it will be better to kill the job. Also there is a race > condition between submit the spark job and query/operation cancellation, it > will be better to check driverContext.isShutdown right after submit the spark > job. This will guarantee the job being killed no matter when shutdown is > called. It is similar as HIVE-15997. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16448) Vectorization: Vectorized order_null.q fails with deserialize EOF exception below TEZ ReduceRecordSource.processVectorGroup
[
https://issues.apache.org/jira/browse/HIVE-16448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969574#comment-15969574
]
Matt McCline commented on HIVE-16448:
-
Committed to master. Thank you Gopal.
> Vectorization: Vectorized order_null.q fails with deserialize EOF exception
> below TEZ ReduceRecordSource.processVectorGroup
> ---
>
> Key: HIVE-16448
> URL: https://issues.apache.org/jira/browse/HIVE-16448
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-16448.01.patch, HIVE-16448.02.patch,
> HIVE-16448.03.patch
>
>
> For query "SELECT x.* FROM src_null x ORDER BY b asc, a asc nulls last" here
> is the stack trace:
> {code}
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
> attempt_1492136345968_0001_40_01_00_1:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:389)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:245)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:317)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> ... 15 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:421)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:380)
> ... 18 more
> Caused by: java.io.EOFException
> at
> org.apache.hadoop.hive.serde2.binarysortable.InputByteBuffer.read(InputByteBuffer.java:54)
> at
> org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableDeserializeRead.readNextField(BinarySortableDeserializeRead.java:205)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorDeserializeRow.deserialize(VectorDeserializeRow.java:751)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:417)
> ... 19 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (HIVE-16448) Vectorization: Vectorized order_null.q fails with deserialize EOF exception below TEZ ReduceRecordSource.processVectorGroup
[
https://issues.apache.org/jira/browse/HIVE-16448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline resolved HIVE-16448.
-
Resolution: Fixed
> Vectorization: Vectorized order_null.q fails with deserialize EOF exception
> below TEZ ReduceRecordSource.processVectorGroup
> ---
>
> Key: HIVE-16448
> URL: https://issues.apache.org/jira/browse/HIVE-16448
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-16448.01.patch, HIVE-16448.02.patch,
> HIVE-16448.03.patch
>
>
> For query "SELECT x.* FROM src_null x ORDER BY b asc, a asc nulls last" here
> is the stack trace:
> {code}
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
> attempt_1492136345968_0001_40_01_00_1:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:389)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:245)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:317)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> ... 15 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:421)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:380)
> ... 18 more
> Caused by: java.io.EOFException
> at
> org.apache.hadoop.hive.serde2.binarysortable.InputByteBuffer.read(InputByteBuffer.java:54)
> at
> org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableDeserializeRead.readNextField(BinarySortableDeserializeRead.java:205)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorDeserializeRow.deserialize(VectorDeserializeRow.java:751)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:417)
> ... 19 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16448) Vectorization: Vectorized order_null.q fails with deserialize EOF exception below TEZ ReduceRecordSource.processVectorGroup
[
https://issues.apache.org/jira/browse/HIVE-16448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-16448:
Attachment: HIVE-16448.03.patch
Add Q file output.
> Vectorization: Vectorized order_null.q fails with deserialize EOF exception
> below TEZ ReduceRecordSource.processVectorGroup
> ---
>
> Key: HIVE-16448
> URL: https://issues.apache.org/jira/browse/HIVE-16448
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-16448.01.patch, HIVE-16448.02.patch,
> HIVE-16448.03.patch
>
>
> For query "SELECT x.* FROM src_null x ORDER BY b asc, a asc nulls last" here
> is the stack trace:
> {code}
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
> attempt_1492136345968_0001_40_01_00_1:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:389)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:245)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:317)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> ... 15 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:421)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:380)
> ... 18 more
> Caused by: java.io.EOFException
> at
> org.apache.hadoop.hive.serde2.binarysortable.InputByteBuffer.read(InputByteBuffer.java:54)
> at
> org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableDeserializeRead.readNextField(BinarySortableDeserializeRead.java:205)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorDeserializeRow.deserialize(VectorDeserializeRow.java:751)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:417)
> ... 19 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16448) Vectorization: Vectorized order_null.q fails with deserialize EOF exception below TEZ ReduceRecordSource.processVectorGroup
[
https://issues.apache.org/jira/browse/HIVE-16448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-16448:
Status: In Progress (was: Patch Available)
> Vectorization: Vectorized order_null.q fails with deserialize EOF exception
> below TEZ ReduceRecordSource.processVectorGroup
> ---
>
> Key: HIVE-16448
> URL: https://issues.apache.org/jira/browse/HIVE-16448
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-16448.01.patch, HIVE-16448.02.patch,
> HIVE-16448.03.patch
>
>
> For query "SELECT x.* FROM src_null x ORDER BY b asc, a asc nulls last" here
> is the stack trace:
> {code}
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
> attempt_1492136345968_0001_40_01_00_1:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:389)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:245)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:317)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> ... 15 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:421)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:380)
> ... 18 more
> Caused by: java.io.EOFException
> at
> org.apache.hadoop.hive.serde2.binarysortable.InputByteBuffer.read(InputByteBuffer.java:54)
> at
> org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableDeserializeRead.readNextField(BinarySortableDeserializeRead.java:205)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorDeserializeRow.deserialize(VectorDeserializeRow.java:751)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:417)
> ... 19 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16451) Race condition between HiveStatement.getQueryLog and HiveStatement.runAsyncOnServer
[
https://issues.apache.org/jira/browse/HIVE-16451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-16451:
--
Attachment: HIVE-16451.02.patch
Addressed test failures
> Race condition between HiveStatement.getQueryLog and
> HiveStatement.runAsyncOnServer
> ---
>
> Key: HIVE-16451
> URL: https://issues.apache.org/jira/browse/HIVE-16451
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16451.02.patch, HIVE-16451.patch
>
>
> During the BeeLineDriver testing I have met the following race condition:
> - Run the query asynchronously through BeeLine
> - Querying the logs in the BeeLine
> In the following code:
> {code:title=HiveStatement.runAsyncOnServer}
> private void runAsyncOnServer(String sql) throws SQLException {
> checkConnection("execute");
> closeClientOperation();
> initFlags();
> [..]
> }
> {code}
> {code:title=HiveStatement.getQueryLog}
> public List getQueryLog(boolean incremental, int fetchSize)
> throws SQLException, ClosedOrCancelledStatementException {
> [..]
> try {
> if (stmtHandle != null) {
> [..]
> } else {
> if (isQueryClosed) {
> throw new ClosedOrCancelledStatementException("Method getQueryLog()
> failed. The " +
> "statement has been closed or cancelled.");
> } else {
> return logs;
> }
> }
> } catch (SQLException e) {
> [..]
> }
> [..]
> }
> {code}
> The runAsyncOnServer {{closeClientOperation}} sets {{isQueryClosed}} flag to
> true:
> {code:title=HiveStatement.closeClientOperation}
> void closeClientOperation() throws SQLException {
> [..]
> isQueryClosed = true;
> isExecuteStatementFailed = false;
> stmtHandle = null;
> }
> {code}
> The {{initFlags}} sets it to false:
> {code}
> private void initFlags() {
> isCancelled = false;
> isQueryClosed = false;
> isLogBeingGenerated = true;
> isExecuteStatementFailed = false;
> isOperationComplete = false;
> }
> {code}
> If the {{getQueryLog}} is called after the {{closeClientOperation}}, but
> before the {{initFlags}}, then we will have a following warning if verbose
> mode is set to true in BeeLine:
> {code}
> Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method
> getQueryLog() failed. The statement has been closed or cancelled.
> (state=,code=0)
> {code}
> This caused this fail:
> https://builds.apache.org/job/PreCommit-HIVE-Build/4691/testReport/org.apache.hadoop.hive.cli/TestBeeLineDriver/testCliDriver_smb_mapjoin_11_/
> {code}
> Error Message
> Client result comparison failed with error code = 1 while executing
> fname=smb_mapjoin_11
> 16a17
> > Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method
> > getQueryLog() failed. The statement has been closed or cancelled.
> > (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16448) Vectorization: Vectorized order_null.q fails with deserialize EOF exception below TEZ ReduceRecordSource.processVectorGroup
[
https://issues.apache.org/jira/browse/HIVE-16448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969538#comment-15969538
]
Hive QA commented on HIVE-16448:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863498/HIVE-16448.02.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10578 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_order_null]
(batchId=27)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4698/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4698/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4698/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863498 - PreCommit-HIVE-Build
> Vectorization: Vectorized order_null.q fails with deserialize EOF exception
> below TEZ ReduceRecordSource.processVectorGroup
> ---
>
> Key: HIVE-16448
> URL: https://issues.apache.org/jira/browse/HIVE-16448
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-16448.01.patch, HIVE-16448.02.patch
>
>
> For query "SELECT x.* FROM src_null x ORDER BY b asc, a asc nulls last" here
> is the stack trace:
> {code}
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
> attempt_1492136345968_0001_40_01_00_1:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:389)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:245)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:317)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> ... 15 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to
[jira] [Updated] (HIVE-16455) ADD JAR command leaks JAR Files
[
https://issues.apache.org/jira/browse/HIVE-16455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-16455:
Status: Patch Available (was: Open)
> ADD JAR command leaks JAR Files
> ---
>
> Key: HIVE-16455
> URL: https://issues.apache.org/jira/browse/HIVE-16455
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-16455.1.patch
>
>
> HiveServer2 is leaking file handles when using ADD JAR statement and the JAR
> file added is not used in the query itself.
> {noformat}
> beeline> !connect jdbc:hive2://localhost:1 admin
> 0: jdbc:hive2://localhost:1> create table test_leak (a int);
> 0: jdbc:hive2://localhost:1> insert into test_leak Values (1);
> -- Exit beeline terminal; Find PID of HiveServer2
> [root@host-10-17-80-111 ~]# lsof -p 29588 | grep "(deleted)" | wc -l
> 0
> [root@host-10-17-80-111 ~]# beeline -u jdbc:hive2://localhost:1/default
> -n admin
> And run the command "ADD JAR hdfs:///tmp/hive-contrib.jar; select * from
> test_leak"
> [root@host-10-17-80-111 ~]# lsof -p 29588 | grep "(deleted)" | wc -l
> 1
> java29588 hive 391u REG 252,3125987 2099944
> /tmp/57d98f5b-1e53-44e2-876b-6b4323ac24db_resources/hive-contrib.jar (deleted)
> java29588 hive 392u REG 252,3125987 2099946
> /tmp/eb3184ad-7f15-4a77-a10d-87717ae634d1_resources/hive-contrib.jar (deleted)
> java29588 hive 393r REG 252,3125987 2099825
> /tmp/e29dccfc-5708-4254-addb-7a8988fc0500_resources/hive-contrib.jar (deleted)
> java29588 hive 394r REG 252,3125987 2099833
> /tmp/5153dd4a-a606-4f53-b02c-d606e7e56985_resources/hive-contrib.jar (deleted)
> java29588 hive 395r REG 252,3125987 2099827
> /tmp/ff3cdb05-917f-43c0-830a-b293bf397a23_resources/hive-contrib.jar (deleted)
> java29588 hive 396r REG 252,3125987 2099822
> /tmp/60531b66-5985-421e-8eb5-eeac31fdf964_resources/hive-contrib.jar (deleted)
> java29588 hive 397r REG 252,3125987 2099831
> /tmp/78878921-455c-438c-9735-447566ed8381_resources/hive-contrib.jar (deleted)
> java29588 hive 399r REG 252,3125987 2099835
> /tmp/0e5d7990-30cc-4248-9058-587f7f1ff211_resources/hive-contrib.jar (deleted)
> {noformat}
> You can see the the session directory (and therefore anything in it) is set
> to delete only on exit.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16455) ADD JAR command leaks JAR Files
[
https://issues.apache.org/jira/browse/HIVE-16455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969520#comment-15969520
]
Aihua Xu commented on HIVE-16455:
-
[~rdsr] and [~jdere] Do you guys know why exactly the jars are not get
released? Thanks.
> ADD JAR command leaks JAR Files
> ---
>
> Key: HIVE-16455
> URL: https://issues.apache.org/jira/browse/HIVE-16455
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-16455.1.patch
>
>
> HiveServer2 is leaking file handles when using ADD JAR statement and the JAR
> file added is not used in the query itself.
> {noformat}
> beeline> !connect jdbc:hive2://localhost:1 admin
> 0: jdbc:hive2://localhost:1> create table test_leak (a int);
> 0: jdbc:hive2://localhost:1> insert into test_leak Values (1);
> -- Exit beeline terminal; Find PID of HiveServer2
> [root@host-10-17-80-111 ~]# lsof -p 29588 | grep "(deleted)" | wc -l
> 0
> [root@host-10-17-80-111 ~]# beeline -u jdbc:hive2://localhost:1/default
> -n admin
> And run the command "ADD JAR hdfs:///tmp/hive-contrib.jar; select * from
> test_leak"
> [root@host-10-17-80-111 ~]# lsof -p 29588 | grep "(deleted)" | wc -l
> 1
> java29588 hive 391u REG 252,3125987 2099944
> /tmp/57d98f5b-1e53-44e2-876b-6b4323ac24db_resources/hive-contrib.jar (deleted)
> java29588 hive 392u REG 252,3125987 2099946
> /tmp/eb3184ad-7f15-4a77-a10d-87717ae634d1_resources/hive-contrib.jar (deleted)
> java29588 hive 393r REG 252,3125987 2099825
> /tmp/e29dccfc-5708-4254-addb-7a8988fc0500_resources/hive-contrib.jar (deleted)
> java29588 hive 394r REG 252,3125987 2099833
> /tmp/5153dd4a-a606-4f53-b02c-d606e7e56985_resources/hive-contrib.jar (deleted)
> java29588 hive 395r REG 252,3125987 2099827
> /tmp/ff3cdb05-917f-43c0-830a-b293bf397a23_resources/hive-contrib.jar (deleted)
> java29588 hive 396r REG 252,3125987 2099822
> /tmp/60531b66-5985-421e-8eb5-eeac31fdf964_resources/hive-contrib.jar (deleted)
> java29588 hive 397r REG 252,3125987 2099831
> /tmp/78878921-455c-438c-9735-447566ed8381_resources/hive-contrib.jar (deleted)
> java29588 hive 399r REG 252,3125987 2099835
> /tmp/0e5d7990-30cc-4248-9058-587f7f1ff211_resources/hive-contrib.jar (deleted)
> {noformat}
> You can see the the session directory (and therefore anything in it) is set
> to delete only on exit.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16455) ADD JAR command leaks JAR Files
[
https://issues.apache.org/jira/browse/HIVE-16455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-16455:
Attachment: HIVE-16455.1.patch
patch-1: we need to explicitly remove jars from the classpath of the
classloader. Looks like closing the classLoader is not necessarily releasing
the references to the jar.
> ADD JAR command leaks JAR Files
> ---
>
> Key: HIVE-16455
> URL: https://issues.apache.org/jira/browse/HIVE-16455
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-16455.1.patch
>
>
> HiveServer2 is leaking file handles when using ADD JAR statement and the JAR
> file added is not used in the query itself.
> {noformat}
> beeline> !connect jdbc:hive2://localhost:1 admin
> 0: jdbc:hive2://localhost:1> create table test_leak (a int);
> 0: jdbc:hive2://localhost:1> insert into test_leak Values (1);
> -- Exit beeline terminal; Find PID of HiveServer2
> [root@host-10-17-80-111 ~]# lsof -p 29588 | grep "(deleted)" | wc -l
> 0
> [root@host-10-17-80-111 ~]# beeline -u jdbc:hive2://localhost:1/default
> -n admin
> And run the command "ADD JAR hdfs:///tmp/hive-contrib.jar; select * from
> test_leak"
> [root@host-10-17-80-111 ~]# lsof -p 29588 | grep "(deleted)" | wc -l
> 1
> java29588 hive 391u REG 252,3125987 2099944
> /tmp/57d98f5b-1e53-44e2-876b-6b4323ac24db_resources/hive-contrib.jar (deleted)
> java29588 hive 392u REG 252,3125987 2099946
> /tmp/eb3184ad-7f15-4a77-a10d-87717ae634d1_resources/hive-contrib.jar (deleted)
> java29588 hive 393r REG 252,3125987 2099825
> /tmp/e29dccfc-5708-4254-addb-7a8988fc0500_resources/hive-contrib.jar (deleted)
> java29588 hive 394r REG 252,3125987 2099833
> /tmp/5153dd4a-a606-4f53-b02c-d606e7e56985_resources/hive-contrib.jar (deleted)
> java29588 hive 395r REG 252,3125987 2099827
> /tmp/ff3cdb05-917f-43c0-830a-b293bf397a23_resources/hive-contrib.jar (deleted)
> java29588 hive 396r REG 252,3125987 2099822
> /tmp/60531b66-5985-421e-8eb5-eeac31fdf964_resources/hive-contrib.jar (deleted)
> java29588 hive 397r REG 252,3125987 2099831
> /tmp/78878921-455c-438c-9735-447566ed8381_resources/hive-contrib.jar (deleted)
> java29588 hive 399r REG 252,3125987 2099835
> /tmp/0e5d7990-30cc-4248-9058-587f7f1ff211_resources/hive-contrib.jar (deleted)
> {noformat}
> You can see the the session directory (and therefore anything in it) is set
> to delete only on exit.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16454) Add blobstore tests for inserting empty into dynamic partition/list bucket tables & inserting cross blobstore tables
[ https://issues.apache.org/jira/browse/HIVE-16454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969490#comment-15969490 ] Rentao Wu commented on HIVE-16454: -- Attached patch and reviewboard > Add blobstore tests for inserting empty into dynamic partition/list bucket > tables & inserting cross blobstore tables > > > Key: HIVE-16454 > URL: https://issues.apache.org/jira/browse/HIVE-16454 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.1.1 >Reporter: Rentao Wu >Assignee: Rentao Wu > Fix For: 2.3.0, 3.0.0 > > Attachments: HIVE-16454.patch > > > This patch introduces two regression tests into the hive-blobstore qtest > module: insert_empty_into_blobstore.q and insert_blobstore_to_blobstore.q. > These tests the following cases: > 1. Insert empty data into dynamic partitioned and list bucketed tables. > 2. Insert data from a blobstore table to another blobstore table. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16455) ADD JAR command leaks JAR Files
[
https://issues.apache.org/jira/browse/HIVE-16455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu reassigned HIVE-16455:
---
> ADD JAR command leaks JAR Files
> ---
>
> Key: HIVE-16455
> URL: https://issues.apache.org/jira/browse/HIVE-16455
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>
> HiveServer2 is leaking file handles when using ADD JAR statement and the JAR
> file added is not used in the query itself.
> {noformat}
> beeline> !connect jdbc:hive2://localhost:1 admin
> 0: jdbc:hive2://localhost:1> create table test_leak (a int);
> 0: jdbc:hive2://localhost:1> insert into test_leak Values (1);
> -- Exit beeline terminal; Find PID of HiveServer2
> [root@host-10-17-80-111 ~]# lsof -p 29588 | grep "(deleted)" | wc -l
> 0
> [root@host-10-17-80-111 ~]# beeline -u jdbc:hive2://localhost:1/default
> -n admin
> And run the command "ADD JAR hdfs:///tmp/hive-contrib.jar; select * from
> test_leak"
> [root@host-10-17-80-111 ~]# lsof -p 29588 | grep "(deleted)" | wc -l
> 1
> java29588 hive 391u REG 252,3125987 2099944
> /tmp/57d98f5b-1e53-44e2-876b-6b4323ac24db_resources/hive-contrib.jar (deleted)
> java29588 hive 392u REG 252,3125987 2099946
> /tmp/eb3184ad-7f15-4a77-a10d-87717ae634d1_resources/hive-contrib.jar (deleted)
> java29588 hive 393r REG 252,3125987 2099825
> /tmp/e29dccfc-5708-4254-addb-7a8988fc0500_resources/hive-contrib.jar (deleted)
> java29588 hive 394r REG 252,3125987 2099833
> /tmp/5153dd4a-a606-4f53-b02c-d606e7e56985_resources/hive-contrib.jar (deleted)
> java29588 hive 395r REG 252,3125987 2099827
> /tmp/ff3cdb05-917f-43c0-830a-b293bf397a23_resources/hive-contrib.jar (deleted)
> java29588 hive 396r REG 252,3125987 2099822
> /tmp/60531b66-5985-421e-8eb5-eeac31fdf964_resources/hive-contrib.jar (deleted)
> java29588 hive 397r REG 252,3125987 2099831
> /tmp/78878921-455c-438c-9735-447566ed8381_resources/hive-contrib.jar (deleted)
> java29588 hive 399r REG 252,3125987 2099835
> /tmp/0e5d7990-30cc-4248-9058-587f7f1ff211_resources/hive-contrib.jar (deleted)
> {noformat}
> You can see the the session directory (and therefore anything in it) is set
> to delete only on exit.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16448) Vectorization: Vectorized order_null.q fails with deserialize EOF exception below TEZ ReduceRecordSource.processVectorGroup
[
https://issues.apache.org/jira/browse/HIVE-16448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-16448:
Attachment: HIVE-16448.02.patch
> Vectorization: Vectorized order_null.q fails with deserialize EOF exception
> below TEZ ReduceRecordSource.processVectorGroup
> ---
>
> Key: HIVE-16448
> URL: https://issues.apache.org/jira/browse/HIVE-16448
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-16448.01.patch, HIVE-16448.02.patch
>
>
> For query "SELECT x.* FROM src_null x ORDER BY b asc, a asc nulls last" here
> is the stack trace:
> {code}
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
> attempt_1492136345968_0001_40_01_00_1:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:389)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:245)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:317)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> ... 15 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:421)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:380)
> ... 18 more
> Caused by: java.io.EOFException
> at
> org.apache.hadoop.hive.serde2.binarysortable.InputByteBuffer.read(InputByteBuffer.java:54)
> at
> org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableDeserializeRead.readNextField(BinarySortableDeserializeRead.java:205)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorDeserializeRow.deserialize(VectorDeserializeRow.java:751)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:417)
> ... 19 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16448) Vectorization: Vectorized order_null.q fails with deserialize EOF exception below TEZ ReduceRecordSource.processVectorGroup
[
https://issues.apache.org/jira/browse/HIVE-16448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-16448:
Status: Patch Available (was: In Progress)
> Vectorization: Vectorized order_null.q fails with deserialize EOF exception
> below TEZ ReduceRecordSource.processVectorGroup
> ---
>
> Key: HIVE-16448
> URL: https://issues.apache.org/jira/browse/HIVE-16448
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-16448.01.patch, HIVE-16448.02.patch
>
>
> For query "SELECT x.* FROM src_null x ORDER BY b asc, a asc nulls last" here
> is the stack trace:
> {code}
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
> attempt_1492136345968_0001_40_01_00_1:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:389)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:245)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:317)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> ... 15 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:421)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:380)
> ... 18 more
> Caused by: java.io.EOFException
> at
> org.apache.hadoop.hive.serde2.binarysortable.InputByteBuffer.read(InputByteBuffer.java:54)
> at
> org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableDeserializeRead.readNextField(BinarySortableDeserializeRead.java:205)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorDeserializeRow.deserialize(VectorDeserializeRow.java:751)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:417)
> ... 19 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16213) ObjectStore can leak Queries when rollbackTransaction throws an exception
[
https://issues.apache.org/jira/browse/HIVE-16213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969460#comment-15969460
]
Vihang Karajgaonkar commented on HIVE-16213:
Hi [~akolb] I looked into the possibility of using QueryWrapper which
implements Autocloseable and then use try-with-resources in Objectstore. For
that I think I will need to create another QueryWrapperBuilder class which is
used to create the QueryWrapper object based on the various query arguments
like filter, result, ordering, unique etc. Once the queryBuilder is initialized
I can do something like this :
{noformat}
try (QueryWrapper query = queryBuilder.build()) {
...
query.execute();
} finally {
if(!successful) {
rollback();
}
}
{noformat}
If we use this approach the patch is becoming very big since there are ~90
instances in ObjectStore.java which uses Query. We will have to replace all
these instances with a QueryWrapper. Given that JDO 3.2 is going to implement
Autocloseable for Query (https://issues.apache.org/jira/browse/JDO-735) I am
not sure if advantages of using this approach are really worth the effort. How
about we use the HIVE-16213.01.patch for now until JDO 3.2 is released. Once
JDO 3.2 is released I can create another JIRA to start consuming it and then
use try-with-resources directly on the query object.
What do think?
> ObjectStore can leak Queries when rollbackTransaction throws an exception
> -
>
> Key: HIVE-16213
> URL: https://issues.apache.org/jira/browse/HIVE-16213
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Alexander Kolbasov
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-16213.01.patch
>
>
> In ObjectStore.java there are a few places with the code similar to:
> {code}
> Query query = null;
> try {
> openTransaction();
> query = pm.newQuery(Something.class);
> ...
> commited = commitTransaction();
> } finally {
> if (!commited) {
> rollbackTransaction();
> }
> if (query != null) {
> query.closeAll();
> }
> }
> {code}
> The problem is that rollbackTransaction() may throw an exception in which
> case query.closeAll() wouldn't be executed.
> The fix would be to wrap rollbackTransaction in its own try-catch block.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16454) Add blobstore tests for inserting empty into dynamic partition/list bucket tables & inserting cross blobstore tables
[ https://issues.apache.org/jira/browse/HIVE-16454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rentao Wu updated HIVE-16454: - Attachment: HIVE-16454.patch > Add blobstore tests for inserting empty into dynamic partition/list bucket > tables & inserting cross blobstore tables > > > Key: HIVE-16454 > URL: https://issues.apache.org/jira/browse/HIVE-16454 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.1.1 >Reporter: Rentao Wu >Assignee: Rentao Wu > Fix For: 2.3.0, 3.0.0 > > Attachments: HIVE-16454.patch > > > This patch introduces two regression tests into the hive-blobstore qtest > module: insert_empty_into_blobstore.q and insert_blobstore_to_blobstore.q. > These tests the following cases: > 1. Insert empty data into dynamic partitioned and list bucketed tables. > 2. Insert data from a blobstore table to another blobstore table. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16415) Add tests covering single inserts of zero rows
[ https://issues.apache.org/jira/browse/HIVE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-16415: --- Fix Version/s: 2.2.0 > Add tests covering single inserts of zero rows > -- > > Key: HIVE-16415 > URL: https://issues.apache.org/jira/browse/HIVE-16415 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.1.1 >Reporter: Thomas Poepping >Assignee: Thomas Poepping > Fix For: 2.2.0, 2.3.0, 3.0.0 > > Attachments: HIVE-16415.01.patch, HIVE-16415.patch > > > This patch introduces two regression tests into the hive-blobstore qtest > module: zero_rows_hdfs.q and zero_rows_blobstore.q. These test doing INSERT > commands with a WHERE clause where the condition of the WHERE clause causes > zero rows to be considered. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16454) Add blobstore tests for inserting empty into dynamic partition/list bucket tables & inserting cross blobstore tables
[ https://issues.apache.org/jira/browse/HIVE-16454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rentao Wu reassigned HIVE-16454: > Add blobstore tests for inserting empty into dynamic partition/list bucket > tables & inserting cross blobstore tables > > > Key: HIVE-16454 > URL: https://issues.apache.org/jira/browse/HIVE-16454 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.1.1 >Reporter: Rentao Wu >Assignee: Rentao Wu > Fix For: 2.3.0, 3.0.0 > > > This patch introduces two regression tests into the hive-blobstore qtest > module: insert_empty_into_blobstore.q and insert_blobstore_to_blobstore.q. > These tests the following cases: > 1. Insert empty data into dynamic partitioned and list bucketed tables. > 2. Insert data from a blobstore table to another blobstore table. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HIVE-14798) MSCK REPAIR TABLE throws null pointer exception
[ https://issues.apache.org/jira/browse/HIVE-14798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anbu Cheeralan resolved HIVE-14798. --- Resolution: Fixed Fix Version/s: 2.1.1 This issue is no longer happening in Hive 2.1.1 > MSCK REPAIR TABLE throws null pointer exception > --- > > Key: HIVE-14798 > URL: https://issues.apache.org/jira/browse/HIVE-14798 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.1.0 >Reporter: Anbu Cheeralan > Fix For: 2.1.1 > > > MSCK REPAIR TABLE statement throws null pointer exception in Hive 2.1 > I have tested the same against external/internal tables created both in HDFS > and in Google Cloud. > The error shown in beeline/sql client > Error: Error while processing statement: FAILED: Execution Error, return code > 1 from org.apache.hadoop.hive.ql.exec.DDLTask (state=08S01,code=1) > Hive Logs: > 2016-09-20T17:28:00,717 ERROR [HiveServer2-Background-Pool: Thread-92]: > metadata.HiveMetaStoreChecker (:()) - java.lang.NullPointerException > 2016-09-20T17:28:00,717 WARN [HiveServer2-Background-Pool: Thread-92]: > exec.DDLTask (:()) - Failed to run metacheck: > org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.getAllLeafDirs(HiveMetaStoreChecker.java:444) > at > org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.getAllLeafDirs(HiveMetaStoreChecker.java:388) > at > org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.findUnknownPartitions(HiveMetaStoreChecker.java:309) > at > org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkTable(HiveMetaStoreChecker.java:285) > at > org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkTable(HiveMetaStoreChecker.java:230) > at > org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkMetastore(HiveMetaStoreChecker.java:109) > at org.apache.hadoop.hive.ql.exec.DDLTask.msck(DDLTask.java:1814) > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:403) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1858) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1562) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1313) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1084) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1077) > at > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:235) > at > org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:90) > at > org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:299) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at > org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:312) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.NullPointerException > at > java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011) > at > java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006) > at > org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call(HiveMetaStoreChecker.java:432) > at > org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call(HiveMetaStoreChecker.java:418) > ... 4 more > Here are the steps to recreate this issue: > use default; > DROP TABLE IF EXISTS repairtable; > CREATE TABLE repairtable(col STRING) PARTITIONED BY (p1 STRING, p2 STRING); > MSCK REPAIR TABLE default.repairtable; -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-14798) MSCK REPAIR TABLE throws null pointer exception
[ https://issues.apache.org/jira/browse/HIVE-14798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969375#comment-15969375 ] Anbu Cheeralan commented on HIVE-14798: --- [~premal] Sorry for the late reply. As mentioned in the comment section by [~chetna] in that ticket, this issue was not in the master branch then (Hive 2.1.1) and I verified in Hive 2.1.1 that this issue is not happening. > MSCK REPAIR TABLE throws null pointer exception > --- > > Key: HIVE-14798 > URL: https://issues.apache.org/jira/browse/HIVE-14798 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.1.0 >Reporter: Anbu Cheeralan > > MSCK REPAIR TABLE statement throws null pointer exception in Hive 2.1 > I have tested the same against external/internal tables created both in HDFS > and in Google Cloud. > The error shown in beeline/sql client > Error: Error while processing statement: FAILED: Execution Error, return code > 1 from org.apache.hadoop.hive.ql.exec.DDLTask (state=08S01,code=1) > Hive Logs: > 2016-09-20T17:28:00,717 ERROR [HiveServer2-Background-Pool: Thread-92]: > metadata.HiveMetaStoreChecker (:()) - java.lang.NullPointerException > 2016-09-20T17:28:00,717 WARN [HiveServer2-Background-Pool: Thread-92]: > exec.DDLTask (:()) - Failed to run metacheck: > org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.getAllLeafDirs(HiveMetaStoreChecker.java:444) > at > org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.getAllLeafDirs(HiveMetaStoreChecker.java:388) > at > org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.findUnknownPartitions(HiveMetaStoreChecker.java:309) > at > org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkTable(HiveMetaStoreChecker.java:285) > at > org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkTable(HiveMetaStoreChecker.java:230) > at > org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkMetastore(HiveMetaStoreChecker.java:109) > at org.apache.hadoop.hive.ql.exec.DDLTask.msck(DDLTask.java:1814) > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:403) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1858) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1562) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1313) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1084) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1077) > at > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:235) > at > org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:90) > at > org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:299) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at > org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:312) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.NullPointerException > at > java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011) > at > java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006) > at > org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call(HiveMetaStoreChecker.java:432) > at > org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call(HiveMetaStoreChecker.java:418) > ... 4 more > Here are the steps to recreate this issue: > use default; > DROP TABLE IF EXISTS repairtable; > CREATE TABLE repairtable(col STRING) PARTITIONED BY (p1 STRING, p2 STRING); > MSCK REPAIR TABLE default.repairtable; -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16415) Add tests covering single inserts of zero rows
[ https://issues.apache.org/jira/browse/HIVE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969361#comment-15969361 ] Thomas Poepping commented on HIVE-16415: I haven't seen any conversation on the dev list about moving 2.2 anytime soon, so I don't see why we shouldn't target that. > Add tests covering single inserts of zero rows > -- > > Key: HIVE-16415 > URL: https://issues.apache.org/jira/browse/HIVE-16415 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.1.1 >Reporter: Thomas Poepping >Assignee: Thomas Poepping > Fix For: 2.3.0, 3.0.0 > > Attachments: HIVE-16415.01.patch, HIVE-16415.patch > > > This patch introduces two regression tests into the hive-blobstore qtest > module: zero_rows_hdfs.q and zero_rows_blobstore.q. These test doing INSERT > commands with a WHERE clause where the condition of the WHERE clause causes > zero rows to be considered. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16415) Add tests covering single inserts of zero rows
[ https://issues.apache.org/jira/browse/HIVE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969358#comment-15969358 ] Sergio Peña commented on HIVE-16415: [~poeppt] do you want me to push this patch to 2.2 or 2.3 is fine? i see the target is set to 2.2 > Add tests covering single inserts of zero rows > -- > > Key: HIVE-16415 > URL: https://issues.apache.org/jira/browse/HIVE-16415 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.1.1 >Reporter: Thomas Poepping >Assignee: Thomas Poepping > Fix For: 2.3.0, 3.0.0 > > Attachments: HIVE-16415.01.patch, HIVE-16415.patch > > > This patch introduces two regression tests into the hive-blobstore qtest > module: zero_rows_hdfs.q and zero_rows_blobstore.q. These test doing INSERT > commands with a WHERE clause where the condition of the WHERE clause causes > zero rows to be considered. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16415) Add tests covering single inserts of zero rows
[ https://issues.apache.org/jira/browse/HIVE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-16415: --- Resolution: Fixed Fix Version/s: 3.0.0 2.3.0 Status: Resolved (was: Patch Available) > Add tests covering single inserts of zero rows > -- > > Key: HIVE-16415 > URL: https://issues.apache.org/jira/browse/HIVE-16415 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.1.1 >Reporter: Thomas Poepping >Assignee: Thomas Poepping > Fix For: 2.3.0, 3.0.0 > > Attachments: HIVE-16415.01.patch, HIVE-16415.patch > > > This patch introduces two regression tests into the hive-blobstore qtest > module: zero_rows_hdfs.q and zero_rows_blobstore.q. These test doing INSERT > commands with a WHERE clause where the condition of the WHERE clause causes > zero rows to be considered. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16415) Add tests covering single inserts of zero rows
[ https://issues.apache.org/jira/browse/HIVE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969354#comment-15969354 ] Thomas Poepping commented on HIVE-16415: Thank you Sergio! > Add tests covering single inserts of zero rows > -- > > Key: HIVE-16415 > URL: https://issues.apache.org/jira/browse/HIVE-16415 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.1.1 >Reporter: Thomas Poepping >Assignee: Thomas Poepping > Attachments: HIVE-16415.01.patch, HIVE-16415.patch > > > This patch introduces two regression tests into the hive-blobstore qtest > module: zero_rows_hdfs.q and zero_rows_blobstore.q. These test doing INSERT > commands with a WHERE clause where the condition of the WHERE clause causes > zero rows to be considered. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16415) Add tests covering single inserts of zero rows
[ https://issues.apache.org/jira/browse/HIVE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969350#comment-15969350 ] Sergio Peña commented on HIVE-16415: Great. thanks [~poeppt] for the change I committed this to master. I will add it to branch-2 as well. > Add tests covering single inserts of zero rows > -- > > Key: HIVE-16415 > URL: https://issues.apache.org/jira/browse/HIVE-16415 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.1.1 >Reporter: Thomas Poepping >Assignee: Thomas Poepping > Attachments: HIVE-16415.01.patch, HIVE-16415.patch > > > This patch introduces two regression tests into the hive-blobstore qtest > module: zero_rows_hdfs.q and zero_rows_blobstore.q. These test doing INSERT > commands with a WHERE clause where the condition of the WHERE clause causes > zero rows to be considered. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16415) Add tests covering single inserts of zero rows
[ https://issues.apache.org/jira/browse/HIVE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-16415: --- Summary: Add tests covering single inserts of zero rows (was: Add blobstore tests for insertion of zero rows) > Add tests covering single inserts of zero rows > -- > > Key: HIVE-16415 > URL: https://issues.apache.org/jira/browse/HIVE-16415 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.1.1 >Reporter: Thomas Poepping >Assignee: Thomas Poepping > Attachments: HIVE-16415.01.patch, HIVE-16415.patch > > > This patch introduces two regression tests into the hive-blobstore qtest > module: zero_rows_hdfs.q and zero_rows_blobstore.q. These test doing INSERT > commands with a WHERE clause where the condition of the WHERE clause causes > zero rows to be considered. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16451) Race condition between HiveStatement.getQueryLog and HiveStatement.runAsyncOnServer
[
https://issues.apache.org/jira/browse/HIVE-16451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969348#comment-15969348
]
Hive QA commented on HIVE-16451:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863475/HIVE-16451.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 10551 tests
executed
*Failed tests:*
{noformat}
TestSSL - did not produce a TEST-*.xml file (likely timed out) (batchId=220)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
(batchId=100)
org.apache.hadoop.hive.ql.TestMetaStoreLimitPartitionRequest.testMoreComplexQueryWithDirectSqlTooManyPartitions
(batchId=214)
org.apache.hadoop.hive.ql.TestMetaStoreLimitPartitionRequest.testQueryWithFallbackToORMTooManyPartitions1
(batchId=214)
org.apache.hadoop.hive.ql.TestMetaStoreLimitPartitionRequest.testQueryWithFallbackToORMTooManyPartitions2
(batchId=214)
org.apache.hadoop.hive.ql.TestMetaStoreLimitPartitionRequest.testQueryWithFallbackToORMTooManyPartitions3
(batchId=214)
org.apache.hadoop.hive.ql.TestMetaStoreLimitPartitionRequest.testQueryWithFallbackToORMTooManyPartitions4
(batchId=214)
org.apache.hadoop.hive.ql.TestMetaStoreLimitPartitionRequest.testQueryWithInWithFallbackToORMTooManyPartitions
(batchId=214)
org.apache.hadoop.hive.ql.TestMetaStoreLimitPartitionRequest.testQueryWithInWithFallbackToORMTooManyPartitions2
(batchId=214)
org.apache.hadoop.hive.ql.TestMetaStoreLimitPartitionRequest.testQueryWithLikeWithFallbackToORMTooManyPartitions
(batchId=214)
org.apache.hadoop.hive.ql.TestMetaStoreLimitPartitionRequest.testSimpleQueryWithDirectSqlTooManyPartitions
(batchId=214)
org.apache.hive.jdbc.authorization.TestJdbcWithSQLAuthUDFBlacklist.testBlackListedUdfUsage
(batchId=223)
org.apache.hive.jdbc.authorization.TestJdbcWithSQLAuthorization.testBlackListedUdfUsage
(batchId=223)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4697/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4697/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4697/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863475 - PreCommit-HIVE-Build
> Race condition between HiveStatement.getQueryLog and
> HiveStatement.runAsyncOnServer
> ---
>
> Key: HIVE-16451
> URL: https://issues.apache.org/jira/browse/HIVE-16451
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16451.patch
>
>
> During the BeeLineDriver testing I have met the following race condition:
> - Run the query asynchronously through BeeLine
> - Querying the logs in the BeeLine
> In the following code:
> {code:title=HiveStatement.runAsyncOnServer}
> private void runAsyncOnServer(String sql) throws SQLException {
> checkConnection("execute");
> closeClientOperation();
> initFlags();
> [..]
> }
> {code}
> {code:title=HiveStatement.getQueryLog}
> public List getQueryLog(boolean incremental, int fetchSize)
> throws SQLException, ClosedOrCancelledStatementException {
> [..]
> try {
> if (stmtHandle != null) {
> [..]
> } else {
> if (isQueryClosed) {
> throw new ClosedOrCancelledStatementException("Method getQueryLog()
> failed. The " +
> "statement has been closed or cancelled.");
> } else {
> return logs;
> }
> }
> } catch (SQLException e) {
> [..]
> }
> [..]
> }
> {code}
> The runAsyncOnServer {{closeClientOperation}} sets {{isQueryClosed}} flag to
> true:
> {code:title=HiveStatement.closeClientOperation}
> void closeClientOperation() throws SQLException {
> [..]
> isQueryClosed = true;
> isExecuteStatementFailed = false;
> stmtHandle = null;
> }
> {code}
> The {{initFlags}} sets it to false:
> {code}
> private void initFlags() {
> isCancelled = false;
> isQueryClosed = false;
> isLogBeingGenerated = true;
> isExecuteStatementFailed = false;
> isOperationComplete = false;
> }
> {code}
> If the {{getQueryLog}} is called after the {{closeClientOperation}}, but
> before the {{initFlags}}, then we will have a following warning if verbose
[jira] [Assigned] (HIVE-12156) expanding view doesn't quote reserved keyword
[ https://issues.apache.org/jira/browse/HIVE-12156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] p h reassigned HIVE-12156: -- Assignee: (was: p h) > expanding view doesn't quote reserved keyword > - > > Key: HIVE-12156 > URL: https://issues.apache.org/jira/browse/HIVE-12156 > Project: Hive > Issue Type: Bug > Components: Parser >Affects Versions: 1.2.1 > Environment: hadoop 2.7 > hive 1.2.1 >Reporter: Jay Lee > Fix For: 2.0.0 > > Attachments: HIVE-12156.01.patch, HIVE-12156.02.patch, > HIVE-12156.03.patch > > > hive> create table testreserved (data struct<`end`:string, id: string>); > OK > Time taken: 0.274 seconds > hive> create view testreservedview as select data.`end` as data_end, data.id > as data_id from testreserved; > OK > Time taken: 0.769 seconds > hive> select data.`end` from testreserved; > OK > Time taken: 1.852 seconds > hive> select data_id from testreservedview; > NoViableAltException(98@[]) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.identifier(HiveParser_IdentifiersParser.java:10858) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6438) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:6768) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:6828) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7012) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7172) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7332) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAmpersandExpression(HiveParser_IdentifiersParser.java:7483) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseOrExpression(HiveParser_IdentifiersParser.java:7634) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceEqualExpression(HiveParser_IdentifiersParser.java:8164) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceNotExpression(HiveParser_IdentifiersParser.java:9177) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAndExpression(HiveParser_IdentifiersParser.java:9296) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceOrExpression(HiveParser_IdentifiersParser.java:9455) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.expression(HiveParser_IdentifiersParser.java:6105) > at > org.apache.hadoop.hive.ql.parse.HiveParser.expression(HiveParser.java:45840) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectItem(HiveParser_SelectClauseParser.java:2907) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectList(HiveParser_SelectClauseParser.java:1373) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1128) > at > org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:45827) > at > org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41495) > at > org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402) > at > org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413) > at > org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283) > at > org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1590) > at > org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1109) > at > org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202) > ... > FAILED: SemanticException line 1:29 cannot recognize input near 'end' 'as' > 'data_end' in expression specification in definition of VIEW testreservedview > [ > select `testreserved`.`data`.end as `data_end`, `testreserved`.`data`.id as > `data_id` from `test`.`testreserved` > ] used as testreservedview at Line 1:20 > When view is expanded, field should be quote with backquote. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-12156) expanding view doesn't quote reserved keyword
[ https://issues.apache.org/jira/browse/HIVE-12156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969336#comment-15969336 ] Perry Hook commented on HIVE-12156: --- this was a mistake. must have misclicked. can't undo > expanding view doesn't quote reserved keyword > - > > Key: HIVE-12156 > URL: https://issues.apache.org/jira/browse/HIVE-12156 > Project: Hive > Issue Type: Bug > Components: Parser >Affects Versions: 1.2.1 > Environment: hadoop 2.7 > hive 1.2.1 >Reporter: Jay Lee >Assignee: Perry Hook > Fix For: 2.0.0 > > Attachments: HIVE-12156.01.patch, HIVE-12156.02.patch, > HIVE-12156.03.patch > > > hive> create table testreserved (data struct<`end`:string, id: string>); > OK > Time taken: 0.274 seconds > hive> create view testreservedview as select data.`end` as data_end, data.id > as data_id from testreserved; > OK > Time taken: 0.769 seconds > hive> select data.`end` from testreserved; > OK > Time taken: 1.852 seconds > hive> select data_id from testreservedview; > NoViableAltException(98@[]) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.identifier(HiveParser_IdentifiersParser.java:10858) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6438) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:6768) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:6828) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7012) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7172) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7332) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAmpersandExpression(HiveParser_IdentifiersParser.java:7483) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseOrExpression(HiveParser_IdentifiersParser.java:7634) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceEqualExpression(HiveParser_IdentifiersParser.java:8164) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceNotExpression(HiveParser_IdentifiersParser.java:9177) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAndExpression(HiveParser_IdentifiersParser.java:9296) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceOrExpression(HiveParser_IdentifiersParser.java:9455) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.expression(HiveParser_IdentifiersParser.java:6105) > at > org.apache.hadoop.hive.ql.parse.HiveParser.expression(HiveParser.java:45840) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectItem(HiveParser_SelectClauseParser.java:2907) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectList(HiveParser_SelectClauseParser.java:1373) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1128) > at > org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:45827) > at > org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41495) > at > org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402) > at > org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413) > at > org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283) > at > org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1590) > at > org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1109) > at > org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202) > ... > FAILED: SemanticException line 1:29 cannot recognize input near 'end' 'as' > 'data_end' in expression specification in definition of VIEW testreservedview > [ > select `testreserved`.`data`.end as `data_end`, `testreserved`.`data`.id as > `data_id` from `test`.`testreserved` > ] used as testreservedview at Line 1:20 > When view is expanded, field should be quote with backquote. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-12156) expanding view doesn't quote reserved keyword
[ https://issues.apache.org/jira/browse/HIVE-12156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Perry Hook reassigned HIVE-12156: - Assignee: Perry Hook (was: Pengcheng Xiong) > expanding view doesn't quote reserved keyword > - > > Key: HIVE-12156 > URL: https://issues.apache.org/jira/browse/HIVE-12156 > Project: Hive > Issue Type: Bug > Components: Parser >Affects Versions: 1.2.1 > Environment: hadoop 2.7 > hive 1.2.1 >Reporter: Jay Lee >Assignee: Perry Hook > Fix For: 2.0.0 > > Attachments: HIVE-12156.01.patch, HIVE-12156.02.patch, > HIVE-12156.03.patch > > > hive> create table testreserved (data struct<`end`:string, id: string>); > OK > Time taken: 0.274 seconds > hive> create view testreservedview as select data.`end` as data_end, data.id > as data_id from testreserved; > OK > Time taken: 0.769 seconds > hive> select data.`end` from testreserved; > OK > Time taken: 1.852 seconds > hive> select data_id from testreservedview; > NoViableAltException(98@[]) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.identifier(HiveParser_IdentifiersParser.java:10858) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6438) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:6768) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:6828) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7012) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7172) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7332) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAmpersandExpression(HiveParser_IdentifiersParser.java:7483) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseOrExpression(HiveParser_IdentifiersParser.java:7634) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceEqualExpression(HiveParser_IdentifiersParser.java:8164) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceNotExpression(HiveParser_IdentifiersParser.java:9177) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAndExpression(HiveParser_IdentifiersParser.java:9296) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceOrExpression(HiveParser_IdentifiersParser.java:9455) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.expression(HiveParser_IdentifiersParser.java:6105) > at > org.apache.hadoop.hive.ql.parse.HiveParser.expression(HiveParser.java:45840) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectItem(HiveParser_SelectClauseParser.java:2907) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectList(HiveParser_SelectClauseParser.java:1373) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1128) > at > org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:45827) > at > org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41495) > at > org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402) > at > org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413) > at > org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283) > at > org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1590) > at > org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1109) > at > org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202) > ... > FAILED: SemanticException line 1:29 cannot recognize input near 'end' 'as' > 'data_end' in expression specification in definition of VIEW testreservedview > [ > select `testreserved`.`data`.end as `data_end`, `testreserved`.`data`.id as > `data_id` from `test`.`testreserved` > ] used as testreservedview at Line 1:20 > When view is expanded, field should be quote with backquote. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16287) Alter table partition rename with location - moves partition back to hive warehouse
[
https://issues.apache.org/jira/browse/HIVE-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969315#comment-15969315
]
Sergio Peña commented on HIVE-16287:
It looks good.
+1
> Alter table partition rename with location - moves partition back to hive
> warehouse
> ---
>
> Key: HIVE-16287
> URL: https://issues.apache.org/jira/browse/HIVE-16287
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.1.0
> Environment: RHEL 6.8
>Reporter: Ying Chen
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-16287.01.patch, HIVE-16287.02.patch,
> HIVE-16287.03.patch, HIVE-16287.04.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> I was renaming my partition in a table that I've created using the location
> clause, and noticed that when after rename is completed, my partition is
> moved to the hive warehouse (hive.metastore.warehouse.dir).
> {quote}
> create table test_local_part (col1 int) partitioned by (col2 int) location
> '/tmp/testtable/test_local_part';
> insert into test_local_part partition (col2=1) values (1),(3);
> insert into test_local_part partition (col2=2) values (3);
> alter table test_local_part partition (col2='1') rename to partition
> (col2='4');
> {quote}
> Running:
>describe formatted test_local_part partition (col2='2')
> # Detailed Partition Information
> Partition Value: [2]
> Database: default
> Table:test_local_part
> CreateTime: Mon Mar 20 13:25:28 PDT 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Location:
> *hdfs://my.server.com:8020/tmp/testtable/test_local_part/col2=2*
> Running:
>describe formatted test_local_part partition (col2='4')
> # Detailed Partition Information
> Partition Value: [4]
> Database: default
> Table:test_local_part
> CreateTime: Mon Mar 20 13:24:53 PDT 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Location:
> *hdfs://my.server.com:8020/apps/hive/warehouse/test_local_part/col2=4*
> ---
> Per Sergio's comment - "The rename should create the new partition name in
> the same location of the table. "
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16287) Alter table partition rename with location - moves partition back to hive warehouse
[
https://issues.apache.org/jira/browse/HIVE-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969271#comment-15969271
]
Vihang Karajgaonkar commented on HIVE-16287:
[~spena] Can you please review?
> Alter table partition rename with location - moves partition back to hive
> warehouse
> ---
>
> Key: HIVE-16287
> URL: https://issues.apache.org/jira/browse/HIVE-16287
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.1.0
> Environment: RHEL 6.8
>Reporter: Ying Chen
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-16287.01.patch, HIVE-16287.02.patch,
> HIVE-16287.03.patch, HIVE-16287.04.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> I was renaming my partition in a table that I've created using the location
> clause, and noticed that when after rename is completed, my partition is
> moved to the hive warehouse (hive.metastore.warehouse.dir).
> {quote}
> create table test_local_part (col1 int) partitioned by (col2 int) location
> '/tmp/testtable/test_local_part';
> insert into test_local_part partition (col2=1) values (1),(3);
> insert into test_local_part partition (col2=2) values (3);
> alter table test_local_part partition (col2='1') rename to partition
> (col2='4');
> {quote}
> Running:
>describe formatted test_local_part partition (col2='2')
> # Detailed Partition Information
> Partition Value: [2]
> Database: default
> Table:test_local_part
> CreateTime: Mon Mar 20 13:25:28 PDT 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Location:
> *hdfs://my.server.com:8020/tmp/testtable/test_local_part/col2=2*
> Running:
>describe formatted test_local_part partition (col2='4')
> # Detailed Partition Information
> Partition Value: [4]
> Database: default
> Table:test_local_part
> CreateTime: Mon Mar 20 13:24:53 PDT 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Location:
> *hdfs://my.server.com:8020/apps/hive/warehouse/test_local_part/col2=4*
> ---
> Per Sergio's comment - "The rename should create the new partition name in
> the same location of the table. "
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16452) Database UUID for metastore DB
[ https://issues.apache.org/jira/browse/HIVE-16452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar reassigned HIVE-16452: -- > Database UUID for metastore DB > -- > > Key: HIVE-16452 > URL: https://issues.apache.org/jira/browse/HIVE-16452 > Project: Hive > Issue Type: New Feature > Components: Metastore >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar > > In cloud environments it is possible that a same database instance is used as > the long running metadata persistence layer and multiple HMS access this > database. These HMS instances could be running the same time or in case of > transient workloads come up on an on-demand basis. HMS is used by multiple > projects in the Hadoop eco-system as the de-facto metadata keeper for various > SQL engines on the cluster. Currently, there is no way to uniquely identify > the database instance which is backing the HMS. For example, if there are two > instances of HMS running on top of same metastore DB, there is no way to > identify that data received from both the metastore clients is coming from > the same database. Similarly, if there in case of transient workloads > multiple HMS services come up and go, a external application which is > fetching data from a HMS has no way to identify that these multiple instances > of HMS are in fact returning the same data. > We can potentially use the combination of javax.jdo.option.ConnectionURL, > javax.jdo.option.ConnectionDriverName configuration of each HMS instance but > this is approach may not be very robust. If the database is migrated to > another server for some reason the ConnectionURL can change. Having a UUID in > the metastore DB which can be queried using a Thrift API can help solve this > problem. This way any application talking to multiple HMS instances can > recognize if the data is coming the same backing database. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16449) BeeLineDriver should handle query result sorting
[
https://issues.apache.org/jira/browse/HIVE-16449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969263#comment-15969263
]
Hive QA commented on HIVE-16449:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863474/HIVE-16449.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10574 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4696/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4696/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4696/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863474 - PreCommit-HIVE-Build
> BeeLineDriver should handle query result sorting
>
>
> Key: HIVE-16449
> URL: https://issues.apache.org/jira/browse/HIVE-16449
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16449.02.patch, HIVE-16449.03.patch,
> HIVE-16449.patch
>
>
> The CLI driver supports the following features:
> -- SORT_QUERY_RESULTS
> -- HASH_QUERY_RESULTS
> -- SORT_AND_HASH_QUERY_RESULTS
> BeeLineDriver should find a way to support these
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15302) Relax the requirement that HoS needs Spark built w/o Hive
[
https://issues.apache.org/jira/browse/HIVE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969242#comment-15969242
]
Marcelo Vanzin commented on HIVE-15302:
---
I'm not sure which approach you're trying here, but I'd really discourage you
from trying to manually figure out the list of needed jars like that... that's
what maven is for.
I've done something like this for Livy in the past by having a fake
spark-submit script that does everything that Livy needs:
https://github.com/cloudera/livy/commit/3c314b11777459e10984ab408aaf2cbd47edf6db
The test code in Livy provides the needed classpath
({{System.getProperty("java.class.path")}}), and it all works out. You could
even expand on that idea to do this outside of tests too, by adding features to
the fake spark-submit script.
> Relax the requirement that HoS needs Spark built w/o Hive
> -
>
> Key: HIVE-15302
> URL: https://issues.apache.org/jira/browse/HIVE-15302
> Project: Hive
> Issue Type: Improvement
>Reporter: Rui Li
>Assignee: Rui Li
>
> This requirement becomes more and more unacceptable as SparkSQL becomes
> widely adopted. Let's use this JIRA to find out how we can relax the
> limitation.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16450) Some metastore operations are not retried even with desired underlining exceptions
[
https://issues.apache.org/jira/browse/HIVE-16450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969216#comment-15969216
]
Hive QA commented on HIVE-16450:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863473/HIVE-16450.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10560 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
(batchId=100)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4695/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4695/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4695/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863473 - PreCommit-HIVE-Build
> Some metastore operations are not retried even with desired underlining
> exceptions
> --
>
> Key: HIVE-16450
> URL: https://issues.apache.org/jira/browse/HIVE-16450
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-16450.1.patch
>
>
> In RetryingHMSHandler class, we are expecting the operations should retry
> when the cause of MetaException is JDOException or NucleusException.
> {noformat}
> if (e.getCause() instanceof MetaException && e.getCause().getCause()
> != null) {
> if (e.getCause().getCause() instanceof javax.jdo.JDOException ||
> e.getCause().getCause() instanceof NucleusException) {
> // The JDOException or the Nucleus Exception may be wrapped
> further in a MetaException
> caughtException = e.getCause().getCause();
>}
> {noformat}
> While in ObjectStore, many places we are only throwing new MetaException(msg)
> without the cause, so we are missing retrying for some cases. e.g., with the
> following JDOException, we should retry but it's ignored.
> {noformat}
> 2017-04-04 17:28:21,602 ERROR metastore.ObjectStore
> (ObjectStore.java:getMTableColumnStatistics(6555)) - Error retrieving
> statistics via jdo
> javax.jdo.JDOException: Exception thrown when executing query
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
> at
> org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:321)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTableColumnStatistics(ObjectStore.java:6546)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$1200(ObjectStore.java:171)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$9.getJdoResult(ObjectStore.java:6606)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$9.getJdoResult(ObjectStore.java:6595)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2633)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatisticsInternal(ObjectStore.java:6594)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatistics(ObjectStore.java:6588)
> at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:103)
> at com.sun.proxy.$Proxy0.getTableColumnStatistics(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTableUpdateTableColumnStats(HiveAlterHandler.java:787)
> at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTable(HiveAlterHandler.java:247)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:3809)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_with_environment_context(HiveMetaStore.java:3779)
> at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
>
[jira] [Updated] (HIVE-16431) Support Parquet StatsNoJobTask for Spark & Tez engine
[ https://issues.apache.org/jira/browse/HIVE-16431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HIVE-16431: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master (forgot to add the JIRA # but added in an addendum commit). Thanks Xuefu for the review. > Support Parquet StatsNoJobTask for Spark & Tez engine > - > > Key: HIVE-16431 > URL: https://issues.apache.org/jira/browse/HIVE-16431 > Project: Hive > Issue Type: Improvement >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Minor > Fix For: 3.0.0 > > Attachments: HIVE-16431.1.patch > > > It seems only MR uses StatsNoJobTask for Parquet input format when computing > stats. We should add it to Tez & Spark as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16429) Should call invokeFailureHooks in handleInterruption to track failed query execution due to interrupted command.
[ https://issues.apache.org/jira/browse/HIVE-16429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-16429: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Thanks Zhihai for the patch. Pushed to the master branch. > Should call invokeFailureHooks in handleInterruption to track failed query > execution due to interrupted command. > > > Key: HIVE-16429 > URL: https://issues.apache.org/jira/browse/HIVE-16429 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Fix For: 3.0.0 > > Attachments: HIVE-16429.000.patch, HIVE-16429.001.patch > > > Should call invokeFailureHooks in handleInterruption to track failed query > execution due to interrupted command. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16433) Not nullify variable "rj" to avoid NPE due to race condition in ExecDriver.
[
https://issues.apache.org/jira/browse/HIVE-16433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HIVE-16433:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Thanks Zhihai for the patch. Pushed to the master branch.
> Not nullify variable "rj" to avoid NPE due to race condition in ExecDriver.
> ---
>
> Key: HIVE-16433
> URL: https://issues.apache.org/jira/browse/HIVE-16433
> Project: Hive
> Issue Type: Bug
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HIVE-16433.000.patch
>
>
> Not nullify variable {{rj}} to avoid NPE due to race condition in ExecDriver.
> currently {{rj}} is set to null in ExecDriver.shutdown which is called from
> other thread for query cancellation. It can happen at any time. There is a
> potential race condition, the {{rj}} is still accessed after shutdown is
> called. For example: if the following code is executed right after
> ExecDriver.shutdown is called.
> {code}
> this.jobID = rj.getJobID();
> updateStatusInQueryDisplay();
> returnVal = jobExecHelper.progress(rj, jc, ctx);
> {code}
> Currently the purpose of nullifying {{rj}} is mainly to make sure
> {{rj.killJob()}} is only called once.
> I will add a flag {{jobKilled}} to make sure {{rj.killJob()}} will be only
> called once.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16451) Race condition between HiveStatement.getQueryLog and HiveStatement.runAsyncOnServer
[
https://issues.apache.org/jira/browse/HIVE-16451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-16451:
--
Status: Patch Available (was: Open)
> Race condition between HiveStatement.getQueryLog and
> HiveStatement.runAsyncOnServer
> ---
>
> Key: HIVE-16451
> URL: https://issues.apache.org/jira/browse/HIVE-16451
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16451.patch
>
>
> During the BeeLineDriver testing I have met the following race condition:
> - Run the query asynchronously through BeeLine
> - Querying the logs in the BeeLine
> In the following code:
> {code:title=HiveStatement.runAsyncOnServer}
> private void runAsyncOnServer(String sql) throws SQLException {
> checkConnection("execute");
> closeClientOperation();
> initFlags();
> [..]
> }
> {code}
> {code:title=HiveStatement.getQueryLog}
> public List getQueryLog(boolean incremental, int fetchSize)
> throws SQLException, ClosedOrCancelledStatementException {
> [..]
> try {
> if (stmtHandle != null) {
> [..]
> } else {
> if (isQueryClosed) {
> throw new ClosedOrCancelledStatementException("Method getQueryLog()
> failed. The " +
> "statement has been closed or cancelled.");
> } else {
> return logs;
> }
> }
> } catch (SQLException e) {
> [..]
> }
> [..]
> }
> {code}
> The runAsyncOnServer {{closeClientOperation}} sets {{isQueryClosed}} flag to
> true:
> {code:title=HiveStatement.closeClientOperation}
> void closeClientOperation() throws SQLException {
> [..]
> isQueryClosed = true;
> isExecuteStatementFailed = false;
> stmtHandle = null;
> }
> {code}
> The {{initFlags}} sets it to false:
> {code}
> private void initFlags() {
> isCancelled = false;
> isQueryClosed = false;
> isLogBeingGenerated = true;
> isExecuteStatementFailed = false;
> isOperationComplete = false;
> }
> {code}
> If the {{getQueryLog}} is called after the {{closeClientOperation}}, but
> before the {{initFlags}}, then we will have a following warning if verbose
> mode is set to true in BeeLine:
> {code}
> Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method
> getQueryLog() failed. The statement has been closed or cancelled.
> (state=,code=0)
> {code}
> This caused this fail:
> https://builds.apache.org/job/PreCommit-HIVE-Build/4691/testReport/org.apache.hadoop.hive.cli/TestBeeLineDriver/testCliDriver_smb_mapjoin_11_/
> {code}
> Error Message
> Client result comparison failed with error code = 1 while executing
> fname=smb_mapjoin_11
> 16a17
> > Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method
> > getQueryLog() failed. The statement has been closed or cancelled.
> > (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16451) Race condition between HiveStatement.getQueryLog and HiveStatement.runAsyncOnServer
[
https://issues.apache.org/jira/browse/HIVE-16451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-16451:
--
Attachment: HIVE-16451.patch
A patch to solve the current problem.
Create a new method {{closeStatementIfNeeded}}, which only closes the
statement, but does not touch the flags.
Using this method should solve the problem, since the flags are not
flip-flopping any more.
> Race condition between HiveStatement.getQueryLog and
> HiveStatement.runAsyncOnServer
> ---
>
> Key: HIVE-16451
> URL: https://issues.apache.org/jira/browse/HIVE-16451
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16451.patch
>
>
> During the BeeLineDriver testing I have met the following race condition:
> - Run the query asynchronously through BeeLine
> - Querying the logs in the BeeLine
> In the following code:
> {code:title=HiveStatement.runAsyncOnServer}
> private void runAsyncOnServer(String sql) throws SQLException {
> checkConnection("execute");
> closeClientOperation();
> initFlags();
> [..]
> }
> {code}
> {code:title=HiveStatement.getQueryLog}
> public List getQueryLog(boolean incremental, int fetchSize)
> throws SQLException, ClosedOrCancelledStatementException {
> [..]
> try {
> if (stmtHandle != null) {
> [..]
> } else {
> if (isQueryClosed) {
> throw new ClosedOrCancelledStatementException("Method getQueryLog()
> failed. The " +
> "statement has been closed or cancelled.");
> } else {
> return logs;
> }
> }
> } catch (SQLException e) {
> [..]
> }
> [..]
> }
> {code}
> The runAsyncOnServer {{closeClientOperation}} sets {{isQueryClosed}} flag to
> true:
> {code:title=HiveStatement.closeClientOperation}
> void closeClientOperation() throws SQLException {
> [..]
> isQueryClosed = true;
> isExecuteStatementFailed = false;
> stmtHandle = null;
> }
> {code}
> The {{initFlags}} sets it to false:
> {code}
> private void initFlags() {
> isCancelled = false;
> isQueryClosed = false;
> isLogBeingGenerated = true;
> isExecuteStatementFailed = false;
> isOperationComplete = false;
> }
> {code}
> If the {{getQueryLog}} is called after the {{closeClientOperation}}, but
> before the {{initFlags}}, then we will have a following warning if verbose
> mode is set to true in BeeLine:
> {code}
> Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method
> getQueryLog() failed. The statement has been closed or cancelled.
> (state=,code=0)
> {code}
> This caused this fail:
> https://builds.apache.org/job/PreCommit-HIVE-Build/4691/testReport/org.apache.hadoop.hive.cli/TestBeeLineDriver/testCliDriver_smb_mapjoin_11_/
> {code}
> Error Message
> Client result comparison failed with error code = 1 while executing
> fname=smb_mapjoin_11
> 16a17
> > Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method
> > getQueryLog() failed. The statement has been closed or cancelled.
> > (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16449) BeeLineDriver should handle query result sorting
[ https://issues.apache.org/jira/browse/HIVE-16449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-16449: -- Attachment: HIVE-16449.03.patch Added null check, which caused test failures. > BeeLineDriver should handle query result sorting > > > Key: HIVE-16449 > URL: https://issues.apache.org/jira/browse/HIVE-16449 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 3.0.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16449.02.patch, HIVE-16449.03.patch, > HIVE-16449.patch > > > The CLI driver supports the following features: > -- SORT_QUERY_RESULTS > -- HASH_QUERY_RESULTS > -- SORT_AND_HASH_QUERY_RESULTS > BeeLineDriver should find a way to support these -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16450) Some metastore operations are not retried even with desired underlining exceptions
[
https://issues.apache.org/jira/browse/HIVE-16450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-16450:
Status: Patch Available (was: Open)
> Some metastore operations are not retried even with desired underlining
> exceptions
> --
>
> Key: HIVE-16450
> URL: https://issues.apache.org/jira/browse/HIVE-16450
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-16450.1.patch
>
>
> In RetryingHMSHandler class, we are expecting the operations should retry
> when the cause of MetaException is JDOException or NucleusException.
> {noformat}
> if (e.getCause() instanceof MetaException && e.getCause().getCause()
> != null) {
> if (e.getCause().getCause() instanceof javax.jdo.JDOException ||
> e.getCause().getCause() instanceof NucleusException) {
> // The JDOException or the Nucleus Exception may be wrapped
> further in a MetaException
> caughtException = e.getCause().getCause();
>}
> {noformat}
> While in ObjectStore, many places we are only throwing new MetaException(msg)
> without the cause, so we are missing retrying for some cases. e.g., with the
> following JDOException, we should retry but it's ignored.
> {noformat}
> 2017-04-04 17:28:21,602 ERROR metastore.ObjectStore
> (ObjectStore.java:getMTableColumnStatistics(6555)) - Error retrieving
> statistics via jdo
> javax.jdo.JDOException: Exception thrown when executing query
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
> at
> org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:321)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTableColumnStatistics(ObjectStore.java:6546)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$1200(ObjectStore.java:171)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$9.getJdoResult(ObjectStore.java:6606)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$9.getJdoResult(ObjectStore.java:6595)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2633)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatisticsInternal(ObjectStore.java:6594)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatistics(ObjectStore.java:6588)
> at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:103)
> at com.sun.proxy.$Proxy0.getTableColumnStatistics(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTableUpdateTableColumnStats(HiveAlterHandler.java:787)
> at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTable(HiveAlterHandler.java:247)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:3809)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_with_environment_context(HiveMetaStore.java:3779)
> at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
> at com.sun.proxy.$Proxy3.alter_table_with_environment_context(Unknown
> Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:9617)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:9601)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
> at
>
[jira] [Commented] (HIVE-16450) Some metastore operations are not retried even with desired underlining exceptions
[
https://issues.apache.org/jira/browse/HIVE-16450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969110#comment-15969110
]
Aihua Xu commented on HIVE-16450:
-
patch-1: set the exception cause to MetaException so RetryingHMSHandler will
retry if it's JDOException or NucleusException. Since no direct JDOException
will be thrown from {{method.invoke(baseHandler, args)}}, removed such catch.
> Some metastore operations are not retried even with desired underlining
> exceptions
> --
>
> Key: HIVE-16450
> URL: https://issues.apache.org/jira/browse/HIVE-16450
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-16450.1.patch
>
>
> In RetryingHMSHandler class, we are expecting the operations should retry
> when the cause of MetaException is JDOException or NucleusException.
> {noformat}
> if (e.getCause() instanceof MetaException && e.getCause().getCause()
> != null) {
> if (e.getCause().getCause() instanceof javax.jdo.JDOException ||
> e.getCause().getCause() instanceof NucleusException) {
> // The JDOException or the Nucleus Exception may be wrapped
> further in a MetaException
> caughtException = e.getCause().getCause();
>}
> {noformat}
> While in ObjectStore, many places we are only throwing new MetaException(msg)
> without the cause, so we are missing retrying for some cases. e.g., with the
> following JDOException, we should retry but it's ignored.
> {noformat}
> 2017-04-04 17:28:21,602 ERROR metastore.ObjectStore
> (ObjectStore.java:getMTableColumnStatistics(6555)) - Error retrieving
> statistics via jdo
> javax.jdo.JDOException: Exception thrown when executing query
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
> at
> org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:321)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTableColumnStatistics(ObjectStore.java:6546)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$1200(ObjectStore.java:171)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$9.getJdoResult(ObjectStore.java:6606)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$9.getJdoResult(ObjectStore.java:6595)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2633)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatisticsInternal(ObjectStore.java:6594)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatistics(ObjectStore.java:6588)
> at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:103)
> at com.sun.proxy.$Proxy0.getTableColumnStatistics(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTableUpdateTableColumnStats(HiveAlterHandler.java:787)
> at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTable(HiveAlterHandler.java:247)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:3809)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_with_environment_context(HiveMetaStore.java:3779)
> at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
> at com.sun.proxy.$Proxy3.alter_table_with_environment_context(Unknown
> Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:9617)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:9601)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> at
[jira] [Updated] (HIVE-16450) Some metastore operations are not retried even with desired underlining exceptions
[
https://issues.apache.org/jira/browse/HIVE-16450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-16450:
Attachment: HIVE-16450.1.patch
> Some metastore operations are not retried even with desired underlining
> exceptions
> --
>
> Key: HIVE-16450
> URL: https://issues.apache.org/jira/browse/HIVE-16450
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-16450.1.patch
>
>
> In RetryingHMSHandler class, we are expecting the operations should retry
> when the cause of MetaException is JDOException or NucleusException.
> {noformat}
> if (e.getCause() instanceof MetaException && e.getCause().getCause()
> != null) {
> if (e.getCause().getCause() instanceof javax.jdo.JDOException ||
> e.getCause().getCause() instanceof NucleusException) {
> // The JDOException or the Nucleus Exception may be wrapped
> further in a MetaException
> caughtException = e.getCause().getCause();
>}
> {noformat}
> While in ObjectStore, many places we are only throwing new MetaException(msg)
> without the cause, so we are missing retrying for some cases. e.g., with the
> following JDOException, we should retry but it's ignored.
> {noformat}
> 2017-04-04 17:28:21,602 ERROR metastore.ObjectStore
> (ObjectStore.java:getMTableColumnStatistics(6555)) - Error retrieving
> statistics via jdo
> javax.jdo.JDOException: Exception thrown when executing query
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
> at
> org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:321)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTableColumnStatistics(ObjectStore.java:6546)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$1200(ObjectStore.java:171)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$9.getJdoResult(ObjectStore.java:6606)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$9.getJdoResult(ObjectStore.java:6595)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2633)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatisticsInternal(ObjectStore.java:6594)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatistics(ObjectStore.java:6588)
> at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:103)
> at com.sun.proxy.$Proxy0.getTableColumnStatistics(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTableUpdateTableColumnStats(HiveAlterHandler.java:787)
> at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTable(HiveAlterHandler.java:247)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:3809)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_with_environment_context(HiveMetaStore.java:3779)
> at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
> at com.sun.proxy.$Proxy3.alter_table_with_environment_context(Unknown
> Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:9617)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:9601)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
> at
>
[jira] [Commented] (HIVE-16451) Race condition between HiveStatement.getQueryLog and HiveStatement.runAsyncOnServer
[
https://issues.apache.org/jira/browse/HIVE-16451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969081#comment-15969081
]
Peter Vary commented on HIVE-16451:
---
Confirmed by adding sleep between the two method, like this:
{code:title=HiveStatement.runAsyncOnServer}
private void runAsyncOnServer(String sql) throws SQLException {
checkConnection("execute");
closeClientOperation();
try {
Thread.currentThread().sleep(8000L);
} catch (Exception e) {
e.printStackTrace();
}
initFlags();
[..]
}
{code}
This reproduced the errors.
> Race condition between HiveStatement.getQueryLog and
> HiveStatement.runAsyncOnServer
> ---
>
> Key: HIVE-16451
> URL: https://issues.apache.org/jira/browse/HIVE-16451
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
>
> During the BeeLineDriver testing I have met the following race condition:
> - Run the query asynchronously through BeeLine
> - Querying the logs in the BeeLine
> In the following code:
> {code:title=HiveStatement.runAsyncOnServer}
> private void runAsyncOnServer(String sql) throws SQLException {
> checkConnection("execute");
> closeClientOperation();
> initFlags();
> [..]
> }
> {code}
> {code:title=HiveStatement.getQueryLog}
> public List getQueryLog(boolean incremental, int fetchSize)
> throws SQLException, ClosedOrCancelledStatementException {
> [..]
> try {
> if (stmtHandle != null) {
> [..]
> } else {
> if (isQueryClosed) {
> throw new ClosedOrCancelledStatementException("Method getQueryLog()
> failed. The " +
> "statement has been closed or cancelled.");
> } else {
> return logs;
> }
> }
> } catch (SQLException e) {
> [..]
> }
> [..]
> }
> {code}
> The runAsyncOnServer {{closeClientOperation}} sets {{isQueryClosed}} flag to
> true:
> {code:title=HiveStatement.closeClientOperation}
> void closeClientOperation() throws SQLException {
> [..]
> isQueryClosed = true;
> isExecuteStatementFailed = false;
> stmtHandle = null;
> }
> {code}
> The {{initFlags}} sets it to false:
> {code}
> private void initFlags() {
> isCancelled = false;
> isQueryClosed = false;
> isLogBeingGenerated = true;
> isExecuteStatementFailed = false;
> isOperationComplete = false;
> }
> {code}
> If the {{getQueryLog}} is called after the {{closeClientOperation}}, but
> before the {{initFlags}}, then we will have a following warning if verbose
> mode is set to true in BeeLine:
> {code}
> Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method
> getQueryLog() failed. The statement has been closed or cancelled.
> (state=,code=0)
> {code}
> This caused this fail:
> https://builds.apache.org/job/PreCommit-HIVE-Build/4691/testReport/org.apache.hadoop.hive.cli/TestBeeLineDriver/testCliDriver_smb_mapjoin_11_/
> {code}
> Error Message
> Client result comparison failed with error code = 1 while executing
> fname=smb_mapjoin_11
> 16a17
> > Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method
> > getQueryLog() failed. The statement has been closed or cancelled.
> > (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16146) If possible find a better way to filter the TestBeeLineDriver output
[ https://issues.apache.org/jira/browse/HIVE-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969075#comment-15969075 ] Peter Vary commented on HIVE-16146: --- Checked the TestBeeLineDriver.testCliDriver[smb_mapjoin_11] error. It is caused by this: - HIVE-16451- Race condition between HiveStatement.getQueryLog and HiveStatement.runAsyncOnServer So the errors are unrelated > If possible find a better way to filter the TestBeeLineDriver output > > > Key: HIVE-16146 > URL: https://issues.apache.org/jira/browse/HIVE-16146 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16146.02.patch, HIVE-16146.03.patch, > HIVE-16146.04.patch, HIVE-16146.05.patch, HIVE-16146.06.patch, > HIVE-16146.patch > > > Currently we apply a blacklist to filter the output of the BeeLine Qtest runs. > It might be a good idea to go thorough of the possibilities and find a better > way, if possible. > I think our main goal could be for the TestBeeLineDriver test output to match > the TestCliDriver output of the came query file. Or if it is not possible, > then at least a similar one > CC: [~vihangk1] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16451) Race condition between HiveStatement.getQueryLog and HiveStatement.runAsyncOnServer
[
https://issues.apache.org/jira/browse/HIVE-16451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary reassigned HIVE-16451:
-
> Race condition between HiveStatement.getQueryLog and
> HiveStatement.runAsyncOnServer
> ---
>
> Key: HIVE-16451
> URL: https://issues.apache.org/jira/browse/HIVE-16451
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
>
> During the BeeLineDriver testing I have met the following race condition:
> - Run the query asynchronously through BeeLine
> - Querying the logs in the BeeLine
> In the following code:
> {code:title=HiveStatement.runAsyncOnServer}
> private void runAsyncOnServer(String sql) throws SQLException {
> checkConnection("execute");
> closeClientOperation();
> initFlags();
> [..]
> }
> {code}
> {code:title=HiveStatement.getQueryLog}
> public List getQueryLog(boolean incremental, int fetchSize)
> throws SQLException, ClosedOrCancelledStatementException {
> [..]
> try {
> if (stmtHandle != null) {
> [..]
> } else {
> if (isQueryClosed) {
> throw new ClosedOrCancelledStatementException("Method getQueryLog()
> failed. The " +
> "statement has been closed or cancelled.");
> } else {
> return logs;
> }
> }
> } catch (SQLException e) {
> [..]
> }
> [..]
> }
> {code}
> The runAsyncOnServer {{closeClientOperation}} sets {{isQueryClosed}} flag to
> true:
> {code:title=HiveStatement.closeClientOperation}
> void closeClientOperation() throws SQLException {
> [..]
> isQueryClosed = true;
> isExecuteStatementFailed = false;
> stmtHandle = null;
> }
> {code}
> The {{initFlags}} sets it to false:
> {code}
> private void initFlags() {
> isCancelled = false;
> isQueryClosed = false;
> isLogBeingGenerated = true;
> isExecuteStatementFailed = false;
> isOperationComplete = false;
> }
> {code}
> If the {{getQueryLog}} is called after the {{closeClientOperation}}, but
> before the {{initFlags}}, then we will have a following warning if verbose
> mode is set to true in BeeLine:
> {code}
> Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method
> getQueryLog() failed. The statement has been closed or cancelled.
> (state=,code=0)
> {code}
> This caused this fail:
> https://builds.apache.org/job/PreCommit-HIVE-Build/4691/testReport/org.apache.hadoop.hive.cli/TestBeeLineDriver/testCliDriver_smb_mapjoin_11_/
> {code}
> Error Message
> Client result comparison failed with error code = 1 while executing
> fname=smb_mapjoin_11
> 16a17
> > Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method
> > getQueryLog() failed. The statement has been closed or cancelled.
> > (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16430) Add log to show the cancelled query id when cancelOperation is called.
[ https://issues.apache.org/jira/browse/HIVE-16430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-16430: --- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to master. Thanks, Zhihai! > Add log to show the cancelled query id when cancelOperation is called. > -- > > Key: HIVE-16430 > URL: https://issues.apache.org/jira/browse/HIVE-16430 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Trivial > Fix For: 3.0.0 > > Attachments: HIVE-16430.000.patch, HIVE-16430.001.patch > > > Add log to show the cancelled query id when cancelOperation is called. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16450) Some metastore operations are not retried even with desired underlining exceptions
[
https://issues.apache.org/jira/browse/HIVE-16450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu reassigned HIVE-16450:
---
> Some metastore operations are not retried even with desired underlining
> exceptions
> --
>
> Key: HIVE-16450
> URL: https://issues.apache.org/jira/browse/HIVE-16450
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>
> In RetryingHMSHandler class, we are expecting the operations should retry
> when the cause of MetaException is JDOException or NucleusException.
> {noformat}
> if (e.getCause() instanceof MetaException && e.getCause().getCause()
> != null) {
> if (e.getCause().getCause() instanceof javax.jdo.JDOException ||
> e.getCause().getCause() instanceof NucleusException) {
> // The JDOException or the Nucleus Exception may be wrapped
> further in a MetaException
> caughtException = e.getCause().getCause();
>}
> {noformat}
> While in ObjectStore, many places we are only throwing new MetaException(msg)
> without the cause, so we are missing retrying for some cases. e.g., with the
> following JDOException, we should retry but it's ignored.
> {noformat}
> 2017-04-04 17:28:21,602 ERROR metastore.ObjectStore
> (ObjectStore.java:getMTableColumnStatistics(6555)) - Error retrieving
> statistics via jdo
> javax.jdo.JDOException: Exception thrown when executing query
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
> at
> org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:321)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTableColumnStatistics(ObjectStore.java:6546)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$1200(ObjectStore.java:171)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$9.getJdoResult(ObjectStore.java:6606)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$9.getJdoResult(ObjectStore.java:6595)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2633)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatisticsInternal(ObjectStore.java:6594)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatistics(ObjectStore.java:6588)
> at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:103)
> at com.sun.proxy.$Proxy0.getTableColumnStatistics(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTableUpdateTableColumnStats(HiveAlterHandler.java:787)
> at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTable(HiveAlterHandler.java:247)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:3809)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_with_environment_context(HiveMetaStore.java:3779)
> at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
> at com.sun.proxy.$Proxy3.alter_table_with_environment_context(Unknown
> Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:9617)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:9601)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
> at
>
[jira] [Commented] (HIVE-16449) BeeLineDriver should handle query result sorting
[
https://issues.apache.org/jira/browse/HIVE-16449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969062#comment-15969062
]
Hive QA commented on HIVE-16449:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863459/HIVE-16449.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 57 failed/errored test(s), 10574 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
org.apache.hive.beeline.TestBeeLineWithArgs.testBackslashInLiteral (batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testBeelineHiveConfVariable
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testBeelineHiveVariable
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testBeelineMultiHiveVariable
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testBeelineWithForce (batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testCSV2OutputWithDoubleQuotes
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testCSVOutput (batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testCSVOutputWithDoubleQuotes
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testConnectionUrlWithSemiColon
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testConnectionWithURLParams
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testDSVOutput (batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testDSVOutputWithDoubleQuotes
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testEmbeddedBeelineConnection
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testEscapeSemiColonInEnclosedQuery
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testEscapeSemiColonInQueries
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testHiveVarSubstitution
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testLastLineCmdInScriptFile
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testMultiCommandsInOneEnclosedQuery
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testMultiCommandsInOneline
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testNullDefault (batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testNullEmpty (batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testNullEmptyCmdArg (batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testNullNonEmpty (batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testOneCommandInMultiLines
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testPositiveScriptFile (batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testSelectQueryWithNonEscapedSemiColon
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testTSV2Output (batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testTSV2OutputWithDoubleQuotes
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testTSVOutput (batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testTSVOutputWithDoubleQuotes
(batchId=217)
org.apache.hive.beeline.TestBeelinePasswordOption.testNoPasswordPrompt
(batchId=217)
org.apache.hive.beeline.TestBeelinePasswordOption.testNoPasswordPrompt2
(batchId=217)
org.apache.hive.beeline.TestBeelinePasswordOption.testPromptPassOptionLastWithBeelineOpts
(batchId=217)
org.apache.hive.beeline.TestBeelinePasswordOption.testPromptPasswordOptionWithBeelineOpts
(batchId=217)
org.apache.hive.beeline.TestBeelinePasswordOption.testPromptPasswordOptionWithOtherOptions
(batchId=217)
org.apache.hive.beeline.TestBeelinePasswordOption.testPromptPasswordVerifyBeelineOpts
(batchId=217)
org.apache.hive.beeline.TestBeelinePasswordOption.testPromptPasswordWithHiveConf
(batchId=217)
org.apache.hive.beeline.cli.TestHiveCli.testCmd (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testDatabaseOptions (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testSetHeaderValue (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testSetPromptValue (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testSourceCmd (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testSourceCmd2 (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testSourceCmd3 (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testSqlFromCmd (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testSqlFromCmdWithDBName (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testVariables (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testVariablesForSource (batchId=173)
org.apache.hive.beeline.hs2connection.TestBeelineConnectionUsingHiveSite.testBeelineConnectionHttp
(batchId=217)
org.apache.hive.beeline.hs2connection.TestBeelineConnectionUsingHiveSite.testBeelineConnectionNoAuth
(batchId=217)
[jira] [Commented] (HIVE-16449) BeeLineDriver should handle query result sorting
[
https://issues.apache.org/jira/browse/HIVE-16449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969043#comment-15969043
]
Hive QA commented on HIVE-16449:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863459/HIVE-16449.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 57 failed/errored test(s), 10574 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.beeline.TestBeeLineWithArgs.testBackslashInLiteral (batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testBeelineHiveConfVariable
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testBeelineHiveVariable
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testBeelineMultiHiveVariable
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testBeelineWithForce (batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testCSV2OutputWithDoubleQuotes
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testCSVOutput (batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testCSVOutputWithDoubleQuotes
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testConnectionUrlWithSemiColon
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testConnectionWithURLParams
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testDSVOutput (batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testDSVOutputWithDoubleQuotes
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testEmbeddedBeelineConnection
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testEscapeSemiColonInEnclosedQuery
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testEscapeSemiColonInQueries
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testHiveVarSubstitution
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testLastLineCmdInScriptFile
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testMultiCommandsInOneEnclosedQuery
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testMultiCommandsInOneline
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testNullDefault (batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testNullEmpty (batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testNullEmptyCmdArg (batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testNullNonEmpty (batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testOneCommandInMultiLines
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testPositiveScriptFile (batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testSelectQueryWithNonEscapedSemiColon
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testTSV2Output (batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testTSV2OutputWithDoubleQuotes
(batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testTSVOutput (batchId=217)
org.apache.hive.beeline.TestBeeLineWithArgs.testTSVOutputWithDoubleQuotes
(batchId=217)
org.apache.hive.beeline.TestBeelinePasswordOption.testNoPasswordPrompt
(batchId=217)
org.apache.hive.beeline.TestBeelinePasswordOption.testNoPasswordPrompt2
(batchId=217)
org.apache.hive.beeline.TestBeelinePasswordOption.testPromptPassOptionLastWithBeelineOpts
(batchId=217)
org.apache.hive.beeline.TestBeelinePasswordOption.testPromptPasswordOptionWithBeelineOpts
(batchId=217)
org.apache.hive.beeline.TestBeelinePasswordOption.testPromptPasswordOptionWithOtherOptions
(batchId=217)
org.apache.hive.beeline.TestBeelinePasswordOption.testPromptPasswordVerifyBeelineOpts
(batchId=217)
org.apache.hive.beeline.TestBeelinePasswordOption.testPromptPasswordWithHiveConf
(batchId=217)
org.apache.hive.beeline.cli.TestHiveCli.testCmd (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testDatabaseOptions (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testSetHeaderValue (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testSetPromptValue (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testSourceCmd (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testSourceCmd2 (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testSourceCmd3 (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testSqlFromCmd (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testSqlFromCmdWithDBName (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testVariables (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testVariablesForSource (batchId=173)
org.apache.hive.beeline.hs2connection.TestBeelineConnectionUsingHiveSite.testBeelineConnectionHttp
(batchId=217)
org.apache.hive.beeline.hs2connection.TestBeelineConnectionUsingHiveSite.testBeelineConnectionNoAuth
(batchId=217)
org.apache.hive.beeline.hs2connection.TestBeelineConnectionUsingHiveSite.testBeelineConnectionSSL
(batchId=217)
[jira] [Commented] (HIVE-15302) Relax the requirement that HoS needs Spark built w/o Hive
[
https://issues.apache.org/jira/browse/HIVE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969031#comment-15969031
]
Rui Li commented on HIVE-15302:
---
Following is the identified min set of required jars:
{noformat}
chill-java-0.8.0.jar kryo-shaded-3.0.3.jar
spark-network-common_2.11-2.0.0.jar
chill_2.11-0.8.0.jar mesos-0.21.1-shaded-protobuf.jar
spark-network-shuffle_2.11-2.0.0.jar
jackson-module-paranamer-2.6.5.jar minlog-1.3.0.jar
spark-unsafe_2.11-2.0.0.jar
jackson-module-scala_2.11-2.6.5.jar scala-library-2.11.8.jar
spark-yarn_2.11-2.0.0.jar
jersey-container-servlet-core-2.22.2.jar scala-xml_2.11-1.0.2.jar
xbean-asm5-shaded-4.4.jar
jersey-server-2.22.2.jar spark-core_2.11-2.0.0.jar
json4s-ast_2.11-3.2.11.jar spark-launcher_2.11-2.0.0.jar
{noformat}
I'll run some more thorough tests with it. Meanwhile, I'd appreciate it if
anyone can help verify it.
> Relax the requirement that HoS needs Spark built w/o Hive
> -
>
> Key: HIVE-15302
> URL: https://issues.apache.org/jira/browse/HIVE-15302
> Project: Hive
> Issue Type: Improvement
>Reporter: Rui Li
>Assignee: Rui Li
>
> This requirement becomes more and more unacceptable as SparkSQL becomes
> widely adopted. Let's use this JIRA to find out how we can relax the
> limitation.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16146) If possible find a better way to filter the TestBeeLineDriver output
[ https://issues.apache.org/jira/browse/HIVE-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968956#comment-15968956 ] Peter Vary commented on HIVE-16146: --- Not related failures: - HIVE-15535 - Flaky test : TestHS2HttpServer.testContextRootUrlRewrite - HIVE-15776 - Flaky test: TestMiniLlapLocalCliDriver vector_if_expr - HIVE-15289 - Flaky test: TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (setup) Maybe related: - TestBeeLineDriver.testCliDriver[smb_mapjoin_11] Will check the last one, but I do not think it is related to this test. For the first glance it seems to me a more general race condition instead. > If possible find a better way to filter the TestBeeLineDriver output > > > Key: HIVE-16146 > URL: https://issues.apache.org/jira/browse/HIVE-16146 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16146.02.patch, HIVE-16146.03.patch, > HIVE-16146.04.patch, HIVE-16146.05.patch, HIVE-16146.06.patch, > HIVE-16146.patch > > > Currently we apply a blacklist to filter the output of the BeeLine Qtest runs. > It might be a good idea to go thorough of the possibilities and find a better > way, if possible. > I think our main goal could be for the TestBeeLineDriver test output to match > the TestCliDriver output of the came query file. Or if it is not possible, > then at least a similar one > CC: [~vihangk1] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16418) Allow HiveKey to skip some bytes for comparison
[
https://issues.apache.org/jira/browse/HIVE-16418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968955#comment-15968955
]
Rui Li commented on HIVE-16418:
---
[~xuefuz] has talked about this with me offline. Let me use an example for the
discussion. Suppose the system TZ is GMT+5. Now a user stores a TimestampTZ of
'2017-04-14 18:00:00 GMT+8'. We have the following choices to store it:
# Store as '2017-04-14 18:00:00 GMT+8'. This is my original plan, which I think
is closest to user's expectation - you store some TimestampTZ and when you
select, you get the same data displayed. It fixes both {{to_utc_timestamp}} and
{{from_utc_timestamp}}. But this way, we need to store the TZ part and thus
requires all the complexity.
# Store as '2017-04-14 10:00:00 GMT'. This means all TimestampTZ values will
display using the GMT timezone. It's much simpler because we don't have to
store the TZ, and we can reuse most of the code like TimestampWritable.
Shortcoming is we discard the TZ info in user's input. And more importantly,
it's difficult to fix the {{from_utc_timestamp}} UDF. This UDF converts a
timestamp in UTC to a user-specified timezone. The return type of course should
be TimestampTZ. But if all TimestampTZ display in UTC, the UDF effectively
becomes useless. So in this way, I guess we have to leave
{{from_utc_timestamp}} as is.
# Store as '2017-04-14 15:00:00 GMT+5'. It's similar to #2 but uses the system
TZ.
If #1 is unacceptable due to the complexity, I prefer #2. #3 seems to introduce
unnecessary ambiguity - if you run same query on clusters in different TZ,
you'll get different results. What do you guys think?
> Allow HiveKey to skip some bytes for comparison
> ---
>
> Key: HIVE-16418
> URL: https://issues.apache.org/jira/browse/HIVE-16418
> Project: Hive
> Issue Type: New Feature
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-16418.1.patch
>
>
> The feature is required when we have to serialize some fields and prevent
> them from being used in comparison, e.g. HIVE-14412.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16427) Fix multi-insert query and write qtests
[
https://issues.apache.org/jira/browse/HIVE-16427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968951#comment-15968951
]
Yongzhi Chen commented on HIVE-16427:
-
The failure is not related.
> Fix multi-insert query and write qtests
> ---
>
> Key: HIVE-16427
> URL: https://issues.apache.org/jira/browse/HIVE-16427
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Reporter: Thomas Poepping
>Assignee: Yongzhi Chen
> Attachments: HIVE-16427.1.patch
>
>
> On HIVE-16415, it was found that the bug reported to be fixed in HIVE-14519
> was not actually fixed.
> This task is to find the problem, fix it, and add qtests to verify no future
> regression.
> Specifically, the following query does not produce correct answers:
> {code}
> From (select * from src) a
> insert overwrite directory '/tmp/emp/dir1/'
> select key, value
> insert overwrite directory '/tmp/emp/dir2/'
> select 'header'
> limit 0
> insert overwrite directory '/tmp/emp/dir3/'
> select key, value
> where key = 100;
> {code}
> This gives incorrect result in master. All dirs end up with 0 rows instead of
> just dir2.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16426) Query cancel: improve the way to handle files
[
https://issues.apache.org/jira/browse/HIVE-16426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968949#comment-15968949
]
Yongzhi Chen commented on HIVE-16426:
-
The failure is not related.
> Query cancel: improve the way to handle files
> -
>
> Key: HIVE-16426
> URL: https://issues.apache.org/jira/browse/HIVE-16426
> Project: Hive
> Issue Type: Improvement
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-16426.1.patch
>
>
> 1. Add data structure support to make it is easy to check query cancel status.
> 2. Handle query cancel more gracefully. Remove possible file leaks caused by
> query cancel as shown in following stack:
> {noformat}
> 2017-04-11 09:57:30,727 WARN org.apache.hadoop.hive.ql.exec.Utilities:
> [HiveServer2-Background-Pool: Thread-149]: Failed to clean-up tmp directories.
> java.io.InterruptedIOException: Call interrupted
> at org.apache.hadoop.ipc.Client.call(Client.java:1496)
> at org.apache.hadoop.ipc.Client.call(Client.java:1439)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
> at com.sun.proxy.$Proxy20.delete(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:535)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
> at com.sun.proxy.$Proxy21.delete(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2059)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:675)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:671)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:671)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.clearWork(Utilities.java:277)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:463)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:142)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1978)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1691)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1423)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1207)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1202)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:238)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:303)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:316)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> 3. Add checkpoints to related file operations to improve response time for
> query cancelling.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16448) Vectorization: Vectorized order_null.q fails with deserialize EOF exception below TEZ ReduceRecordSource.processVectorGroup
[
https://issues.apache.org/jira/browse/HIVE-16448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968942#comment-15968942
]
Hive QA commented on HIVE-16448:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863454/HIVE-16448.01.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 45 failed/errored test(s), 10561 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[partition_wise_fileformat6]
(batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_order_null]
(batchId=27)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_id3]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_leftsemi_mapjoin]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_outer_join1]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_outer_join2]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join1]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join2]
(batchId=165)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
(batchId=104)
org.apache.hadoop.hive.ql.exec.vector.TestVectorSerDeRow.testVectorBinarySortableDeserializeRow
(batchId=268)
org.apache.hadoop.hive.ql.exec.vector.TestVectorSerDeRow.testVectorBinarySortableSerializeRow
(batchId=268)
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.TestVectorMapJoinFastLongHashMap.testExpand
(batchId=269)
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.TestVectorMapJoinFastLongHashMap.testFullMap
(batchId=269)
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.TestVectorMapJoinFastLongHashMap.testGetNonExistent
(batchId=269)
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.TestVectorMapJoinFastLongHashMap.testLargeAndExpand
(batchId=269)
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.TestVectorMapJoinFastLongHashMap.testMultipleKeysMultipleValue
(batchId=269)
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.TestVectorMapJoinFastLongHashMap.testMultipleKeysSingleValue
(batchId=269)
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.TestVectorMapJoinFastLongHashMap.testOneKey
(batchId=269)
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.TestVectorMapJoinFastLongHashMultiSet.testExpand
(batchId=269)
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.TestVectorMapJoinFastLongHashMultiSet.testFullMap
(batchId=269)
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.TestVectorMapJoinFastLongHashMultiSet.testGetNonExistent
(batchId=269)
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.TestVectorMapJoinFastLongHashMultiSet.testLargeAndExpand
(batchId=269)
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.TestVectorMapJoinFastLongHashMultiSet.testMultipleKeysMultipleValue
(batchId=269)
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.TestVectorMapJoinFastLongHashMultiSet.testMultipleKeysSingleValue
(batchId=269)
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.TestVectorMapJoinFastLongHashMultiSet.testOneKey
(batchId=269)
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.TestVectorMapJoinFastLongHashSet.testExpand
(batchId=269)
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.TestVectorMapJoinFastLongHashSet.testFullMap
(batchId=269)
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.TestVectorMapJoinFastLongHashSet.testGetNonExistent
(batchId=269)
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.TestVectorMapJoinFastLongHashSet.testLargeAndExpand
(batchId=269)
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.TestVectorMapJoinFastLongHashSet.testMultipleKeysMultipleValue
(batchId=269)
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.TestVectorMapJoinFastLongHashSet.testMultipleKeysSingleValue
(batchId=269)
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.TestVectorMapJoinFastLongHashSet.testOneKey
(batchId=269)
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.TestVectorMapJoinFastRowHashMap.testBigIntRows
(batchId=268)
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.TestVectorMapJoinFastRowHashMap.testBigIntRowsClipped
(batchId=268)
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.TestVectorMapJoinFastRowHashMap.testBigIntRowsClippedExact
(batchId=268)
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.TestVectorMapJoinFastRowHashMap.testBigIntRowsExact
(batchId=268)
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.TestVectorMapJoinFastRowHashMap.testIntRows
(batchId=268)
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.TestVectorMapJoinFastRowHashMap.testIntRowsClipped
[jira] [Updated] (HIVE-16449) BeeLineDriver should handle query result sorting
[ https://issues.apache.org/jira/browse/HIVE-16449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-16449: -- Description: The CLI driver supports the following features: -- SORT_QUERY_RESULTS -- HASH_QUERY_RESULTS -- SORT_AND_HASH_QUERY_RESULTS BeeLineDriver should find a way to support these was: The CLI driver supports the following features: -- SORT_BEFORE_DIFF -- SORT_QUERY_RESULTS -- HASH_QUERY_RESULTS -- SORT_AND_HASH_QUERY_RESULTS BeeLineDriver should find a way to support these > BeeLineDriver should handle query result sorting > > > Key: HIVE-16449 > URL: https://issues.apache.org/jira/browse/HIVE-16449 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 3.0.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16449.02.patch, HIVE-16449.patch > > > The CLI driver supports the following features: > -- SORT_QUERY_RESULTS > -- HASH_QUERY_RESULTS > -- SORT_AND_HASH_QUERY_RESULTS > BeeLineDriver should find a way to support these -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16449) BeeLineDriver should handle query result sorting
[ https://issues.apache.org/jira/browse/HIVE-16449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-16449: -- Attachment: HIVE-16449.02.patch Removed – SORT_BEFORE_DIFF, since it is implemented differently > BeeLineDriver should handle query result sorting > > > Key: HIVE-16449 > URL: https://issues.apache.org/jira/browse/HIVE-16449 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 3.0.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16449.02.patch, HIVE-16449.patch > > > The CLI driver supports the following features: > -- SORT_BEFORE_DIFF > -- SORT_QUERY_RESULTS > -- HASH_QUERY_RESULTS > -- SORT_AND_HASH_QUERY_RESULTS > BeeLineDriver should find a way to support these -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16449) BeeLineDriver should handle query result sorting
[ https://issues.apache.org/jira/browse/HIVE-16449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-16449: -- Status: Patch Available (was: Open) Let's see the PreCommit test results > BeeLineDriver should handle query result sorting > > > Key: HIVE-16449 > URL: https://issues.apache.org/jira/browse/HIVE-16449 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 3.0.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16449.patch > > > The CLI driver supports the following features: > -- SORT_BEFORE_DIFF > -- SORT_QUERY_RESULTS > -- HASH_QUERY_RESULTS > -- SORT_AND_HASH_QUERY_RESULTS > BeeLineDriver should find a way to support these -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16449) BeeLineDriver should handle query result sorting
[ https://issues.apache.org/jira/browse/HIVE-16449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-16449: -- Attachment: HIVE-16449.patch Added the possibility to the OutputFile, to use the existing converters on the BeeLine results > BeeLineDriver should handle query result sorting > > > Key: HIVE-16449 > URL: https://issues.apache.org/jira/browse/HIVE-16449 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 3.0.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16449.patch > > > The CLI driver supports the following features: > -- SORT_BEFORE_DIFF > -- SORT_QUERY_RESULTS > -- HASH_QUERY_RESULTS > -- SORT_AND_HASH_QUERY_RESULTS > BeeLineDriver should find a way to support these -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16448) Vectorization: Vectorized order_null.q fails with deserialize EOF exception below TEZ ReduceRecordSource.processVectorGroup
[
https://issues.apache.org/jira/browse/HIVE-16448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968910#comment-15968910
]
Gopal V commented on HIVE-16448:
+1 tests pending.
Quick suggestion - is columnNullMarker always the opposite of
columnNotNullMarker?
The current patch makes it possible to pass ZERO, ZERO to both, which is
probably not a valid situation.
> Vectorization: Vectorized order_null.q fails with deserialize EOF exception
> below TEZ ReduceRecordSource.processVectorGroup
> ---
>
> Key: HIVE-16448
> URL: https://issues.apache.org/jira/browse/HIVE-16448
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-16448.01.patch
>
>
> For query "SELECT x.* FROM src_null x ORDER BY b asc, a asc nulls last" here
> is the stack trace:
> {code}
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
> attempt_1492136345968_0001_40_01_00_1:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:389)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:245)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:317)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> ... 15 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:421)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:380)
> ... 18 more
> Caused by: java.io.EOFException
> at
> org.apache.hadoop.hive.serde2.binarysortable.InputByteBuffer.read(InputByteBuffer.java:54)
> at
> org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableDeserializeRead.readNextField(BinarySortableDeserializeRead.java:205)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorDeserializeRow.deserialize(VectorDeserializeRow.java:751)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:417)
> ... 19 more
> {code}
--
This message was
[jira] [Updated] (HIVE-16448) Vectorization: Vectorized order_null.q fails with deserialize EOF exception below TEZ ReduceRecordSource.processVectorGroup
[
https://issues.apache.org/jira/browse/HIVE-16448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-16448:
Status: Patch Available (was: Open)
> Vectorization: Vectorized order_null.q fails with deserialize EOF exception
> below TEZ ReduceRecordSource.processVectorGroup
> ---
>
> Key: HIVE-16448
> URL: https://issues.apache.org/jira/browse/HIVE-16448
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-16448.01.patch
>
>
> For query "SELECT x.* FROM src_null x ORDER BY b asc, a asc nulls last" here
> is the stack trace:
> {code}
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
> attempt_1492136345968_0001_40_01_00_1:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:389)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:245)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:317)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> ... 15 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:421)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:380)
> ... 18 more
> Caused by: java.io.EOFException
> at
> org.apache.hadoop.hive.serde2.binarysortable.InputByteBuffer.read(InputByteBuffer.java:54)
> at
> org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableDeserializeRead.readNextField(BinarySortableDeserializeRead.java:205)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorDeserializeRow.deserialize(VectorDeserializeRow.java:751)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:417)
> ... 19 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16448) Vectorization: Vectorized order_null.q fails with deserialize EOF exception below TEZ ReduceRecordSource.processVectorGroup
[
https://issues.apache.org/jira/browse/HIVE-16448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-16448:
Attachment: HIVE-16448.01.patch
> Vectorization: Vectorized order_null.q fails with deserialize EOF exception
> below TEZ ReduceRecordSource.processVectorGroup
> ---
>
> Key: HIVE-16448
> URL: https://issues.apache.org/jira/browse/HIVE-16448
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-16448.01.patch
>
>
> For query "SELECT x.* FROM src_null x ORDER BY b asc, a asc nulls last" here
> is the stack trace:
> {code}
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
> attempt_1492136345968_0001_40_01_00_1:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:389)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:245)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:317)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> ... 15 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:421)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:380)
> ... 18 more
> Caused by: java.io.EOFException
> at
> org.apache.hadoop.hive.serde2.binarysortable.InputByteBuffer.read(InputByteBuffer.java:54)
> at
> org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableDeserializeRead.readNextField(BinarySortableDeserializeRead.java:205)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorDeserializeRow.deserialize(VectorDeserializeRow.java:751)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:417)
> ... 19 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16146) If possible find a better way to filter the TestBeeLineDriver output
[
https://issues.apache.org/jira/browse/HIVE-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968812#comment-15968812
]
Hive QA commented on HIVE-16146:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863431/HIVE-16146.06.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10561 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_11]
(batchId=234)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
(batchId=99)
org.apache.hive.service.server.TestHS2HttpServer.testContextRootUrlRewrite
(batchId=191)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4691/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4691/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4691/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863431 - PreCommit-HIVE-Build
> If possible find a better way to filter the TestBeeLineDriver output
>
>
> Key: HIVE-16146
> URL: https://issues.apache.org/jira/browse/HIVE-16146
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Affects Versions: 2.2.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16146.02.patch, HIVE-16146.03.patch,
> HIVE-16146.04.patch, HIVE-16146.05.patch, HIVE-16146.06.patch,
> HIVE-16146.patch
>
>
> Currently we apply a blacklist to filter the output of the BeeLine Qtest runs.
> It might be a good idea to go thorough of the possibilities and find a better
> way, if possible.
> I think our main goal could be for the TestBeeLineDriver test output to match
> the TestCliDriver output of the came query file. Or if it is not possible,
> then at least a similar one
> CC: [~vihangk1]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16287) Alter table partition rename with location - moves partition back to hive warehouse
[
https://issues.apache.org/jira/browse/HIVE-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968759#comment-15968759
]
Hive QA commented on HIVE-16287:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863427/HIVE-16287.04.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 10574 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4690/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4690/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4690/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863427 - PreCommit-HIVE-Build
> Alter table partition rename with location - moves partition back to hive
> warehouse
> ---
>
> Key: HIVE-16287
> URL: https://issues.apache.org/jira/browse/HIVE-16287
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.1.0
> Environment: RHEL 6.8
>Reporter: Ying Chen
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-16287.01.patch, HIVE-16287.02.patch,
> HIVE-16287.03.patch, HIVE-16287.04.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> I was renaming my partition in a table that I've created using the location
> clause, and noticed that when after rename is completed, my partition is
> moved to the hive warehouse (hive.metastore.warehouse.dir).
> {quote}
> create table test_local_part (col1 int) partitioned by (col2 int) location
> '/tmp/testtable/test_local_part';
> insert into test_local_part partition (col2=1) values (1),(3);
> insert into test_local_part partition (col2=2) values (3);
> alter table test_local_part partition (col2='1') rename to partition
> (col2='4');
> {quote}
> Running:
>describe formatted test_local_part partition (col2='2')
> # Detailed Partition Information
> Partition Value: [2]
> Database: default
> Table:test_local_part
> CreateTime: Mon Mar 20 13:25:28 PDT 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Location:
> *hdfs://my.server.com:8020/tmp/testtable/test_local_part/col2=2*
> Running:
>describe formatted test_local_part partition (col2='4')
> # Detailed Partition Information
> Partition Value: [4]
> Database: default
> Table:test_local_part
> CreateTime: Mon Mar 20 13:24:53 PDT 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Location:
> *hdfs://my.server.com:8020/apps/hive/warehouse/test_local_part/col2=4*
> ---
> Per Sergio's comment - "The rename should create the new partition name in
> the same location of the table. "
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (HIVE-16320) Flaky test: TestBeeLineDriver.testCliDriver[escape_comments]
[ https://issues.apache.org/jira/browse/HIVE-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary resolved HIVE-16320. --- Resolution: Done Fix Version/s: 3.0.0 Is should be solved by now... > Flaky test: TestBeeLineDriver.testCliDriver[escape_comments] > > > Key: HIVE-16320 > URL: https://issues.apache.org/jira/browse/HIVE-16320 > Project: Hive > Issue Type: Sub-task >Reporter: Sahil Takiar >Assignee: Peter Vary > Fix For: 3.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16427) Fix multi-insert query and write qtests
[
https://issues.apache.org/jira/browse/HIVE-16427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968717#comment-15968717
]
Hive QA commented on HIVE-16427:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863423/HIVE-16427.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10574 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel
(batchId=217)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4689/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4689/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4689/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863423 - PreCommit-HIVE-Build
> Fix multi-insert query and write qtests
> ---
>
> Key: HIVE-16427
> URL: https://issues.apache.org/jira/browse/HIVE-16427
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Reporter: Thomas Poepping
>Assignee: Yongzhi Chen
> Attachments: HIVE-16427.1.patch
>
>
> On HIVE-16415, it was found that the bug reported to be fixed in HIVE-14519
> was not actually fixed.
> This task is to find the problem, fix it, and add qtests to verify no future
> regression.
> Specifically, the following query does not produce correct answers:
> {code}
> From (select * from src) a
> insert overwrite directory '/tmp/emp/dir1/'
> select key, value
> insert overwrite directory '/tmp/emp/dir2/'
> select 'header'
> limit 0
> insert overwrite directory '/tmp/emp/dir3/'
> select key, value
> where key = 100;
> {code}
> This gives incorrect result in master. All dirs end up with 0 rows instead of
> just dir2.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16355) Service: embedded mode should only be available if service is loaded onto the classpath
[
https://issues.apache.org/jira/browse/HIVE-16355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968710#comment-15968710
]
Peter Vary commented on HIVE-16355:
---
Hi [~kgyrtkirk],
How complicated would be to use serviceloader as stated in the comment? I would
prefer that solution above using reflection.
Thanks,
Peter
> Service: embedded mode should only be available if service is loaded onto the
> classpath
> ---
>
> Key: HIVE-16355
> URL: https://issues.apache.org/jira/browse/HIVE-16355
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore, Server Infrastructure
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-16355.1.patch, HIVE-16355.2.patch,
> HIVE-16355.2.patch
>
>
> I would like to relax the hard reference to
> {{EmbeddedThriftBinaryCLIService}} to be only used in case {{service}} module
> is loaded onto the classpath.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16416) Service: move constants out from HiveAuthFactory
[ https://issues.apache.org/jira/browse/HIVE-16416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968706#comment-15968706 ] Peter Vary commented on HIVE-16416: --- +1 LGTM (non-binding) > Service: move constants out from HiveAuthFactory > > > Key: HIVE-16416 > URL: https://issues.apache.org/jira/browse/HIVE-16416 > Project: Hive > Issue Type: Sub-task > Components: Server Infrastructure >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Attachments: HIVE-16416.1.patch > > > It took me a while to notice that there are only some constants which are > keep pulling in this class :) > it contains a tricky dependency to the whole ql module; but in client mode > that part is totally unused - moving the constants out from it, enables the > client to operate without the factory. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16146) If possible find a better way to filter the TestBeeLineDriver output
[ https://issues.apache.org/jira/browse/HIVE-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-16146: -- Attachment: HIVE-16146.06.patch Rebased the patch. Since after the rebase the TestBeeLineDriver runs more query files, more replace patterns are needed: - The output contains the original queries, so we should rewrite (default.src->src) for the output file as well - More static regex for hiding the file locations, and serdes to match the CliDriver output - More specific regex for hiding the database change in the explain results Followup jira (HIVE-16449) is created to handle -- SORT_BEFORE_DIFF. Otherwise the query output files are the same as the CliDriver version output > If possible find a better way to filter the TestBeeLineDriver output > > > Key: HIVE-16146 > URL: https://issues.apache.org/jira/browse/HIVE-16146 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16146.02.patch, HIVE-16146.03.patch, > HIVE-16146.04.patch, HIVE-16146.05.patch, HIVE-16146.06.patch, > HIVE-16146.patch > > > Currently we apply a blacklist to filter the output of the BeeLine Qtest runs. > It might be a good idea to go thorough of the possibilities and find a better > way, if possible. > I think our main goal could be for the TestBeeLineDriver test output to match > the TestCliDriver output of the came query file. Or if it is not possible, > then at least a similar one > CC: [~vihangk1] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16449) BeeLineDriver should handle query result sorting
[ https://issues.apache.org/jira/browse/HIVE-16449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary reassigned HIVE-16449: - > BeeLineDriver should handle query result sorting > > > Key: HIVE-16449 > URL: https://issues.apache.org/jira/browse/HIVE-16449 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 3.0.0 >Reporter: Peter Vary >Assignee: Peter Vary > > The CLI driver supports the following features: > -- SORT_BEFORE_DIFF > -- SORT_QUERY_RESULTS > -- HASH_QUERY_RESULTS > -- SORT_AND_HASH_QUERY_RESULTS > BeeLineDriver should find a way to support these -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16287) Alter table partition rename with location - moves partition back to hive warehouse
[
https://issues.apache.org/jira/browse/HIVE-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-16287:
---
Attachment: HIVE-16287.04.patch
TestHBaseMetastoreSql and TestMiniLlapCliDriver[orc_llap_counters] tests are
working for me locally. Resubmitting the patch one more time to make sure they
are not flaky test failures. The third test failure is known flaky test failing
on other pre-commit runs as well.
> Alter table partition rename with location - moves partition back to hive
> warehouse
> ---
>
> Key: HIVE-16287
> URL: https://issues.apache.org/jira/browse/HIVE-16287
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.1.0
> Environment: RHEL 6.8
>Reporter: Ying Chen
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-16287.01.patch, HIVE-16287.02.patch,
> HIVE-16287.03.patch, HIVE-16287.04.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> I was renaming my partition in a table that I've created using the location
> clause, and noticed that when after rename is completed, my partition is
> moved to the hive warehouse (hive.metastore.warehouse.dir).
> {quote}
> create table test_local_part (col1 int) partitioned by (col2 int) location
> '/tmp/testtable/test_local_part';
> insert into test_local_part partition (col2=1) values (1),(3);
> insert into test_local_part partition (col2=2) values (3);
> alter table test_local_part partition (col2='1') rename to partition
> (col2='4');
> {quote}
> Running:
>describe formatted test_local_part partition (col2='2')
> # Detailed Partition Information
> Partition Value: [2]
> Database: default
> Table:test_local_part
> CreateTime: Mon Mar 20 13:25:28 PDT 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Location:
> *hdfs://my.server.com:8020/tmp/testtable/test_local_part/col2=2*
> Running:
>describe formatted test_local_part partition (col2='4')
> # Detailed Partition Information
> Partition Value: [4]
> Database: default
> Table:test_local_part
> CreateTime: Mon Mar 20 13:24:53 PDT 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Location:
> *hdfs://my.server.com:8020/apps/hive/warehouse/test_local_part/col2=4*
> ---
> Per Sergio's comment - "The rename should create the new partition name in
> the same location of the table. "
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-13567) Auto-gather column stats - phase 2
[
https://issues.apache.org/jira/browse/HIVE-13567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968679#comment-15968679
]
Hive QA commented on HIVE-13567:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863414/HIVE-13567.06.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 74 failed/errored test(s), 10557 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_11]
(batchId=234)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_12]
(batchId=234)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_13]
(batchId=234)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_16]
(batchId=234)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_7]
(batchId=234)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark1]
(batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark2]
(batchId=2)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark3]
(batchId=42)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin5]
(batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin_negative2]
(batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin_negative]
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_3]
(batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_4]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cast1] (batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynamic_rdd_cache]
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby11] (batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby12] (batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby1] (batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby1_noskew]
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby2_noskew]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby2_noskew_multi_distinct]
(batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby4] (batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby4_noskew]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby6] (batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby6_noskew]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby7_map] (batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby7_map_skew]
(batchId=41)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby8_map_skew]
(batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby9] (batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_cube_multi_gby]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_position]
(batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_ppr] (batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_ppr_multi_distinct]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort_grouping_operators]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input8] (batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input9] (batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input_part10] (batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join43] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_7]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_dyn_part1]
(batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_dyn_part8]
(batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[multi_insert_mixed]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[partition_coltype_literals]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_constant_expr]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_join_partition_key]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_11]
(batchId=2)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_20]
(batchId=69)
