[jira] [Commented] (HIVE-16446) org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified by setting t
[ https://issues.apache.org/jira/browse/HIVE-16446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968672#comment-15968672 ] Vihang Karajgaonkar commented on HIVE-16446: I can take a look at this .. Does it work if you use fs.s3a.secret.key and fs.s3a.access.key? > org.apache.hadoop.hive.ql.exec.DDLTask. > MetaException(message:java.lang.IllegalArgumentException: AWS Access Key ID > and Secret Access Key must be specified by setting the fs.s3n.awsAccessKeyId > and fs.s3n.awsSecretAccessKey properties > - > > Key: HIVE-16446 > URL: https://issues.apache.org/jira/browse/HIVE-16446 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.1.0 >Reporter: Kalexin Baoerjiin > > After upgrading our Cloudera cluster to CDH 5.10.1 we are experiencing the > following problem during some Hive DDL. > > SET fs.s3n.awsSecretAccessKey=; > SET fs.s3n.awsAccessKeyId=; > > ALTER TABLE hive_1k_partitions ADD IF NOT EXISTS partition (year='2014', > month='2014-01', dt='2014-01-01', hours='00', minutes='16', seconds='22') > location 's3n://' > > Stack trace I was able to recover: > [ Message content over the limit has been removed. ] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:383) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:318) > at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:416) > at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:432) > at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:726) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:693) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:628) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > Job Submission failed with exception ‘java.lang.IllegalArgumentException(AWS > Access Key ID and Secret Access Key must be specified by setting the > fs.s3n.awsAccessKeyId and fs.s3n.awsSecretAccessKey properties > (respectively).)’ > FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.mr.MapRedTask > [9:31] > Logging initialized using configuration in > jar:file:/opt/cloudera/parcels/CDH-5.10.1-1.cdh5.10.1.p0.10/jars/hive-common-1.1.0-cdh5.10.1.jar!/hive-log4j.properties > In the past we did not have to set s3 key and ID in core-site.xml because we > were using them dynamically inside our hive DDL scripts. > After setting S3 secret key and Access ID in core-site.xml this problem goes > away. However this is an incompatibility change from the previous Hive > shipped in CDH 5.9. > Cloudera 5.10.x release note mentioned (HIVE-14269 : Enhanced write > performance for Hive tables stored on Amazon S3.) is the only Hive related > changes. > https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_rn_new_in_cdh_510.html > https://issues.apache.org/jira/browse/HIVE-14269 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16446) org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified by setting th
[ https://issues.apache.org/jira/browse/HIVE-16446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar reassigned HIVE-16446: -- Assignee: Vihang Karajgaonkar > org.apache.hadoop.hive.ql.exec.DDLTask. > MetaException(message:java.lang.IllegalArgumentException: AWS Access Key ID > and Secret Access Key must be specified by setting the fs.s3n.awsAccessKeyId > and fs.s3n.awsSecretAccessKey properties > - > > Key: HIVE-16446 > URL: https://issues.apache.org/jira/browse/HIVE-16446 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.1.0 >Reporter: Kalexin Baoerjiin >Assignee: Vihang Karajgaonkar > > After upgrading our Cloudera cluster to CDH 5.10.1 we are experiencing the > following problem during some Hive DDL. > > SET fs.s3n.awsSecretAccessKey=; > SET fs.s3n.awsAccessKeyId=; > > ALTER TABLE hive_1k_partitions ADD IF NOT EXISTS partition (year='2014', > month='2014-01', dt='2014-01-01', hours='00', minutes='16', seconds='22') > location 's3n://' > > Stack trace I was able to recover: > [ Message content over the limit has been removed. ] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:383) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:318) > at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:416) > at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:432) > at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:726) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:693) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:628) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > Job Submission failed with exception ‘java.lang.IllegalArgumentException(AWS > Access Key ID and Secret Access Key must be specified by setting the > fs.s3n.awsAccessKeyId and fs.s3n.awsSecretAccessKey properties > (respectively).)’ > FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.mr.MapRedTask > [9:31] > Logging initialized using configuration in > jar:file:/opt/cloudera/parcels/CDH-5.10.1-1.cdh5.10.1.p0.10/jars/hive-common-1.1.0-cdh5.10.1.jar!/hive-log4j.properties > In the past we did not have to set s3 key and ID in core-site.xml because we > were using them dynamically inside our hive DDL scripts. > After setting S3 secret key and Access ID in core-site.xml this problem goes > away. However this is an incompatibility change from the previous Hive > shipped in CDH 5.9. > Cloudera 5.10.x release note mentioned (HIVE-14269 : Enhanced write > performance for Hive tables stored on Amazon S3.) is the only Hive related > changes. > https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_rn_new_in_cdh_510.html > https://issues.apache.org/jira/browse/HIVE-14269 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16427) Fix multi-insert query and write qtests
[
https://issues.apache.org/jira/browse/HIVE-16427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-16427:
Attachment: HIVE-16427.1.patch
Process the case for limit 0
> Fix multi-insert query and write qtests
> ---
>
> Key: HIVE-16427
> URL: https://issues.apache.org/jira/browse/HIVE-16427
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Reporter: Thomas Poepping
>Assignee: Yongzhi Chen
> Attachments: HIVE-16427.1.patch
>
>
> On HIVE-16415, it was found that the bug reported to be fixed in HIVE-14519
> was not actually fixed.
> This task is to find the problem, fix it, and add qtests to verify no future
> regression.
> Specifically, the following query does not produce correct answers:
> {code}
> From (select * from src) a
> insert overwrite directory '/tmp/emp/dir1/'
> select key, value
> insert overwrite directory '/tmp/emp/dir2/'
> select 'header'
> limit 0
> insert overwrite directory '/tmp/emp/dir3/'
> select key, value
> where key = 100;
> {code}
> This gives incorrect result in master. All dirs end up with 0 rows instead of
> just dir2.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16427) Fix multi-insert query and write qtests
[
https://issues.apache.org/jira/browse/HIVE-16427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-16427:
Status: Patch Available (was: Open)
> Fix multi-insert query and write qtests
> ---
>
> Key: HIVE-16427
> URL: https://issues.apache.org/jira/browse/HIVE-16427
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Reporter: Thomas Poepping
>Assignee: Yongzhi Chen
> Attachments: HIVE-16427.1.patch
>
>
> On HIVE-16415, it was found that the bug reported to be fixed in HIVE-14519
> was not actually fixed.
> This task is to find the problem, fix it, and add qtests to verify no future
> regression.
> Specifically, the following query does not produce correct answers:
> {code}
> From (select * from src) a
> insert overwrite directory '/tmp/emp/dir1/'
> select key, value
> insert overwrite directory '/tmp/emp/dir2/'
> select 'header'
> limit 0
> insert overwrite directory '/tmp/emp/dir3/'
> select key, value
> where key = 100;
> {code}
> This gives incorrect result in master. All dirs end up with 0 rows instead of
> just dir2.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16427) Fix multi-insert query and write qtests
[
https://issues.apache.org/jira/browse/HIVE-16427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen reassigned HIVE-16427:
---
Assignee: Yongzhi Chen
> Fix multi-insert query and write qtests
> ---
>
> Key: HIVE-16427
> URL: https://issues.apache.org/jira/browse/HIVE-16427
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Reporter: Thomas Poepping
>Assignee: Yongzhi Chen
>
> On HIVE-16415, it was found that the bug reported to be fixed in HIVE-14519
> was not actually fixed.
> This task is to find the problem, fix it, and add qtests to verify no future
> regression.
> Specifically, the following query does not produce correct answers:
> {code}
> From (select * from src) a
> insert overwrite directory '/tmp/emp/dir1/'
> select key, value
> insert overwrite directory '/tmp/emp/dir2/'
> select 'header'
> limit 0
> insert overwrite directory '/tmp/emp/dir3/'
> select key, value
> where key = 100;
> {code}
> This gives incorrect result in master. All dirs end up with 0 rows instead of
> just dir2.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (HIVE-14568) Hive Decimal Returns NULL
[ https://issues.apache.org/jira/browse/HIVE-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503847#comment-15503847 ] Akhil Chalamalasetty edited comment on HIVE-14568 at 4/14/17 5:25 AM: -- Thanks for the elaborate explanation Zhang. We will workaround this issue by casting the column to a lower precision & scale. Since we have a few developers migrating from ORACLE and Postgres SQL, we thought this would be a feature request to ease the usage of Hive. Please let us know if there is a way to introduce such a mode on Hive and if that would have any performance impacts once implemented. Regards, Akhil was (Author: akhilnaidu): Thanks for the elaborate explanation Zhang. We will workaround this issue by casting the column to a lower precision & scale. Since we have a few developers migrating from ORACLE and Postgres SQL, we thought this would be a feature request to ease the usage of Hive. Please let us know if there is a way to introduce such a mode on Hive and if that would have a any performance impacts once implemented. Regards, Akhil > Hive Decimal Returns NULL > - > > Key: HIVE-14568 > URL: https://issues.apache.org/jira/browse/HIVE-14568 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.0.0, 1.2.0 > Environment: Centos 6.7, Hadoop 2.7.2,hive 1.0.0,2.0 >Reporter: gurmukh singh >Assignee: Xuefu Zhang > > Hi > I was under the impression that the bug: > https://issues.apache.org/jira/browse/HIVE-5022 got fixed. But, I see the > same issue in Hive 1.0 and hive 1.2 as well. > hive> desc mul_table; > OK > prc decimal(38,28) > vol decimal(38,10) > Time taken: 0.068 seconds, Fetched: 2 row(s) > hive> select prc, vol, prc*vol as cost from mul_table; > OK > 1.2 200 NULL > 1.44 200 NULL > 2.14 100 NULL > 3.004 50 NULL > 1.2 200 NULL > Time taken: 0.048 seconds, Fetched: 5 row(s) > Rather then returning NULL, it should give error or round off. > I understand that, I can use Double instead of decimal or can cast it, but > still returning "Null" will make many things go unnoticed. > hive> desc mul_table2; > OK > prc double > vol decimal(14,10) > Time taken: 0.049 seconds, Fetched: 2 row(s) > hive> select * from mul_table2; > OK > 1.4 200 > 1.34 200 > 7.34 100 > 7454533.354544100 > Time taken: 0.028 seconds, Fetched: 4 row(s) > hive> select prc, vol, prc*vol as cost from mul_table3; > OK > 7.34 100 734.0 > 7.34 10007340.0 > 1.000410001000.4 > 7454533.354544100 7.454533354544E8 <- Wrong result > 7454533.35454410007.454533354544E9 <- Wrong result > Time taken: 0.025 seconds, Fetched: 5 row(s) > Casting: > hive> select prc, vol, cast(prc*vol as decimal(38,38)) as cost from > mul_table3; > OK > 7.34 100 NULL > 7.34 1000NULL > 1.00041000NULL > 7454533.354544100 NULL > 7454533.3545441000NULL > Time taken: 0.033 seconds, Fetched: 5 row(s) > hive> select prc, vol, cast(prc*vol as decimal(38,10)) as cost from > mul_table3; > OK > 7.34 100 734 > 7.34 10007340 > 1.000410001000.4 > 7454533.354544100 745453335.4544 > 7454533.35454410007454533354.544 > Time taken: 0.026 seconds, Fetched: 5 row(s) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HIVE-14568) Hive Decimal Returns NULL
[ https://issues.apache.org/jira/browse/HIVE-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503847#comment-15503847 ] Akhil Chalamalasetty edited comment on HIVE-14568 at 4/14/17 5:25 AM: -- Thank you for the elaborate explanation Zhang. We will workaround this issue by casting the column to a lower precision & scale. Since we have a few developers migrating from ORACLE and Postgres SQL, we thought this would be a feature request to ease the usage of Hive. Please let us know if there is a way to introduce such a mode on Hive and if that would have any performance impacts once implemented. Regards, Akhil was (Author: akhilnaidu): Thanks for the elaborate explanation Zhang. We will workaround this issue by casting the column to a lower precision & scale. Since we have a few developers migrating from ORACLE and Postgres SQL, we thought this would be a feature request to ease the usage of Hive. Please let us know if there is a way to introduce such a mode on Hive and if that would have any performance impacts once implemented. Regards, Akhil > Hive Decimal Returns NULL > - > > Key: HIVE-14568 > URL: https://issues.apache.org/jira/browse/HIVE-14568 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.0.0, 1.2.0 > Environment: Centos 6.7, Hadoop 2.7.2,hive 1.0.0,2.0 >Reporter: gurmukh singh >Assignee: Xuefu Zhang > > Hi > I was under the impression that the bug: > https://issues.apache.org/jira/browse/HIVE-5022 got fixed. But, I see the > same issue in Hive 1.0 and hive 1.2 as well. > hive> desc mul_table; > OK > prc decimal(38,28) > vol decimal(38,10) > Time taken: 0.068 seconds, Fetched: 2 row(s) > hive> select prc, vol, prc*vol as cost from mul_table; > OK > 1.2 200 NULL > 1.44 200 NULL > 2.14 100 NULL > 3.004 50 NULL > 1.2 200 NULL > Time taken: 0.048 seconds, Fetched: 5 row(s) > Rather then returning NULL, it should give error or round off. > I understand that, I can use Double instead of decimal or can cast it, but > still returning "Null" will make many things go unnoticed. > hive> desc mul_table2; > OK > prc double > vol decimal(14,10) > Time taken: 0.049 seconds, Fetched: 2 row(s) > hive> select * from mul_table2; > OK > 1.4 200 > 1.34 200 > 7.34 100 > 7454533.354544100 > Time taken: 0.028 seconds, Fetched: 4 row(s) > hive> select prc, vol, prc*vol as cost from mul_table3; > OK > 7.34 100 734.0 > 7.34 10007340.0 > 1.000410001000.4 > 7454533.354544100 7.454533354544E8 <- Wrong result > 7454533.35454410007.454533354544E9 <- Wrong result > Time taken: 0.025 seconds, Fetched: 5 row(s) > Casting: > hive> select prc, vol, cast(prc*vol as decimal(38,38)) as cost from > mul_table3; > OK > 7.34 100 NULL > 7.34 1000NULL > 1.00041000NULL > 7454533.354544100 NULL > 7454533.3545441000NULL > Time taken: 0.033 seconds, Fetched: 5 row(s) > hive> select prc, vol, cast(prc*vol as decimal(38,10)) as cost from > mul_table3; > OK > 7.34 100 734 > 7.34 10007340 > 1.000410001000.4 > 7454533.354544100 745453335.4544 > 7454533.35454410007454533354.544 > Time taken: 0.026 seconds, Fetched: 5 row(s) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16426) Query cancel: improve the way to handle files
[
https://issues.apache.org/jira/browse/HIVE-16426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968632#comment-15968632
]
Hive QA commented on HIVE-16426:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863412/HIVE-16426.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10574 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4687/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4687/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4687/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863412 - PreCommit-HIVE-Build
> Query cancel: improve the way to handle files
> -
>
> Key: HIVE-16426
> URL: https://issues.apache.org/jira/browse/HIVE-16426
> Project: Hive
> Issue Type: Improvement
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-16426.1.patch
>
>
> 1. Add data structure support to make it is easy to check query cancel status.
> 2. Handle query cancel more gracefully. Remove possible file leaks caused by
> query cancel as shown in following stack:
> {noformat}
> 2017-04-11 09:57:30,727 WARN org.apache.hadoop.hive.ql.exec.Utilities:
> [HiveServer2-Background-Pool: Thread-149]: Failed to clean-up tmp directories.
> java.io.InterruptedIOException: Call interrupted
> at org.apache.hadoop.ipc.Client.call(Client.java:1496)
> at org.apache.hadoop.ipc.Client.call(Client.java:1439)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
> at com.sun.proxy.$Proxy20.delete(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:535)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
> at com.sun.proxy.$Proxy21.delete(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2059)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:675)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:671)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:671)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.clearWork(Utilities.java:277)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:463)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:142)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1978)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1691)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1423)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1207)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1202)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:238)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:303)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
> at
>
[jira] [Updated] (HIVE-16436) Response times in "Task Execution Summary" at the end of the job is not correct
[
https://issues.apache.org/jira/browse/HIVE-16436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-16436:
-
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Thanks [~rajesh.balamohan]! Committed .2 to master.
> Response times in "Task Execution Summary" at the end of the job is not
> correct
> ---
>
> Key: HIVE-16436
> URL: https://issues.apache.org/jira/browse/HIVE-16436
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Fix For: 3.0.0
>
> Attachments: HIVE-16436.1.patch, HIVE-16436.2.patch
>
>
> "Task execution summary" is printed at the of running a hive query. E.g
> {noformat}
> Task Execution Summary
> --
> VERTICES DURATION(ms) CPU_TIME(ms)GC_TIME(ms) INPUT_RECORDS
> OUTPUT_RECORDS
> --
> Map 1 277869.00 0 0 1,500,000,000
> 1,500,000,000
> Map 2 277868.00 0 0 5,999,989,709
>31,162,299
> Reducer 3 59875.00 0 0 1,531,162,299
> 2,018
> Reducer 4 2436.00 0 0 2,018
> 2
> Reducer 5375.00 0 0 2
> 0
> --
> {noformat}
> Response times reported here for Map-1/Map-2 is not correct. Not sure if
> this is broken due to any other patch. Creating this jira for tracking
> purpose.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16415) Add blobstore tests for insertion of zero rows
[
https://issues.apache.org/jira/browse/HIVE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968583#comment-15968583
]
Hive QA commented on HIVE-16415:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863399/HIVE-16415.01.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10560 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
(batchId=102)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4686/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4686/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4686/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863399 - PreCommit-HIVE-Build
> Add blobstore tests for insertion of zero rows
> --
>
> Key: HIVE-16415
> URL: https://issues.apache.org/jira/browse/HIVE-16415
> Project: Hive
> Issue Type: Test
> Components: Tests
>Affects Versions: 2.1.1
>Reporter: Thomas Poepping
>Assignee: Thomas Poepping
> Attachments: HIVE-16415.01.patch, HIVE-16415.patch
>
>
> This patch introduces two regression tests into the hive-blobstore qtest
> module: zero_rows_hdfs.q and zero_rows_blobstore.q. These test doing INSERT
> commands with a WHERE clause where the condition of the WHERE clause causes
> zero rows to be considered.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15229) 'like any' and 'like all' operators in hive
[
https://issues.apache.org/jira/browse/HIVE-15229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968570#comment-15968570
]
Carter Shanklin commented on HIVE-15229:
[~simanchal] [~vgarg] [~pxiong]
This appears to be a Teradata extension to the SQL standard (not a bad one I'd
say). My question is whether this will contribute to generic quantified
comparison predicate support, possibly as a followup.
First, on the Teradata specific stuff, I'm referring to Teradata document "SQL
Functions, Operators, Expressions and Predicates", Release 14.10, page 928.
Teradata supports subqueries in addition to pattern expressions. I didn't see
any tests with subqueries so it's not clear to me that subqueries are
supported. In my experience with quantified predicates, they are typically used
with subqueries. That may be different for this TD specific extension but with
standard features it is almost always the case.
Also: With quantified comparison predicates SOME and ANY are aliases. The
Teradata documentation confirms that a user can specify either SOME or ANY.
I expect that Hive will eventually support full quantified comparison
predicates, all the commercial analytical databases already do. For an example:
where x = any ( select y from table where table.field = x );
IOW instead of like I could say =, <>, >, <, >=, <=. Based on the patch it
looks like the any/all is tied specifically to like. Maybe that's the Hive way
I honestly don't know since I hardly ever get below the SQL layer. All I want
to ask is whether this sets us down the right path to supporting these
comparison operators.
> 'like any' and 'like all' operators in hive
> ---
>
> Key: HIVE-15229
> URL: https://issues.apache.org/jira/browse/HIVE-15229
> Project: Hive
> Issue Type: New Feature
> Components: Operators
>Reporter: Simanchal Das
>Assignee: Simanchal Das
>Priority: Minor
> Attachments: HIVE-15229.1.patch, HIVE-15229.2.patch,
> HIVE-15229.3.patch, HIVE-15229.4.patch, HIVE-15229.5.patch
>
>
> In Teradata 'like any' and 'like all' operators are mostly used when we are
> matching a text field with numbers of patterns.
> 'like any' and 'like all' operator are equivalents of multiple like operator
> like example below.
> {noformat}
> --like any
> select col1 from table1 where col2 like any ('%accountant%', '%accounting%',
> '%retail%', '%bank%', '%insurance%');
> --Can be written using multiple like condition
> select col1 from table1 where col2 like '%accountant%' or col2 like
> '%accounting%' or col2 like '%retail%' or col2 like '%bank%' or col2 like
> '%insurance%' ;
> --like all
> select col1 from table1 where col2 like all ('%accountant%', '%accounting%',
> '%retail%', '%bank%', '%insurance%');
> --Can be written using multiple like operator
> select col1 from table1 where col2 like '%accountant%' and col2 like
> '%accounting%' and col2 like '%retail%' and col2 like '%bank%' and col2 like
> '%insurance%' ;
> {noformat}
> Problem statement:
> Now a days so many data warehouse projects are being migrated from Teradata
> to Hive.
> Always Data engineer and Business analyst are searching for these two
> operator.
> If we introduce these two operator in hive then so many scripts will be
> migrated smoothly instead of converting these operators to multiple like
> operators.
> Result:
> 1. 'LIKE ANY' operator return true if a text(column value) matches to any
> pattern.
> 2. 'LIKE ALL' operator return true if a text(column value) matches to all
> patterns.
> 3. 'LIKE ANY' and 'LIKE ALL' returns NULL not only if the expression on the
> left hand side is NULL, but also if one of the pattern in the list is NULL.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-13567) Auto-gather column stats - phase 2
[ https://issues.apache.org/jira/browse/HIVE-13567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13567: --- Status: Patch Available (was: Open) > Auto-gather column stats - phase 2 > -- > > Key: HIVE-13567 > URL: https://issues.apache.org/jira/browse/HIVE-13567 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13567.01.patch, HIVE-13567.02.patch, > HIVE-13567.03.patch, HIVE-13567.04.patch, HIVE-13567.05.patch, > HIVE-13567.06.patch > > > in phase 2, we are going to set auto-gather column on as default. This needs > to update golden files. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-13567) Auto-gather column stats - phase 2
[ https://issues.apache.org/jira/browse/HIVE-13567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13567: --- Attachment: HIVE-13567.06.patch > Auto-gather column stats - phase 2 > -- > > Key: HIVE-13567 > URL: https://issues.apache.org/jira/browse/HIVE-13567 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13567.01.patch, HIVE-13567.02.patch, > HIVE-13567.03.patch, HIVE-13567.04.patch, HIVE-13567.05.patch, > HIVE-13567.06.patch > > > in phase 2, we are going to set auto-gather column on as default. This needs > to update golden files. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-13567) Auto-gather column stats - phase 2
[ https://issues.apache.org/jira/browse/HIVE-13567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13567: --- Status: Open (was: Patch Available) > Auto-gather column stats - phase 2 > -- > > Key: HIVE-13567 > URL: https://issues.apache.org/jira/browse/HIVE-13567 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13567.01.patch, HIVE-13567.02.patch, > HIVE-13567.03.patch, HIVE-13567.04.patch, HIVE-13567.05.patch, > HIVE-13567.06.patch > > > in phase 2, we are going to set auto-gather column on as default. This needs > to update golden files. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16440) Fix failing test columnstats_partlvl_invalid_values when autogather column stats is on
[ https://issues.apache.org/jira/browse/HIVE-16440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-16440: --- Labels: 2.3.0 (was: ) > Fix failing test columnstats_partlvl_invalid_values when autogather column > stats is on > -- > > Key: HIVE-16440 > URL: https://issues.apache.org/jira/browse/HIVE-16440 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.1.0, 2.3.0 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Labels: 2.3.0 > Fix For: 2.3.0 > > Attachments: HIVE-16440.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16440) Fix failing test columnstats_partlvl_invalid_values when autogather column stats is on
[ https://issues.apache.org/jira/browse/HIVE-16440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-16440: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Fix failing test columnstats_partlvl_invalid_values when autogather column > stats is on > -- > > Key: HIVE-16440 > URL: https://issues.apache.org/jira/browse/HIVE-16440 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.1.0, 2.3.0 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Labels: 2.3.0 > Fix For: 2.3.0 > > Attachments: HIVE-16440.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16440) Fix failing test columnstats_partlvl_invalid_values when autogather column stats is on
[ https://issues.apache.org/jira/browse/HIVE-16440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-16440: --- Fix Version/s: 2.3.0 > Fix failing test columnstats_partlvl_invalid_values when autogather column > stats is on > -- > > Key: HIVE-16440 > URL: https://issues.apache.org/jira/browse/HIVE-16440 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.1.0, 2.3.0 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Labels: 2.3.0 > Fix For: 2.3.0 > > Attachments: HIVE-16440.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16440) Fix failing test columnstats_partlvl_invalid_values when autogather column stats is on
[ https://issues.apache.org/jira/browse/HIVE-16440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-16440: --- Affects Version/s: 2.3.0 2.1.0 > Fix failing test columnstats_partlvl_invalid_values when autogather column > stats is on > -- > > Key: HIVE-16440 > URL: https://issues.apache.org/jira/browse/HIVE-16440 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.1.0, 2.3.0 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Labels: 2.3.0 > Fix For: 2.3.0 > > Attachments: HIVE-16440.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15986) Support "is [not] distinct from"
[ https://issues.apache.org/jira/browse/HIVE-15986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-15986: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks, Vineet! > Support "is [not] distinct from" > > > Key: HIVE-15986 > URL: https://issues.apache.org/jira/browse/HIVE-15986 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Carter Shanklin >Assignee: Vineet Garg > Fix For: 3.0.0 > > Attachments: HIVE-15986.1.patch, HIVE-15986.2.patch, > HIVE-15986.3.patch, HIVE-15986.4.patch, HIVE-15986.5.patch, > HIVE-15986.5.patch, HIVE-15986.7.patch > > > Support standard "is [not] distinct from" syntax. For example this gives a > standard way to do a comparison to null safe join: select * from t1 join t2 > on t1.x is not distinct from t2.y. SQL standard reference Section 8.15 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15986) Support "is [not] distinct from"
[
https://issues.apache.org/jira/browse/HIVE-15986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968555#comment-15968555
]
Hive QA commented on HIVE-15986:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863371/HIVE-15986.7.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10573 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4685/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4685/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4685/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863371 - PreCommit-HIVE-Build
> Support "is [not] distinct from"
>
>
> Key: HIVE-15986
> URL: https://issues.apache.org/jira/browse/HIVE-15986
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Reporter: Carter Shanklin
>Assignee: Vineet Garg
> Attachments: HIVE-15986.1.patch, HIVE-15986.2.patch,
> HIVE-15986.3.patch, HIVE-15986.4.patch, HIVE-15986.5.patch,
> HIVE-15986.5.patch, HIVE-15986.7.patch
>
>
> Support standard "is [not] distinct from" syntax. For example this gives a
> standard way to do a comparison to null safe join: select * from t1 join t2
> on t1.x is not distinct from t2.y. SQL standard reference Section 8.15
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16440) Fix failing test columnstats_partlvl_invalid_values when autogather column stats is on
[ https://issues.apache.org/jira/browse/HIVE-16440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968554#comment-15968554 ] Ashutosh Chauhan commented on HIVE-16440: - +1 > Fix failing test columnstats_partlvl_invalid_values when autogather column > stats is on > -- > > Key: HIVE-16440 > URL: https://issues.apache.org/jira/browse/HIVE-16440 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-16440.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16426) Query cancel: improve the way to handle files
[
https://issues.apache.org/jira/browse/HIVE-16426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-16426:
Attachment: HIVE-16426.1.patch
> Query cancel: improve the way to handle files
> -
>
> Key: HIVE-16426
> URL: https://issues.apache.org/jira/browse/HIVE-16426
> Project: Hive
> Issue Type: Improvement
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-16426.1.patch
>
>
> 1. Add data structure support to make it is easy to check query cancel status.
> 2. Handle query cancel more gracefully. Remove possible file leaks caused by
> query cancel as shown in following stack:
> {noformat}
> 2017-04-11 09:57:30,727 WARN org.apache.hadoop.hive.ql.exec.Utilities:
> [HiveServer2-Background-Pool: Thread-149]: Failed to clean-up tmp directories.
> java.io.InterruptedIOException: Call interrupted
> at org.apache.hadoop.ipc.Client.call(Client.java:1496)
> at org.apache.hadoop.ipc.Client.call(Client.java:1439)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
> at com.sun.proxy.$Proxy20.delete(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:535)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
> at com.sun.proxy.$Proxy21.delete(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2059)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:675)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:671)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:671)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.clearWork(Utilities.java:277)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:463)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:142)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1978)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1691)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1423)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1207)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1202)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:238)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:303)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:316)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> 3. Add checkpoints to related file operations to improve response time for
> query cancelling.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16426) Query cancel: improve the way to handle files
[
https://issues.apache.org/jira/browse/HIVE-16426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-16426:
Attachment: (was: HIVE-16426.1.patch)
> Query cancel: improve the way to handle files
> -
>
> Key: HIVE-16426
> URL: https://issues.apache.org/jira/browse/HIVE-16426
> Project: Hive
> Issue Type: Improvement
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-16426.1.patch
>
>
> 1. Add data structure support to make it is easy to check query cancel status.
> 2. Handle query cancel more gracefully. Remove possible file leaks caused by
> query cancel as shown in following stack:
> {noformat}
> 2017-04-11 09:57:30,727 WARN org.apache.hadoop.hive.ql.exec.Utilities:
> [HiveServer2-Background-Pool: Thread-149]: Failed to clean-up tmp directories.
> java.io.InterruptedIOException: Call interrupted
> at org.apache.hadoop.ipc.Client.call(Client.java:1496)
> at org.apache.hadoop.ipc.Client.call(Client.java:1439)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
> at com.sun.proxy.$Proxy20.delete(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:535)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
> at com.sun.proxy.$Proxy21.delete(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2059)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:675)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:671)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:671)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.clearWork(Utilities.java:277)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:463)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:142)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1978)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1691)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1423)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1207)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1202)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:238)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:303)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:316)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> 3. Add checkpoints to related file operations to improve response time for
> query cancelling.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16418) Allow HiveKey to skip some bytes for comparison
[
https://issues.apache.org/jira/browse/HIVE-16418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968551#comment-15968551
]
Rui Li commented on HIVE-16418:
---
Thanks guys for your inputs!
To [~gopalv]'s question:
bq. That's the part that confuses me, what are in these bytes?
These bytes are the timezone info (offset from GMT) of each TimestampTZ field.
The reason why I wanted to store the TZ info is to solve HIVE-14305. UDFs
{{to_utc_timestamp}} and {{from_utc_timestamp}} essentially require the
TimestampTZ data type. But if we don't store TZ part, the timestamp will
display using the default TZ. But the default TZ may skip some timestamp due to
DST, and thus gives wrong result.
I'm not sure how to solve the issue w/o storing TZ. Any suggestions?
> Allow HiveKey to skip some bytes for comparison
> ---
>
> Key: HIVE-16418
> URL: https://issues.apache.org/jira/browse/HIVE-16418
> Project: Hive
> Issue Type: New Feature
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-16418.1.patch
>
>
> The feature is required when we have to serialize some fields and prevent
> them from being used in comparison, e.g. HIVE-14412.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-14412) Add a timezone-aware timestamp
[ https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968546#comment-15968546 ] Rui Li commented on HIVE-14412: --- Hi [~ashutoshc], thanks for the suggestions. bq. Timestamp with Timezone should behave as ZonedDateTime and Timestamp should behave as LocalDateTime Agreed. But ZonedDateTime is "the combination of a LocalDateTime and a ZoneId", which means we still have to store the ZoneId right? Btw, shall we move the discussion to HIVE-16418 - other guys are commenting there :) > Add a timezone-aware timestamp > -- > > Key: HIVE-14412 > URL: https://issues.apache.org/jira/browse/HIVE-14412 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Rui Li >Assignee: Rui Li > Attachments: HIVE-14412.1.patch, HIVE-14412.2.patch, > HIVE-14412.3.patch, HIVE-14412.4.patch, HIVE-14412.5.patch, > HIVE-14412.6.patch, HIVE-14412.7.patch, HIVE-14412.8.patch > > > Java's Timestamp stores the time elapsed since the epoch. While it's by > itself unambiguous, ambiguity comes when we parse a string into timestamp, or > convert a timestamp to string, causing problems like HIVE-14305. > To solve the issue, I think we should make timestamp aware of timezone. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16440) Fix failing test columnstats_partlvl_invalid_values when autogather column stats is on
[
https://issues.apache.org/jira/browse/HIVE-16440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968539#comment-15968539
]
Pengcheng Xiong commented on HIVE-16440:
[~ashutoshc], could u take a look? The main change is to change
{code}
(!conf.getBoolVar(ConfVars.HIVESTATSCOLAUTOGATHER))
{code}
with context!=null because we should depends on the context rather than the
configuration to determine.
> Fix failing test columnstats_partlvl_invalid_values when autogather column
> stats is on
> --
>
> Key: HIVE-16440
> URL: https://issues.apache.org/jira/browse/HIVE-16440
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-16440.01.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16423) Add hint to enforce semi join optimization
[
https://issues.apache.org/jira/browse/HIVE-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968515#comment-15968515
]
Hive QA commented on HIVE-16423:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863366/HIVE-16423.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10572 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4684/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4684/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4684/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863366 - PreCommit-HIVE-Build
> Add hint to enforce semi join optimization
> --
>
> Key: HIVE-16423
> URL: https://issues.apache.org/jira/browse/HIVE-16423
> Project: Hive
> Issue Type: Task
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
> Attachments: HIVE-16423.1.patch, HIVE-16423.2.patch
>
>
> Add hints in semijoin to enforce particular semi join optimization.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16448) Vectorization: Vectorized order_null.q fails with deserialize EOF exception below TEZ ReduceRecordSource.processVectorGroup
[
https://issues.apache.org/jira/browse/HIVE-16448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-16448:
Description:
For query "SELECT x.* FROM src_null x ORDER BY b asc, a asc nulls last" here is
the stack trace:
{code}
], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
attempt_1492136345968_0001_40_01_00_1:java.lang.RuntimeException:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
DeserializeRead details: Reading inputByteBuffer of length 3 at start offset 0
for length 3 to read 2 fields with types [string, int]. Read field #1 at field
start position 1 current read offset 3 column sort order [false, false]
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at
org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException:
DeserializeRead details: Reading inputByteBuffer of length 3 at start offset 0
for length 3 to read 2 fields with types [string, int]. Read field #1 at field
start position 1 current read offset 3 column sort order [false, false]
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:389)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:245)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:317)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
... 15 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
DeserializeRead details: Reading inputByteBuffer of length 3 at start offset 0
for length 3 to read 2 fields with types [string, int]. Read field #1 at field
start position 1 current read offset 3 column sort order [false, false]
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:421)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:380)
... 18 more
Caused by: java.io.EOFException
at
org.apache.hadoop.hive.serde2.binarysortable.InputByteBuffer.read(InputByteBuffer.java:54)
at
org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableDeserializeRead.readNextField(BinarySortableDeserializeRead.java:205)
at
org.apache.hadoop.hive.ql.exec.vector.VectorDeserializeRow.deserialize(VectorDeserializeRow.java:751)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:417)
... 19 more
{code}
was:
Here is the stack trace:
{code}
], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
attempt_1492136345968_0001_40_01_00_1:java.lang.RuntimeException:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
DeserializeRead details: Reading inputByteBuffer of length 3 at start offset 0
for length 3 to read 2 fields with types [string, int]. Read field #1 at field
start position 1 current read offset 3 column sort order [false, false]
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at
[jira] [Assigned] (HIVE-16448) Vectorization: Vectorized order_null.q fails with deserialize EOF exception below TEZ ReduceRecordSource.processVectorGroup
[
https://issues.apache.org/jira/browse/HIVE-16448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline reassigned HIVE-16448:
---
> Vectorization: Vectorized order_null.q fails with deserialize EOF exception
> below TEZ ReduceRecordSource.processVectorGroup
> ---
>
> Key: HIVE-16448
> URL: https://issues.apache.org/jira/browse/HIVE-16448
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
>
> Here is the stack trace:
> {code}
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
> attempt_1492136345968_0001_40_01_00_1:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:389)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:245)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:317)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> ... 15 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> DeserializeRead details: Reading inputByteBuffer of length 3 at start offset
> 0 for length 3 to read 2 fields with types [string, int]. Read field #1 at
> field start position 1 current read offset 3 column sort order [false, false]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:421)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:380)
> ... 18 more
> Caused by: java.io.EOFException
> at
> org.apache.hadoop.hive.serde2.binarysortable.InputByteBuffer.read(InputByteBuffer.java:54)
> at
> org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableDeserializeRead.readNextField(BinarySortableDeserializeRead.java:205)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorDeserializeRow.deserialize(VectorDeserializeRow.java:751)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:417)
> ... 19 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16426) Query cancel: improve the way to handle files
[
https://issues.apache.org/jira/browse/HIVE-16426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968483#comment-15968483
]
Hive QA commented on HIVE-16426:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863358/HIVE-16426.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10568 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=84)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4683/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4683/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4683/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863358 - PreCommit-HIVE-Build
> Query cancel: improve the way to handle files
> -
>
> Key: HIVE-16426
> URL: https://issues.apache.org/jira/browse/HIVE-16426
> Project: Hive
> Issue Type: Improvement
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-16426.1.patch
>
>
> 1. Add data structure support to make it is easy to check query cancel status.
> 2. Handle query cancel more gracefully. Remove possible file leaks caused by
> query cancel as shown in following stack:
> {noformat}
> 2017-04-11 09:57:30,727 WARN org.apache.hadoop.hive.ql.exec.Utilities:
> [HiveServer2-Background-Pool: Thread-149]: Failed to clean-up tmp directories.
> java.io.InterruptedIOException: Call interrupted
> at org.apache.hadoop.ipc.Client.call(Client.java:1496)
> at org.apache.hadoop.ipc.Client.call(Client.java:1439)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
> at com.sun.proxy.$Proxy20.delete(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:535)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
> at com.sun.proxy.$Proxy21.delete(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2059)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:675)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:671)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:671)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.clearWork(Utilities.java:277)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:463)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:142)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1978)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1691)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1423)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1207)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1202)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:238)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:303)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
>
[jira] [Comment Edited] (HIVE-16311) Improve the performance for FastHiveDecimalImpl.fastDivide
[ https://issues.apache.org/jira/browse/HIVE-16311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968467#comment-15968467 ] Colin Ma edited comment on HIVE-16311 at 4/14/17 1:14 AM: -- [~mmccline], [~xuefuz], any comments for the latest patch? Can you help to commit the patch to the upstream, thanks for your help. was (Author: colinma): [~mmccline], [~xuefuz], any comments for the latest patch? > Improve the performance for FastHiveDecimalImpl.fastDivide > -- > > Key: HIVE-16311 > URL: https://issues.apache.org/jira/browse/HIVE-16311 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.2.0 >Reporter: Colin Ma >Assignee: Colin Ma > Fix For: 3.0.0 > > Attachments: HIVE-16311.001.patch, HIVE-16311.002.patch, > HIVE-16311.003.patch, HIVE-16311.004.patch, HIVE-16311.005.patch, > HIVE-16311.006.patch, HIVE-16311.007.patch, HIVE-16311.008.patch, > HIVE-16311.withTrailingZero.patch > > > FastHiveDecimalImpl.fastDivide is poor performance when evaluate the > expression as 12345.67/123.45 > There are 2 points can be improved: > 1. Don't always use HiveDecimal.MAX_SCALE as scale when do the > BigDecimal.divide. > 2. Get the precision for BigInteger in a fast way if possible. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16311) Improve the performance for FastHiveDecimalImpl.fastDivide
[ https://issues.apache.org/jira/browse/HIVE-16311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968467#comment-15968467 ] Colin Ma commented on HIVE-16311: - [~mmccline], [~xuefuz], any comments for the latest patch? > Improve the performance for FastHiveDecimalImpl.fastDivide > -- > > Key: HIVE-16311 > URL: https://issues.apache.org/jira/browse/HIVE-16311 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.2.0 >Reporter: Colin Ma >Assignee: Colin Ma > Fix For: 3.0.0 > > Attachments: HIVE-16311.001.patch, HIVE-16311.002.patch, > HIVE-16311.003.patch, HIVE-16311.004.patch, HIVE-16311.005.patch, > HIVE-16311.006.patch, HIVE-16311.007.patch, HIVE-16311.008.patch, > HIVE-16311.withTrailingZero.patch > > > FastHiveDecimalImpl.fastDivide is poor performance when evaluate the > expression as 12345.67/123.45 > There are 2 points can be improved: > 1. Don't always use HiveDecimal.MAX_SCALE as scale when do the > BigDecimal.divide. > 2. Get the precision for BigInteger in a fast way if possible. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-13567) Auto-gather column stats - phase 2
[
https://issues.apache.org/jira/browse/HIVE-13567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968456#comment-15968456
]
Hive QA commented on HIVE-13567:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863352/HIVE-13567.05.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4682/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4682/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4682/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-04-14 00:45:48.351
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-4682/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-04-14 00:45:48.354
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at be59e02 HIVE-15708 : Upgrade calcite version to 1.12 (Remus
Rusanu via Ashutosh Chauhan)
+ git clean -f -d
Removing
ql/src/test/queries/clientnegative/columnstats_partlvl_invalid_values_autogather.q
Removing
ql/src/test/results/clientnegative/columnstats_partlvl_invalid_values_autogather.q.out
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at be59e02 HIVE-15708 : Upgrade calcite version to 1.12 (Remus
Rusanu via Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-04-14 00:45:53.913
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/accumulo-handler/src/test/results/positive/accumulo_queries.q.out: No
such file or directory
error:
a/accumulo-handler/src/test/results/positive/accumulo_single_sourced_multi_insert.q.out:
No such file or directory
error: a/contrib/src/test/results/clientpositive/serde_typedbytes.q.out: No
such file or directory
error: a/contrib/src/test/results/clientpositive/serde_typedbytes2.q.out: No
such file or directory
error: a/contrib/src/test/results/clientpositive/serde_typedbytes3.q.out: No
such file or directory
error: a/contrib/src/test/results/clientpositive/serde_typedbytes4.q.out: No
such file or directory
error: a/contrib/src/test/results/clientpositive/serde_typedbytes5.q.out: No
such file or directory
error: a/data/conf/hive-site.xml: No such file or directory
error: a/hbase-handler/src/test/results/positive/hbase_queries.q.out: No such
file or directory
error:
a/hbase-handler/src/test/results/positive/hbase_single_sourced_multi_insert.q.out:
No such file or directory
error: a/hbase-handler/src/test/results/positive/hbasestats.q.out: No such file
or directory
error:
a/ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java:
No such file or directory
error: a/ql/src/test/queries/clientpositive/bucket_num_reducers.q: No such file
or directory
error: a/ql/src/test/queries/clientpositive/combine1.q: No such file or
directory
error:
a/ql/src/test/queries/clientpositive/encryption_join_with_different_encryption_keys.q:
No such file or directory
error:
a/ql/src/test/queries/clientpositive/infer_bucket_sort_reducers_power_two.q: No
such file or directory
error: a/ql/src/test/queries/clientpositive/orc_wide_table.q: No such file or
directory
error: a/ql/src/test/queries/clientpositive/udf_round_2.q: No such file or
directory
error: a/ql/src/test/results/clientnegative/fileformat_void_input.q.out: No
such file or directory
error:
a/ql/src/test/results/clientpositive/alter_numbuckets_partitioned_table2_h23.q.out:
No such file or directory
error:
[jira] [Commented] (HIVE-16440) Fix failing test columnstats_partlvl_invalid_values when autogather column stats is on
[
https://issues.apache.org/jira/browse/HIVE-16440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968455#comment-15968455
]
Hive QA commented on HIVE-16440:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863350/HIVE-16440.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10572 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4681/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4681/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4681/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863350 - PreCommit-HIVE-Build
> Fix failing test columnstats_partlvl_invalid_values when autogather column
> stats is on
> --
>
> Key: HIVE-16440
> URL: https://issues.apache.org/jira/browse/HIVE-16440
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-16440.01.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-14412) Add a timezone-aware timestamp
[ https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968447#comment-15968447 ] Ashutosh Chauhan commented on HIVE-14412: - One concern I have is storing TZ in storage doesn't seem useful. Its adding implementation complexity (and wasting disk space) without any benefit. Normalizing timestamps into UTC and then storing number of millis (alongwith fractional part) will retain full fidelity. We can display that either in original TZ (or in user's session timezone). w.r.t semantics my understanding is Timestamp with Timezone should behave as [ZonedTimeZone | https://docs.oracle.com/javase/8/docs/api/java/time/ZonedDateTime.html] and Timestamp should behave as [LocalDateTime | https://docs.oracle.com/javase/8/docs/api/java/time/LocalDateTime.html] and for this reason instead of using java.sql.Timestamp we should use java.time.ZonedTimeZone and java.time.LocalDateTime as in memory representations for TS w TZ and TS w/o TZ types. > Add a timezone-aware timestamp > -- > > Key: HIVE-14412 > URL: https://issues.apache.org/jira/browse/HIVE-14412 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Rui Li >Assignee: Rui Li > Attachments: HIVE-14412.1.patch, HIVE-14412.2.patch, > HIVE-14412.3.patch, HIVE-14412.4.patch, HIVE-14412.5.patch, > HIVE-14412.6.patch, HIVE-14412.7.patch, HIVE-14412.8.patch > > > Java's Timestamp stores the time elapsed since the epoch. While it's by > itself unambiguous, ambiguity comes when we parse a string into timestamp, or > convert a timestamp to string, causing problems like HIVE-14305. > To solve the issue, I think we should make timestamp aware of timezone. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16443) HiveOperation doesn't have operations for Update, Delete, Merge
[
https://issues.apache.org/jira/browse/HIVE-16443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-16443:
--
Description:
ideally it should have with proper privileges specified
SQLUPDATE("UPDATE", null, null, true, false),
SQLDELETE("DELETE", null, null, true, false),
SQLMERGE("MERGE", null, null, true, false);
It would also be useful to have INSERT and SELECT
all of these are currently QUERY is not informative
see how VIEW related stuff in SemanticAnalyzerFactory to set more specific
operation type
SELECT can be determined by
{noformat}
private boolean isReadOnly(ASTNode ast) {
if(ast == null) {
return false;
}
if(ast.getType() == HiveParser.TOK_QUERY) {
return isReadOnly((ASTNode)
ast.getFirstChildWithType(HiveParser.TOK_INSERT));
}
if(ast.getType() == HiveParser.TOK_INSERT) {
return
isReadOnly((ASTNode)ast.getFirstChildWithType(HiveParser.TOK_DESTINATION));
}
if(ast.getType() == HiveParser.TOK_DESTINATION) {
return null != ast.getFirstChildWithType(HiveParser.TOK_DIR);
}
return false;
}
{noformat}
was:
ideally it should have with proper privileges specified
SQLUPDATE("UPDATE", null, null, true, false),
SQLDELETE("DELETE", null, null, true, false),
SQLMERGE("MERGE", null, null, true, false);
It would also be useful to have INSERT and SELECT
all of these are currently QUERY is not informative
see how VIEW related stuff in SemanticAnalyzerFactory to set more specific
operation type
> HiveOperation doesn't have operations for Update, Delete, Merge
> ---
>
> Key: HIVE-16443
> URL: https://issues.apache.org/jira/browse/HIVE-16443
> Project: Hive
> Issue Type: Bug
> Components: Query Planning, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> ideally it should have with proper privileges specified
> SQLUPDATE("UPDATE", null, null, true, false),
> SQLDELETE("DELETE", null, null, true, false),
> SQLMERGE("MERGE", null, null, true, false);
> It would also be useful to have INSERT and SELECT
> all of these are currently QUERY is not informative
> see how VIEW related stuff in SemanticAnalyzerFactory to set more specific
> operation type
> SELECT can be determined by
> {noformat}
> private boolean isReadOnly(ASTNode ast) {
> if(ast == null) {
> return false;
> }
> if(ast.getType() == HiveParser.TOK_QUERY) {
> return isReadOnly((ASTNode)
> ast.getFirstChildWithType(HiveParser.TOK_INSERT));
> }
> if(ast.getType() == HiveParser.TOK_INSERT) {
> return
> isReadOnly((ASTNode)ast.getFirstChildWithType(HiveParser.TOK_DESTINATION));
> }
> if(ast.getType() == HiveParser.TOK_DESTINATION) {
> return null != ast.getFirstChildWithType(HiveParser.TOK_DIR);
> }
> return false;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16321) Possible deadlock in metastore with Acid enabled
[
https://issues.apache.org/jira/browse/HIVE-16321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968426#comment-15968426
]
Wei Zheng commented on HIVE-16321:
--
Typo in setConf(): getCommectionTimeoutMs. Also we may use a constant instead
of hardcoded number for this timeout. Should we add "if (connPoolMutex ==
null)" before initializing connPoolMutex?
In setupJdbcConnectionPool(): for hikaricp block, there are two
setMaximumPoolSize. One should be removed.
> Possible deadlock in metastore with Acid enabled
>
>
> Key: HIVE-16321
> URL: https://issues.apache.org/jira/browse/HIVE-16321
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-16321.01.patch
>
>
> TxnStore.MutexAPI is a mechanism how different Metastore instances can
> coordinate their operations. It uses a JDBCConnection to achieve it.
> In some cases this may lead to deadlock. TxnHandler uses a connection pool
> of fixed size. Suppose you have X simultaneous calls to TxnHandler.lock(),
> where X is >= size of the pool. This take all connections form the pool, so
> when
> {noformat}
> handle = getMutexAPI().acquireLock(MUTEX_KEY.CheckLock.name());
> {noformat}
> is executed in _TxnHandler.checkLock(Connection dbConn, long extLockId)_ the
> pool is empty and the system is deadlocked.
> MutexAPI can't use the same connection as the operation it's protecting.
> (TxnHandler.checkLock(Connection dbConn, long extLockId) is an example).
> We could make MutexAPI use a separate connection pool (size > 'primary' conn
> pool).
> Or we could make TxnHandler.lock(LockRequest rqst) return immediately after
> enqueueing the lock with the expectation that the caller will always follow
> up with a call to checkLock(CheckLockRequest rqst).
> cc [~f1sherox]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16445) enable Acid by default in the parent patch and run build bot
[ https://issues.apache.org/jira/browse/HIVE-16445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-16445: - > enable Acid by default in the parent patch and run build bot > > > Key: HIVE-16445 > URL: https://issues.apache.org/jira/browse/HIVE-16445 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16369) Vectorization: Support PTF (Part 1: No Custom Window Framing -- Default Only)
[ https://issues.apache.org/jira/browse/HIVE-16369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-16369: Status: In Progress (was: Patch Available) > Vectorization: Support PTF (Part 1: No Custom Window Framing -- Default Only) > - > > Key: HIVE-16369 > URL: https://issues.apache.org/jira/browse/HIVE-16369 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-16369.01.patch, HIVE-16369.02.patch > > > Support vectorization of PTF that doesn't include custom PRECEDING / > FOLLOWING window frame clauses. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16287) Alter table partition rename with location - moves partition back to hive warehouse
[
https://issues.apache.org/jira/browse/HIVE-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968402#comment-15968402
]
Hive QA commented on HIVE-16287:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863344/HIVE-16287.03.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10571 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters]
(batchId=141)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
org.apache.hadoop.hive.metastore.hbase.TestHBaseMetastoreSql.partitionedTable
(batchId=201)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4680/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4680/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4680/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863344 - PreCommit-HIVE-Build
> Alter table partition rename with location - moves partition back to hive
> warehouse
> ---
>
> Key: HIVE-16287
> URL: https://issues.apache.org/jira/browse/HIVE-16287
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.1.0
> Environment: RHEL 6.8
>Reporter: Ying Chen
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-16287.01.patch, HIVE-16287.02.patch,
> HIVE-16287.03.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> I was renaming my partition in a table that I've created using the location
> clause, and noticed that when after rename is completed, my partition is
> moved to the hive warehouse (hive.metastore.warehouse.dir).
> {quote}
> create table test_local_part (col1 int) partitioned by (col2 int) location
> '/tmp/testtable/test_local_part';
> insert into test_local_part partition (col2=1) values (1),(3);
> insert into test_local_part partition (col2=2) values (3);
> alter table test_local_part partition (col2='1') rename to partition
> (col2='4');
> {quote}
> Running:
>describe formatted test_local_part partition (col2='2')
> # Detailed Partition Information
> Partition Value: [2]
> Database: default
> Table:test_local_part
> CreateTime: Mon Mar 20 13:25:28 PDT 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Location:
> *hdfs://my.server.com:8020/tmp/testtable/test_local_part/col2=2*
> Running:
>describe formatted test_local_part partition (col2='4')
> # Detailed Partition Information
> Partition Value: [4]
> Database: default
> Table:test_local_part
> CreateTime: Mon Mar 20 13:24:53 PDT 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Location:
> *hdfs://my.server.com:8020/apps/hive/warehouse/test_local_part/col2=4*
> ---
> Per Sergio's comment - "The rename should create the new partition name in
> the same location of the table. "
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16415) Add blobstore tests for insertion of zero rows
[ https://issues.apache.org/jira/browse/HIVE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Poepping updated HIVE-16415: --- Attachment: HIVE-16415.01.patch > Add blobstore tests for insertion of zero rows > -- > > Key: HIVE-16415 > URL: https://issues.apache.org/jira/browse/HIVE-16415 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.1.1 >Reporter: Thomas Poepping >Assignee: Thomas Poepping > Attachments: HIVE-16415.01.patch, HIVE-16415.patch > > > This patch introduces two regression tests into the hive-blobstore qtest > module: zero_rows_hdfs.q and zero_rows_blobstore.q. These test doing INSERT > commands with a WHERE clause where the condition of the WHERE clause causes > zero rows to be considered. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16415) Add blobstore tests for insertion of zero rows
[ https://issues.apache.org/jira/browse/HIVE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Poepping updated HIVE-16415: --- Status: Open (was: Patch Available) > Add blobstore tests for insertion of zero rows > -- > > Key: HIVE-16415 > URL: https://issues.apache.org/jira/browse/HIVE-16415 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.1.1 >Reporter: Thomas Poepping >Assignee: Thomas Poepping > Attachments: HIVE-16415.01.patch, HIVE-16415.patch > > > This patch introduces two regression tests into the hive-blobstore qtest > module: zero_rows_hdfs.q and zero_rows_blobstore.q. These test doing INSERT > commands with a WHERE clause where the condition of the WHERE clause causes > zero rows to be considered. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16415) Add blobstore tests for insertion of zero rows
[ https://issues.apache.org/jira/browse/HIVE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Poepping updated HIVE-16415: --- Status: Patch Available (was: Open) > Add blobstore tests for insertion of zero rows > -- > > Key: HIVE-16415 > URL: https://issues.apache.org/jira/browse/HIVE-16415 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.1.1 >Reporter: Thomas Poepping >Assignee: Thomas Poepping > Attachments: HIVE-16415.01.patch, HIVE-16415.patch > > > This patch introduces two regression tests into the hive-blobstore qtest > module: zero_rows_hdfs.q and zero_rows_blobstore.q. These test doing INSERT > commands with a WHERE clause where the condition of the WHERE clause causes > zero rows to be considered. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16436) Response times in "Task Execution Summary" at the end of the job is not correct
[
https://issues.apache.org/jira/browse/HIVE-16436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968393#comment-15968393
]
Rajesh Balamohan commented on HIVE-16436:
-
[~prasanth_j] - I have not seen negative values in the response times. I tried
with .2 patch and it works fine. I am fine with either of the patches. :)
> Response times in "Task Execution Summary" at the end of the job is not
> correct
> ---
>
> Key: HIVE-16436
> URL: https://issues.apache.org/jira/browse/HIVE-16436
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: HIVE-16436.1.patch, HIVE-16436.2.patch
>
>
> "Task execution summary" is printed at the of running a hive query. E.g
> {noformat}
> Task Execution Summary
> --
> VERTICES DURATION(ms) CPU_TIME(ms)GC_TIME(ms) INPUT_RECORDS
> OUTPUT_RECORDS
> --
> Map 1 277869.00 0 0 1,500,000,000
> 1,500,000,000
> Map 2 277868.00 0 0 5,999,989,709
>31,162,299
> Reducer 3 59875.00 0 0 1,531,162,299
> 2,018
> Reducer 4 2436.00 0 0 2,018
> 2
> Reducer 5375.00 0 0 2
> 0
> --
> {noformat}
> Response times reported here for Map-1/Map-2 is not correct. Not sure if
> this is broken due to any other patch. Creating this jira for tracking
> purpose.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16415) Add blobstore tests for insertion of zero rows
[ https://issues.apache.org/jira/browse/HIVE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968391#comment-15968391 ] Thomas Poepping commented on HIVE-16415: [~spena] I like that idea. Do you think the commit messages deserves to be edited with that addition? "Add tests covering insertion of zero rows", maybe even "Add tests covering single inserts of zero rows" to differentiate from the separate issue [~ashutoshc] found. I have a new diff, I'll upload now > Add blobstore tests for insertion of zero rows > -- > > Key: HIVE-16415 > URL: https://issues.apache.org/jira/browse/HIVE-16415 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.1.1 >Reporter: Thomas Poepping >Assignee: Thomas Poepping > Attachments: HIVE-16415.patch > > > This patch introduces two regression tests into the hive-blobstore qtest > module: zero_rows_hdfs.q and zero_rows_blobstore.q. These test doing INSERT > commands with a WHERE clause where the condition of the WHERE clause causes > zero rows to be considered. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16443) HiveOperation doesn't have operations for Update, Delete, Merge
[
https://issues.apache.org/jira/browse/HIVE-16443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-16443:
--
Description:
ideally it should have with proper privileges specified
SQLUPDATE("UPDATE", null, null, true, false),
SQLDELETE("DELETE", null, null, true, false),
SQLMERGE("MERGE", null, null, true, false);
It would also be useful to have INSERT and SELECT
all of these are currently QUERY is not informative
see how VIEW related stuff in SemanticAnalyzerFactory to set more specific
operation type
was:
ideally it should have with proper privileges specified
SQLUPDATE("UPDATE", null, null, true, false),
SQLDELETE("DELETE", null, null, true, false),
SQLMERGE("MERGE", null, null, true, false);
It would also be useful to have INSERT and SELECT
all of these are currently QUERY is not informative
> HiveOperation doesn't have operations for Update, Delete, Merge
> ---
>
> Key: HIVE-16443
> URL: https://issues.apache.org/jira/browse/HIVE-16443
> Project: Hive
> Issue Type: Bug
> Components: Query Planning, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> ideally it should have with proper privileges specified
> SQLUPDATE("UPDATE", null, null, true, false),
> SQLDELETE("DELETE", null, null, true, false),
> SQLMERGE("MERGE", null, null, true, false);
> It would also be useful to have INSERT and SELECT
> all of these are currently QUERY is not informative
> see how VIEW related stuff in SemanticAnalyzerFactory to set more specific
> operation type
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16443) HiveOperation doesn't have operations for Update, Delete, Merge
[
https://issues.apache.org/jira/browse/HIVE-16443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-16443:
--
Description:
ideally it should have with proper privileges specified
SQLUPDATE("UPDATE", null, null, true, false),
SQLDELETE("DELETE", null, null, true, false),
SQLMERGE("MERGE", null, null, true, false);
It would also be useful to have INSERT and SELECT
all of these are currently QUERY is not informative
was:
ideally it should have with proper privileges specified
SQLUPDATE("UPDATE", null, null, true, false),
SQLDELETE("DELETE", null, null, true, false),
SQLMERGE("MERGE", null, null, true, false);
> HiveOperation doesn't have operations for Update, Delete, Merge
> ---
>
> Key: HIVE-16443
> URL: https://issues.apache.org/jira/browse/HIVE-16443
> Project: Hive
> Issue Type: Bug
> Components: Query Planning, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> ideally it should have with proper privileges specified
> SQLUPDATE("UPDATE", null, null, true, false),
> SQLDELETE("DELETE", null, null, true, false),
> SQLMERGE("MERGE", null, null, true, false);
> It would also be useful to have INSERT and SELECT
> all of these are currently QUERY is not informative
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16443) HiveOperation doesn't have operations for Update, Delete, Merge
[
https://issues.apache.org/jira/browse/HIVE-16443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman reassigned HIVE-16443:
-
> HiveOperation doesn't have operations for Update, Delete, Merge
> ---
>
> Key: HIVE-16443
> URL: https://issues.apache.org/jira/browse/HIVE-16443
> Project: Hive
> Issue Type: Bug
> Components: Query Planning, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> ideally it should have with proper privileges specified
> SQLUPDATE("UPDATE", null, null, true, false),
> SQLDELETE("DELETE", null, null, true, false),
> SQLMERGE("MERGE", null, null, true, false);
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16439) Exclude older v2 version of jackson lib from dependent jars in pom.xml
[
https://issues.apache.org/jira/browse/HIVE-16439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968355#comment-15968355
]
Hive QA commented on HIVE-16439:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863340/HIVE-16439.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10571 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.spark.client.rpc.TestRpc.testEncryption (batchId=280)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4679/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4679/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4679/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863340 - PreCommit-HIVE-Build
> Exclude older v2 version of jackson lib from dependent jars in pom.xml
> ---
>
> Key: HIVE-16439
> URL: https://issues.apache.org/jira/browse/HIVE-16439
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-16439.1.patch
>
>
> There are multiple versions of jackson libs included in the dependent jars
> like spark-client and metrics-json. That causes older versions of jackson
> libs to be used.
> We need to exclude them from the dependencies and use the explicit one
> (currently 2.6.5).
> com.fasterxml.jackson.core
> jackson-databind
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16442) ExplainSemanticAnalyzer shares QueryState between statements
[
https://issues.apache.org/jira/browse/HIVE-16442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-16442:
--
Component/s: Query Planning
> ExplainSemanticAnalyzer shares QueryState between statements
>
>
> Key: HIVE-16442
> URL: https://issues.apache.org/jira/browse/HIVE-16442
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Eugene Koifman
>
> explain analyze .
> will call the Driver recursively to actually execute the query
> when it does
> {noformat}
> BaseSemanticAnalyzer sem = SemanticAnalyzerFactory.get(queryState, input);
> {noformat}
> it ends up sharing QueryState between different query executions
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16361) Automatically kill runaway client processes
[ https://issues.apache.org/jira/browse/HIVE-16361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-16361: -- Summary: Automatically kill runaway client processes (was: Automatically kill runaway processes ) > Automatically kill runaway client processes > > > Key: HIVE-16361 > URL: https://issues.apache.org/jira/browse/HIVE-16361 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Priority: Critical > > HIVE-13249 added an enforceable limit on how many transactions can be opened > concurrently where the system starts to reject new work to prevent the system > getting to a point where it cannot manage the load. > Another condition to guard against is a runaway process (which would usually > be some app (e.g. Storm) using Streaming Ingest API) that create a very large > number of transactions very quickly all of which immediately get aborted due > to some misconfiguration. This can cause large amount of metatdata to > accumulate in the ACID system slowing everything down and causing instability. > Now that we have TXNS.TXN_AGENT_INFO information we could probably use that > refuse work from a client even before we open any txns if it passes some > "runaway client" heuristic. > This is like an unintentional DOS attack -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16321) Possible deadlock in metastore with Acid enabled
[
https://issues.apache.org/jira/browse/HIVE-16321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968315#comment-15968315
]
Eugene Koifman commented on HIVE-16321:
---
wow, a clean run!
[~wzheng] could you review please
> Possible deadlock in metastore with Acid enabled
>
>
> Key: HIVE-16321
> URL: https://issues.apache.org/jira/browse/HIVE-16321
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-16321.01.patch
>
>
> TxnStore.MutexAPI is a mechanism how different Metastore instances can
> coordinate their operations. It uses a JDBCConnection to achieve it.
> In some cases this may lead to deadlock. TxnHandler uses a connection pool
> of fixed size. Suppose you have X simultaneous calls to TxnHandler.lock(),
> where X is >= size of the pool. This take all connections form the pool, so
> when
> {noformat}
> handle = getMutexAPI().acquireLock(MUTEX_KEY.CheckLock.name());
> {noformat}
> is executed in _TxnHandler.checkLock(Connection dbConn, long extLockId)_ the
> pool is empty and the system is deadlocked.
> MutexAPI can't use the same connection as the operation it's protecting.
> (TxnHandler.checkLock(Connection dbConn, long extLockId) is an example).
> We could make MutexAPI use a separate connection pool (size > 'primary' conn
> pool).
> Or we could make TxnHandler.lock(LockRequest rqst) return immediately after
> enqueueing the lock with the expectation that the caller will always follow
> up with a call to checkLock(CheckLockRequest rqst).
> cc [~f1sherox]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15986) Support "is [not] distinct from"
[ https://issues.apache.org/jira/browse/HIVE-15986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968297#comment-15968297 ] Ashutosh Chauhan commented on HIVE-15986: - +1 > Support "is [not] distinct from" > > > Key: HIVE-15986 > URL: https://issues.apache.org/jira/browse/HIVE-15986 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Carter Shanklin >Assignee: Vineet Garg > Attachments: HIVE-15986.1.patch, HIVE-15986.2.patch, > HIVE-15986.3.patch, HIVE-15986.4.patch, HIVE-15986.5.patch, > HIVE-15986.5.patch, HIVE-15986.7.patch > > > Support standard "is [not] distinct from" syntax. For example this gives a > standard way to do a comparison to null safe join: select * from t1 join t2 > on t1.x is not distinct from t2.y. SQL standard reference Section 8.15 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16321) Possible deadlock in metastore with Acid enabled
[
https://issues.apache.org/jira/browse/HIVE-16321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968294#comment-15968294
]
Hive QA commented on HIVE-16321:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863338/HIVE-16321.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 10571 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4678/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4678/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4678/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863338 - PreCommit-HIVE-Build
> Possible deadlock in metastore with Acid enabled
>
>
> Key: HIVE-16321
> URL: https://issues.apache.org/jira/browse/HIVE-16321
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-16321.01.patch
>
>
> TxnStore.MutexAPI is a mechanism how different Metastore instances can
> coordinate their operations. It uses a JDBCConnection to achieve it.
> In some cases this may lead to deadlock. TxnHandler uses a connection pool
> of fixed size. Suppose you have X simultaneous calls to TxnHandler.lock(),
> where X is >= size of the pool. This take all connections form the pool, so
> when
> {noformat}
> handle = getMutexAPI().acquireLock(MUTEX_KEY.CheckLock.name());
> {noformat}
> is executed in _TxnHandler.checkLock(Connection dbConn, long extLockId)_ the
> pool is empty and the system is deadlocked.
> MutexAPI can't use the same connection as the operation it's protecting.
> (TxnHandler.checkLock(Connection dbConn, long extLockId) is an example).
> We could make MutexAPI use a separate connection pool (size > 'primary' conn
> pool).
> Or we could make TxnHandler.lock(LockRequest rqst) return immediately after
> enqueueing the lock with the expectation that the caller will always follow
> up with a call to checkLock(CheckLockRequest rqst).
> cc [~f1sherox]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16431) Support Parquet StatsNoJobTask for Spark & Tez engine
[ https://issues.apache.org/jira/browse/HIVE-16431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968286#comment-15968286 ] Xuefu Zhang commented on HIVE-16431: Cool then! +1 > Support Parquet StatsNoJobTask for Spark & Tez engine > - > > Key: HIVE-16431 > URL: https://issues.apache.org/jira/browse/HIVE-16431 > Project: Hive > Issue Type: Improvement >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Minor > Attachments: HIVE-16431.1.patch > > > It seems only MR uses StatsNoJobTask for Parquet input format when computing > stats. We should add it to Tez & Spark as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16431) Support Parquet StatsNoJobTask for Spark & Tez engine
[ https://issues.apache.org/jira/browse/HIVE-16431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968264#comment-15968264 ] Chao Sun commented on HIVE-16431: - [~xuefuz] That is done in HIVE-14858: https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java#L94 > Support Parquet StatsNoJobTask for Spark & Tez engine > - > > Key: HIVE-16431 > URL: https://issues.apache.org/jira/browse/HIVE-16431 > Project: Hive > Issue Type: Improvement >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Minor > Attachments: HIVE-16431.1.patch > > > It seems only MR uses StatsNoJobTask for Parquet input format when computing > stats. We should add it to Tez & Spark as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16431) Support Parquet StatsNoJobTask for Spark & Tez engine
[ https://issues.apache.org/jira/browse/HIVE-16431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968261#comment-15968261 ] Xuefu Zhang commented on HIVE-16431: I was looking at GenMRTableScan1.java. My code could be old though. > Support Parquet StatsNoJobTask for Spark & Tez engine > - > > Key: HIVE-16431 > URL: https://issues.apache.org/jira/browse/HIVE-16431 > Project: Hive > Issue Type: Improvement >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Minor > Attachments: HIVE-16431.1.patch > > > It seems only MR uses StatsNoJobTask for Parquet input format when computing > stats. We should add it to Tez & Spark as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15986) Support "is [not] distinct from"
[ https://issues.apache.org/jira/browse/HIVE-15986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-15986: --- Status: Patch Available (was: Open) > Support "is [not] distinct from" > > > Key: HIVE-15986 > URL: https://issues.apache.org/jira/browse/HIVE-15986 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Carter Shanklin >Assignee: Vineet Garg > Attachments: HIVE-15986.1.patch, HIVE-15986.2.patch, > HIVE-15986.3.patch, HIVE-15986.4.patch, HIVE-15986.5.patch, > HIVE-15986.5.patch, HIVE-15986.7.patch > > > Support standard "is [not] distinct from" syntax. For example this gives a > standard way to do a comparison to null safe join: select * from t1 join t2 > on t1.x is not distinct from t2.y. SQL standard reference Section 8.15 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15986) Support "is [not] distinct from"
[ https://issues.apache.org/jira/browse/HIVE-15986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-15986: --- Status: Open (was: Patch Available) > Support "is [not] distinct from" > > > Key: HIVE-15986 > URL: https://issues.apache.org/jira/browse/HIVE-15986 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Carter Shanklin >Assignee: Vineet Garg > Attachments: HIVE-15986.1.patch, HIVE-15986.2.patch, > HIVE-15986.3.patch, HIVE-15986.4.patch, HIVE-15986.5.patch, > HIVE-15986.5.patch, HIVE-15986.7.patch > > > Support standard "is [not] distinct from" syntax. For example this gives a > standard way to do a comparison to null safe join: select * from t1 join t2 > on t1.x is not distinct from t2.y. SQL standard reference Section 8.15 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15986) Support "is [not] distinct from"
[ https://issues.apache.org/jira/browse/HIVE-15986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-15986: --- Attachment: HIVE-15986.7.patch > Support "is [not] distinct from" > > > Key: HIVE-15986 > URL: https://issues.apache.org/jira/browse/HIVE-15986 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Carter Shanklin >Assignee: Vineet Garg > Attachments: HIVE-15986.1.patch, HIVE-15986.2.patch, > HIVE-15986.3.patch, HIVE-15986.4.patch, HIVE-15986.5.patch, > HIVE-15986.5.patch, HIVE-15986.7.patch > > > Support standard "is [not] distinct from" syntax. For example this gives a > standard way to do a comparison to null safe join: select * from t1 join t2 > on t1.x is not distinct from t2.y. SQL standard reference Section 8.15 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16418) Allow HiveKey to skip some bytes for comparison
[ https://issues.apache.org/jira/browse/HIVE-16418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968256#comment-15968256 ] Eugene Koifman commented on HIVE-16418: --- It seems that storing TZ will cause more confusion than it's worth. I'm not even sure it's a good idea to automatically convert UTC to local TZ. This should be done by the application displaying the value or through an explicit conversion/formatting SQL function. This way it's always obvious what the default comparison/sorting is and what modifications to those operation any particular query is requesting. > Allow HiveKey to skip some bytes for comparison > --- > > Key: HIVE-16418 > URL: https://issues.apache.org/jira/browse/HIVE-16418 > Project: Hive > Issue Type: New Feature >Reporter: Rui Li >Assignee: Rui Li > Attachments: HIVE-16418.1.patch > > > The feature is required when we have to serialize some fields and prevent > them from being used in comparison, e.g. HIVE-14412. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-10299) Enable new cost model for Tez execution engine
[ https://issues.apache.org/jira/browse/HIVE-10299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-10299: --- Status: Open (was: Patch Available) > Enable new cost model for Tez execution engine > -- > > Key: HIVE-10299 > URL: https://issues.apache.org/jira/browse/HIVE-10299 > Project: Hive > Issue Type: Task >Reporter: Jesus Camacho Rodriguez >Assignee: Vineet Garg > Attachments: HIVE-10299.2.patch, HIVE-10299.3.patch, HIVE-10299.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-10299) Enable new cost model for Tez execution engine
[ https://issues.apache.org/jira/browse/HIVE-10299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-10299: --- Status: Patch Available (was: Open) > Enable new cost model for Tez execution engine > -- > > Key: HIVE-10299 > URL: https://issues.apache.org/jira/browse/HIVE-10299 > Project: Hive > Issue Type: Task >Reporter: Jesus Camacho Rodriguez >Assignee: Vineet Garg > Attachments: HIVE-10299.2.patch, HIVE-10299.3.patch, HIVE-10299.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16431) Support Parquet StatsNoJobTask for Spark & Tez engine
[ https://issues.apache.org/jira/browse/HIVE-16431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968240#comment-15968240 ] Chao Sun commented on HIVE-16431: - [~xuefuz] Hmm... MR should already been done. Which class you were looking at? > Support Parquet StatsNoJobTask for Spark & Tez engine > - > > Key: HIVE-16431 > URL: https://issues.apache.org/jira/browse/HIVE-16431 > Project: Hive > Issue Type: Improvement >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Minor > Attachments: HIVE-16431.1.patch > > > It seems only MR uses StatsNoJobTask for Parquet input format when computing > stats. We should add it to Tez & Spark as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16431) Support Parquet StatsNoJobTask for Spark & Tez engine
[ https://issues.apache.org/jira/browse/HIVE-16431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968233#comment-15968233 ] Xuefu Zhang commented on HIVE-16431: Patch looks good. I checked the same thing for MR and found over there equal check (instead of is assignable from) is in place. Should we be consistent in all three places? > Support Parquet StatsNoJobTask for Spark & Tez engine > - > > Key: HIVE-16431 > URL: https://issues.apache.org/jira/browse/HIVE-16431 > Project: Hive > Issue Type: Improvement >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Minor > Attachments: HIVE-16431.1.patch > > > It seems only MR uses StatsNoJobTask for Parquet input format when computing > stats. We should add it to Tez & Spark as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16423) Add hint to enforce semi join optimization
[ https://issues.apache.org/jira/browse/HIVE-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-16423: -- Attachment: HIVE-16423.2.patch Segregated the patch to only focus on hints. > Add hint to enforce semi join optimization > -- > > Key: HIVE-16423 > URL: https://issues.apache.org/jira/browse/HIVE-16423 > Project: Hive > Issue Type: Task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Attachments: HIVE-16423.1.patch, HIVE-16423.2.patch > > > Add hints in semijoin to enforce particular semi join optimization. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Work started] (HIVE-16441) De-duplicate semijoin branches in n-way joins
[ https://issues.apache.org/jira/browse/HIVE-16441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-16441 started by Deepak Jaiswal. - > De-duplicate semijoin branches in n-way joins > - > > Key: HIVE-16441 > URL: https://issues.apache.org/jira/browse/HIVE-16441 > Project: Hive > Issue Type: Task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > > Currently in n-way joins, semi join optimization creates n branches on same > key. Instead it should reuse one branch for all the joins. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16441) De-duplicate semijoin branches in n-way joins
[ https://issues.apache.org/jira/browse/HIVE-16441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal reassigned HIVE-16441: - > De-duplicate semijoin branches in n-way joins > - > > Key: HIVE-16441 > URL: https://issues.apache.org/jira/browse/HIVE-16441 > Project: Hive > Issue Type: Task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > > Currently in n-way joins, semi join optimization creates n branches on same > key. Instead it should reuse one branch for all the joins. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16427) Fix multi-insert query and write qtests
[
https://issues.apache.org/jira/browse/HIVE-16427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968214#comment-15968214
]
Yongzhi Chen commented on HIVE-16427:
-
Yes, it is a different case, HIVE-14519 fixed the case that null return caused
by filter. This test case is the null value caused by limit statement.
> Fix multi-insert query and write qtests
> ---
>
> Key: HIVE-16427
> URL: https://issues.apache.org/jira/browse/HIVE-16427
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Reporter: Thomas Poepping
>
> On HIVE-16415, it was found that the bug reported to be fixed in HIVE-14519
> was not actually fixed.
> This task is to find the problem, fix it, and add qtests to verify no future
> regression.
> Specifically, the following query does not produce correct answers:
> {code}
> From (select * from src) a
> insert overwrite directory '/tmp/emp/dir1/'
> select key, value
> insert overwrite directory '/tmp/emp/dir2/'
> select 'header'
> limit 0
> insert overwrite directory '/tmp/emp/dir3/'
> select key, value
> where key = 100;
> {code}
> This gives incorrect result in master. All dirs end up with 0 rows instead of
> just dir2.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16423) Add hint to enforce semi join optimization
[ https://issues.apache.org/jira/browse/HIVE-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-16423: -- Description: Add hints in semijoin to enforce particular semi join optimization. (was: Currently in an n-way join, a semi join branch is created n times. Instead, it should reuse the same branch. Add hints in semijoin to enforce particular semi join optimization.) > Add hint to enforce semi join optimization > -- > > Key: HIVE-16423 > URL: https://issues.apache.org/jira/browse/HIVE-16423 > Project: Hive > Issue Type: Task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Attachments: HIVE-16423.1.patch > > > Add hints in semijoin to enforce particular semi join optimization. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16423) Add hint to enforce semi join optimization
[ https://issues.apache.org/jira/browse/HIVE-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-16423: -- Summary: Add hint to enforce semi join optimization (was: De-duplicate semijoin branches and add hint to enforce semi join optimization) > Add hint to enforce semi join optimization > -- > > Key: HIVE-16423 > URL: https://issues.apache.org/jira/browse/HIVE-16423 > Project: Hive > Issue Type: Task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Attachments: HIVE-16423.1.patch > > > Currently in an n-way join, a semi join branch is created n times. Instead, > it should reuse the same branch. > Add hints in semijoin to enforce particular semi join optimization. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16429) Should call invokeFailureHooks in handleInterruption to track failed query execution due to interrupted command.
[
https://issues.apache.org/jira/browse/HIVE-16429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968203#comment-15968203
]
Hive QA commented on HIVE-16429:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863332/HIVE-16429.001.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 10571 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4677/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4677/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4677/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863332 - PreCommit-HIVE-Build
> Should call invokeFailureHooks in handleInterruption to track failed query
> execution due to interrupted command.
>
>
> Key: HIVE-16429
> URL: https://issues.apache.org/jira/browse/HIVE-16429
> Project: Hive
> Issue Type: Improvement
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Attachments: HIVE-16429.000.patch, HIVE-16429.001.patch
>
>
> Should call invokeFailureHooks in handleInterruption to track failed query
> execution due to interrupted command.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16426) Query cancel: improve the way to handle files
[
https://issues.apache.org/jira/browse/HIVE-16426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-16426:
Status: Patch Available (was: Open)
> Query cancel: improve the way to handle files
> -
>
> Key: HIVE-16426
> URL: https://issues.apache.org/jira/browse/HIVE-16426
> Project: Hive
> Issue Type: Improvement
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-16426.1.patch
>
>
> 1. Add data structure support to make it is easy to check query cancel status.
> 2. Handle query cancel more gracefully. Remove possible file leaks caused by
> query cancel as shown in following stack:
> {noformat}
> 2017-04-11 09:57:30,727 WARN org.apache.hadoop.hive.ql.exec.Utilities:
> [HiveServer2-Background-Pool: Thread-149]: Failed to clean-up tmp directories.
> java.io.InterruptedIOException: Call interrupted
> at org.apache.hadoop.ipc.Client.call(Client.java:1496)
> at org.apache.hadoop.ipc.Client.call(Client.java:1439)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
> at com.sun.proxy.$Proxy20.delete(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:535)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
> at com.sun.proxy.$Proxy21.delete(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2059)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:675)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:671)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:671)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.clearWork(Utilities.java:277)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:463)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:142)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1978)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1691)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1423)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1207)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1202)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:238)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:303)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:316)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> 3. Add checkpoints to related file operations to improve response time for
> query cancelling.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15708) Upgrade calcite version to 1.12
[ https://issues.apache.org/jira/browse/HIVE-15708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-15708: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Failures not reproducible. Pushed to master. Thanks, Remus! > Upgrade calcite version to 1.12 > --- > > Key: HIVE-15708 > URL: https://issues.apache.org/jira/browse/HIVE-15708 > Project: Hive > Issue Type: Task > Components: CBO, Logical Optimizer >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan >Assignee: Remus Rusanu > Fix For: 3.0.0 > > Attachments: HIVE-15708.01.patch, HIVE-15708.02.patch, > HIVE-15708.03.patch, HIVE-15708.04.patch, HIVE-15708.05.patch, > HIVE-15708.06.patch, HIVE-15708.07.patch, HIVE-15708.08.patch, > HIVE-15708.09.patch, HIVE-15708.10.patch, HIVE-15708.11.patch, > HIVE-15708.12.patch, HIVE-15708.13.patch, HIVE-15708.14.patch, > HIVE-15708.15.patch, HIVE-15708.15.patch, HIVE-15708.16.patch, > HIVE-15708.17.patch, HIVE-15708.18.patch, HIVE-15708.19.patch, > HIVe-15708.20.patch, HIVE-15708.21.patch, HIVE-15708.22.patch, > HIVE-15708.23.patch > > > Currently we are on 1.10 Need to upgrade calcite version to 1.11 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16426) Query cancel: improve the way to handle files
[
https://issues.apache.org/jira/browse/HIVE-16426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-16426:
Attachment: HIVE-16426.1.patch
1. Use threadlocal variable to store cancel state to make it is accessible
without being passed around by parameters.
2. Add checkpoints for file operations.
3. Remove backgroundHandle.cancel to avoid failed file cleanup because of the
interruption. By what I observed that the method seems not very effective for
scheduled operation, for example, the on going HMS API calls.
> Query cancel: improve the way to handle files
> -
>
> Key: HIVE-16426
> URL: https://issues.apache.org/jira/browse/HIVE-16426
> Project: Hive
> Issue Type: Improvement
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-16426.1.patch
>
>
> 1. Add data structure support to make it is easy to check query cancel status.
> 2. Handle query cancel more gracefully. Remove possible file leaks caused by
> query cancel as shown in following stack:
> {noformat}
> 2017-04-11 09:57:30,727 WARN org.apache.hadoop.hive.ql.exec.Utilities:
> [HiveServer2-Background-Pool: Thread-149]: Failed to clean-up tmp directories.
> java.io.InterruptedIOException: Call interrupted
> at org.apache.hadoop.ipc.Client.call(Client.java:1496)
> at org.apache.hadoop.ipc.Client.call(Client.java:1439)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
> at com.sun.proxy.$Proxy20.delete(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:535)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
> at com.sun.proxy.$Proxy21.delete(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2059)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:675)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:671)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:671)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.clearWork(Utilities.java:277)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:463)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:142)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1978)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1691)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1423)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1207)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1202)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:238)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:303)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:316)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> 3. Add checkpoints to related file operations to improve response time for
> query cancelling.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11418) Dropping a database in an encryption zone with CASCADE and trash enabled fails
[
https://issues.apache.org/jira/browse/HIVE-11418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968180#comment-15968180
]
Ashutosh Chauhan commented on HIVE-11418:
-
yeah adding a syntax for a workaround of a bug will be confusing. Better to not
introduce it and leave it with saying known issue, fixed in later version.
> Dropping a database in an encryption zone with CASCADE and trash enabled fails
> --
>
> Key: HIVE-11418
> URL: https://issues.apache.org/jira/browse/HIVE-11418
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 1.2.0
>Reporter: Sergio Peña
>Assignee: Sahil Takiar
> Attachments: HIVE-11418.1.patch, HIVE-11418.2.patch
>
>
> Here's the query that fails:
> {noformat}
> hive> CREATE DATABASE db;
> hive> USE db;
> hive> CREATE TABLE a(id int);
> hive> SET fs.trash.interval=1;
> hive> DROP DATABASE db CASCADE;
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Unable to drop
> db.a because it is in an encryption zone and trash
> is enabled. Use PURGE option to skip trash.)
> {noformat}
> DROP DATABASE does not support PURGE, so we have to remove the tables one by
> one, and then drop the database.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11418) Dropping a database in an encryption zone with CASCADE and trash enabled fails
[
https://issues.apache.org/jira/browse/HIVE-11418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968168#comment-15968168
]
Sergio Peña commented on HIVE-11418:
The PURGE does not exist on the DROP DATABASE yet. We were thinking if adding
it on branch-2 in order to delete an encrypted table without sending it to
trash as this patch with hadoop 2.8 will work only on 3.0.
The current workaround is to delete one table at a time when they're encrypted
and trash is enabled. So one option (if we don't want to include PURGE then
remove it on 3.0) is to mark it as a known bug on Hive versions older than 3.0.
Does it make sense?
> Dropping a database in an encryption zone with CASCADE and trash enabled fails
> --
>
> Key: HIVE-11418
> URL: https://issues.apache.org/jira/browse/HIVE-11418
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 1.2.0
>Reporter: Sergio Peña
>Assignee: Sahil Takiar
> Attachments: HIVE-11418.1.patch, HIVE-11418.2.patch
>
>
> Here's the query that fails:
> {noformat}
> hive> CREATE DATABASE db;
> hive> USE db;
> hive> CREATE TABLE a(id int);
> hive> SET fs.trash.interval=1;
> hive> DROP DATABASE db CASCADE;
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Unable to drop
> db.a because it is in an encryption zone and trash
> is enabled. Use PURGE option to skip trash.)
> {noformat}
> DROP DATABASE does not support PURGE, so we have to remove the tables one by
> one, and then drop the database.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15708) Upgrade calcite version to 1.12
[
https://issues.apache.org/jira/browse/HIVE-15708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968153#comment-15968153
]
Hive QA commented on HIVE-15708:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863330/HIVE-15708.23.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10571 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.parse.TestParseNegativeDriver.testCliDriver[unknown_function4]
(batchId=233)
org.apache.hadoop.hive.ql.parse.TestParseNegativeDriver.testCliDriver[wrong_distinct2]
(batchId=233)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4676/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4676/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4676/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863330 - PreCommit-HIVE-Build
> Upgrade calcite version to 1.12
> ---
>
> Key: HIVE-15708
> URL: https://issues.apache.org/jira/browse/HIVE-15708
> Project: Hive
> Issue Type: Task
> Components: CBO, Logical Optimizer
>Affects Versions: 2.2.0
>Reporter: Ashutosh Chauhan
>Assignee: Remus Rusanu
> Attachments: HIVE-15708.01.patch, HIVE-15708.02.patch,
> HIVE-15708.03.patch, HIVE-15708.04.patch, HIVE-15708.05.patch,
> HIVE-15708.06.patch, HIVE-15708.07.patch, HIVE-15708.08.patch,
> HIVE-15708.09.patch, HIVE-15708.10.patch, HIVE-15708.11.patch,
> HIVE-15708.12.patch, HIVE-15708.13.patch, HIVE-15708.14.patch,
> HIVE-15708.15.patch, HIVE-15708.15.patch, HIVE-15708.16.patch,
> HIVE-15708.17.patch, HIVE-15708.18.patch, HIVE-15708.19.patch,
> HIVe-15708.20.patch, HIVE-15708.21.patch, HIVE-15708.22.patch,
> HIVE-15708.23.patch
>
>
> Currently we are on 1.10 Need to upgrade calcite version to 1.11
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-13567) Auto-gather column stats - phase 2
[ https://issues.apache.org/jira/browse/HIVE-13567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13567: --- Status: Patch Available (was: Open) > Auto-gather column stats - phase 2 > -- > > Key: HIVE-13567 > URL: https://issues.apache.org/jira/browse/HIVE-13567 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13567.01.patch, HIVE-13567.02.patch, > HIVE-13567.03.patch, HIVE-13567.04.patch, HIVE-13567.05.patch > > > in phase 2, we are going to set auto-gather column on as default. This needs > to update golden files. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-13567) Auto-gather column stats - phase 2
[ https://issues.apache.org/jira/browse/HIVE-13567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13567: --- Attachment: HIVE-13567.05.patch > Auto-gather column stats - phase 2 > -- > > Key: HIVE-13567 > URL: https://issues.apache.org/jira/browse/HIVE-13567 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13567.01.patch, HIVE-13567.02.patch, > HIVE-13567.03.patch, HIVE-13567.04.patch, HIVE-13567.05.patch > > > in phase 2, we are going to set auto-gather column on as default. This needs > to update golden files. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-13567) Auto-gather column stats - phase 2
[ https://issues.apache.org/jira/browse/HIVE-13567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13567: --- Status: Open (was: Patch Available) > Auto-gather column stats - phase 2 > -- > > Key: HIVE-13567 > URL: https://issues.apache.org/jira/browse/HIVE-13567 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13567.01.patch, HIVE-13567.02.patch, > HIVE-13567.03.patch, HIVE-13567.04.patch, HIVE-13567.05.patch > > > in phase 2, we are going to set auto-gather column on as default. This needs > to update golden files. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16440) Fix failing test columnstats_partlvl_invalid_values when autogather column stats is on
[ https://issues.apache.org/jira/browse/HIVE-16440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-16440: --- Attachment: HIVE-16440.01.patch > Fix failing test columnstats_partlvl_invalid_values when autogather column > stats is on > -- > > Key: HIVE-16440 > URL: https://issues.apache.org/jira/browse/HIVE-16440 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-16440.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16440) Fix failing test columnstats_partlvl_invalid_values when autogather column stats is on
[ https://issues.apache.org/jira/browse/HIVE-16440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-16440: --- Status: Patch Available (was: Open) > Fix failing test columnstats_partlvl_invalid_values when autogather column > stats is on > -- > > Key: HIVE-16440 > URL: https://issues.apache.org/jira/browse/HIVE-16440 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-16440.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16440) Fix failing test columnstats_partlvl_invalid_values when autogather column stats is on
[ https://issues.apache.org/jira/browse/HIVE-16440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong reassigned HIVE-16440: -- > Fix failing test columnstats_partlvl_invalid_values when autogather column > stats is on > -- > > Key: HIVE-16440 > URL: https://issues.apache.org/jira/browse/HIVE-16440 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16355) Service: embedded mode should only be available if service is loaded onto the classpath
[
https://issues.apache.org/jira/browse/HIVE-16355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968133#comment-15968133
]
Zoltan Haindrich commented on HIVE-16355:
-
failures are not related
> Service: embedded mode should only be available if service is loaded onto the
> classpath
> ---
>
> Key: HIVE-16355
> URL: https://issues.apache.org/jira/browse/HIVE-16355
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore, Server Infrastructure
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-16355.1.patch, HIVE-16355.2.patch,
> HIVE-16355.2.patch
>
>
> I would like to relax the hard reference to
> {{EmbeddedThriftBinaryCLIService}} to be only used in case {{service}} module
> is loaded onto the classpath.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16416) Service: move constants out from HiveAuthFactory
[ https://issues.apache.org/jira/browse/HIVE-16416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968131#comment-15968131 ] Zoltan Haindrich commented on HIVE-16416: - test failures are not related > Service: move constants out from HiveAuthFactory > > > Key: HIVE-16416 > URL: https://issues.apache.org/jira/browse/HIVE-16416 > Project: Hive > Issue Type: Sub-task > Components: Server Infrastructure >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Attachments: HIVE-16416.1.patch > > > It took me a while to notice that there are only some constants which are > keep pulling in this class :) > it contains a tricky dependency to the whole ql module; but in client mode > that part is totally unused - moving the constants out from it, enables the > client to operate without the factory. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16287) Alter table partition rename with location - moves partition back to hive warehouse
[
https://issues.apache.org/jira/browse/HIVE-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-16287:
---
Attachment: HIVE-16287.03.patch
Updating the patch for the test failures.
> Alter table partition rename with location - moves partition back to hive
> warehouse
> ---
>
> Key: HIVE-16287
> URL: https://issues.apache.org/jira/browse/HIVE-16287
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.1.0
> Environment: RHEL 6.8
>Reporter: Ying Chen
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-16287.01.patch, HIVE-16287.02.patch,
> HIVE-16287.03.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> I was renaming my partition in a table that I've created using the location
> clause, and noticed that when after rename is completed, my partition is
> moved to the hive warehouse (hive.metastore.warehouse.dir).
> {quote}
> create table test_local_part (col1 int) partitioned by (col2 int) location
> '/tmp/testtable/test_local_part';
> insert into test_local_part partition (col2=1) values (1),(3);
> insert into test_local_part partition (col2=2) values (3);
> alter table test_local_part partition (col2='1') rename to partition
> (col2='4');
> {quote}
> Running:
>describe formatted test_local_part partition (col2='2')
> # Detailed Partition Information
> Partition Value: [2]
> Database: default
> Table:test_local_part
> CreateTime: Mon Mar 20 13:25:28 PDT 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Location:
> *hdfs://my.server.com:8020/tmp/testtable/test_local_part/col2=2*
> Running:
>describe formatted test_local_part partition (col2='4')
> # Detailed Partition Information
> Partition Value: [4]
> Database: default
> Table:test_local_part
> CreateTime: Mon Mar 20 13:24:53 PDT 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Location:
> *hdfs://my.server.com:8020/apps/hive/warehouse/test_local_part/col2=4*
> ---
> Per Sergio's comment - "The rename should create the new partition name in
> the same location of the table. "
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16415) Add blobstore tests for insertion of zero rows
[ https://issues.apache.org/jira/browse/HIVE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968115#comment-15968115 ] Sergio Peña commented on HIVE-16415: [~poeppt] can we add this zero rows tests on TestCliDriver as well? I think they're should be good to have. > Add blobstore tests for insertion of zero rows > -- > > Key: HIVE-16415 > URL: https://issues.apache.org/jira/browse/HIVE-16415 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.1.1 >Reporter: Thomas Poepping >Assignee: Thomas Poepping > Attachments: HIVE-16415.patch > > > This patch introduces two regression tests into the hive-blobstore qtest > module: zero_rows_hdfs.q and zero_rows_blobstore.q. These test doing INSERT > commands with a WHERE clause where the condition of the WHERE clause causes > zero rows to be considered. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16427) Fix multi-insert query and write qtests
[
https://issues.apache.org/jira/browse/HIVE-16427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968111#comment-15968111
]
Sergio Peña commented on HIVE-16427:
[~ychena] Seems the patch on HIVE-14519 fixed the issue partially. Here's
another test case that is causing multi-insert query to fail.
> Fix multi-insert query and write qtests
> ---
>
> Key: HIVE-16427
> URL: https://issues.apache.org/jira/browse/HIVE-16427
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Reporter: Thomas Poepping
>
> On HIVE-16415, it was found that the bug reported to be fixed in HIVE-14519
> was not actually fixed.
> This task is to find the problem, fix it, and add qtests to verify no future
> regression.
> Specifically, the following query does not produce correct answers:
> {code}
> From (select * from src) a
> insert overwrite directory '/tmp/emp/dir1/'
> select key, value
> insert overwrite directory '/tmp/emp/dir2/'
> select 'header'
> limit 0
> insert overwrite directory '/tmp/emp/dir3/'
> select key, value
> where key = 100;
> {code}
> This gives incorrect result in master. All dirs end up with 0 rows instead of
> just dir2.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16287) Alter table partition rename with location - moves partition back to hive warehouse
[
https://issues.apache.org/jira/browse/HIVE-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968107#comment-15968107
]
Vihang Karajgaonkar commented on HIVE-16287:
Thanks [~ying1] for reviewing. I ran the tests locally and found the same. I
will do some tests and update the patch a in few minutes.
> Alter table partition rename with location - moves partition back to hive
> warehouse
> ---
>
> Key: HIVE-16287
> URL: https://issues.apache.org/jira/browse/HIVE-16287
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.1.0
> Environment: RHEL 6.8
>Reporter: Ying Chen
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-16287.01.patch, HIVE-16287.02.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> I was renaming my partition in a table that I've created using the location
> clause, and noticed that when after rename is completed, my partition is
> moved to the hive warehouse (hive.metastore.warehouse.dir).
> {quote}
> create table test_local_part (col1 int) partitioned by (col2 int) location
> '/tmp/testtable/test_local_part';
> insert into test_local_part partition (col2=1) values (1),(3);
> insert into test_local_part partition (col2=2) values (3);
> alter table test_local_part partition (col2='1') rename to partition
> (col2='4');
> {quote}
> Running:
>describe formatted test_local_part partition (col2='2')
> # Detailed Partition Information
> Partition Value: [2]
> Database: default
> Table:test_local_part
> CreateTime: Mon Mar 20 13:25:28 PDT 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Location:
> *hdfs://my.server.com:8020/tmp/testtable/test_local_part/col2=2*
> Running:
>describe formatted test_local_part partition (col2='4')
> # Detailed Partition Information
> Partition Value: [4]
> Database: default
> Table:test_local_part
> CreateTime: Mon Mar 20 13:24:53 PDT 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Location:
> *hdfs://my.server.com:8020/apps/hive/warehouse/test_local_part/col2=4*
> ---
> Per Sergio's comment - "The rename should create the new partition name in
> the same location of the table. "
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16427) Fix multi-insert query and write qtests
[
https://issues.apache.org/jira/browse/HIVE-16427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-16427:
---
Description:
On HIVE-16415, it was found that the bug reported to be fixed in HIVE-14519 was
not actually fixed.
This task is to find the problem, fix it, and add qtests to verify no future
regression.
Specifically, the following query does not produce correct answers:
{code}
>From (select * from src) a
insert overwrite directory '/tmp/emp/dir1/'
select key, value
insert overwrite directory '/tmp/emp/dir2/'
select 'header'
limit 0
insert overwrite directory '/tmp/emp/dir3/'
select key, value
where key = 100;
{code}
This gives incorrect result in master. All dirs end up with 0 rows instead of
just dir2.
was:
On HIVE-16415, it was found that the bug reported to be fixed in HIVE-14519 was
not actually fixed.
This task is to find the problem, fix it, and add qtests to verify no future
regression.
Specifically, the following query does not produce correct answers:
{code}
>From (select * from src) a
insert overwrite directory '/tmp/emp/dir1/'
select key, value
insert overwrite directory '/tmp/emp/dir2/'
select 'header'
limit 0
insert overwrite directory '/tmp/emp/dir3/'
select key, value
where key = 100;
{code}
> Fix multi-insert query and write qtests
> ---
>
> Key: HIVE-16427
> URL: https://issues.apache.org/jira/browse/HIVE-16427
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Reporter: Thomas Poepping
>
> On HIVE-16415, it was found that the bug reported to be fixed in HIVE-14519
> was not actually fixed.
> This task is to find the problem, fix it, and add qtests to verify no future
> regression.
> Specifically, the following query does not produce correct answers:
> {code}
> From (select * from src) a
> insert overwrite directory '/tmp/emp/dir1/'
> select key, value
> insert overwrite directory '/tmp/emp/dir2/'
> select 'header'
> limit 0
> insert overwrite directory '/tmp/emp/dir3/'
> select key, value
> where key = 100;
> {code}
> This gives incorrect result in master. All dirs end up with 0 rows instead of
> just dir2.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16436) Response times in "Task Execution Summary" at the end of the job is not correct
[
https://issues.apache.org/jira/browse/HIVE-16436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968101#comment-15968101
]
Hive QA commented on HIVE-16436:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863322/HIVE-16436.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10571 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4675/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4675/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4675/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863322 - PreCommit-HIVE-Build
> Response times in "Task Execution Summary" at the end of the job is not
> correct
> ---
>
> Key: HIVE-16436
> URL: https://issues.apache.org/jira/browse/HIVE-16436
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: HIVE-16436.1.patch, HIVE-16436.2.patch
>
>
> "Task execution summary" is printed at the of running a hive query. E.g
> {noformat}
> Task Execution Summary
> --
> VERTICES DURATION(ms) CPU_TIME(ms)GC_TIME(ms) INPUT_RECORDS
> OUTPUT_RECORDS
> --
> Map 1 277869.00 0 0 1,500,000,000
> 1,500,000,000
> Map 2 277868.00 0 0 5,999,989,709
>31,162,299
> Reducer 3 59875.00 0 0 1,531,162,299
> 2,018
> Reducer 4 2436.00 0 0 2,018
> 2
> Reducer 5375.00 0 0 2
> 0
> --
> {noformat}
> Response times reported here for Map-1/Map-2 is not correct. Not sure if
> this is broken due to any other patch. Creating this jira for tracking
> purpose.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16287) Alter table partition rename with location - moves partition back to hive warehouse
[
https://issues.apache.org/jira/browse/HIVE-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968090#comment-15968090
]
Ying Chen commented on HIVE-16287:
--
In the change for HiveAlterHandler - " destPath =
wh.getPartitionPath(msdb.getDatabase(dbname), tbl, part_vals);"
Shouldn't new_part.getValues() be used instead of part_vals ?
> Alter table partition rename with location - moves partition back to hive
> warehouse
> ---
>
> Key: HIVE-16287
> URL: https://issues.apache.org/jira/browse/HIVE-16287
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.1.0
> Environment: RHEL 6.8
>Reporter: Ying Chen
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-16287.01.patch, HIVE-16287.02.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> I was renaming my partition in a table that I've created using the location
> clause, and noticed that when after rename is completed, my partition is
> moved to the hive warehouse (hive.metastore.warehouse.dir).
> {quote}
> create table test_local_part (col1 int) partitioned by (col2 int) location
> '/tmp/testtable/test_local_part';
> insert into test_local_part partition (col2=1) values (1),(3);
> insert into test_local_part partition (col2=2) values (3);
> alter table test_local_part partition (col2='1') rename to partition
> (col2='4');
> {quote}
> Running:
>describe formatted test_local_part partition (col2='2')
> # Detailed Partition Information
> Partition Value: [2]
> Database: default
> Table:test_local_part
> CreateTime: Mon Mar 20 13:25:28 PDT 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Location:
> *hdfs://my.server.com:8020/tmp/testtable/test_local_part/col2=2*
> Running:
>describe formatted test_local_part partition (col2='4')
> # Detailed Partition Information
> Partition Value: [4]
> Database: default
> Table:test_local_part
> CreateTime: Mon Mar 20 13:24:53 PDT 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Location:
> *hdfs://my.server.com:8020/apps/hive/warehouse/test_local_part/col2=4*
> ---
> Per Sergio's comment - "The rename should create the new partition name in
> the same location of the table. "
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16439) Exclude older v2 version of jackson lib from dependent jars in pom.xml
[ https://issues.apache.org/jira/browse/HIVE-16439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-16439: Attachment: (was: HIVE-16439.1.patch) > Exclude older v2 version of jackson lib from dependent jars in pom.xml > --- > > Key: HIVE-16439 > URL: https://issues.apache.org/jira/browse/HIVE-16439 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-16439.1.patch > > > There are multiple versions of jackson libs included in the dependent jars > like spark-client and metrics-json. That causes older versions of jackson > libs to be used. > We need to exclude them from the dependencies and use the explicit one > (currently 2.6.5). > com.fasterxml.jackson.core > jackson-databind -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16439) Exclude older v2 version of jackson lib from dependent jars in pom.xml
[ https://issues.apache.org/jira/browse/HIVE-16439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-16439: Attachment: HIVE-16439.1.patch > Exclude older v2 version of jackson lib from dependent jars in pom.xml > --- > > Key: HIVE-16439 > URL: https://issues.apache.org/jira/browse/HIVE-16439 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-16439.1.patch > > > There are multiple versions of jackson libs included in the dependent jars > like spark-client and metrics-json. That causes older versions of jackson > libs to be used. > We need to exclude them from the dependencies and use the explicit one > (currently 2.6.5). > com.fasterxml.jackson.core > jackson-databind -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16439) Exclude older v2 version of jackson lib from dependent jars in pom.xml
[ https://issues.apache.org/jira/browse/HIVE-16439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-16439: Status: Patch Available (was: Open) > Exclude older v2 version of jackson lib from dependent jars in pom.xml > --- > > Key: HIVE-16439 > URL: https://issues.apache.org/jira/browse/HIVE-16439 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-16439.1.patch > > > There are multiple versions of jackson libs included in the dependent jars > like spark-client and metrics-json. That causes older versions of jackson > libs to be used. > We need to exclude them from the dependencies and use the explicit one > (currently 2.6.5). > com.fasterxml.jackson.core > jackson-databind -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16321) Possible deadlock in metastore with Acid enabled
[
https://issues.apache.org/jira/browse/HIVE-16321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-16321:
--
Status: Patch Available (was: Open)
> Possible deadlock in metastore with Acid enabled
>
>
> Key: HIVE-16321
> URL: https://issues.apache.org/jira/browse/HIVE-16321
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-16321.01.patch
>
>
> TxnStore.MutexAPI is a mechanism how different Metastore instances can
> coordinate their operations. It uses a JDBCConnection to achieve it.
> In some cases this may lead to deadlock. TxnHandler uses a connection pool
> of fixed size. Suppose you have X simultaneous calls to TxnHandler.lock(),
> where X is >= size of the pool. This take all connections form the pool, so
> when
> {noformat}
> handle = getMutexAPI().acquireLock(MUTEX_KEY.CheckLock.name());
> {noformat}
> is executed in _TxnHandler.checkLock(Connection dbConn, long extLockId)_ the
> pool is empty and the system is deadlocked.
> MutexAPI can't use the same connection as the operation it's protecting.
> (TxnHandler.checkLock(Connection dbConn, long extLockId) is an example).
> We could make MutexAPI use a separate connection pool (size > 'primary' conn
> pool).
> Or we could make TxnHandler.lock(LockRequest rqst) return immediately after
> enqueueing the lock with the expectation that the caller will always follow
> up with a call to checkLock(CheckLockRequest rqst).
> cc [~f1sherox]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16439) Exclude older v2 version of jackson lib from dependent jars in pom.xml
[ https://issues.apache.org/jira/browse/HIVE-16439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-16439: Attachment: HIVE-16439.1.patch patch-1: exclude the dependencies from common and spark-client projects. > Exclude older v2 version of jackson lib from dependent jars in pom.xml > --- > > Key: HIVE-16439 > URL: https://issues.apache.org/jira/browse/HIVE-16439 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-16439.1.patch > > > There are multiple versions of jackson libs included in the dependent jars > like spark-client and metrics-json. That causes older versions of jackson > libs to be used. > We need to exclude them from the dependencies and use the explicit one > (currently 2.6.5). > com.fasterxml.jackson.core > jackson-databind -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16321) Possible deadlock in metastore with Acid enabled
[
https://issues.apache.org/jira/browse/HIVE-16321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-16321:
--
Attachment: HIVE-16321.01.patch
> Possible deadlock in metastore with Acid enabled
>
>
> Key: HIVE-16321
> URL: https://issues.apache.org/jira/browse/HIVE-16321
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-16321.01.patch
>
>
> TxnStore.MutexAPI is a mechanism how different Metastore instances can
> coordinate their operations. It uses a JDBCConnection to achieve it.
> In some cases this may lead to deadlock. TxnHandler uses a connection pool
> of fixed size. Suppose you have X simultaneous calls to TxnHandler.lock(),
> where X is >= size of the pool. This take all connections form the pool, so
> when
> {noformat}
> handle = getMutexAPI().acquireLock(MUTEX_KEY.CheckLock.name());
> {noformat}
> is executed in _TxnHandler.checkLock(Connection dbConn, long extLockId)_ the
> pool is empty and the system is deadlocked.
> MutexAPI can't use the same connection as the operation it's protecting.
> (TxnHandler.checkLock(Connection dbConn, long extLockId) is an example).
> We could make MutexAPI use a separate connection pool (size > 'primary' conn
> pool).
> Or we could make TxnHandler.lock(LockRequest rqst) return immediately after
> enqueueing the lock with the expectation that the caller will always follow
> up with a call to checkLock(CheckLockRequest rqst).
> cc [~f1sherox]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16439) Exclude older v2 version of jackson lib from pom.xml
[ https://issues.apache.org/jira/browse/HIVE-16439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu reassigned HIVE-16439: --- > Exclude older v2 version of jackson lib from pom.xml > - > > Key: HIVE-16439 > URL: https://issues.apache.org/jira/browse/HIVE-16439 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Aihua Xu >Assignee: Aihua Xu > > There are multiple versions of jackson libs included in the dependent jars > like spark-client and metrics-json. That causes older versions of jackson > libs to be used. > We need to exclude them from the dependencies and use the explicit one > (currently 2.6.5). > com.fasterxml.jackson.core > jackson-databind -- This message was sent by Atlassian JIRA (v6.3.15#6346)
