[jira] [Updated] (HIVE-13502) Beeline doesnt support session parameters in JDBC URL as documentation states.
[ https://issues.apache.org/jira/browse/HIVE-13502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-13502: - Attachment: HIVE-13502.4.patch > Beeline doesnt support session parameters in JDBC URL as documentation states. > -- > > Key: HIVE-13502 > URL: https://issues.apache.org/jira/browse/HIVE-13502 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 1.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-13502.1.patch, HIVE-13502.2.patch, > HIVE-13502.3.patch, HIVE-13502.3.patch, HIVE-13502.4.patch, HIVE-13502.patch > > > https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-ConnectionURLs > documents that sessions variables like credentials etc are accepted as part > of the URL. However, Beeline does not support such URLs today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13502) Beeline doesnt support session parameters in JDBC URL as documentation states.
[ https://issues.apache.org/jira/browse/HIVE-13502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-13502: - Status: Patch Available (was: Open) The previous patch caused a build failure in the test class when resolving a conflict during rebase. > Beeline doesnt support session parameters in JDBC URL as documentation states. > -- > > Key: HIVE-13502 > URL: https://issues.apache.org/jira/browse/HIVE-13502 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 1.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-13502.1.patch, HIVE-13502.2.patch, > HIVE-13502.3.patch, HIVE-13502.3.patch, HIVE-13502.4.patch, HIVE-13502.patch > > > https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-ConnectionURLs > documents that sessions variables like credentials etc are accepted as part > of the URL. However, Beeline does not support such URLs today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13502) Beeline doesnt support session parameters in JDBC URL as documentation states.
[ https://issues.apache.org/jira/browse/HIVE-13502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-13502: - Status: Open (was: Patch Available) > Beeline doesnt support session parameters in JDBC URL as documentation states. > -- > > Key: HIVE-13502 > URL: https://issues.apache.org/jira/browse/HIVE-13502 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 1.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-13502.1.patch, HIVE-13502.2.patch, > HIVE-13502.3.patch, HIVE-13502.3.patch, HIVE-13502.patch > > > https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-ConnectionURLs > documents that sessions variables like credentials etc are accepted as part > of the URL. However, Beeline does not support such URLs today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13449) LLAP: HS2 should get the token directly, rather than from LLAP
[ https://issues.apache.org/jira/browse/HIVE-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286075#comment-15286075 ] Lefty Leverenz commented on HIVE-13449: --- +1 for the parameter description in HiveConf.java (hive.llap.create.token.locally). > LLAP: HS2 should get the token directly, rather than from LLAP > -- > > Key: HIVE-13449 > URL: https://issues.apache.org/jira/browse/HIVE-13449 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13449.01.patch, HIVE-13449.02.WIP.patch, > HIVE-13449.02.patch, HIVE-13449.03.patch, HIVE-13449.04.patch, > HIVE-13449.patch > > > HS2 doesn't need a roundtrip to LLAP; it can instantiate the SecretManager > directly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13691) No record with CQ_ID=0 found in COMPACTION_QUEUE
[
https://issues.apache.org/jira/browse/HIVE-13691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-13691:
--
Priority: Critical (was: Major)
> No record with CQ_ID=0 found in COMPACTION_QUEUE
>
>
> Key: HIVE-13691
> URL: https://issues.apache.org/jira/browse/HIVE-13691
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Critical
> Fix For: 1.3.0, 2.1.0
>
> Attachments: HIVE-13691.patch
>
>
> {noformat}
> 2016-04-29 18:49:31,594 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(141)) - Caught exception while trying to determine if we
> should compact id:0,dbname:default,tableName:service_logs_v2,par
> tName:ds=2016-04-21,state:^@,type:null,runAs:null,tooManyAborts:false,highestTxnId:0.
> Marking clean to avoid repeated failures, MetaException(message:Timeout
> when executing method: getTable)
> at
> org.apache.hadoop.hive.metastore.Deadline.newMetaException(Deadline.java:187)
> at org.apache.hadoop.hive.metastore.Deadline.check(Deadline.java:177)
> at
> org.apache.hadoop.hive.metastore.Deadline.checkTimeout(Deadline.java:160)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToParts(ObjectStore.java:1839)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsViaOrmFilter(ObjectStore.java:2255)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$300(ObjectStore.java:165)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$3.getJdoResult(ObjectStore.java:2051)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$3.getJdoResult(ObjectStore.java:2043)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2400)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNamesInternal(ObjectStore.java:2043)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNames(ObjectStore.java:2037)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy0.getPartitionsByNames(Unknown Source)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorThread.resolvePartition(CompactorThread.java:111)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:129)
> Caused by: org.apache.hadoop.hive.metastore.DeadlineException: Timeout when
> executing method: getTable
> at org.apache.hadoop.hive.metastore.Deadline.check(Deadline.java:174)
> ... 16 more
> 2016-04-29 18:49:31,595 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(154)) - Initiator loop caught unexpected exception this
> time through the loop: java.lang.IllegalStateException: No record with
> CQ_ID=0 found in COMPACTION_QUEUE
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:861)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:144)
> {noformat}
> {noformat}
> 2016-04-29 18:49:31,595 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(154)) - Initiator loop caught unexpected exception this
> time through the loop: java.lang.IllegalStateException: No record with
> CQ_ID=0 found in COMPACTION_QUEUE
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:861)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:144)
> {noformat}
> is triggered by _DeadlineException: Timeout when executing method_ but is
> nonetheless an issue.
> We should be able to record an entry in completed_compaction_queue to
> represent a failed compaction even if an entry in compaction_queue was never
> made, as is the case here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13691) No record with CQ_ID=0 found in COMPACTION_QUEUE
[
https://issues.apache.org/jira/browse/HIVE-13691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-13691:
--
Status: Patch Available (was: Reopened)
> No record with CQ_ID=0 found in COMPACTION_QUEUE
>
>
> Key: HIVE-13691
> URL: https://issues.apache.org/jira/browse/HIVE-13691
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 1.3.0, 2.1.0
>
> Attachments: HIVE-13691.patch
>
>
> {noformat}
> 2016-04-29 18:49:31,594 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(141)) - Caught exception while trying to determine if we
> should compact id:0,dbname:default,tableName:service_logs_v2,par
> tName:ds=2016-04-21,state:^@,type:null,runAs:null,tooManyAborts:false,highestTxnId:0.
> Marking clean to avoid repeated failures, MetaException(message:Timeout
> when executing method: getTable)
> at
> org.apache.hadoop.hive.metastore.Deadline.newMetaException(Deadline.java:187)
> at org.apache.hadoop.hive.metastore.Deadline.check(Deadline.java:177)
> at
> org.apache.hadoop.hive.metastore.Deadline.checkTimeout(Deadline.java:160)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToParts(ObjectStore.java:1839)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsViaOrmFilter(ObjectStore.java:2255)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$300(ObjectStore.java:165)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$3.getJdoResult(ObjectStore.java:2051)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$3.getJdoResult(ObjectStore.java:2043)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2400)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNamesInternal(ObjectStore.java:2043)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNames(ObjectStore.java:2037)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy0.getPartitionsByNames(Unknown Source)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorThread.resolvePartition(CompactorThread.java:111)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:129)
> Caused by: org.apache.hadoop.hive.metastore.DeadlineException: Timeout when
> executing method: getTable
> at org.apache.hadoop.hive.metastore.Deadline.check(Deadline.java:174)
> ... 16 more
> 2016-04-29 18:49:31,595 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(154)) - Initiator loop caught unexpected exception this
> time through the loop: java.lang.IllegalStateException: No record with
> CQ_ID=0 found in COMPACTION_QUEUE
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:861)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:144)
> {noformat}
> {noformat}
> 2016-04-29 18:49:31,595 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(154)) - Initiator loop caught unexpected exception this
> time through the loop: java.lang.IllegalStateException: No record with
> CQ_ID=0 found in COMPACTION_QUEUE
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:861)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:144)
> {noformat}
> is triggered by _DeadlineException: Timeout when executing method_ but is
> nonetheless an issue.
> We should be able to record an entry in completed_compaction_queue to
> represent a failed compaction even if an entry in compaction_queue was never
> made, as is the case here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13691) No record with CQ_ID=0 found in COMPACTION_QUEUE
[
https://issues.apache.org/jira/browse/HIVE-13691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-13691:
--
Attachment: HIVE-13691.patch
> No record with CQ_ID=0 found in COMPACTION_QUEUE
>
>
> Key: HIVE-13691
> URL: https://issues.apache.org/jira/browse/HIVE-13691
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 1.3.0, 2.1.0
>
> Attachments: HIVE-13691.patch
>
>
> {noformat}
> 2016-04-29 18:49:31,594 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(141)) - Caught exception while trying to determine if we
> should compact id:0,dbname:default,tableName:service_logs_v2,par
> tName:ds=2016-04-21,state:^@,type:null,runAs:null,tooManyAborts:false,highestTxnId:0.
> Marking clean to avoid repeated failures, MetaException(message:Timeout
> when executing method: getTable)
> at
> org.apache.hadoop.hive.metastore.Deadline.newMetaException(Deadline.java:187)
> at org.apache.hadoop.hive.metastore.Deadline.check(Deadline.java:177)
> at
> org.apache.hadoop.hive.metastore.Deadline.checkTimeout(Deadline.java:160)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToParts(ObjectStore.java:1839)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsViaOrmFilter(ObjectStore.java:2255)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$300(ObjectStore.java:165)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$3.getJdoResult(ObjectStore.java:2051)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$3.getJdoResult(ObjectStore.java:2043)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2400)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNamesInternal(ObjectStore.java:2043)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNames(ObjectStore.java:2037)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy0.getPartitionsByNames(Unknown Source)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorThread.resolvePartition(CompactorThread.java:111)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:129)
> Caused by: org.apache.hadoop.hive.metastore.DeadlineException: Timeout when
> executing method: getTable
> at org.apache.hadoop.hive.metastore.Deadline.check(Deadline.java:174)
> ... 16 more
> 2016-04-29 18:49:31,595 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(154)) - Initiator loop caught unexpected exception this
> time through the loop: java.lang.IllegalStateException: No record with
> CQ_ID=0 found in COMPACTION_QUEUE
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:861)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:144)
> {noformat}
> {noformat}
> 2016-04-29 18:49:31,595 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(154)) - Initiator loop caught unexpected exception this
> time through the loop: java.lang.IllegalStateException: No record with
> CQ_ID=0 found in COMPACTION_QUEUE
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:861)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:144)
> {noformat}
> is triggered by _DeadlineException: Timeout when executing method_ but is
> nonetheless an issue.
> We should be able to record an entry in completed_compaction_queue to
> represent a failed compaction even if an entry in compaction_queue was never
> made, as is the case here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13691) No record with CQ_ID=0 found in COMPACTION_QUEUE
[
https://issues.apache.org/jira/browse/HIVE-13691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-13691:
--
Target Version/s: 1.3.0, 2.1.0
> No record with CQ_ID=0 found in COMPACTION_QUEUE
>
>
> Key: HIVE-13691
> URL: https://issues.apache.org/jira/browse/HIVE-13691
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 1.3.0, 2.1.0
>
> Attachments: HIVE-13691.patch
>
>
> {noformat}
> 2016-04-29 18:49:31,594 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(141)) - Caught exception while trying to determine if we
> should compact id:0,dbname:default,tableName:service_logs_v2,par
> tName:ds=2016-04-21,state:^@,type:null,runAs:null,tooManyAborts:false,highestTxnId:0.
> Marking clean to avoid repeated failures, MetaException(message:Timeout
> when executing method: getTable)
> at
> org.apache.hadoop.hive.metastore.Deadline.newMetaException(Deadline.java:187)
> at org.apache.hadoop.hive.metastore.Deadline.check(Deadline.java:177)
> at
> org.apache.hadoop.hive.metastore.Deadline.checkTimeout(Deadline.java:160)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToParts(ObjectStore.java:1839)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsViaOrmFilter(ObjectStore.java:2255)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$300(ObjectStore.java:165)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$3.getJdoResult(ObjectStore.java:2051)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$3.getJdoResult(ObjectStore.java:2043)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2400)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNamesInternal(ObjectStore.java:2043)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNames(ObjectStore.java:2037)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy0.getPartitionsByNames(Unknown Source)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorThread.resolvePartition(CompactorThread.java:111)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:129)
> Caused by: org.apache.hadoop.hive.metastore.DeadlineException: Timeout when
> executing method: getTable
> at org.apache.hadoop.hive.metastore.Deadline.check(Deadline.java:174)
> ... 16 more
> 2016-04-29 18:49:31,595 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(154)) - Initiator loop caught unexpected exception this
> time through the loop: java.lang.IllegalStateException: No record with
> CQ_ID=0 found in COMPACTION_QUEUE
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:861)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:144)
> {noformat}
> {noformat}
> 2016-04-29 18:49:31,595 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(154)) - Initiator loop caught unexpected exception this
> time through the loop: java.lang.IllegalStateException: No record with
> CQ_ID=0 found in COMPACTION_QUEUE
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:861)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:144)
> {noformat}
> is triggered by _DeadlineException: Timeout when executing method_ but is
> nonetheless an issue.
> We should be able to record an entry in completed_compaction_queue to
> represent a failed compaction even if an entry in compaction_queue was never
> made, as is the case here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13745) UDF current_date、current_timestamp、unix_timestamp NPE
[
https://issues.apache.org/jira/browse/HIVE-13745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286059#comment-15286059
]
Bill Wailliam commented on HIVE-13745:
--
2016-05-17 10:54:32,779 FATAL [main]
org.apache.hadoop.hive.ql.exec.mr.ExecMapper:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row {"time":"2016-05-17 10:43:54.09","offset":100}
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:545)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDFUnixTimeStamp.initializeInput(GenericUDFUnixTimeStamp.java:50)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDFToUnixTimeStamp.initialize(GenericUDFToUnixTimeStamp.java:66)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:139)
at
org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:145)
at
org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:139)
at
org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:139)
at
org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:139)
at
org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:76)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:164)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:535)
... 9 more
> UDF current_date、current_timestamp、unix_timestamp NPE
> -
>
> Key: HIVE-13745
> URL: https://issues.apache.org/jira/browse/HIVE-13745
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Bill Wailliam
>Assignee: Bill Wailliam
> Attachments: HIVE-13745.patch
>
>
> NullPointerException when current_date is used in mapreduce
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13269) Simplify comparison expressions using column stats
[
https://issues.apache.org/jira/browse/HIVE-13269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286057#comment-15286057
]
Hive QA commented on HIVE-13269:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12804195/HIVE-13269.03.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 61 failed/errored test(s), 10011 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-join1.q-mapjoin_decimal.q-union5.q-and-12-more - did not
produce a TEST-*.xml file
TestMiniTezCliDriver-update_orig_table.q-union2.q-bucket4.q-and-12-more - did
not produce a TEST-*.xml file
TestSparkCliDriver-auto_join_reordering_values.q-ptf_seqfile.q-auto_join18.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_disable_merge_for_bucketing
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge9
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_diff_fs
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_reduce_deduplicate
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join5
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join0
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_14
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby5_noskew
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_complex_types
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_map_ppr_multi_distinct
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_input13
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_input18
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_gby3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_mixed
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_16
org.apache.hadoop.hive.llap.daemon.impl.TestLlapDaemonProtocolServerImpl.test
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.exec.tez.TestHostAffinitySplitLocationProvider.testOrcSplitsLocationAffinity
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
[jira] [Commented] (HIVE-13293) Query occurs performance degradation after enabling parallel order by for Hive on Spark
[ https://issues.apache.org/jira/browse/HIVE-13293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286005#comment-15286005 ] Xuefu Zhang commented on HIVE-13293: Please commit. Thanks. > Query occurs performance degradation after enabling parallel order by for > Hive on Spark > --- > > Key: HIVE-13293 > URL: https://issues.apache.org/jira/browse/HIVE-13293 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 2.0.0 >Reporter: Lifeng Wang >Assignee: Rui Li > Attachments: HIVE-13293.1.patch, HIVE-13293.2.patch, > HIVE-13293.3.patch, HIVE-13293.3.patch, HIVE-13293.3.patch > > > I use TPCx-BB to do some performance test on Hive on Spark engine. And found > query 10 has performance degradation when enabling parallel order by. > It seems that sampling cost much time before running the real query. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6131) New columns after table alter result in null values despite data
[ https://issues.apache.org/jira/browse/HIVE-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285949#comment-15285949 ] niklaus xiao commented on HIVE-6131: This is not a bug, you can use `alter table t1 replace columns (c1 string, c2 string) cascade`, see https://issues.apache.org/jira/browse/HIVE-8839 > New columns after table alter result in null values despite data > > > Key: HIVE-6131 > URL: https://issues.apache.org/jira/browse/HIVE-6131 > Project: Hive > Issue Type: Bug >Affects Versions: 0.11.0, 0.12.0, 0.13.0, 1.2.1 >Reporter: James Vaughan >Priority: Critical > Attachments: HIVE-6131.1.patch > > > Hi folks, > I found and verified a bug on our CDH 4.0.3 install of Hive when adding > columns to tables with Partitions using 'REPLACE COLUMNS'. I dug through the > Jira a little bit and didn't see anything for it so hopefully this isn't just > noise on the radar. > Basically, when you alter a table with partitions and then reupload data to > that partition, it doesn't seem to recognize the extra data that actually > exists in HDFS- as in, returns NULL values on the new column despite having > the data and recognizing the new column in the metadata. > Here's some steps to reproduce using a basic table: > 1. Run this hive command: CREATE TABLE jvaughan_test (col1 string) > partitioned by (day string); > 2. Create a simple file on the system with a couple of entries, something > like "hi" and "hi2" separated by newlines. > 3. Run this hive command, pointing it at the file: LOAD DATA LOCAL INPATH > '' OVERWRITE INTO TABLE jvaughan_test PARTITION (day = '2014-01-02'); > 4. Confirm the data with: SELECT * FROM jvaughan_test WHERE day = > '2014-01-02'; > 5. Alter the column definitions: ALTER TABLE jvaughan_test REPLACE COLUMNS > (col1 string, col2 string); > 6. Edit your file and add a second column using the default separator > (ctrl+v, then ctrl+a in Vim) and add two more entries, such as "hi3" on the > first row and "hi4" on the second > 7. Run step 3 again > 8. Check the data again like in step 4 > For me, this is the results that get returned: > hive> select * from jvaughan_test where day = '2014-01-01'; > OK > hiNULL2014-01-02 > hi2 NULL2014-01-02 > This is despite the fact that there is data in the file stored by the > partition in HDFS. > Let me know if you need any other information. The only workaround for me > currently is to drop partitions for any I'm replacing data in and THEN > reupload the new data file. > Thanks, > -James -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13293) Query occurs performance degradation after enabling parallel order by for Hive on Spark
[ https://issues.apache.org/jira/browse/HIVE-13293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285944#comment-15285944 ] Rui Li commented on HIVE-13293: --- Hi [~xuefuz], any further comments on this one? > Query occurs performance degradation after enabling parallel order by for > Hive on Spark > --- > > Key: HIVE-13293 > URL: https://issues.apache.org/jira/browse/HIVE-13293 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 2.0.0 >Reporter: Lifeng Wang >Assignee: Rui Li > Attachments: HIVE-13293.1.patch, HIVE-13293.2.patch, > HIVE-13293.3.patch, HIVE-13293.3.patch, HIVE-13293.3.patch > > > I use TPCx-BB to do some performance test on Hive on Spark engine. And found > query 10 has performance degradation when enabling parallel order by. > It seems that sampling cost much time before running the real query. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10815) Let HiveMetaStoreClient Choose MetaStore Randomly
[ https://issues.apache.org/jira/browse/HIVE-10815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285941#comment-15285941 ] Nemon Lou commented on HIVE-10815: -- Seems that it fail to attract any volunteer to review. Shall I rebase it to master? > Let HiveMetaStoreClient Choose MetaStore Randomly > - > > Key: HIVE-10815 > URL: https://issues.apache.org/jira/browse/HIVE-10815 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Metastore >Affects Versions: 1.2.0 >Reporter: Nemon Lou >Assignee: Nemon Lou > Attachments: HIVE-10815.patch > > > Currently HiveMetaStoreClient using a fixed order to choose MetaStore URIs > when multiple metastores configured. > Choosing MetaStore Randomly will be good for load balance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13622) WriteSet tracking optimizations
[
https://issues.apache.org/jira/browse/HIVE-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285939#comment-15285939
]
Eugene Koifman commented on HIVE-13622:
---
Test Results
{noformat}
Test Name Duration Age
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
5 min 32 sec1
org.apache.hadoop.hive.llap.tez.TestLlapDaemonProtocolClientProxy.testSingleInvocationPerNode
5 sec 1
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
10 sec 1
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testConnections 87 ms
1
org.apache.hive.jdbc.TestSSL.testSSLConnectionWithURL 32 sec 1
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testPreemptionQueueComparator
5 sec 1
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority
5.3 sec 4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_orig_table_use_metadata
41 sec 6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32 8.4 sec 6
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static
1 min 57 sec7
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
1 min 15 sec7
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForMemoryTokenStore
1.2 sec 7
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForDBTokenStore
0.32 sec7
org.apache.hive.minikdc.TestMiniHiveKdc.testLogin 1 min 49 sec7
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
54 ms 27
org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle
28 sec 27
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
5.1 sec 40
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_selectindate13 sec
59
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avrocountemptytbl
9.6 sec 59
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order_null 46 sec
59
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys
1 min 36 sec59
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
9.7 sec 59
org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver
1 min 16 sec59
{noformat}
Failures are not related
> WriteSet tracking optimizations
> ---
>
> Key: HIVE-13622
> URL: https://issues.apache.org/jira/browse/HIVE-13622
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.3.0, 2.1.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Critical
> Attachments: HIVE-13622.2.patch, HIVE-13622.3.patch,
> HIVE-13622.4.patch
>
>
> HIVE-13395 solves the the lost update problem with some inefficiencies.
> 1. TxhHandler.OperationType is currently derived from LockType. This doesn't
> distinguish between Update and Delete but would be useful. See comments in
> TxnHandler. Should be able to pass in Insert/Update/Delete info from client
> into TxnHandler.
> 2. TxnHandler.addDynamicPartitions() should know the OperationType as well
> from the client. It currently extrapolates it from TXN_COMPONENTS. This
> works but requires extra SQL statements and is thus less performant. It will
> not work multi-stmt txns. See comments in the code.
> 3. TxnHandler.checkLock() see more comments around
> "isPartOfDynamicPartitionInsert". If TxnHandler knew whether it is being
> called as part of an op running with dynamic partitions, it could be more
> efficient. In that case we don't have to write to TXN_COMPONENTS at all
> during lock acquisition. Conversely, if not running with DynPart then, we
> can kill current txn on lock grant rather than wait until commit time.
> 4. TxnHandler.addDynamicPartitions() - the insert stmt here should combing
> multiple rows into single SQL stmt (but with a limit for extreme cases)
> 5. TxnHandler.enqueueLockWithRetry() - this currently adds components that
> are only being read to TXN_COMPONENTS. This is useless at best since read
> op don't generate anything to compact. For example, delete from T where t1
> in (select c1 from C) - no reason to add C to txn_components but we do.
>
> All of these require some Thrift changes
> Once done, re-enable TestDbTxnHandler2.testWriteSetTracking11()
> Also see comments in
>
[jira] [Commented] (HIVE-13622) WriteSet tracking optimizations
[ https://issues.apache.org/jira/browse/HIVE-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285940#comment-15285940 ] Eugene Koifman commented on HIVE-13622: --- [~alangates] could you review please > WriteSet tracking optimizations > --- > > Key: HIVE-13622 > URL: https://issues.apache.org/jira/browse/HIVE-13622 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0, 2.1.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-13622.2.patch, HIVE-13622.3.patch, > HIVE-13622.4.patch > > > HIVE-13395 solves the the lost update problem with some inefficiencies. > 1. TxhHandler.OperationType is currently derived from LockType. This doesn't > distinguish between Update and Delete but would be useful. See comments in > TxnHandler. Should be able to pass in Insert/Update/Delete info from client > into TxnHandler. > 2. TxnHandler.addDynamicPartitions() should know the OperationType as well > from the client. It currently extrapolates it from TXN_COMPONENTS. This > works but requires extra SQL statements and is thus less performant. It will > not work multi-stmt txns. See comments in the code. > 3. TxnHandler.checkLock() see more comments around > "isPartOfDynamicPartitionInsert". If TxnHandler knew whether it is being > called as part of an op running with dynamic partitions, it could be more > efficient. In that case we don't have to write to TXN_COMPONENTS at all > during lock acquisition. Conversely, if not running with DynPart then, we > can kill current txn on lock grant rather than wait until commit time. > 4. TxnHandler.addDynamicPartitions() - the insert stmt here should combing > multiple rows into single SQL stmt (but with a limit for extreme cases) > 5. TxnHandler.enqueueLockWithRetry() - this currently adds components that > are only being read to TXN_COMPONENTS. This is useless at best since read > op don't generate anything to compact. For example, delete from T where t1 > in (select c1 from C) - no reason to add C to txn_components but we do. > > All of these require some Thrift changes > Once done, re-enable TestDbTxnHandler2.testWriteSetTracking11() > Also see comments in > [here|https://issues.apache.org/jira/browse/HIVE-13395?focusedCommentId=15271712=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15271712] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13774) add actual access checks to GenericUDTFGetSplits
[ https://issues.apache.org/jira/browse/HIVE-13774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285872#comment-15285872 ] Sergey Shelukhin commented on HIVE-13774: - cc [~jdere] > add actual access checks to GenericUDTFGetSplits > > > Key: HIVE-13774 > URL: https://issues.apache.org/jira/browse/HIVE-13774 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin > > GenericUDTFGetSplits generates splits without proper Hive compilation. We > need to actually perform security checks (Driver.doAuthorization) before we > sign and return the splits -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13750) Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer when possible
[
https://issues.apache.org/jira/browse/HIVE-13750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285853#comment-15285853
]
Hive QA commented on HIVE-13750:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12804193/HIVE-13750.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/302/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/302/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-302/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-maven-3.0.5/bin:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-maven-3.0.5/bin:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-302/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at e738914 Remove unintended import that caused build failure for
JDK 8 in commit 4533d21b0be487e1f11fcc95578a2ba103e72a64 HIVE-13682:
EOFException with fast hashtable (Matt McCline, reviewed by Sergey Shelukhin)
+ git clean -f -d
+ git checkout master
Already on 'master'
+ git reset --hard origin/master
HEAD is now at e738914 Remove unintended import that caused build failure for
JDK 8 in commit 4533d21b0be487e1f11fcc95578a2ba103e72a64 HIVE-13682:
EOFException with fast hashtable (Matt McCline, reviewed by Sergey Shelukhin)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12804193 - PreCommit-HIVE-MASTER-Build
> Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer
> when possible
> --
>
> Key: HIVE-13750
> URL: https://issues.apache.org/jira/browse/HIVE-13750
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13750.patch, HIVE-13750.patch
>
>
> Extend ReduceDedup to remove additional shuffle stage created by sorted
> dynamic partition optimizer when possible, thus avoiding unnecessary work.
> By [~ashutoshc]:
> {quote}
> Currently, if config is on Sorted Dynamic Partition Optimizer (SDPO)
> unconditionally adds an extra shuffle stage. If sort columns of previous
> shuffle and partitioning columns of table match, reduce sink deduplication
> optimizer removes extra shuffle stage, thus bringing down overhead to zero.
> However, if they don’t match, we end up doing extra shuffle. This can be
> improved since we can add table partition columns as a sort columns on
> earlier shuffle and avoid this extra shuffle. This ensures that in cases
> query already has a shuffle stage, we are not
[jira] [Commented] (HIVE-9405) Upgrade to Hadoop 2.7 when released
[
https://issues.apache.org/jira/browse/HIVE-9405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285848#comment-15285848
]
Hive QA commented on HIVE-9405:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12804190/HIVE-9405.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 604 failed/errored test(s), 10055 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-explainuser_4.q-update_after_multiple_inserts.q-mapreduce2.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-join1.q-mapjoin_decimal.q-union5.q-and-12-more - did not
produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_move_tbl
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_add_part_multiple
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_alter_merge_orc
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_alter_merge_stats_orc
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_annotate_stats_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join0
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join11
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join12
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join13
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join14
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join15
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join16
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join17
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join18
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join18_multi_distinct
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join19
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join20
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join21
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join22
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join23
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join24
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join26
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join27
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join28
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join29
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join30
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join31
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join4
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join5
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join6
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join7
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join8
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join9
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_filters
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_nulls
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_reordering_values
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_stats
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_stats2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_without_localtask
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_smb_mapjoin_14
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_12
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_13
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_14
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_15
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_16
[jira] [Commented] (HIVE-10815) Let HiveMetaStoreClient Choose MetaStore Randomly
[ https://issues.apache.org/jira/browse/HIVE-10815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285834#comment-15285834 ] Sergey Shelukhin commented on HIVE-10815: - Has this patch been abandoned? > Let HiveMetaStoreClient Choose MetaStore Randomly > - > > Key: HIVE-10815 > URL: https://issues.apache.org/jira/browse/HIVE-10815 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Metastore >Affects Versions: 1.2.0 >Reporter: Nemon Lou >Assignee: Nemon Lou > Attachments: HIVE-10815.patch > > > Currently HiveMetaStoreClient using a fixed order to choose MetaStore URIs > when multiple metastores configured. > Choosing MetaStore Randomly will be good for load balance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13449) LLAP: HS2 should get the token directly, rather than from LLAP
[ https://issues.apache.org/jira/browse/HIVE-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13449: Attachment: HIVE-13449.04.patch Recent review feedback > LLAP: HS2 should get the token directly, rather than from LLAP > -- > > Key: HIVE-13449 > URL: https://issues.apache.org/jira/browse/HIVE-13449 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13449.01.patch, HIVE-13449.02.WIP.patch, > HIVE-13449.02.patch, HIVE-13449.03.patch, HIVE-13449.04.patch, > HIVE-13449.patch > > > HS2 doesn't need a roundtrip to LLAP; it can instantiate the SecretManager > directly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13698) refactor TezSession-s to support shared state
[ https://issues.apache.org/jira/browse/HIVE-13698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285764#comment-15285764 ] Sergey Shelukhin commented on HIVE-13698: - cc [~sseth] > refactor TezSession-s to support shared state > - > > Key: HIVE-13698 > URL: https://issues.apache.org/jira/browse/HIVE-13698 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin > > It would be nice to have shared state for the sessions, to reuse things like > clients to get tokens, etc. > Since the shared state is only useful in multi-session scenario, i.e. HS2, > the existing session creation paths thru the session pool can just be made > cleaner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13306) Better Decimal vectorization
[
https://issues.apache.org/jira/browse/HIVE-13306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Teddy Choi updated HIVE-13306:
--
Attachment: HIVE-13306.1.patch
It's a working draft. It shows 70x addition performance, 3x multiplication and
2x division performance regarding to existing implementations. I will modify
this code further for wider use cases and more performance and more
readability. Thanks. :)
{noformat}
# Run complete. Total time: 00:02:30
Benchmark
Mode SamplesScore Error Units
o.a.h.b.v.VectorizedArithmeticBench.DecimalColAddDecimalColColumnBench.bench
avgt2 4012665235.500 ± NaN ns/op
o.a.h.b.v.VectorizedArithmeticBench.DecimalColDivideDecimalColColumnBench.bench
avgt2 19167315269.000 ± NaN ns/op
o.a.h.b.v.VectorizedArithmeticBench.DecimalColMultiplyDecimalColColumnBench.bench
avgt2 3391096996.500 ± NaN ns/op
o.a.h.b.v.VectorizedArithmeticBench.DecimalV2ColAddDecimalColColumnBench.bench
avgt2 56848247.500 ± NaN ns/op
o.a.h.b.v.VectorizedArithmeticBench.DecimalV2ColDivideDecimalColColumnBench.bench
avgt2 9162374089.500 ± NaN ns/op
o.a.h.b.v.VectorizedArithmeticBench.DecimalV2ColMultiplyDecimalColColumnBench.bench
avgt2 1146261770.500 ± NaN ns/op
{noformat}
> Better Decimal vectorization
>
>
> Key: HIVE-13306
> URL: https://issues.apache.org/jira/browse/HIVE-13306
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Teddy Choi
>Priority: Critical
> Attachments: HIVE-13306.1.patch
>
>
> Decimal Vectorization Requirements
> • Today, the LongColumnVector, DoubleColumnVector, BytesColumnVector,
> TimestampColumnVector classes store the data as primitive Java data types
> long, double, or byte arrays for efficiency.
> • DecimalColumnVector is different - it has an array of Object references
> to HiveDecimal objects.
> • The HiveDecimal object uses an internal object BigDecimal for its
> implementation. Further, BigDecimal itself uses an internal object
> BigInteger for its implementation, and BigInteger uses an int array. 4
> objects total.
> • And, HiveDecimal is an immutable object which means arithmetic and
> other operations produce new HiveDecimal object with 3 new objects underneath.
> • A major reason Vectorization is fast is the ColumnVector classes except
> DecimalColumnVector do not have to allocate additional memory per row. This
> avoids memory fragmentation and pressure on the Java Garbage Collector that
> DecimalColumnVector can generate. It is very significant.
> • What can be done with DecimalColumnVector to make it much more
> efficient?
> o Design several new decimal classes that allow the caller to manage the
> decimal storage.
> o If it takes N int values to store a decimal (e.g. N=1..5), then a new
> DecimalColumnVector would have an int[] of length N*1024 (where 1024 is the
> default column vector size).
> o Why store a decimal in separate int values?
> • Java does not support 128 bit integers.
> • Java does not support unsigned integers.
> • In order to do multiplication of a decimal represented in a long you
> need twice the storage (i.e. 128 bits). So you need to represent parts in 32
> bit integers.
> • But really since we do not have unsigned, really you can only do
> multiplications on N-1 bits or 31 bits.
> • So, 5 ints are needed for decimal storage... of 38 digits.
> o It makes sense to have just one algorithm for decimals rather than one
> for HiveDecimal and another for DecimalColumnVector. So, make HiveDecimal
> store N int values, too.
> o A lower level primitive decimal class would accept decimals stored as
> int arrays and produces results into int arrays. It would be used by
> HiveDecimal and DecimalColumnVector.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HIVE-13306) Better Decimal vectorization
[ https://issues.apache.org/jira/browse/HIVE-13306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-13306 started by Teddy Choi. - > Better Decimal vectorization > > > Key: HIVE-13306 > URL: https://issues.apache.org/jira/browse/HIVE-13306 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Teddy Choi >Priority: Critical > > Decimal Vectorization Requirements > • Today, the LongColumnVector, DoubleColumnVector, BytesColumnVector, > TimestampColumnVector classes store the data as primitive Java data types > long, double, or byte arrays for efficiency. > • DecimalColumnVector is different - it has an array of Object references > to HiveDecimal objects. > • The HiveDecimal object uses an internal object BigDecimal for its > implementation. Further, BigDecimal itself uses an internal object > BigInteger for its implementation, and BigInteger uses an int array. 4 > objects total. > • And, HiveDecimal is an immutable object which means arithmetic and > other operations produce new HiveDecimal object with 3 new objects underneath. > • A major reason Vectorization is fast is the ColumnVector classes except > DecimalColumnVector do not have to allocate additional memory per row. This > avoids memory fragmentation and pressure on the Java Garbage Collector that > DecimalColumnVector can generate. It is very significant. > • What can be done with DecimalColumnVector to make it much more > efficient? > o Design several new decimal classes that allow the caller to manage the > decimal storage. > o If it takes N int values to store a decimal (e.g. N=1..5), then a new > DecimalColumnVector would have an int[] of length N*1024 (where 1024 is the > default column vector size). > o Why store a decimal in separate int values? > • Java does not support 128 bit integers. > • Java does not support unsigned integers. > • In order to do multiplication of a decimal represented in a long you > need twice the storage (i.e. 128 bits). So you need to represent parts in 32 > bit integers. > • But really since we do not have unsigned, really you can only do > multiplications on N-1 bits or 31 bits. > • So, 5 ints are needed for decimal storage... of 38 digits. > o It makes sense to have just one algorithm for decimals rather than one > for HiveDecimal and another for DecimalColumnVector. So, make HiveDecimal > store N int values, too. > o A lower level primitive decimal class would accept decimals stored as > int arrays and produces results into int arrays. It would be used by > HiveDecimal and DecimalColumnVector. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12331) Remove hive.enforce.bucketing & hive.enforce.sorting configs
[ https://issues.apache.org/jira/browse/HIVE-12331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285696#comment-15285696 ] Shannon Ladymon commented on HIVE-12331: Doc done. Removed TODOC 2.0 label. > Remove hive.enforce.bucketing & hive.enforce.sorting configs > > > Key: HIVE-12331 > URL: https://issues.apache.org/jira/browse/HIVE-12331 > Project: Hive > Issue Type: Improvement > Components: Configuration >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Fix For: 2.0.0 > > Attachments: HIVE-12331.1.patch, HIVE-12331.patch > > > If table is created as bucketed and/or sorted and this config is set to > false, you will insert data in wrong buckets and/or sort order and then if > you use these tables subsequently in BMJ or SMBJ you will get wrong results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12331) Remove hive.enforce.bucketing & hive.enforce.sorting configs
[ https://issues.apache.org/jira/browse/HIVE-12331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-12331: -- Labels: (was: TODOC2.0) > Remove hive.enforce.bucketing & hive.enforce.sorting configs > > > Key: HIVE-12331 > URL: https://issues.apache.org/jira/browse/HIVE-12331 > Project: Hive > Issue Type: Improvement > Components: Configuration >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Fix For: 2.0.0 > > Attachments: HIVE-12331.1.patch, HIVE-12331.patch > > > If table is created as bucketed and/or sorted and this config is set to > false, you will insert data in wrong buckets and/or sort order and then if > you use these tables subsequently in BMJ or SMBJ you will get wrong results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13549) Remove jdk version specific out files from Hive2
[ https://issues.apache.org/jira/browse/HIVE-13549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285691#comment-15285691 ] Mohit Sabharwal commented on HIVE-13549: Confirmed that test failures exclude version specific files. > Remove jdk version specific out files from Hive2 > > > Key: HIVE-13549 > URL: https://issues.apache.org/jira/browse/HIVE-13549 > Project: Hive > Issue Type: Sub-task >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-13549-1-java8.patch, HIVE-13549-java8.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13708) Create table should verify datatypes supported by the serde
[
https://issues.apache.org/jira/browse/HIVE-13708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13708:
-
Status: Patch Available (was: Open)
> Create table should verify datatypes supported by the serde
> ---
>
> Key: HIVE-13708
> URL: https://issues.apache.org/jira/browse/HIVE-13708
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Thejas M Nair
>Assignee: Hari Sankar Sivarama Subramaniyan
>Priority: Critical
> Attachments: HIVE-13708.1.patch, HIVE-13708.2.patch
>
>
> As [~Goldshuv] mentioned in HIVE-.
> Create table with serde such as OpenCSVSerde allows for creation of table
> with columns of arbitrary types. But 'describe table' would still return
> string datatypes, and so does selects on the table.
> This is misleading and would result in users not getting intended results.
> The create table ideally should disallow the creation of such tables with
> unsupported types.
> Example posted by [~Goldshuv] in HIVE- -
> {noformat}
> CREATE EXTERNAL TABLE test (totalprice DECIMAL(38,10))
> ROW FORMAT SERDE 'com.bizo.hive.serde.csv.CSVSerde' with
> serdeproperties ("separatorChar" = ",","quoteChar"= "'","escapeChar"= "\\")
> STORED AS TEXTFILE
> LOCATION ''
> tblproperties ("skip.header.line.count"="1");
> {noformat}
> Now consider this sql:
> hive> select min(totalprice) from test;
> in this case given my data, the result should have been 874.89, but the
> actual result became 11.57 (as it is first according to byte ordering of
> a string type). this is a wrong result.
> hive> desc extended test;
> OK
> o_totalprice string from deserializer
> ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13708) Create table should verify datatypes supported by the serde
[
https://issues.apache.org/jira/browse/HIVE-13708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13708:
-
Attachment: HIVE-13708.2.patch
> Create table should verify datatypes supported by the serde
> ---
>
> Key: HIVE-13708
> URL: https://issues.apache.org/jira/browse/HIVE-13708
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Thejas M Nair
>Assignee: Hari Sankar Sivarama Subramaniyan
>Priority: Critical
> Attachments: HIVE-13708.1.patch, HIVE-13708.2.patch
>
>
> As [~Goldshuv] mentioned in HIVE-.
> Create table with serde such as OpenCSVSerde allows for creation of table
> with columns of arbitrary types. But 'describe table' would still return
> string datatypes, and so does selects on the table.
> This is misleading and would result in users not getting intended results.
> The create table ideally should disallow the creation of such tables with
> unsupported types.
> Example posted by [~Goldshuv] in HIVE- -
> {noformat}
> CREATE EXTERNAL TABLE test (totalprice DECIMAL(38,10))
> ROW FORMAT SERDE 'com.bizo.hive.serde.csv.CSVSerde' with
> serdeproperties ("separatorChar" = ",","quoteChar"= "'","escapeChar"= "\\")
> STORED AS TEXTFILE
> LOCATION ''
> tblproperties ("skip.header.line.count"="1");
> {noformat}
> Now consider this sql:
> hive> select min(totalprice) from test;
> in this case given my data, the result should have been 874.89, but the
> actual result became 11.57 (as it is first according to byte ordering of
> a string type). this is a wrong result.
> hive> desc extended test;
> OK
> o_totalprice string from deserializer
> ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13708) Create table should verify datatypes supported by the serde
[
https://issues.apache.org/jira/browse/HIVE-13708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13708:
-
Status: Open (was: Patch Available)
> Create table should verify datatypes supported by the serde
> ---
>
> Key: HIVE-13708
> URL: https://issues.apache.org/jira/browse/HIVE-13708
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Thejas M Nair
>Assignee: Hari Sankar Sivarama Subramaniyan
>Priority: Critical
> Attachments: HIVE-13708.1.patch
>
>
> As [~Goldshuv] mentioned in HIVE-.
> Create table with serde such as OpenCSVSerde allows for creation of table
> with columns of arbitrary types. But 'describe table' would still return
> string datatypes, and so does selects on the table.
> This is misleading and would result in users not getting intended results.
> The create table ideally should disallow the creation of such tables with
> unsupported types.
> Example posted by [~Goldshuv] in HIVE- -
> {noformat}
> CREATE EXTERNAL TABLE test (totalprice DECIMAL(38,10))
> ROW FORMAT SERDE 'com.bizo.hive.serde.csv.CSVSerde' with
> serdeproperties ("separatorChar" = ",","quoteChar"= "'","escapeChar"= "\\")
> STORED AS TEXTFILE
> LOCATION ''
> tblproperties ("skip.header.line.count"="1");
> {noformat}
> Now consider this sql:
> hive> select min(totalprice) from test;
> in this case given my data, the result should have been 874.89, but the
> actual result became 11.57 (as it is first according to byte ordering of
> a string type). this is a wrong result.
> hive> desc extended test;
> OK
> o_totalprice string from deserializer
> ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (HIVE-13691) No record with CQ_ID=0 found in COMPACTION_QUEUE
[
https://issues.apache.org/jira/browse/HIVE-13691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman reopened HIVE-13691:
---
> No record with CQ_ID=0 found in COMPACTION_QUEUE
>
>
> Key: HIVE-13691
> URL: https://issues.apache.org/jira/browse/HIVE-13691
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 1.3.0, 2.1.0
>
>
> {noformat}
> 2016-04-29 18:49:31,594 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(141)) - Caught exception while trying to determine if we
> should compact id:0,dbname:default,tableName:service_logs_v2,par
> tName:ds=2016-04-21,state:^@,type:null,runAs:null,tooManyAborts:false,highestTxnId:0.
> Marking clean to avoid repeated failures, MetaException(message:Timeout
> when executing method: getTable)
> at
> org.apache.hadoop.hive.metastore.Deadline.newMetaException(Deadline.java:187)
> at org.apache.hadoop.hive.metastore.Deadline.check(Deadline.java:177)
> at
> org.apache.hadoop.hive.metastore.Deadline.checkTimeout(Deadline.java:160)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToParts(ObjectStore.java:1839)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsViaOrmFilter(ObjectStore.java:2255)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$300(ObjectStore.java:165)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$3.getJdoResult(ObjectStore.java:2051)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$3.getJdoResult(ObjectStore.java:2043)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2400)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNamesInternal(ObjectStore.java:2043)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNames(ObjectStore.java:2037)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy0.getPartitionsByNames(Unknown Source)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorThread.resolvePartition(CompactorThread.java:111)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:129)
> Caused by: org.apache.hadoop.hive.metastore.DeadlineException: Timeout when
> executing method: getTable
> at org.apache.hadoop.hive.metastore.Deadline.check(Deadline.java:174)
> ... 16 more
> 2016-04-29 18:49:31,595 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(154)) - Initiator loop caught unexpected exception this
> time through the loop: java.lang.IllegalStateException: No record with
> CQ_ID=0 found in COMPACTION_QUEUE
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:861)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:144)
> {noformat}
> {noformat}
> 2016-04-29 18:49:31,595 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(154)) - Initiator loop caught unexpected exception this
> time through the loop: java.lang.IllegalStateException: No record with
> CQ_ID=0 found in COMPACTION_QUEUE
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:861)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:144)
> {noformat}
> is triggered by _DeadlineException: Timeout when executing method_ but is
> nonetheless an issue.
> We should be able to record an entry in completed_compaction_queue to
> represent a failed compaction even if an entry in compaction_queue was never
> made, as is the case here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-13691) No record with CQ_ID=0 found in COMPACTION_QUEUE
[
https://issues.apache.org/jira/browse/HIVE-13691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman resolved HIVE-13691.
---
Resolution: Fixed
> No record with CQ_ID=0 found in COMPACTION_QUEUE
>
>
> Key: HIVE-13691
> URL: https://issues.apache.org/jira/browse/HIVE-13691
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 1.3.0, 2.1.0
>
>
> {noformat}
> 2016-04-29 18:49:31,594 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(141)) - Caught exception while trying to determine if we
> should compact id:0,dbname:default,tableName:service_logs_v2,par
> tName:ds=2016-04-21,state:^@,type:null,runAs:null,tooManyAborts:false,highestTxnId:0.
> Marking clean to avoid repeated failures, MetaException(message:Timeout
> when executing method: getTable)
> at
> org.apache.hadoop.hive.metastore.Deadline.newMetaException(Deadline.java:187)
> at org.apache.hadoop.hive.metastore.Deadline.check(Deadline.java:177)
> at
> org.apache.hadoop.hive.metastore.Deadline.checkTimeout(Deadline.java:160)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToParts(ObjectStore.java:1839)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsViaOrmFilter(ObjectStore.java:2255)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$300(ObjectStore.java:165)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$3.getJdoResult(ObjectStore.java:2051)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$3.getJdoResult(ObjectStore.java:2043)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2400)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNamesInternal(ObjectStore.java:2043)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNames(ObjectStore.java:2037)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy0.getPartitionsByNames(Unknown Source)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorThread.resolvePartition(CompactorThread.java:111)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:129)
> Caused by: org.apache.hadoop.hive.metastore.DeadlineException: Timeout when
> executing method: getTable
> at org.apache.hadoop.hive.metastore.Deadline.check(Deadline.java:174)
> ... 16 more
> 2016-04-29 18:49:31,595 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(154)) - Initiator loop caught unexpected exception this
> time through the loop: java.lang.IllegalStateException: No record with
> CQ_ID=0 found in COMPACTION_QUEUE
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:861)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:144)
> {noformat}
> {noformat}
> 2016-04-29 18:49:31,595 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(154)) - Initiator loop caught unexpected exception this
> time through the loop: java.lang.IllegalStateException: No record with
> CQ_ID=0 found in COMPACTION_QUEUE
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:861)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:144)
> {noformat}
> is triggered by _DeadlineException: Timeout when executing method_ but is
> nonetheless an issue.
> We should be able to record an entry in completed_compaction_queue to
> represent a failed compaction even if an entry in compaction_queue was never
> made, as is the case here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work stopped] (HIVE-13691) No record with CQ_ID=0 found in COMPACTION_QUEUE
[
https://issues.apache.org/jira/browse/HIVE-13691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-13691 stopped by Eugene Koifman.
-
> No record with CQ_ID=0 found in COMPACTION_QUEUE
>
>
> Key: HIVE-13691
> URL: https://issues.apache.org/jira/browse/HIVE-13691
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 1.3.0, 2.1.0
>
>
> {noformat}
> 2016-04-29 18:49:31,594 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(141)) - Caught exception while trying to determine if we
> should compact id:0,dbname:default,tableName:service_logs_v2,par
> tName:ds=2016-04-21,state:^@,type:null,runAs:null,tooManyAborts:false,highestTxnId:0.
> Marking clean to avoid repeated failures, MetaException(message:Timeout
> when executing method: getTable)
> at
> org.apache.hadoop.hive.metastore.Deadline.newMetaException(Deadline.java:187)
> at org.apache.hadoop.hive.metastore.Deadline.check(Deadline.java:177)
> at
> org.apache.hadoop.hive.metastore.Deadline.checkTimeout(Deadline.java:160)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToParts(ObjectStore.java:1839)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsViaOrmFilter(ObjectStore.java:2255)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$300(ObjectStore.java:165)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$3.getJdoResult(ObjectStore.java:2051)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$3.getJdoResult(ObjectStore.java:2043)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2400)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNamesInternal(ObjectStore.java:2043)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNames(ObjectStore.java:2037)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy0.getPartitionsByNames(Unknown Source)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorThread.resolvePartition(CompactorThread.java:111)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:129)
> Caused by: org.apache.hadoop.hive.metastore.DeadlineException: Timeout when
> executing method: getTable
> at org.apache.hadoop.hive.metastore.Deadline.check(Deadline.java:174)
> ... 16 more
> 2016-04-29 18:49:31,595 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(154)) - Initiator loop caught unexpected exception this
> time through the loop: java.lang.IllegalStateException: No record with
> CQ_ID=0 found in COMPACTION_QUEUE
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:861)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:144)
> {noformat}
> {noformat}
> 2016-04-29 18:49:31,595 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(154)) - Initiator loop caught unexpected exception this
> time through the loop: java.lang.IllegalStateException: No record with
> CQ_ID=0 found in COMPACTION_QUEUE
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:861)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:144)
> {noformat}
> is triggered by _DeadlineException: Timeout when executing method_ but is
> nonetheless an issue.
> We should be able to record an entry in completed_compaction_queue to
> represent a failed compaction even if an entry in compaction_queue was never
> made, as is the case here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HIVE-13691) No record with CQ_ID=0 found in COMPACTION_QUEUE
[
https://issues.apache.org/jira/browse/HIVE-13691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-13691 started by Eugene Koifman.
-
> No record with CQ_ID=0 found in COMPACTION_QUEUE
>
>
> Key: HIVE-13691
> URL: https://issues.apache.org/jira/browse/HIVE-13691
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 1.3.0, 2.1.0
>
>
> {noformat}
> 2016-04-29 18:49:31,594 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(141)) - Caught exception while trying to determine if we
> should compact id:0,dbname:default,tableName:service_logs_v2,par
> tName:ds=2016-04-21,state:^@,type:null,runAs:null,tooManyAborts:false,highestTxnId:0.
> Marking clean to avoid repeated failures, MetaException(message:Timeout
> when executing method: getTable)
> at
> org.apache.hadoop.hive.metastore.Deadline.newMetaException(Deadline.java:187)
> at org.apache.hadoop.hive.metastore.Deadline.check(Deadline.java:177)
> at
> org.apache.hadoop.hive.metastore.Deadline.checkTimeout(Deadline.java:160)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToParts(ObjectStore.java:1839)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsViaOrmFilter(ObjectStore.java:2255)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$300(ObjectStore.java:165)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$3.getJdoResult(ObjectStore.java:2051)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$3.getJdoResult(ObjectStore.java:2043)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2400)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNamesInternal(ObjectStore.java:2043)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNames(ObjectStore.java:2037)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy0.getPartitionsByNames(Unknown Source)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorThread.resolvePartition(CompactorThread.java:111)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:129)
> Caused by: org.apache.hadoop.hive.metastore.DeadlineException: Timeout when
> executing method: getTable
> at org.apache.hadoop.hive.metastore.Deadline.check(Deadline.java:174)
> ... 16 more
> 2016-04-29 18:49:31,595 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(154)) - Initiator loop caught unexpected exception this
> time through the loop: java.lang.IllegalStateException: No record with
> CQ_ID=0 found in COMPACTION_QUEUE
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:861)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:144)
> {noformat}
> {noformat}
> 2016-04-29 18:49:31,595 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(154)) - Initiator loop caught unexpected exception this
> time through the loop: java.lang.IllegalStateException: No record with
> CQ_ID=0 found in COMPACTION_QUEUE
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:861)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:144)
> {noformat}
> is triggered by _DeadlineException: Timeout when executing method_ but is
> nonetheless an issue.
> We should be able to record an entry in completed_compaction_queue to
> represent a failed compaction even if an entry in compaction_queue was never
> made, as is the case here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13651) LlapBaseInputFormat: figure out where credentials come from
[
https://issues.apache.org/jira/browse/HIVE-13651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285635#comment-15285635
]
Sergey Shelukhin commented on HIVE-13651:
-
Actually, I think this is better done separately. LLAP token and non-LLAP
tokens come from different places
> LlapBaseInputFormat: figure out where credentials come from
> ---
>
> Key: HIVE-13651
> URL: https://issues.apache.org/jira/browse/HIVE-13651
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Jason Dere
>
> todo in LlapBaseInputFormat.constructSubmitWorkRequestProto()
> {code}
> // TODO Figure out where credentials will come from. Normally Hive sets up
> // URLs on the tez dag, for which Tez acquires credentials.
> //taskCredentials.addAll(getContext().getCredentials());
> //
> Preconditions.checkState(currentQueryIdentifierProto.getDagIdentifier() ==
> //
> taskSpec.getTaskAttemptID().getTaskID().getVertexID().getDAGId().getId());
> //ByteBuffer credentialsBinary =
> credentialMap.get(currentQueryIdentifierProto);
> //if (credentialsBinary == null) {
> // credentialsBinary =
> serializeCredentials(getContext().getCredentials());
> // credentialMap.putIfAbsent(currentQueryIdentifierProto,
> credentialsBinary.duplicate());
> //} else {
> // credentialsBinary = credentialsBinary.duplicate();
> //}
> //
> builder.setCredentialsBinary(ByteString.copyFrom(credentialsBinary));
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13651) LlapBaseInputFormat: figure out where credentials come from
[
https://issues.apache.org/jira/browse/HIVE-13651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-13651:
Assignee: (was: Sergey Shelukhin)
> LlapBaseInputFormat: figure out where credentials come from
> ---
>
> Key: HIVE-13651
> URL: https://issues.apache.org/jira/browse/HIVE-13651
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Jason Dere
>
> todo in LlapBaseInputFormat.constructSubmitWorkRequestProto()
> {code}
> // TODO Figure out where credentials will come from. Normally Hive sets up
> // URLs on the tez dag, for which Tez acquires credentials.
> //taskCredentials.addAll(getContext().getCredentials());
> //
> Preconditions.checkState(currentQueryIdentifierProto.getDagIdentifier() ==
> //
> taskSpec.getTaskAttemptID().getTaskID().getVertexID().getDAGId().getId());
> //ByteBuffer credentialsBinary =
> credentialMap.get(currentQueryIdentifierProto);
> //if (credentialsBinary == null) {
> // credentialsBinary =
> serializeCredentials(getContext().getCredentials());
> // credentialMap.putIfAbsent(currentQueryIdentifierProto,
> credentialsBinary.duplicate());
> //} else {
> // credentialsBinary = credentialsBinary.duplicate();
> //}
> //
> builder.setCredentialsBinary(ByteString.copyFrom(credentialsBinary));
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (HIVE-13731) LLAP: return LLAP token with the splits
[ https://issues.apache.org/jira/browse/HIVE-13731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reopened HIVE-13731: - > LLAP: return LLAP token with the splits > --- > > Key: HIVE-13731 > URL: https://issues.apache.org/jira/browse/HIVE-13731 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > Need to return the token with the splits, then take it in LLAPIF and make > sure it's used when talking to LLAP -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13731) LLAP: return LLAP token with the splits
[ https://issues.apache.org/jira/browse/HIVE-13731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-13731: --- Assignee: Sergey Shelukhin > LLAP: return LLAP token with the splits > --- > > Key: HIVE-13731 > URL: https://issues.apache.org/jira/browse/HIVE-13731 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > Need to return the token with the splits, then take it in LLAPIF and make > sure it's used when talking to LLAP -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13651) LlapBaseInputFormat: figure out where credentials come from
[
https://issues.apache.org/jira/browse/HIVE-13651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin reassigned HIVE-13651:
---
Assignee: Sergey Shelukhin
> LlapBaseInputFormat: figure out where credentials come from
> ---
>
> Key: HIVE-13651
> URL: https://issues.apache.org/jira/browse/HIVE-13651
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Jason Dere
>Assignee: Sergey Shelukhin
>
> todo in LlapBaseInputFormat.constructSubmitWorkRequestProto()
> {code}
> // TODO Figure out where credentials will come from. Normally Hive sets up
> // URLs on the tez dag, for which Tez acquires credentials.
> //taskCredentials.addAll(getContext().getCredentials());
> //
> Preconditions.checkState(currentQueryIdentifierProto.getDagIdentifier() ==
> //
> taskSpec.getTaskAttemptID().getTaskID().getVertexID().getDAGId().getId());
> //ByteBuffer credentialsBinary =
> credentialMap.get(currentQueryIdentifierProto);
> //if (credentialsBinary == null) {
> // credentialsBinary =
> serializeCredentials(getContext().getCredentials());
> // credentialMap.putIfAbsent(currentQueryIdentifierProto,
> credentialsBinary.duplicate());
> //} else {
> // credentialsBinary = credentialsBinary.duplicate();
> //}
> //
> builder.setCredentialsBinary(ByteString.copyFrom(credentialsBinary));
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13771) LLAPIF: generate app ID
[ https://issues.apache.org/jira/browse/HIVE-13771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285603#comment-15285603 ] Sergey Shelukhin commented on HIVE-13771: - cc [~jdere] > LLAPIF: generate app ID > --- > > Key: HIVE-13771 > URL: https://issues.apache.org/jira/browse/HIVE-13771 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin > > See comments in the HIVE-13675 patch. The uniqueness needs to be ensured; the > user may be allowed to supply a prefix (e.g. his YARN app Id, if any) for > ease of tracking -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13675) LLAP: add HMAC signatures to LLAPIF splits
[ https://issues.apache.org/jira/browse/HIVE-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13675: Assignee: Sergey Shelukhin Status: Patch Available (was: Open) > LLAP: add HMAC signatures to LLAPIF splits > -- > > Key: HIVE-13675 > URL: https://issues.apache.org/jira/browse/HIVE-13675 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13675.WIP.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13675) LLAP: add HMAC signatures to LLAPIF splits
[ https://issues.apache.org/jira/browse/HIVE-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13675: Attachment: HIVE-13675.WIP.patch Most of the old patches were inapplicable due to various changes during branch merge and in other JIRAs. Here's a mostly-finished patch on top of HIVE-13444. There are too many patches outstanding to complete this before they are committed. cc [~jdere] [~sseth] > LLAP: add HMAC signatures to LLAPIF splits > -- > > Key: HIVE-13675 > URL: https://issues.apache.org/jira/browse/HIVE-13675 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin > Attachments: HIVE-13675.WIP.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13680) HiveServer2: Provide a way to compress ResultSets
[ https://issues.apache.org/jira/browse/HIVE-13680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285570#comment-15285570 ] Kevin Liew commented on HIVE-13680: --- Ah, that makes more sense. I was apprehensive of that approach which is why I researched and included points about security options. I'll revise the document. > HiveServer2: Provide a way to compress ResultSets > - > > Key: HIVE-13680 > URL: https://issues.apache.org/jira/browse/HIVE-13680 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Reporter: Vaibhav Gumashta >Assignee: Kevin Liew > Attachments: proposal.pdf > > > With HIVE-12049 in, we can provide an option to compress ResultSets before > writing to disk. The user can specify a compression library via a config > param which can be used in the tasks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13502) Beeline doesnt support session parameters in JDBC URL as documentation states.
[
https://issues.apache.org/jira/browse/HIVE-13502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285568#comment-15285568
]
Hive QA commented on HIVE-13502:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12804179/HIVE-13502.3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/300/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/300/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-300/
Messages:
{noformat}
This message was trimmed, see log for full details
[WARNING]
/data/hive-ptest/working/apache-github-source-source/itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java:
Some input files use or override a deprecated API.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java:
Recompile with -Xlint:deprecation for details.
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @
hive-it-util ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/hive-ptest/working/apache-github-source-source/itests/util/src/test/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-it-util ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/itests/util/target/tmp
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/itests/util/target/warehouse
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/itests/util/target/tmp/conf
[copy] Copying 15 files to
/data/hive-ptest/working/apache-github-source-source/itests/util/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
hive-it-util ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-it-util ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-it-util ---
[INFO] Building jar:
/data/hive-ptest/working/apache-github-source-source/itests/util/target/hive-it-util-2.1.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
hive-it-util ---
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-it-util ---
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/itests/util/target/hive-it-util-2.1.0-SNAPSHOT.jar
to
/home/hiveptest/.m2/repository/org/apache/hive/hive-it-util/2.1.0-SNAPSHOT/hive-it-util-2.1.0-SNAPSHOT.jar
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/itests/util/pom.xml to
/home/hiveptest/.m2/repository/org/apache/hive/hive-it-util/2.1.0-SNAPSHOT/hive-it-util-2.1.0-SNAPSHOT.pom
[INFO]
[INFO]
[INFO] Building Hive Integration - Unit Tests 2.1.0-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-it-unit ---
[INFO] Deleting
/data/hive-ptest/working/apache-github-source-source/itests/hive-unit/target
[INFO] Deleting
/data/hive-ptest/working/apache-github-source-source/itests/hive-unit (includes
= [datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @
hive-it-unit ---
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (download-spark) @ hive-it-unit ---
[INFO] Executing tasks
main:
[exec] + /bin/pwd
[exec] + BASE_DIR=./target
[exec] + HIVE_ROOT=./target/../../../
[exec] + DOWNLOAD_DIR=./../thirdparty
[exec] + mkdir -p ./../thirdparty
[exec]
/data/hive-ptest/working/apache-github-source-source/itests/hive-unit
[exec] + download
http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz
spark
[exec] +
url=http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz
[exec] + finalName=spark
[exec] ++ basename
http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz
[exec] + tarName=spark-1.6.0-bin-hadoop2-without-hive.tgz
[exec] + rm -rf ./target/spark
[exec] + [[ ! -f ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz
]]
[exec] + tar -zxf ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz
-C ./target
[exec] + mv ./target/spark-1.6.0-bin-hadoop2-without-hive
[jira] [Commented] (HIVE-13549) Remove jdk version specific out files from Hive2
[
https://issues.apache.org/jira/browse/HIVE-13549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285565#comment-15285565
]
Hive QA commented on HIVE-13549:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12804217/HIVE-13549-java8.patch
{color:green}SUCCESS:{color} +1 due to 29 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 196 failed/errored test(s), 8819 tests
executed
*Failed tests:*
{noformat}
TestCliDriver-gen_udf_example_add10.q-ppd_join4.q-union27.q-and-12-more - did
not produce a TEST-*.xml file
TestCliDriver-partition_timestamp.q-ppd_random.q-vector_outer_join5.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-ptf_general_queries.q-unionDistinct_1.q-groupby1_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
TestJdbcWithMiniHA - did not produce a TEST-*.xml file
TestJdbcWithMiniMr - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_sortmerge_join_7.q-tez_union_group_by.q-orc_merge9.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-bucket_map_join_tez1.q-auto_sortmerge_join_16.q-skewjoin.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-constprog_dpp.q-dynamic_partition_pruning.q-vectorization_10.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-enforce_order.q-vector_partition_diff_num_cols.q-unionDistinct_1.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-explainuser_4.q-update_after_multiple_inserts.q-mapreduce2.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-groupby2.q-tez_dynpart_hashjoin_1.q-custom_input_output_format.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-insert_values_non_partitioned.q-join1.q-schema_evol_orc_nonvec_mapwork_part.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-mapjoin_mapjoin.q-insert_into1.q-vector_decimal_2.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-schema_evol_text_nonvec_mapwork_table.q-vector_decimal_trailing.q-subquery_in.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-smb_cache.q-transform_ppr2.q-vector_outer_join0.q-and-3-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_auto_smb_mapjoin_14.q-vectorized_distinct_gby.q-union_fast_stats.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_coalesce.q-cbo_windowing.q-tez_join.q-and-12-more -
did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_interval_2.q-schema_evol_text_nonvec_mapwork_part_all_primitive.q-tez_fsstat.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorization_13.q-auto_sortmerge_join_13.q-tez_bmj_schema_evolution.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorized_parquet.q-orc_merge4.q-orc_merge_incompat1.q-and-12-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-bucket6.q-scriptfile1_win.q-quotedid_smb.q-and-1-more - did
not produce a TEST-*.xml file
TestMinimrCliDriver-join1.q-infer_bucket_sort_bucketed_table.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestNegativeCliDriver-udf_invalid.q-nopart_insert.q-insert_into_with_schema.q-and-726-more
- did not produce a TEST-*.xml file
TestOperationLoggingAPIWithTez - did not produce a TEST-*.xml file
TestSparkCliDriver-bucketmapjoin3.q-enforce_order.q-union11.q-and-12-more - did
not produce a TEST-*.xml file
TestSparkCliDriver-bucketsortoptimize_insert_7.q-smb_mapjoin_15.q-mapreduce1.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-groupby2.q-custom_input_output_format.q-join41.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-groupby6_map.q-join13.q-join_reorder3.q-and-12-more - did
not produce a TEST-*.xml file
TestSparkCliDriver-join_cond_pushdown_unqual4.q-bucketmapjoin12.q-avro_decimal_native.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ptf_rcfile.q-bucketmapjoin_negative.q-bucket_map_join_spark2.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-script_pipe.q-stats12.q-auto_join24.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-skewjoin_noskew.q-sample2.q-skewjoinopt10.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-skewjoin_union_remove_2.q-timestamp_null.q-union32.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-skewjoinopt3.q-union27.q-multigroupby_singlemr.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-stats13.q-stats2.q-ppd_gby_join.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-union_remove_23.q-transform_ppr2.q-join20.q-and-12-more -
did not produce a TEST-*.xml file
[jira] [Commented] (HIVE-13197) Add adapted constprog2.q and constprog_partitioner.q tests back
[ https://issues.apache.org/jira/browse/HIVE-13197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285385#comment-15285385 ] Ashutosh Chauhan commented on HIVE-13197: - [~jcamachorodriguez] Can you take a quick look at this one? > Add adapted constprog2.q and constprog_partitioner.q tests back > --- > > Key: HIVE-13197 > URL: https://issues.apache.org/jira/browse/HIVE-13197 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0, 2.0.1 >Reporter: Jesus Camacho Rodriguez >Assignee: Ashutosh Chauhan > Attachments: HIVE-13197.patch > > > HIVE-12749 removes constprog2.q and constprog_partitioner.q tests, as they > did not test constant propagation anymore. Ideally, we should create them > again with compatible types to test constant propagation and constant > propagation in the presence of partitions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13680) HiveServer2: Provide a way to compress ResultSets
[ https://issues.apache.org/jira/browse/HIVE-13680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285383#comment-15285383 ] Vaibhav Gumashta commented on HIVE-13680: - [~kliew] I didn't mean the client should send the plugin jar - just the name. Also, I meant that as part of the opensession call, the server can send back the configured compressor name (snappy for example) which the client can use if available. [~alangates] That's not the proposed path forward. > HiveServer2: Provide a way to compress ResultSets > - > > Key: HIVE-13680 > URL: https://issues.apache.org/jira/browse/HIVE-13680 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Reporter: Vaibhav Gumashta >Assignee: Kevin Liew > Attachments: proposal.pdf > > > With HIVE-12049 in, we can provide an option to compress ResultSets before > writing to disk. The user can specify a compression library via a config > param which can be used in the tasks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13662) Set file permission and ACL in file sink operator
[ https://issues.apache.org/jira/browse/HIVE-13662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285379#comment-15285379 ] Ashutosh Chauhan commented on HIVE-13662: - FileSinkOperator uses HiveFileFormatUtils.getHiveRecordWriter() which calls HiveOutputFormat::getHiveRecordWriter() which will eventually call FileSystem.create() which accepts permissions as an argument. That is where we need to pass in permissions so that we have permissions set at the time of file creation itself, so that we can avoid two calls on Namenode. > Set file permission and ACL in file sink operator > - > > Key: HIVE-13662 > URL: https://issues.apache.org/jira/browse/HIVE-13662 > Project: Hive > Issue Type: Bug >Reporter: Rui Li >Assignee: Pengcheng Xiong > Attachments: HIVE-13662.01.patch > > > As suggested > [here|https://issues.apache.org/jira/browse/HIVE-13572?focusedCommentId=15254438=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15254438]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13680) HiveServer2: Provide a way to compress ResultSets
[ https://issues.apache.org/jira/browse/HIVE-13680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285371#comment-15285371 ] Alan Gates commented on HIVE-13680: --- bq. If the server does not have that compressor plugin, it will request the jar from the client. The client should send the compressor plugin and all dependencies. We should not do this. It is a code injection security issue. We cannot have users sending code into HS2 and the task nodes without full security controls. In this case it makes more sense to make sure that we provide compressors for the standard options Hadoop already supports (snappy, lzo, ...) Can the existing (de)compressors already in use in ORC etc. be used for this rather than re-implementing? > HiveServer2: Provide a way to compress ResultSets > - > > Key: HIVE-13680 > URL: https://issues.apache.org/jira/browse/HIVE-13680 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Reporter: Vaibhav Gumashta >Assignee: Kevin Liew > Attachments: proposal.pdf > > > With HIVE-12049 in, we can provide an option to compress ResultSets before > writing to disk. The user can specify a compression library via a config > param which can be used in the tasks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13566) Auto-gather column stats - phase 1
[ https://issues.apache.org/jira/browse/HIVE-13566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285348#comment-15285348 ] Pengcheng Xiong commented on HIVE-13566: coool, thanks. :) > Auto-gather column stats - phase 1 > -- > > Key: HIVE-13566 > URL: https://issues.apache.org/jira/browse/HIVE-13566 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13566.01.patch > > > This jira adds code and tests for auto-gather column stats. Golden file > update will be done in phase 2 - HIVE-11160 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13566) Auto-gather column stats - phase 1
[ https://issues.apache.org/jira/browse/HIVE-13566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13566: Description: This jira adds code and tests for auto-gather column stats. Golden file update will be done in phase 2 - HIVE-11160 > Auto-gather column stats - phase 1 > -- > > Key: HIVE-13566 > URL: https://issues.apache.org/jira/browse/HIVE-13566 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13566.01.patch > > > This jira adds code and tests for auto-gather column stats. Golden file > update will be done in phase 2 - HIVE-11160 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13566) Auto-gather column stats - phase 1
[ https://issues.apache.org/jira/browse/HIVE-13566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285344#comment-15285344 ] Ashutosh Chauhan commented on HIVE-13566: - Updated title and description to reflect that,. > Auto-gather column stats - phase 1 > -- > > Key: HIVE-13566 > URL: https://issues.apache.org/jira/browse/HIVE-13566 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13566.01.patch > > > This jira adds code and tests for auto-gather column stats. Golden file > update will be done in phase 2 - HIVE-11160 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13566) Auto-gather column stats - phase 1
[ https://issues.apache.org/jira/browse/HIVE-13566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13566: Summary: Auto-gather column stats - phase 1 (was: enable merging of bit vectors for insert into) > Auto-gather column stats - phase 1 > -- > > Key: HIVE-13566 > URL: https://issues.apache.org/jira/browse/HIVE-13566 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13566.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13566) enable merging of bit vectors for insert into
[ https://issues.apache.org/jira/browse/HIVE-13566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285338#comment-15285338 ] Pengcheng Xiong commented on HIVE-13566: [~ashutoshc], that is true (excluding the thrift change, and some other API changes). And I will have a follow-up jira to update golden files. > enable merging of bit vectors for insert into > - > > Key: HIVE-13566 > URL: https://issues.apache.org/jira/browse/HIVE-13566 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13566.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13662) Set file permission and ACL in file sink operator
[ https://issues.apache.org/jira/browse/HIVE-13662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13662: --- Status: Patch Available (was: Open) > Set file permission and ACL in file sink operator > - > > Key: HIVE-13662 > URL: https://issues.apache.org/jira/browse/HIVE-13662 > Project: Hive > Issue Type: Bug >Reporter: Rui Li >Assignee: Pengcheng Xiong > Attachments: HIVE-13662.01.patch > > > As suggested > [here|https://issues.apache.org/jira/browse/HIVE-13572?focusedCommentId=15254438=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15254438]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13662) Set file permission and ACL in file sink operator
[ https://issues.apache.org/jira/browse/HIVE-13662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13662: --- Attachment: HIVE-13662.01.patch > Set file permission and ACL in file sink operator > - > > Key: HIVE-13662 > URL: https://issues.apache.org/jira/browse/HIVE-13662 > Project: Hive > Issue Type: Bug >Reporter: Rui Li >Assignee: Pengcheng Xiong > Attachments: HIVE-13662.01.patch > > > As suggested > [here|https://issues.apache.org/jira/browse/HIVE-13572?focusedCommentId=15254438=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15254438]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13566) enable merging of bit vectors for insert into
[ https://issues.apache.org/jira/browse/HIVE-13566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285330#comment-15285330 ] Ashutosh Chauhan commented on HIVE-13566: - Attached patch seems to indicate its implementing HIVE-11160 instead. > enable merging of bit vectors for insert into > - > > Key: HIVE-13566 > URL: https://issues.apache.org/jira/browse/HIVE-13566 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13566.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13269) Simplify comparison expressions using column stats
[ https://issues.apache.org/jira/browse/HIVE-13269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285299#comment-15285299 ] Ashutosh Chauhan commented on HIVE-13269: - Patch looks good. Few minor comments on RB. > Simplify comparison expressions using column stats > -- > > Key: HIVE-13269 > URL: https://issues.apache.org/jira/browse/HIVE-13269 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13269.01.patch, HIVE-13269.02.patch, > HIVE-13269.03.patch, HIVE-13269.patch, HIVE-13269.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13566) enable merging of bit vectors for insert into
[ https://issues.apache.org/jira/browse/HIVE-13566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13566: --- Status: Patch Available (was: Open) > enable merging of bit vectors for insert into > - > > Key: HIVE-13566 > URL: https://issues.apache.org/jira/browse/HIVE-13566 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13566.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13767) Wrong type inferred in Semijoin condition leads to AssertionError
[
https://issues.apache.org/jira/browse/HIVE-13767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285267#comment-15285267
]
Ashutosh Chauhan commented on HIVE-13767:
-
Thanks for explanation!
+1 pending tests.
> Wrong type inferred in Semijoin condition leads to AssertionError
> -
>
> Key: HIVE-13767
> URL: https://issues.apache.org/jira/browse/HIVE-13767
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13767.patch
>
>
> Following query fails to run:
> {noformat}
> SELECT
> COALESCE(498, LEAD(COALESCE(-973, -684, 515)) OVER (PARTITION BY
> (t2.int_col_10 + t1.smallint_col_50) ORDER BY (t2.int_col_10 +
> t1.smallint_col_50), FLOOR(t1.double_col_16) DESC), 524) AS int_col,
> (t2.int_col_10) + (t1.smallint_col_50) AS int_col_1,
> FLOOR(t1.double_col_16) AS float_col,
> COALESCE(SUM(COALESCE(62, -380, -435)) OVER (PARTITION BY (t2.int_col_10
> + t1.smallint_col_50) ORDER BY (t2.int_col_10 + t1.smallint_col_50) DESC,
> FLOOR(t1.double_col_16) DESC ROWS BETWEEN UNBOUNDED PRECEDING AND 48
> FOLLOWING), 704) AS int_col_2
> FROM table_1 t1
> INNER JOIN table_18 t2 ON (((t2.tinyint_col_15) = (t1.bigint_col_7)) AND
>((t2.decimal2709_col_9) =
> (t1.decimal2016_col_26))) AND
>((t2.tinyint_col_20) = (t1.tinyint_col_3))
> WHERE (t2.smallint_col_19) IN (SELECT
> COALESCE(-92, -994) AS int_col
> FROM table_1 tt1
> INNER JOIN table_18 tt2 ON (tt2.decimal1911_col_16) =
> (tt1.decimal2612_col_77)
> WHERE (t1.timestamp_col_9) = (tt2.timestamp_col_18));
> {noformat}
> Following error is seen in the logs:
> {noformat}
> 2016-04-27T04:32:09,605 WARN [...2a24 HiveServer2-Handler-Pool: Thread-211]:
> thrift.ThriftCLIService (ThriftCLIService.java:ExecuteStatement(501)) - Error
> executing statement:
> org.apache.hive.service.cli.HiveSQLException: Error running query:
> java.lang.AssertionError: mismatched type $8 TIMESTAMP(9)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:178)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:216)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:327)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:458)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:435)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:272)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:492)
> [hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1317)
> [hive-service-rpc-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1302)
> [hive-service-rpc-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> [hive-exec-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> [hive-exec-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> [hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> [hive-exec-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_77]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_77]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> Caused by: java.lang.AssertionError: mismatched type $8 TIMESTAMP(9)
> at
> org.apache.calcite.rex.RexUtil$FixNullabilityShuttle.visitInputRef(RexUtil.java:2042)
> ~[calcite-core-1.6.0.2.5.0.0-248.jar:1.6.0.2.5.0.0-248]
> at
>
[jira] [Updated] (HIVE-13736) View's input/output formats are TEXT by default
[ https://issues.apache.org/jira/browse/HIVE-13736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-13736: Status: Patch Available (was: Open) Need code review. > View's input/output formats are TEXT by default > --- > > Key: HIVE-13736 > URL: https://issues.apache.org/jira/browse/HIVE-13736 > Project: Hive > Issue Type: New Feature > Components: Hive >Affects Versions: 1.2.1 >Reporter: Pavas Garg >Assignee: Yongzhi Chen >Priority: Minor > Attachments: HIVE-13736.1.patch > > > Feature request where Hive View's input/output formats are text by default in > order to help 3rd party compatibility -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13736) View's input/output formats are TEXT by default
[ https://issues.apache.org/jira/browse/HIVE-13736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-13736: Attachment: HIVE-13736.1.patch > View's input/output formats are TEXT by default > --- > > Key: HIVE-13736 > URL: https://issues.apache.org/jira/browse/HIVE-13736 > Project: Hive > Issue Type: New Feature > Components: Hive >Affects Versions: 1.2.1 >Reporter: Pavas Garg >Assignee: Yongzhi Chen >Priority: Minor > Attachments: HIVE-13736.1.patch > > > Feature request where Hive View's input/output formats are text by default in > order to help 3rd party compatibility -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13736) View's input/output formats are TEXT by default
[ https://issues.apache.org/jira/browse/HIVE-13736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285162#comment-15285162 ] Yongzhi Chen commented on HIVE-13736: - In the current implementation, view file format is hard-coded as Sequencefile. I will implement view default format the same way as table and index: Getting the default format for view from hive.default.fileformat and hive.default.fileformat.managed If hive.default.fileformat.managed is not none, use its value as default fileformat, otherwise use hive.default.fileformat's value. The default value for hive.default.fileformat.managed is None The default value for hive.default.fileformat is TextFile. > View's input/output formats are TEXT by default > --- > > Key: HIVE-13736 > URL: https://issues.apache.org/jira/browse/HIVE-13736 > Project: Hive > Issue Type: New Feature > Components: Hive >Affects Versions: 1.2.1 >Reporter: Pavas Garg >Assignee: Yongzhi Chen >Priority: Minor > > Feature request where Hive View's input/output formats are text by default in > order to help 3rd party compatibility -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13562) Enable vector bridge for all non-vectorized udfs
[
https://issues.apache.org/jira/browse/HIVE-13562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-13562:
Status: Patch Available (was: In Progress)
Try again...
> Enable vector bridge for all non-vectorized udfs
>
>
> Key: HIVE-13562
> URL: https://issues.apache.org/jira/browse/HIVE-13562
> Project: Hive
> Issue Type: Improvement
> Components: Vectorization
>Reporter: Ashutosh Chauhan
>Assignee: Matt McCline
> Attachments: HIVE-13562.01.patch, HIVE-13562.02.patch
>
>
> Mechanism already exists for this via {{VectorUDFAdaptor}} but we have
> arbitrarily hand picked few udfs to go through it. I think we should enable
> this by default for all udfs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13562) Enable vector bridge for all non-vectorized udfs
[
https://issues.apache.org/jira/browse/HIVE-13562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-13562:
Attachment: HIVE-13562.02.patch
> Enable vector bridge for all non-vectorized udfs
>
>
> Key: HIVE-13562
> URL: https://issues.apache.org/jira/browse/HIVE-13562
> Project: Hive
> Issue Type: Improvement
> Components: Vectorization
>Reporter: Ashutosh Chauhan
>Assignee: Matt McCline
> Attachments: HIVE-13562.01.patch, HIVE-13562.02.patch
>
>
> Mechanism already exists for this via {{VectorUDFAdaptor}} but we have
> arbitrarily hand picked few udfs to go through it. I think we should enable
> this by default for all udfs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13562) Enable vector bridge for all non-vectorized udfs
[
https://issues.apache.org/jira/browse/HIVE-13562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-13562:
Status: In Progress (was: Patch Available)
> Enable vector bridge for all non-vectorized udfs
>
>
> Key: HIVE-13562
> URL: https://issues.apache.org/jira/browse/HIVE-13562
> Project: Hive
> Issue Type: Improvement
> Components: Vectorization
>Reporter: Ashutosh Chauhan
>Assignee: Matt McCline
> Attachments: HIVE-13562.01.patch, HIVE-13562.02.patch
>
>
> Mechanism already exists for this via {{VectorUDFAdaptor}} but we have
> arbitrarily hand picked few udfs to go through it. I think we should enable
> this by default for all udfs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[
https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285130#comment-15285130
]
Hive QA commented on HIVE-13068:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12804134/HIVE-13068.04.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 44 failed/errored test(s), 9959 tests
executed
*Failed tests:*
{noformat}
TestFileSinkOperator - did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_sortmerge_join_16.q-skewjoin.q-vectorization_div0.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-constprog_dpp.q-dynamic_partition_pruning.q-vectorization_10.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-join1.q-mapjoin_decimal.q-union5.q-and-12-more - did not
produce a TEST-*.xml file
TestMiniTezCliDriver-schema_evol_text_nonvec_mapwork_table.q-vector_decimal_trailing.q-subquery_in.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-script_pipe.q-vector_decimal_aggregate.q-vector_data_types.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-smb_cache.q-transform_ppr2.q-vector_outer_join0.q-and-5-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_coalesce.q-cbo_windowing.q-tez_join.q-and-12-more -
did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_interval_2.q-schema_evol_text_nonvec_mapwork_part_all_primitive.q-tez_fsstat.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-union_remove_23.q-transform_ppr2.q-join20.q-and-12-more -
did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_annotate_stats_groupby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constant_prop_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_join_preds
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_quotedid_basic
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join17
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_cbo_subq_in
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_disable_merge_for_bucketing
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby2_noskew
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby5_map_skew
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_input_part2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapjoin1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_nullgroup
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_join2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_sample2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_skewjoin_noskew
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats14
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_statsfs
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union6
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_6
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_pushdown
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorized_mapjoin
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/299/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/299/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-299/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 44 tests failed
{noformat}
This message is automatically
[jira] [Commented] (HIVE-13622) WriteSet tracking optimizations
[ https://issues.apache.org/jira/browse/HIVE-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285109#comment-15285109 ] Eugene Koifman commented on HIVE-13622: --- [~teabot] could you look at this patch please? I made changes to Lock and TestLock but they are not perfect. I now need the client to set additional properties on LockComponent. One to indicate the operation type (CRUD) and another to indicate if the resource is acid or not. I made changes in Lock conservatively to make sure old behavior is preserved but ideally it should differentiate all 4 operations. It wasn't obvious to me how to get that information. > WriteSet tracking optimizations > --- > > Key: HIVE-13622 > URL: https://issues.apache.org/jira/browse/HIVE-13622 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0, 2.1.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-13622.2.patch, HIVE-13622.3.patch, > HIVE-13622.4.patch > > > HIVE-13395 solves the the lost update problem with some inefficiencies. > 1. TxhHandler.OperationType is currently derived from LockType. This doesn't > distinguish between Update and Delete but would be useful. See comments in > TxnHandler. Should be able to pass in Insert/Update/Delete info from client > into TxnHandler. > 2. TxnHandler.addDynamicPartitions() should know the OperationType as well > from the client. It currently extrapolates it from TXN_COMPONENTS. This > works but requires extra SQL statements and is thus less performant. It will > not work multi-stmt txns. See comments in the code. > 3. TxnHandler.checkLock() see more comments around > "isPartOfDynamicPartitionInsert". If TxnHandler knew whether it is being > called as part of an op running with dynamic partitions, it could be more > efficient. In that case we don't have to write to TXN_COMPONENTS at all > during lock acquisition. Conversely, if not running with DynPart then, we > can kill current txn on lock grant rather than wait until commit time. > 4. TxnHandler.addDynamicPartitions() - the insert stmt here should combing > multiple rows into single SQL stmt (but with a limit for extreme cases) > 5. TxnHandler.enqueueLockWithRetry() - this currently adds components that > are only being read to TXN_COMPONENTS. This is useless at best since read > op don't generate anything to compact. For example, delete from T where t1 > in (select c1 from C) - no reason to add C to txn_components but we do. > > All of these require some Thrift changes > Once done, re-enable TestDbTxnHandler2.testWriteSetTracking11() > Also see comments in > [here|https://issues.apache.org/jira/browse/HIVE-13395?focusedCommentId=15271712=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15271712] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13652) Import table change order of dynamic partitions
[
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285104#comment-15285104
]
Sushanth Sowmyan commented on HIVE-13652:
-
> In any case current behaviour is the worst possible as it leaves partitioning
> on "higher will"
I laughed at this, and you're definitely right about this.
The rest is merely context behind why this problem has not been tackled earlier
(since we break it for some users at the very least), and honestly, if we don't
tackle it now, we simply kick the ball further down the road, and it will need
tackling one way or another, which we have been doing so far.
> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Lukas Waldmann
>
> Table with multiple dynamic partitions like year,month, day exported using
> "export table" command is imported (using "import table") such a way that
> order of partitions is changed to day, month, year.
> Export DB: Hive 0.14
> Import DB: Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1 to 'path'
> import command:
> import table t1 from 'path'
> HDFS file structure on both original table location and export path keeps the
> original partition order ../year/month/day
> HDFS file structure after import is .../day/month/year
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13622) WriteSet tracking optimizations
[ https://issues.apache.org/jira/browse/HIVE-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-13622: -- Attachment: HIVE-13622.4.patch > WriteSet tracking optimizations > --- > > Key: HIVE-13622 > URL: https://issues.apache.org/jira/browse/HIVE-13622 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0, 2.1.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-13622.2.patch, HIVE-13622.3.patch, > HIVE-13622.4.patch > > > HIVE-13395 solves the the lost update problem with some inefficiencies. > 1. TxhHandler.OperationType is currently derived from LockType. This doesn't > distinguish between Update and Delete but would be useful. See comments in > TxnHandler. Should be able to pass in Insert/Update/Delete info from client > into TxnHandler. > 2. TxnHandler.addDynamicPartitions() should know the OperationType as well > from the client. It currently extrapolates it from TXN_COMPONENTS. This > works but requires extra SQL statements and is thus less performant. It will > not work multi-stmt txns. See comments in the code. > 3. TxnHandler.checkLock() see more comments around > "isPartOfDynamicPartitionInsert". If TxnHandler knew whether it is being > called as part of an op running with dynamic partitions, it could be more > efficient. In that case we don't have to write to TXN_COMPONENTS at all > during lock acquisition. Conversely, if not running with DynPart then, we > can kill current txn on lock grant rather than wait until commit time. > 4. TxnHandler.addDynamicPartitions() - the insert stmt here should combing > multiple rows into single SQL stmt (but with a limit for extreme cases) > 5. TxnHandler.enqueueLockWithRetry() - this currently adds components that > are only being read to TXN_COMPONENTS. This is useless at best since read > op don't generate anything to compact. For example, delete from T where t1 > in (select c1 from C) - no reason to add C to txn_components but we do. > > All of these require some Thrift changes > Once done, re-enable TestDbTxnHandler2.testWriteSetTracking11() > Also see comments in > [here|https://issues.apache.org/jira/browse/HIVE-13395?focusedCommentId=15271712=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15271712] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13760) Add a HIVE_QUERY_TIMEOUT configuration to kill a query if a query is running for more than the configured timeout value.
[ https://issues.apache.org/jira/browse/HIVE-13760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285048#comment-15285048 ] Thejas M Nair commented on HIVE-13760: -- cc [~vgumashta] > Add a HIVE_QUERY_TIMEOUT configuration to kill a query if a query is running > for more than the configured timeout value. > > > Key: HIVE-13760 > URL: https://issues.apache.org/jira/browse/HIVE-13760 > Project: Hive > Issue Type: Improvement > Components: Configuration >Affects Versions: 2.0.0 >Reporter: zhihai xu >Assignee: zhihai xu > > Add a HIVE_QUERY_TIMEOUT configuration to kill a query if a query is running > for more than the configured timeout value. The default value will be -1 , > which means no timeout. This will be useful for user to manage queries with > SLA. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13767) Wrong type inferred in Semijoin condition leads to AssertionError
[
https://issues.apache.org/jira/browse/HIVE-13767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285000#comment-15285000
]
Jesus Camacho Rodriguez commented on HIVE-13767:
The method will push any equi join conditions that are not column references as
Projections on top of the children of the join, and will return the new
condition.
The problem is that the position of the columns is accounted incorrectly, and
thus we end up inferring incorrect type for the column.
Let me go into detail in the code.
Observe the loop in L197-L211. It extracts which conditions are column
references, and which ones need to pushed to the inputs of the join. Assume we
have a condition {{CAST(a)=b AND c=d}}. First conjunct is added to the columns
that need to be added to the inputs (because of CAST), while second conjunct is
added to the columns that do not need to be pushed i.e. they can be directly
referenced.
Then loop in L213-L229 creates the first part of the condition consisting of
the equality conditions that do not need to be pushed i.e. {{c=d}} in our
example. But observe that _leftKey_ and _rightKey_, which are used to inferred
the type, are extracted from _leftJoinKeys_ and _rightJoinKeys_ respectively,
using index _i_ from origColEqConds... This is not right, as _leftKey_ will
reference {{CAST(a)}} and _rightKey_ will reference {{b}}. That is to say, the
condition that do not need to be pushed is at _i=1_.
Thus, we need the keep the positions of _leftJoinKeys_ and _rightJoinKeys_ that
contain conditions that do not need to be pushed: we keep this information in
the new _origColEqCondsPos_ bitset.
> Wrong type inferred in Semijoin condition leads to AssertionError
> -
>
> Key: HIVE-13767
> URL: https://issues.apache.org/jira/browse/HIVE-13767
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13767.patch
>
>
> Following query fails to run:
> {noformat}
> SELECT
> COALESCE(498, LEAD(COALESCE(-973, -684, 515)) OVER (PARTITION BY
> (t2.int_col_10 + t1.smallint_col_50) ORDER BY (t2.int_col_10 +
> t1.smallint_col_50), FLOOR(t1.double_col_16) DESC), 524) AS int_col,
> (t2.int_col_10) + (t1.smallint_col_50) AS int_col_1,
> FLOOR(t1.double_col_16) AS float_col,
> COALESCE(SUM(COALESCE(62, -380, -435)) OVER (PARTITION BY (t2.int_col_10
> + t1.smallint_col_50) ORDER BY (t2.int_col_10 + t1.smallint_col_50) DESC,
> FLOOR(t1.double_col_16) DESC ROWS BETWEEN UNBOUNDED PRECEDING AND 48
> FOLLOWING), 704) AS int_col_2
> FROM table_1 t1
> INNER JOIN table_18 t2 ON (((t2.tinyint_col_15) = (t1.bigint_col_7)) AND
>((t2.decimal2709_col_9) =
> (t1.decimal2016_col_26))) AND
>((t2.tinyint_col_20) = (t1.tinyint_col_3))
> WHERE (t2.smallint_col_19) IN (SELECT
> COALESCE(-92, -994) AS int_col
> FROM table_1 tt1
> INNER JOIN table_18 tt2 ON (tt2.decimal1911_col_16) =
> (tt1.decimal2612_col_77)
> WHERE (t1.timestamp_col_9) = (tt2.timestamp_col_18));
> {noformat}
> Following error is seen in the logs:
> {noformat}
> 2016-04-27T04:32:09,605 WARN [...2a24 HiveServer2-Handler-Pool: Thread-211]:
> thrift.ThriftCLIService (ThriftCLIService.java:ExecuteStatement(501)) - Error
> executing statement:
> org.apache.hive.service.cli.HiveSQLException: Error running query:
> java.lang.AssertionError: mismatched type $8 TIMESTAMP(9)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:178)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:216)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:327)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:458)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:435)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:272)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:492)
> [hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1317)
>
[jira] [Commented] (HIVE-10448) LLAP: Consider replacing BytesBytesMultiHashMap with new fast hash table code of Native Vector Map Join
[ https://issues.apache.org/jira/browse/HIVE-10448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284990#comment-15284990 ] Sergey Shelukhin commented on HIVE-10448: - The intent of this JIRA was to remove BytesBytes completely... is that covered by HIVE-9824? > LLAP: Consider replacing BytesBytesMultiHashMap with new fast hash table code > of Native Vector Map Join > --- > > Key: HIVE-10448 > URL: https://issues.apache.org/jira/browse/HIVE-10448 > Project: Hive > Issue Type: Sub-task >Reporter: Matt McCline >Assignee: Matt McCline > > [~sershe] observed that the new fast hash table code developed for Native > Vector Map Join uses many of the same concepts as BytesBytesMultiHashMap can > could be considered a v2 replacement for it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13622) WriteSet tracking optimizations
[ https://issues.apache.org/jira/browse/HIVE-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-13622: -- Attachment: HIVE-13622.3.patch > WriteSet tracking optimizations > --- > > Key: HIVE-13622 > URL: https://issues.apache.org/jira/browse/HIVE-13622 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0, 2.1.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-13622.2.patch, HIVE-13622.3.patch > > > HIVE-13395 solves the the lost update problem with some inefficiencies. > 1. TxhHandler.OperationType is currently derived from LockType. This doesn't > distinguish between Update and Delete but would be useful. See comments in > TxnHandler. Should be able to pass in Insert/Update/Delete info from client > into TxnHandler. > 2. TxnHandler.addDynamicPartitions() should know the OperationType as well > from the client. It currently extrapolates it from TXN_COMPONENTS. This > works but requires extra SQL statements and is thus less performant. It will > not work multi-stmt txns. See comments in the code. > 3. TxnHandler.checkLock() see more comments around > "isPartOfDynamicPartitionInsert". If TxnHandler knew whether it is being > called as part of an op running with dynamic partitions, it could be more > efficient. In that case we don't have to write to TXN_COMPONENTS at all > during lock acquisition. Conversely, if not running with DynPart then, we > can kill current txn on lock grant rather than wait until commit time. > 4. TxnHandler.addDynamicPartitions() - the insert stmt here should combing > multiple rows into single SQL stmt (but with a limit for extreme cases) > 5. TxnHandler.enqueueLockWithRetry() - this currently adds components that > are only being read to TXN_COMPONENTS. This is useless at best since read > op don't generate anything to compact. For example, delete from T where t1 > in (select c1 from C) - no reason to add C to txn_components but we do. > > All of these require some Thrift changes > Once done, re-enable TestDbTxnHandler2.testWriteSetTracking11() > Also see comments in > [here|https://issues.apache.org/jira/browse/HIVE-13395?focusedCommentId=15271712=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15271712] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13566) enable merging of bit vectors for insert into
[ https://issues.apache.org/jira/browse/HIVE-13566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13566: --- Attachment: HIVE-13566.01.patch > enable merging of bit vectors for insert into > - > > Key: HIVE-13566 > URL: https://issues.apache.org/jira/browse/HIVE-13566 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13566.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13549) Remove jdk version specific out files from Hive2
[ https://issues.apache.org/jira/browse/HIVE-13549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284950#comment-15284950 ] Mohit Sabharwal commented on HIVE-13549: Not sure why the patch is not running pre-commit. Renamed patch to HIVE-13549-java8.patch -- to see is this is a file naming issue. > Remove jdk version specific out files from Hive2 > > > Key: HIVE-13549 > URL: https://issues.apache.org/jira/browse/HIVE-13549 > Project: Hive > Issue Type: Sub-task >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-13549-1-java8.patch, HIVE-13549-java8.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13549) Remove jdk version specific out files from Hive2
[ https://issues.apache.org/jira/browse/HIVE-13549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-13549: --- Attachment: HIVE-13549-java8.patch > Remove jdk version specific out files from Hive2 > > > Key: HIVE-13549 > URL: https://issues.apache.org/jira/browse/HIVE-13549 > Project: Hive > Issue Type: Sub-task >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-13549-1-java8.patch, HIVE-13549-java8.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13343) Need to disable hybrid grace hash join in llap mode except for dynamically partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284939#comment-15284939 ] Vikram Dixit K commented on HIVE-13343: --- Test failures unrelated. > Need to disable hybrid grace hash join in llap mode except for dynamically > partitioned hash join > > > Key: HIVE-13343 > URL: https://issues.apache.org/jira/browse/HIVE-13343 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13343.1.patch, HIVE-13343.2.patch, > HIVE-13343.3.patch, HIVE-13343.4.patch, HIVE-13343.5.patch, > HIVE-13343.6.patch, HIVE-13343.7.patch > > > Due to performance reasons, we should disable use of hybrid grace hash join > in llap when dynamic partition hash join is not used. With dynamic partition > hash join, we need hybrid grace hash join due to the possibility of skews. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13269) Simplify comparison expressions using column stats
[ https://issues.apache.org/jira/browse/HIVE-13269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284931#comment-15284931 ] Ashutosh Chauhan commented on HIVE-13269: - Can you post a link for RB ? > Simplify comparison expressions using column stats > -- > > Key: HIVE-13269 > URL: https://issues.apache.org/jira/browse/HIVE-13269 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13269.01.patch, HIVE-13269.02.patch, > HIVE-13269.03.patch, HIVE-13269.patch, HIVE-13269.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13767) Wrong type inferred in Semijoin condition leads to AssertionError
[
https://issues.apache.org/jira/browse/HIVE-13767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284927#comment-15284927
]
Ashutosh Chauhan commented on HIVE-13767:
-
Can you describe the bug and subsequent fix?
> Wrong type inferred in Semijoin condition leads to AssertionError
> -
>
> Key: HIVE-13767
> URL: https://issues.apache.org/jira/browse/HIVE-13767
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13767.patch
>
>
> Following query fails to run:
> {noformat}
> SELECT
> COALESCE(498, LEAD(COALESCE(-973, -684, 515)) OVER (PARTITION BY
> (t2.int_col_10 + t1.smallint_col_50) ORDER BY (t2.int_col_10 +
> t1.smallint_col_50), FLOOR(t1.double_col_16) DESC), 524) AS int_col,
> (t2.int_col_10) + (t1.smallint_col_50) AS int_col_1,
> FLOOR(t1.double_col_16) AS float_col,
> COALESCE(SUM(COALESCE(62, -380, -435)) OVER (PARTITION BY (t2.int_col_10
> + t1.smallint_col_50) ORDER BY (t2.int_col_10 + t1.smallint_col_50) DESC,
> FLOOR(t1.double_col_16) DESC ROWS BETWEEN UNBOUNDED PRECEDING AND 48
> FOLLOWING), 704) AS int_col_2
> FROM table_1 t1
> INNER JOIN table_18 t2 ON (((t2.tinyint_col_15) = (t1.bigint_col_7)) AND
>((t2.decimal2709_col_9) =
> (t1.decimal2016_col_26))) AND
>((t2.tinyint_col_20) = (t1.tinyint_col_3))
> WHERE (t2.smallint_col_19) IN (SELECT
> COALESCE(-92, -994) AS int_col
> FROM table_1 tt1
> INNER JOIN table_18 tt2 ON (tt2.decimal1911_col_16) =
> (tt1.decimal2612_col_77)
> WHERE (t1.timestamp_col_9) = (tt2.timestamp_col_18));
> {noformat}
> Following error is seen in the logs:
> {noformat}
> 2016-04-27T04:32:09,605 WARN [...2a24 HiveServer2-Handler-Pool: Thread-211]:
> thrift.ThriftCLIService (ThriftCLIService.java:ExecuteStatement(501)) - Error
> executing statement:
> org.apache.hive.service.cli.HiveSQLException: Error running query:
> java.lang.AssertionError: mismatched type $8 TIMESTAMP(9)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:178)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:216)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:327)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:458)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:435)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:272)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:492)
> [hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1317)
> [hive-service-rpc-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1302)
> [hive-service-rpc-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> [hive-exec-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> [hive-exec-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> [hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> [hive-exec-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_77]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_77]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> Caused by: java.lang.AssertionError: mismatched type $8 TIMESTAMP(9)
> at
> org.apache.calcite.rex.RexUtil$FixNullabilityShuttle.visitInputRef(RexUtil.java:2042)
> ~[calcite-core-1.6.0.2.5.0.0-248.jar:1.6.0.2.5.0.0-248]
> at
>
[jira] [Updated] (HIVE-13767) Wrong type inferred in Semijoin condition leads to AssertionError
[
https://issues.apache.org/jira/browse/HIVE-13767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13767:
---
Attachment: HIVE-13767.patch
> Wrong type inferred in Semijoin condition leads to AssertionError
> -
>
> Key: HIVE-13767
> URL: https://issues.apache.org/jira/browse/HIVE-13767
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13767.patch
>
>
> Following query fails to run:
> {noformat}
> SELECT
> COALESCE(498, LEAD(COALESCE(-973, -684, 515)) OVER (PARTITION BY
> (t2.int_col_10 + t1.smallint_col_50) ORDER BY (t2.int_col_10 +
> t1.smallint_col_50), FLOOR(t1.double_col_16) DESC), 524) AS int_col,
> (t2.int_col_10) + (t1.smallint_col_50) AS int_col_1,
> FLOOR(t1.double_col_16) AS float_col,
> COALESCE(SUM(COALESCE(62, -380, -435)) OVER (PARTITION BY (t2.int_col_10
> + t1.smallint_col_50) ORDER BY (t2.int_col_10 + t1.smallint_col_50) DESC,
> FLOOR(t1.double_col_16) DESC ROWS BETWEEN UNBOUNDED PRECEDING AND 48
> FOLLOWING), 704) AS int_col_2
> FROM table_1 t1
> INNER JOIN table_18 t2 ON (((t2.tinyint_col_15) = (t1.bigint_col_7)) AND
>((t2.decimal2709_col_9) =
> (t1.decimal2016_col_26))) AND
>((t2.tinyint_col_20) = (t1.tinyint_col_3))
> WHERE (t2.smallint_col_19) IN (SELECT
> COALESCE(-92, -994) AS int_col
> FROM table_1 tt1
> INNER JOIN table_18 tt2 ON (tt2.decimal1911_col_16) =
> (tt1.decimal2612_col_77)
> WHERE (t1.timestamp_col_9) = (tt2.timestamp_col_18));
> {noformat}
> Following error is seen in the logs:
> {noformat}
> 2016-04-27T04:32:09,605 WARN [...2a24 HiveServer2-Handler-Pool: Thread-211]:
> thrift.ThriftCLIService (ThriftCLIService.java:ExecuteStatement(501)) - Error
> executing statement:
> org.apache.hive.service.cli.HiveSQLException: Error running query:
> java.lang.AssertionError: mismatched type $8 TIMESTAMP(9)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:178)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:216)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:327)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:458)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:435)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:272)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:492)
> [hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1317)
> [hive-service-rpc-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1302)
> [hive-service-rpc-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> [hive-exec-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> [hive-exec-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> [hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> [hive-exec-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_77]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_77]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> Caused by: java.lang.AssertionError: mismatched type $8 TIMESTAMP(9)
> at
> org.apache.calcite.rex.RexUtil$FixNullabilityShuttle.visitInputRef(RexUtil.java:2042)
> ~[calcite-core-1.6.0.2.5.0.0-248.jar:1.6.0.2.5.0.0-248]
> at
> org.apache.calcite.rex.RexUtil$FixNullabilityShuttle.visitInputRef(RexUtil.java:2020)
>
[jira] [Commented] (HIVE-13753) Make metastore client thread safe in DbTxnManager
[ https://issues.apache.org/jira/browse/HIVE-13753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284895#comment-15284895 ] Vaibhav Gumashta commented on HIVE-13753: - Thanks [~wzheng] :) > Make metastore client thread safe in DbTxnManager > - > > Key: HIVE-13753 > URL: https://issues.apache.org/jira/browse/HIVE-13753 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0, 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13753.1.patch, HIVE-13753.2.patch, > HIVE-13753.3.patch > > > The fact that multiple threads sharing the same metastore client which is > used for RPC to Thrift is not thread safe. > Race condition can happen when one sees "out of sequence response" error > message from Thrift server. That means the response from the Thrift server is > for a different request (by a different thread). > Solution will be to synchronize methods from the client side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13753) Make metastore client thread safe in DbTxnManager
[ https://issues.apache.org/jira/browse/HIVE-13753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284890#comment-15284890 ] Wei Zheng commented on HIVE-13753: -- That failure was caused by the issue with repository.apache.org. That has been resolved during the weekend. It has nothing to do with the patch :) > Make metastore client thread safe in DbTxnManager > - > > Key: HIVE-13753 > URL: https://issues.apache.org/jira/browse/HIVE-13753 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0, 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13753.1.patch, HIVE-13753.2.patch, > HIVE-13753.3.patch > > > The fact that multiple threads sharing the same metastore client which is > used for RPC to Thrift is not thread safe. > Race condition can happen when one sees "out of sequence response" error > message from Thrift server. That means the response from the Thrift server is > for a different request (by a different thread). > Solution will be to synchronize methods from the client side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13767) Wrong type inferred in Semijoin condition leads to AssertionError
[
https://issues.apache.org/jira/browse/HIVE-13767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284893#comment-15284893
]
Jesus Camacho Rodriguez commented on HIVE-13767:
[~ashutoshc], could you take a look? Thanks!
> Wrong type inferred in Semijoin condition leads to AssertionError
> -
>
> Key: HIVE-13767
> URL: https://issues.apache.org/jira/browse/HIVE-13767
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13767.patch
>
>
> Following query fails to run:
> {noformat}
> SELECT
> COALESCE(498, LEAD(COALESCE(-973, -684, 515)) OVER (PARTITION BY
> (t2.int_col_10 + t1.smallint_col_50) ORDER BY (t2.int_col_10 +
> t1.smallint_col_50), FLOOR(t1.double_col_16) DESC), 524) AS int_col,
> (t2.int_col_10) + (t1.smallint_col_50) AS int_col_1,
> FLOOR(t1.double_col_16) AS float_col,
> COALESCE(SUM(COALESCE(62, -380, -435)) OVER (PARTITION BY (t2.int_col_10
> + t1.smallint_col_50) ORDER BY (t2.int_col_10 + t1.smallint_col_50) DESC,
> FLOOR(t1.double_col_16) DESC ROWS BETWEEN UNBOUNDED PRECEDING AND 48
> FOLLOWING), 704) AS int_col_2
> FROM table_1 t1
> INNER JOIN table_18 t2 ON (((t2.tinyint_col_15) = (t1.bigint_col_7)) AND
>((t2.decimal2709_col_9) =
> (t1.decimal2016_col_26))) AND
>((t2.tinyint_col_20) = (t1.tinyint_col_3))
> WHERE (t2.smallint_col_19) IN (SELECT
> COALESCE(-92, -994) AS int_col
> FROM table_1 tt1
> INNER JOIN table_18 tt2 ON (tt2.decimal1911_col_16) =
> (tt1.decimal2612_col_77)
> WHERE (t1.timestamp_col_9) = (tt2.timestamp_col_18));
> {noformat}
> Following error is seen in the logs:
> {noformat}
> 2016-04-27T04:32:09,605 WARN [...2a24 HiveServer2-Handler-Pool: Thread-211]:
> thrift.ThriftCLIService (ThriftCLIService.java:ExecuteStatement(501)) - Error
> executing statement:
> org.apache.hive.service.cli.HiveSQLException: Error running query:
> java.lang.AssertionError: mismatched type $8 TIMESTAMP(9)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:178)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:216)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:327)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:458)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:435)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:272)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:492)
> [hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1317)
> [hive-service-rpc-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1302)
> [hive-service-rpc-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> [hive-exec-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> [hive-exec-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> [hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> [hive-exec-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_77]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_77]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> Caused by: java.lang.AssertionError: mismatched type $8 TIMESTAMP(9)
> at
> org.apache.calcite.rex.RexUtil$FixNullabilityShuttle.visitInputRef(RexUtil.java:2042)
> ~[calcite-core-1.6.0.2.5.0.0-248.jar:1.6.0.2.5.0.0-248]
> at
>
[jira] [Updated] (HIVE-13767) Wrong type inferred in Semijoin condition leads to AssertionError
[
https://issues.apache.org/jira/browse/HIVE-13767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13767:
---
Status: Patch Available (was: In Progress)
> Wrong type inferred in Semijoin condition leads to AssertionError
> -
>
> Key: HIVE-13767
> URL: https://issues.apache.org/jira/browse/HIVE-13767
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13767.patch
>
>
> Following query fails to run:
> {noformat}
> SELECT
> COALESCE(498, LEAD(COALESCE(-973, -684, 515)) OVER (PARTITION BY
> (t2.int_col_10 + t1.smallint_col_50) ORDER BY (t2.int_col_10 +
> t1.smallint_col_50), FLOOR(t1.double_col_16) DESC), 524) AS int_col,
> (t2.int_col_10) + (t1.smallint_col_50) AS int_col_1,
> FLOOR(t1.double_col_16) AS float_col,
> COALESCE(SUM(COALESCE(62, -380, -435)) OVER (PARTITION BY (t2.int_col_10
> + t1.smallint_col_50) ORDER BY (t2.int_col_10 + t1.smallint_col_50) DESC,
> FLOOR(t1.double_col_16) DESC ROWS BETWEEN UNBOUNDED PRECEDING AND 48
> FOLLOWING), 704) AS int_col_2
> FROM table_1 t1
> INNER JOIN table_18 t2 ON (((t2.tinyint_col_15) = (t1.bigint_col_7)) AND
>((t2.decimal2709_col_9) =
> (t1.decimal2016_col_26))) AND
>((t2.tinyint_col_20) = (t1.tinyint_col_3))
> WHERE (t2.smallint_col_19) IN (SELECT
> COALESCE(-92, -994) AS int_col
> FROM table_1 tt1
> INNER JOIN table_18 tt2 ON (tt2.decimal1911_col_16) =
> (tt1.decimal2612_col_77)
> WHERE (t1.timestamp_col_9) = (tt2.timestamp_col_18));
> {noformat}
> Following error is seen in the logs:
> {noformat}
> 2016-04-27T04:32:09,605 WARN [...2a24 HiveServer2-Handler-Pool: Thread-211]:
> thrift.ThriftCLIService (ThriftCLIService.java:ExecuteStatement(501)) - Error
> executing statement:
> org.apache.hive.service.cli.HiveSQLException: Error running query:
> java.lang.AssertionError: mismatched type $8 TIMESTAMP(9)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:178)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:216)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:327)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:458)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:435)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:272)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:492)
> [hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1317)
> [hive-service-rpc-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1302)
> [hive-service-rpc-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> [hive-exec-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> [hive-exec-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> [hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> [hive-exec-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_77]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_77]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> Caused by: java.lang.AssertionError: mismatched type $8 TIMESTAMP(9)
> at
> org.apache.calcite.rex.RexUtil$FixNullabilityShuttle.visitInputRef(RexUtil.java:2042)
> ~[calcite-core-1.6.0.2.5.0.0-248.jar:1.6.0.2.5.0.0-248]
> at
> org.apache.calcite.rex.RexUtil$FixNullabilityShuttle.visitInputRef(RexUtil.java:2020)
>
[jira] [Work started] (HIVE-13767) Wrong type inferred in Semijoin condition leads to AssertionError
[
https://issues.apache.org/jira/browse/HIVE-13767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-13767 started by Jesus Camacho Rodriguez.
--
> Wrong type inferred in Semijoin condition leads to AssertionError
> -
>
> Key: HIVE-13767
> URL: https://issues.apache.org/jira/browse/HIVE-13767
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13767.patch
>
>
> Following query fails to run:
> {noformat}
> SELECT
> COALESCE(498, LEAD(COALESCE(-973, -684, 515)) OVER (PARTITION BY
> (t2.int_col_10 + t1.smallint_col_50) ORDER BY (t2.int_col_10 +
> t1.smallint_col_50), FLOOR(t1.double_col_16) DESC), 524) AS int_col,
> (t2.int_col_10) + (t1.smallint_col_50) AS int_col_1,
> FLOOR(t1.double_col_16) AS float_col,
> COALESCE(SUM(COALESCE(62, -380, -435)) OVER (PARTITION BY (t2.int_col_10
> + t1.smallint_col_50) ORDER BY (t2.int_col_10 + t1.smallint_col_50) DESC,
> FLOOR(t1.double_col_16) DESC ROWS BETWEEN UNBOUNDED PRECEDING AND 48
> FOLLOWING), 704) AS int_col_2
> FROM table_1 t1
> INNER JOIN table_18 t2 ON (((t2.tinyint_col_15) = (t1.bigint_col_7)) AND
>((t2.decimal2709_col_9) =
> (t1.decimal2016_col_26))) AND
>((t2.tinyint_col_20) = (t1.tinyint_col_3))
> WHERE (t2.smallint_col_19) IN (SELECT
> COALESCE(-92, -994) AS int_col
> FROM table_1 tt1
> INNER JOIN table_18 tt2 ON (tt2.decimal1911_col_16) =
> (tt1.decimal2612_col_77)
> WHERE (t1.timestamp_col_9) = (tt2.timestamp_col_18));
> {noformat}
> Following error is seen in the logs:
> {noformat}
> 2016-04-27T04:32:09,605 WARN [...2a24 HiveServer2-Handler-Pool: Thread-211]:
> thrift.ThriftCLIService (ThriftCLIService.java:ExecuteStatement(501)) - Error
> executing statement:
> org.apache.hive.service.cli.HiveSQLException: Error running query:
> java.lang.AssertionError: mismatched type $8 TIMESTAMP(9)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:178)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:216)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:327)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:458)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:435)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:272)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:492)
> [hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1317)
> [hive-service-rpc-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1302)
> [hive-service-rpc-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> [hive-exec-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> [hive-exec-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> [hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> [hive-exec-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_77]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_77]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> Caused by: java.lang.AssertionError: mismatched type $8 TIMESTAMP(9)
> at
> org.apache.calcite.rex.RexUtil$FixNullabilityShuttle.visitInputRef(RexUtil.java:2042)
> ~[calcite-core-1.6.0.2.5.0.0-248.jar:1.6.0.2.5.0.0-248]
> at
> org.apache.calcite.rex.RexUtil$FixNullabilityShuttle.visitInputRef(RexUtil.java:2020)
>
[jira] [Commented] (HIVE-13753) Make metastore client thread safe in DbTxnManager
[ https://issues.apache.org/jira/browse/HIVE-13753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284885#comment-15284885 ] Vaibhav Gumashta commented on HIVE-13753: - [~wzheng] Looks like there was a build failure. > Make metastore client thread safe in DbTxnManager > - > > Key: HIVE-13753 > URL: https://issues.apache.org/jira/browse/HIVE-13753 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0, 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13753.1.patch, HIVE-13753.2.patch, > HIVE-13753.3.patch > > > The fact that multiple threads sharing the same metastore client which is > used for RPC to Thrift is not thread safe. > Race condition can happen when one sees "out of sequence response" error > message from Thrift server. That means the response from the Thrift server is > for a different request (by a different thread). > Solution will be to synchronize methods from the client side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13753) Make metastore client thread safe in DbTxnManager
[ https://issues.apache.org/jira/browse/HIVE-13753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13753: - Resolution: Fixed Fix Version/s: 2.1.0 1.3.0 Status: Resolved (was: Patch Available) Committed to master and branch-1. Thanks [~vgumashta] for the review! > Make metastore client thread safe in DbTxnManager > - > > Key: HIVE-13753 > URL: https://issues.apache.org/jira/browse/HIVE-13753 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0, 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13753.1.patch, HIVE-13753.2.patch, > HIVE-13753.3.patch > > > The fact that multiple threads sharing the same metastore client which is > used for RPC to Thrift is not thread safe. > Race condition can happen when one sees "out of sequence response" error > message from Thrift server. That means the response from the Thrift server is > for a different request (by a different thread). > Solution will be to synchronize methods from the client side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13753) Make metastore client thread safe in DbTxnManager
[ https://issues.apache.org/jira/browse/HIVE-13753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284862#comment-15284862 ] Wei Zheng commented on HIVE-13753: -- Test failures are not related. Test Name Duration Age org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_partitioned 26 sec 1 org.apache.hive.hcatalog.listener.TestDbNotificationListener.cleanupNotifs 1 min 1 sec 1 org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation 10 sec 1 org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority 5.2 sec 1 org.apache.hive.spark.client.TestSparkClient.testSyncRpc 33 sec 1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_orig_table_use_metadata 46 sec 3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32 6.1 sec 3 org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static 1 min 37 sec4 org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic 1 min 28 sec4 org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForMemoryTokenStore 1.6 sec 4 org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForDBTokenStore 0.41 sec4 org.apache.hive.minikdc.TestMiniHiveKdc.testLogin 1 min 56 sec4 org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testShowLocksFilterOptions 5.7 sec 6 org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec 44 ms 24 org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle 32 sec 24 org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure 3.4 sec 37 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_selectindate15 sec 56 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avrocountemptytbl 11 sec 56 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order_null 37 sec 56 org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys 1 min 34 sec56 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 9.6 sec 56 org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver 57 sec 56 > Make metastore client thread safe in DbTxnManager > - > > Key: HIVE-13753 > URL: https://issues.apache.org/jira/browse/HIVE-13753 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0, 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13753.1.patch, HIVE-13753.2.patch, > HIVE-13753.3.patch > > > The fact that multiple threads sharing the same metastore client which is > used for RPC to Thrift is not thread safe. > Race condition can happen when one sees "out of sequence response" error > message from Thrift server. That means the response from the Thrift server is > for a different request (by a different thread). > Solution will be to synchronize methods from the client side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13758) "Create table like" command should initialize the basic stats for the table
[ https://issues.apache.org/jira/browse/HIVE-13758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13758: --- Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) > "Create table like" command should initialize the basic stats for the table > --- > > Key: HIVE-13758 > URL: https://issues.apache.org/jira/browse/HIVE-13758 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.1.0 > > Attachments: HIVE-13758.01.patch, HIVE-13758.final.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13758) "Create table like" command should initialize the basic stats for the table
[ https://issues.apache.org/jira/browse/HIVE-13758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13758: --- Attachment: HIVE-13758.final.patch > "Create table like" command should initialize the basic stats for the table > --- > > Key: HIVE-13758 > URL: https://issues.apache.org/jira/browse/HIVE-13758 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13758.01.patch, HIVE-13758.final.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13758) "Create table like" command should initialize the basic stats for the table
[ https://issues.apache.org/jira/browse/HIVE-13758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284846#comment-15284846 ] Pengcheng Xiong commented on HIVE-13758: checked all the test failures, update the golden files, push it to master. Thanks [~ashutoshc] for the review! > "Create table like" command should initialize the basic stats for the table > --- > > Key: HIVE-13758 > URL: https://issues.apache.org/jira/browse/HIVE-13758 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13758.01.patch, HIVE-13758.final.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13730) hybridgrace_hashjoin_1.q test gets stuck
[ https://issues.apache.org/jira/browse/HIVE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13730: - Attachment: HIVE-13730.4.patch Last build had issue with repository.apache.org. Upload patch 4 (same as patch 3). > hybridgrace_hashjoin_1.q test gets stuck > > > Key: HIVE-13730 > URL: https://issues.apache.org/jira/browse/HIVE-13730 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Wei Zheng >Priority: Blocker > Attachments: HIVE-13730.1.patch, HIVE-13730.2.patch, > HIVE-13730.3.patch, HIVE-13730.4.patch > > > I am seeing hybridgrace_hashjoin_1.q getting stuck on master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13354) Add ability to specify Compaction options per table and per request
[ https://issues.apache.org/jira/browse/HIVE-13354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13354: - Status: Patch Available (was: Open) > Add ability to specify Compaction options per table and per request > --- > > Key: HIVE-13354 > URL: https://issues.apache.org/jira/browse/HIVE-13354 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.0.0, 1.3.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Labels: TODOC2.1 > Attachments: HIVE-13354.1.patch, > HIVE-13354.1.withoutSchemaChange.patch > > > Currently the are a few options that determine when automatic compaction is > triggered. They are specified once for the warehouse. > This doesn't make sense - some table may be more important and need to be > compacted more often. > We should allow specifying these on per table basis. > Also, compaction is an MR job launched from within the metastore. There is > currently no way to control job parameters (like memory, for example) except > to specify it in hive-site.xml for metastore which means they are site wide. > Should add a way to specify these per table (perhaps even per compaction if > launched via ALTER TABLE) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13614) Implementation for PreparedStatement#setBigDecimal
[ https://issues.apache.org/jira/browse/HIVE-13614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Iwao AVE! updated HIVE-13614: - Status: Patch Available (was: Open) > Implementation for PreparedStatement#setBigDecimal > -- > > Key: HIVE-13614 > URL: https://issues.apache.org/jira/browse/HIVE-13614 > Project: Hive > Issue Type: Improvement > Components: JDBC >Reporter: Iwao AVE! >Priority: Minor > Attachments: HIVE-13614.patch > > > Hi, > I am a committer of MyBatis project (https://github.com/mybatis) and some of > our users have difficulty with using MyBatis-Migrations (a db migration tool) > with Hive. > https://github.com/mybatis/migrations/issues/25 > Basically, Migrations uses BigDecimal as the ID type and the lack of support > in Hive's JDBC driver prevents our users from using Migrations with Hive. > So, it would be beneficial to both of Hive and MyBatis users if you could > apply the following change to mitigate the situation. > https://github.com/harawata/hive/commit/1910632442ad9b70b41b28e37596843d1b9f7d3f > Please let me know if I should send the modification as a GitHub pull request. > Thanks in advance! > Iwao -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13614) Implementation for PreparedStatement#setBigDecimal
[ https://issues.apache.org/jira/browse/HIVE-13614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Iwao AVE! updated HIVE-13614: - Attachment: HIVE-13614.patch The test failed with some server side issue. Re-attaching the same patch. If it's possible, please port this to branch-1. Thanks! > Implementation for PreparedStatement#setBigDecimal > -- > > Key: HIVE-13614 > URL: https://issues.apache.org/jira/browse/HIVE-13614 > Project: Hive > Issue Type: Improvement > Components: JDBC >Reporter: Iwao AVE! >Priority: Minor > Attachments: HIVE-13614.patch > > > Hi, > I am a committer of MyBatis project (https://github.com/mybatis) and some of > our users have difficulty with using MyBatis-Migrations (a db migration tool) > with Hive. > https://github.com/mybatis/migrations/issues/25 > Basically, Migrations uses BigDecimal as the ID type and the lack of support > in Hive's JDBC driver prevents our users from using Migrations with Hive. > So, it would be beneficial to both of Hive and MyBatis users if you could > apply the following change to mitigate the situation. > https://github.com/harawata/hive/commit/1910632442ad9b70b41b28e37596843d1b9f7d3f > Please let me know if I should send the modification as a GitHub pull request. > Thanks in advance! > Iwao -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13614) Implementation for PreparedStatement#setBigDecimal
[ https://issues.apache.org/jira/browse/HIVE-13614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Iwao AVE! updated HIVE-13614: - Status: Open (was: Patch Available) > Implementation for PreparedStatement#setBigDecimal > -- > > Key: HIVE-13614 > URL: https://issues.apache.org/jira/browse/HIVE-13614 > Project: Hive > Issue Type: Improvement > Components: JDBC >Reporter: Iwao AVE! >Priority: Minor > > Hi, > I am a committer of MyBatis project (https://github.com/mybatis) and some of > our users have difficulty with using MyBatis-Migrations (a db migration tool) > with Hive. > https://github.com/mybatis/migrations/issues/25 > Basically, Migrations uses BigDecimal as the ID type and the lack of support > in Hive's JDBC driver prevents our users from using Migrations with Hive. > So, it would be beneficial to both of Hive and MyBatis users if you could > apply the following change to mitigate the situation. > https://github.com/harawata/hive/commit/1910632442ad9b70b41b28e37596843d1b9f7d3f > Please let me know if I should send the modification as a GitHub pull request. > Thanks in advance! > Iwao -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13614) Implementation for PreparedStatement#setBigDecimal
[ https://issues.apache.org/jira/browse/HIVE-13614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Iwao AVE! updated HIVE-13614: - Attachment: (was: HIVE-13614.patch) > Implementation for PreparedStatement#setBigDecimal > -- > > Key: HIVE-13614 > URL: https://issues.apache.org/jira/browse/HIVE-13614 > Project: Hive > Issue Type: Improvement > Components: JDBC >Reporter: Iwao AVE! >Priority: Minor > > Hi, > I am a committer of MyBatis project (https://github.com/mybatis) and some of > our users have difficulty with using MyBatis-Migrations (a db migration tool) > with Hive. > https://github.com/mybatis/migrations/issues/25 > Basically, Migrations uses BigDecimal as the ID type and the lack of support > in Hive's JDBC driver prevents our users from using Migrations with Hive. > So, it would be beneficial to both of Hive and MyBatis users if you could > apply the following change to mitigate the situation. > https://github.com/harawata/hive/commit/1910632442ad9b70b41b28e37596843d1b9f7d3f > Please let me know if I should send the modification as a GitHub pull request. > Thanks in advance! > Iwao -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13616) Investigate renaming a table without invalidating the column stats
[ https://issues.apache.org/jira/browse/HIVE-13616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284798#comment-15284798 ] Chaoyu Tang commented on HIVE-13616: +1 > Investigate renaming a table without invalidating the column stats > -- > > Key: HIVE-13616 > URL: https://issues.apache.org/jira/browse/HIVE-13616 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-13616.1.patch > > > Right now when we rename a table, we clear the column stats rather than > updating it (HIVE-9720) since ObjectStore uses DN to talk to DB. Investigate > the possibility that if we can achieve updating the stats without rescanning > the whole table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13693) Multi-insert query drops Filter before file output when there is a.val <> b.val
[
https://issues.apache.org/jira/browse/HIVE-13693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284790#comment-15284790
]
Hive QA commented on HIVE-13693:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12804133/HIVE-13693.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 9998 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-enforce_order.q-vector_partition_diff_num_cols.q-unionDistinct_1.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-explainuser_4.q-update_after_multiple_inserts.q-mapreduce2.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-groupby2.q-tez_dynpart_hashjoin_1.q-custom_input_output_format.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-join1.q-mapjoin_decimal.q-union5.q-and-12-more - did not
produce a TEST-*.xml file
TestMiniTezCliDriver-mapjoin_mapjoin.q-insert_into1.q-vector_decimal_2.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-smb_cache.q-transform_ppr2.q-vector_outer_join0.q-and-5-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_transform.q-union_remove_7.q-date_udf.q-and-12-more -
did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_external_table_ppd
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_ppd_key_range
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_pushdown
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_timestamp
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_ppd_key_ranges
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_single_reducer
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_gby
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_udf_min
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/298/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/298/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-298/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 24 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12804133 - PreCommit-HIVE-MASTER-Build
> Multi-insert query drops Filter before file output when there is a.val <>
> b.val
> ---
>
> Key: HIVE-13693
> URL: https://issues.apache.org/jira/browse/HIVE-13693
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.3.0, 2.0.0, 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13693.01.patch, HIVE-13693.01.patch,
> HIVE-13693.patch
>
>
> To reproduce:
> {noformat}
> CREATE TABLE T_A ( id STRING, val STRING );
> CREATE TABLE T_B ( id STRING, val STRING );
> CREATE TABLE join_result_1 ( ida STRING, vala STRING, idb STRING, valb STRING
> );
> CREATE TABLE join_result_3 ( ida STRING, vala STRING, idb STRING, valb STRING
> );
> INSERT INTO TABLE T_A
> VALUES ('Id_1', 'val_101'), ('Id_2', 'val_102'), ('Id_3', 'val_103');
> INSERT INTO TABLE T_B
> VALUES ('Id_1', 'val_103'), ('Id_2', 'val_104');
> explain
> FROM T_A a LEFT JOIN T_B b ON a.id = b.id
> INSERT OVERWRITE TABLE join_result_1
> SELECT a.*, b.*
> WHERE b.id = 'Id_1' AND b.val = 'val_103'
> INSERT OVERWRITE TABLE join_result_3
> SELECT a.*, b.*
> WHERE b.val = 'val_104' AND b.id = 'Id_2' AND a.val <> b.val;
> {noformat}
> The (wrong) plan is the following:
> {noformat}
> STAGE DEPENDENCIES:
> Stage-2 is a root stage
> Stage-3 depends on stages:
