[jira] [Resolved] (HIVE-21257) Vectorized LENGTH UDF doesn't respect the max length of VARCHAR or CHAR in Hive 3+
[ https://issues.apache.org/jira/browse/HIVE-21257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi resolved HIVE-21257. --- Resolution: Not A Problem > Vectorized LENGTH UDF doesn't respect the max length of VARCHAR or CHAR in > Hive 3+ > -- > > Key: HIVE-21257 > URL: https://issues.apache.org/jira/browse/HIVE-21257 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0, 3.1.1 >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > > After HIVE-19951 is fixed, there still are some cases that vectorized length > UDF doesn't respect the max length of VARCHAR or CHAR. StringLength has an > internal bug. Moreover, it's hard to get input data type details in Hive 2, > unlike Hive 3. So separate both implementation to keep code clean in Hive 3 > while the changes minimal in Hive 2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21257) Vectorized LENGTH UDF doesn't respect the max length of VARCHAR or CHAR in Hive 3+
[ https://issues.apache.org/jira/browse/HIVE-21257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-21257: -- Status: Open (was: Patch Available) > Vectorized LENGTH UDF doesn't respect the max length of VARCHAR or CHAR in > Hive 3+ > -- > > Key: HIVE-21257 > URL: https://issues.apache.org/jira/browse/HIVE-21257 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.1, 4.0.0 >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > > After HIVE-19951 is fixed, there still are some cases that vectorized length > UDF doesn't respect the max length of VARCHAR or CHAR. StringLength has an > internal bug. Moreover, it's hard to get input data type details in Hive 2, > unlike Hive 3. So separate both implementation to keep code clean in Hive 3 > while the changes minimal in Hive 2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21264) Improvements Around CharTypeInfo
[
https://issues.apache.org/jira/browse/HIVE-21264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767929#comment-16767929
]
Hive QA commented on HIVE-21264:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} serde in master has 197 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} serde: The patch generated 0 new + 51 unchanged - 2
fixed = 51 total (was 53) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16068/dev-support/hive-personality.sh
|
| git revision | master / 292f6bb |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: serde U: serde |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16068/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improvements Around CharTypeInfo
>
>
> Key: HIVE-21264
> URL: https://issues.apache.org/jira/browse/HIVE-21264
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-21264.1.patch, HIVE-21264.2.patch
>
>
> The {{CharTypeInfo}} stores the type name of the data type (char/varchar) and
> the length (1-255). {{CharTypeInfo}} objects are often getting cached once
> they are created.
> The {{hashcode()}} and {{equals()}} of its sub-classes varchar and char are
> inconsistent.
> * Make hashcode and equals consistent (and fast)
> * Simplify the {{getQualifiedName}} implementation and reduce the scope to
> protected
> * Other related nits
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21269) Mandate -update and -delete as DistCp options to avoid data inconsistency with external tables replication.
[ https://issues.apache.org/jira/browse/HIVE-21269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-21269: Summary: Mandate -update and -delete as DistCp options to avoid data inconsistency with external tables replication. (was: Hive replication should mandate -update and -delete as DistCp options to avoid data inconsistency.) > Mandate -update and -delete as DistCp options to avoid data inconsistency > with external tables replication. > > > Key: HIVE-21269 > URL: https://issues.apache.org/jira/browse/HIVE-21269 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, replication > > Currently, external tables replication, copies the data in directory level. > So, if target directory exist, then DistCp should compare and update or skip > data files in the directory instead of creating new directory inside > pre-existing target directory. > This can be achieved using -update. > Also, -delete option is needed to delete the files missing in source > directory but present in target. > Hive should mandate these DistCp options even if user passes other options. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21269) Mandate -update and -delete as DistCp options to sync data for external tables replication.
[ https://issues.apache.org/jira/browse/HIVE-21269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-21269: Summary: Mandate -update and -delete as DistCp options to sync data for external tables replication. (was: Mandate -update and -delete as DistCp options to avoid data inconsistency with external tables replication.) > Mandate -update and -delete as DistCp options to sync data for external > tables replication. > > > Key: HIVE-21269 > URL: https://issues.apache.org/jira/browse/HIVE-21269 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, replication > > Currently, external tables replication, copies the data in directory level. > So, if target directory exist, then DistCp should compare and update or skip > data files in the directory instead of creating new directory inside > pre-existing target directory. > This can be achieved using -update. > Also, -delete option is needed to delete the files missing in source > directory but present in target. > Hive should mandate these DistCp options even if user passes other options. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21269) Mandate -update and -delete as DistCp options to sync data files for external tables replication.
[ https://issues.apache.org/jira/browse/HIVE-21269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-21269: Summary: Mandate -update and -delete as DistCp options to sync data files for external tables replication. (was: Mandate -update and -delete as DistCp options to sync data for external tables replication.) > Mandate -update and -delete as DistCp options to sync data files for > external tables replication. > -- > > Key: HIVE-21269 > URL: https://issues.apache.org/jira/browse/HIVE-21269 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, replication > > Currently, external tables replication, copies the data in directory level. > So, if target directory exist, then DistCp should compare and update or skip > data files in the directory instead of creating new directory inside > pre-existing target directory. > This can be achieved using -update. > Also, -delete option is needed to delete the files missing in source > directory but present in target. > Hive should mandate these DistCp options even if user passes other options. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21269) Hive replication should mandate -update and -delete as DistCp options to avoid data inconsistency.
[ https://issues.apache.org/jira/browse/HIVE-21269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan reassigned HIVE-21269: --- > Hive replication should mandate -update and -delete as DistCp options to > avoid data inconsistency. > -- > > Key: HIVE-21269 > URL: https://issues.apache.org/jira/browse/HIVE-21269 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, replication > > Currently, external tables replication, copies the data in directory level. > So, if target directory exist, then DistCp should compare and update or skip > data files in the directory instead of creating new directory inside > pre-existing target directory. > This can be achieved using -update. > Also, -delete option is needed to delete the files missing in source > directory but present in target. > Hive should mandate these DistCp options even if user passes other options. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21240) JSON SerDe Deserialize Re-Write
[
https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767920#comment-16767920
]
Hive QA commented on HIVE-21240:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958661/HIVE-21240.5.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 49 failed/errored test(s), 15805 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[json_serde1] (batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[json_serde2] (batchId=22)
org.apache.hadoop.hive.cli.TestMiniHiveKafkaCliDriver.testCliDriver[kafka_storage_handler]
(batchId=275)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_acid_part_update]
(batchId=178)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_acid_part_update_llap_io]
(batchId=181)
org.apache.hive.hcatalog.data.TestJsonSerDe.testRW (batchId=215)
org.apache.hive.hcatalog.data.TestJsonSerDe.testRobustRead (batchId=215)
org.apache.hive.hcatalog.mapreduce.TestHCatDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask[6]
(batchId=211)
org.apache.hive.hcatalog.mapreduce.TestHCatDynamicPartitioned.testHCatDynamicPartitionedTable[6]
(batchId=211)
org.apache.hive.hcatalog.mapreduce.TestHCatExternalDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask[6]
(batchId=213)
org.apache.hive.hcatalog.mapreduce.TestHCatExternalDynamicPartitioned.testHCatDynamicPartitionedTable[6]
(batchId=213)
org.apache.hive.hcatalog.mapreduce.TestHCatExternalDynamicPartitioned.testHCatExternalDynamicCustomLocation[6]
(batchId=213)
org.apache.hive.hcatalog.mapreduce.TestHCatExternalNonPartitioned.testHCatNonPartitionedTable[6]
(batchId=214)
org.apache.hive.hcatalog.mapreduce.TestHCatExternalPartitioned.testHCatPartitionedTable[6]
(batchId=210)
org.apache.hive.hcatalog.mapreduce.TestHCatMutableDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask[6]
(batchId=208)
org.apache.hive.hcatalog.mapreduce.TestHCatMutableDynamicPartitioned.testHCatDynamicPartitionedTable[6]
(batchId=208)
org.apache.hive.hcatalog.mapreduce.TestHCatMutableNonPartitioned.testHCatNonPartitionedTable[6]
(batchId=214)
org.apache.hive.hcatalog.mapreduce.TestHCatMutablePartitioned.testHCatPartitionedTable[6]
(batchId=212)
org.apache.hive.hcatalog.mapreduce.TestHCatNonPartitioned.testHCatNonPartitionedTable[6]
(batchId=209)
org.apache.hive.hcatalog.mapreduce.TestHCatPartitioned.testHCatPartitionedTable[6]
(batchId=209)
org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData[6]
(batchId=207)
org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema[6]
(batchId=207)
org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag[6]
(batchId=207)
org.apache.hive.hcatalog.pig.TestHCatStorer.testBagNStruct[6] (batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testDateCharTypes[6] (batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testDynamicPartitioningMultiPartColsInDataNoSpec[6]
(batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testDynamicPartitioningMultiPartColsInDataPartialSpec[6]
(batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testMultiPartColsInData[6]
(batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testPartColsInData[6] (batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testStoreFuncAllSimpleTypes[6]
(batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testStoreFuncSimple[6] (batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testStoreInPartiitonedTbl[6]
(batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testStoreMultiTables[6]
(batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testStoreWithNoCtorArgs[6]
(batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testStoreWithNoSchema[6]
(batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testWriteChar[6] (batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testWriteDate2[6] (batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testWriteDate3[6] (batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testWriteDate[6] (batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testWriteDecimalXY[6] (batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testWriteDecimalX[6] (batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testWriteDecimal[6] (batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testWriteSmallint[6] (batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testWriteTimestamp[6] (batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testWriteTinyint[6] (batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testWriteVarchar[6] (batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorerMulti.testStoreBasicTable[6]
(batchId=207)
[jira] [Commented] (HIVE-21249) Reduce memory footprint in ObjectStore.refreshPrivileges
[
https://issues.apache.org/jira/browse/HIVE-21249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767903#comment-16767903
]
Sankar Hariappan commented on HIVE-21249:
-
[~daijy], I've few comments.
1. In ObjectStore.listTableColumnGrants
- Need to normalizeIdentifier(columnName) before use.
- Can we change the if-else logic as follows to minimize the number of
comparisons to 2 in any flow.
{code}
if (columnName == null) {
if (authorizer == null {
} else {
}
} else {
if (authorizer == null {
} else {
}
}
{code}
2. Shall remove the unused ObjectStore.listTableAllColumnGrants method.
3. In PrivilegeSynchronizer.run, can we make smaller batch of columns instead
of invoking refresh_privileges for each column? This would avoid too many HMS
api calls.
> Reduce memory footprint in ObjectStore.refreshPrivileges
> -
>
> Key: HIVE-21249
> URL: https://issues.apache.org/jira/browse/HIVE-21249
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-21249.1.patch
>
>
> We found there're could be many records in TBL_COL_PRIVS for a single table
> (a table granted to many users), thus result a OOM in
> ObjectStore.listTableAllColumnGrants. We shall reduce the memory footprint
> for ObjectStore.refreshPrivileges. Here is the stack of OOM:
> {code}
> org.datanucleus.api.jdo.JDOPersistenceManager.retrieveAll(JDOPersistenceManager.java:690)
> org.datanucleus.api.jdo.JDOPersistenceManager.retrieveAll(JDOPersistenceManager.java:710)
> org.apache.hadoop.hive.metastore.ObjectStore.listTableAllColumnGrants(ObjectStore.java:6629)
> org.apache.hadoop.hive.metastore.ObjectStore.refreshPrivileges(ObjectStore.java:6200)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:498)
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> com.sun.proxy.$Proxy32.refreshPrivileges(, line not available)
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.refresh_privileges(HiveMetaStore.java:6507)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:498)
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> com.sun.proxy.$Proxy34.refresh_privileges(, line not available)
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$refresh_privileges.getResult(ThriftHiveMetastore.java:17608)
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$refresh_privileges.getResult(ThriftHiveMetastore.java:17592)
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:636)
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:631)
> java.security.AccessController.doPrivileged(Native method)
> javax.security.auth.Subject.doAs(Subject.java:422)
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:631)
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-21249) Reduce memory footprint in ObjectStore.refreshPrivileges
[
https://issues.apache.org/jira/browse/HIVE-21249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767903#comment-16767903
]
Sankar Hariappan edited comment on HIVE-21249 at 2/14/19 6:31 AM:
--
[~daijy], I've few comments.
1. In ObjectStore.listTableColumnGrants
- Need to normalizeIdentifier(columnName) before use.
- Can we change the if-else logic as follows to minimize the number of
comparisons to 2 in any flow.
{code}
if (columnName == null) {
if (authorizer == null {
} else {
}
} else {
if (authorizer == null {
} else {
}
}
{code}
2. Shall remove the unused ObjectStore.listTableAllColumnGrants method.
3. In PrivilegeSynchronizer.run, can we make smaller batch of columns instead
of invoking refresh_privileges for each column? This would avoid too many HMS
api calls.
was (Author: sankarh):
[~daijy], I've few comments.
1. In ObjectStore.listTableColumnGrants
- Need to normalizeIdentifier(columnName) before use.
- Can we change the if-else logic as follows to minimize the number of
comparisons to 2 in any flow.
{code}
if (columnName == null) {
if (authorizer == null {
} else {
}
} else {
if (authorizer == null {
} else {
}
}
{code}
2. Shall remove the unused ObjectStore.listTableAllColumnGrants method.
3. In PrivilegeSynchronizer.run, can we make smaller batch of columns instead
of invoking refresh_privileges for each column? This would avoid too many HMS
api calls.
> Reduce memory footprint in ObjectStore.refreshPrivileges
> -
>
> Key: HIVE-21249
> URL: https://issues.apache.org/jira/browse/HIVE-21249
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-21249.1.patch
>
>
> We found there're could be many records in TBL_COL_PRIVS for a single table
> (a table granted to many users), thus result a OOM in
> ObjectStore.listTableAllColumnGrants. We shall reduce the memory footprint
> for ObjectStore.refreshPrivileges. Here is the stack of OOM:
> {code}
> org.datanucleus.api.jdo.JDOPersistenceManager.retrieveAll(JDOPersistenceManager.java:690)
> org.datanucleus.api.jdo.JDOPersistenceManager.retrieveAll(JDOPersistenceManager.java:710)
> org.apache.hadoop.hive.metastore.ObjectStore.listTableAllColumnGrants(ObjectStore.java:6629)
> org.apache.hadoop.hive.metastore.ObjectStore.refreshPrivileges(ObjectStore.java:6200)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:498)
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> com.sun.proxy.$Proxy32.refreshPrivileges(, line not available)
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.refresh_privileges(HiveMetaStore.java:6507)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:498)
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> com.sun.proxy.$Proxy34.refresh_privileges(, line not available)
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$refresh_privileges.getResult(ThriftHiveMetastore.java:17608)
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$refresh_privileges.getResult(ThriftHiveMetastore.java:17592)
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:636)
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:631)
> java.security.AccessController.doPrivileged(Native method)
> javax.security.auth.Subject.doAs(Subject.java:422)
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:631)
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
[jira] [Commented] (HIVE-21268) REPL: Repl dump can output - Database, Table, Dir, last_repl_id
[
https://issues.apache.org/jira/browse/HIVE-21268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767900#comment-16767900
]
Gopal V commented on HIVE-21268:
Is there a "repl info ''" or something equivalent to reverse the repl
dump to the source?
> REPL: Repl dump can output - Database, Table, Dir, last_repl_id
> ---
>
> Key: HIVE-21268
> URL: https://issues.apache.org/jira/browse/HIVE-21268
> Project: Hive
> Issue Type: Improvement
>Reporter: Gopal V
>Priority: Major
>
> {code}
> INFO : Completed executing
> command(queryId=root_20190214061031_639e3a52-5c62-40be-a3cd-3e0b18b7b41d);
> Time taken: 0.374 seconds
> INFO : OK
> ++---+
> | dump_dir | last_repl_id |
> ++---+
> | /user/root/repl/a74389d0-7cde-4cf4-aa40-3079a98b80a8 | 1104594 |
> ++---+
> 1 row selected (0.445 seconds)
> {code}
> is somewhat hard to associate back to the table name.
> The logs a couple of lines above actually print the operation detail.
> {code}
> INFO : REPL::TABLE_DUMP:
> {"dbName":"tpcds_bin_partitioned_orc_1000","tableName":"item","tableType":"MANAGED_TABLE","tablesDumpProgress":"1/38","dumpTime":1550124632}
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21268) REPL: Repl dump can output - Database, Table, Dir, last_repl_id
[
https://issues.apache.org/jira/browse/HIVE-21268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767898#comment-16767898
]
anishek commented on HIVE-21268:
yeh there is no table level dump support for now, so if only db level repl is
provided then the dump_dir will be corresponding to the state of the db and
hence no table info. the full db,table,dir,last_repl,id is good only if there
is single table in that dump.
> REPL: Repl dump can output - Database, Table, Dir, last_repl_id
> ---
>
> Key: HIVE-21268
> URL: https://issues.apache.org/jira/browse/HIVE-21268
> Project: Hive
> Issue Type: Improvement
>Reporter: Gopal V
>Priority: Major
>
> {code}
> INFO : Completed executing
> command(queryId=root_20190214061031_639e3a52-5c62-40be-a3cd-3e0b18b7b41d);
> Time taken: 0.374 seconds
> INFO : OK
> ++---+
> | dump_dir | last_repl_id |
> ++---+
> | /user/root/repl/a74389d0-7cde-4cf4-aa40-3079a98b80a8 | 1104594 |
> ++---+
> 1 row selected (0.445 seconds)
> {code}
> is somewhat hard to associate back to the table name.
> The logs a couple of lines above actually print the operation detail.
> {code}
> INFO : REPL::TABLE_DUMP:
> {"dbName":"tpcds_bin_partitioned_orc_1000","tableName":"item","tableType":"MANAGED_TABLE","tablesDumpProgress":"1/38","dumpTime":1550124632}
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21240) JSON SerDe Deserialize Re-Write
[
https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767894#comment-16767894
]
Hive QA commented on HIVE-21240:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
3s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} serde in master has 197 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
54s{color} | {color:blue} ql in master has 2262 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} hcatalog/core in master has 29 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} serde: The patch generated 4 new + 4 unchanged - 25
fixed = 8 total (was 29) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 2 new + 6 unchanged - 5 fixed
= 8 total (was 11) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
48s{color} | {color:red} serde generated 1 new + 193 unchanged - 4 fixed = 194
total (was 197) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 6s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:serde |
| | Boxing/unboxing to parse a primitive
org.apache.hadoop.hive.serde2.json.HiveJsonReader.getColIndex(String) At
HiveJsonReader.java:org.apache.hadoop.hive.serde2.json.HiveJsonReader.getColIndex(String)
At HiveJsonReader.java:[line 488] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16067/dev-support/hive-personality.sh
|
| git revision | master / 292f6bb |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16067/yetus/diff-checkstyle-serde.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16067/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16067/yetus/new-findbugs-serde.html
|
| modules | C: serde ql hcatalog/core U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16067/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> JSON SerDe Deserialize Re-Write
> ---
>
> Key: HIVE-21240
> URL: https://issues.apache.org/jira/browse/HIVE-21240
>
[jira] [Updated] (HIVE-21167) Bucketing: Bucketing version 1 is incorrectly partitioning data
[
https://issues.apache.org/jira/browse/HIVE-21167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deepak Jaiswal updated HIVE-21167:
--
Status: Patch Available (was: Open)
> Bucketing: Bucketing version 1 is incorrectly partitioning data
> ---
>
> Key: HIVE-21167
> URL: https://issues.apache.org/jira/browse/HIVE-21167
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Vaibhav Gumashta
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-21167.1.patch
>
>
> Using murmur hash for bucketing columns was introduced in HIVE-18910,
> following which {{'bucketing_version'='1'}} stands for the old behaviour
> (where for example integer columns were partitioned based on mod values).
> Looks like we have a bug in the old bucketing scheme now. I could repro it
> when modified the existing schema using an alter table add column and adding
> new data. Repro:
> {code}
> 0: jdbc:hive2://localhost:10010> create transactional table acid_ptn_bucket1
> (a int, b int) partitioned by(ds string) clustered by (a) into 2 buckets
> stored as ORC TBLPROPERTIES('bucketing_version'='1', 'transactional'='true',
> 'transactional_properties'='default');
> No rows affected (0.418 seconds)
> 0: jdbc:hive2://localhost:10010> insert into acid_ptn_bucket1 partition (ds)
> values(1,2,'today'),(1,3,'today'),(1,4,'yesterday'),(2,2,'yesterday'),(2,3,'today'),(2,4,'today');
> 6 rows affected (3.695 seconds)
> {code}
> Data from ORC file (data as expected):
> {code}
> /apps/hive/warehouse/acid_ptn_bucket1/ds=today/delta_001_001_/bucket_0
> {"operation": 0, "originalTransaction": 1, "bucket": 536870912, "rowId": 0,
> "currentTransaction": 1, "row": {"a": 2, "b": 4}}
> {"operation": 0, "originalTransaction": 1, "bucket": 536870912, "rowId": 1,
> "currentTransaction": 1, "row": {"a": 2, "b": 3}}
> /apps/hive/warehouse/acid_ptn_bucket1/ds=today/delta_001_001_/bucket_1
> {"operation": 0, "originalTransaction": 1, "bucket": 536936448, "rowId": 0,
> "currentTransaction": 1, "row": {"a": 1, "b": 3}}
> {"operation": 0, "originalTransaction": 1, "bucket": 536936448, "rowId": 1,
> "currentTransaction": 1, "row": {"a": 1, "b": 2}}
> {code}
> Modifying table schema and inserting new data:
> {code}
> 0: jdbc:hive2://localhost:10010> alter table acid_ptn_bucket1 add columns(c
> int);
> No rows affected (0.541 seconds)
> 0: jdbc:hive2://localhost:10010> insert into acid_ptn_bucket1 partition (ds)
> values(3,2,1000,'yesterday'),(3,3,1001,'today'),(3,4,1002,'yesterday'),(4,2,1003,'today'),
> (4,3,1004,'yesterday'),(4,4,1005,'today');
> 6 rows affected (3.699 seconds)
> {code}
> Data from ORC file (wrong partitioning):
> {code}
> /apps/hive/warehouse/acid_ptn_bucket1/ds=today/delta_003_003_/bucket_0
> {"operation": 0, "originalTransaction": 3, "bucket": 536870912, "rowId": 0,
> "currentTransaction": 3, "row": {"a": 3, "b": 3, "c": 1001}}
> /apps/hive/warehouse/acid_ptn_bucket1/ds=today/delta_003_003_/bucket_1
> {"operation": 0, "originalTransaction": 3, "bucket": 536936448, "rowId": 0,
> "currentTransaction": 3, "row": {"a": 4, "b": 4, "c": 1005}}
> {"operation": 0, "originalTransaction": 3, "bucket": 536936448, "rowId": 1,
> "currentTransaction": 3, "row": {"a": 4, "b": 2, "c": 1003}}
> {code}
> As seen above, the expected behaviour is that new data with column 'a' being
> 3 should go to bucket1 and column 'a' being 4 should go to bucket0, but the
> partitioning is wrong.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21167) Bucketing: Bucketing version 1 is incorrectly partitioning data
[
https://issues.apache.org/jira/browse/HIVE-21167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deepak Jaiswal updated HIVE-21167:
--
Attachment: HIVE-21167.1.patch
> Bucketing: Bucketing version 1 is incorrectly partitioning data
> ---
>
> Key: HIVE-21167
> URL: https://issues.apache.org/jira/browse/HIVE-21167
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Vaibhav Gumashta
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-21167.1.patch
>
>
> Using murmur hash for bucketing columns was introduced in HIVE-18910,
> following which {{'bucketing_version'='1'}} stands for the old behaviour
> (where for example integer columns were partitioned based on mod values).
> Looks like we have a bug in the old bucketing scheme now. I could repro it
> when modified the existing schema using an alter table add column and adding
> new data. Repro:
> {code}
> 0: jdbc:hive2://localhost:10010> create transactional table acid_ptn_bucket1
> (a int, b int) partitioned by(ds string) clustered by (a) into 2 buckets
> stored as ORC TBLPROPERTIES('bucketing_version'='1', 'transactional'='true',
> 'transactional_properties'='default');
> No rows affected (0.418 seconds)
> 0: jdbc:hive2://localhost:10010> insert into acid_ptn_bucket1 partition (ds)
> values(1,2,'today'),(1,3,'today'),(1,4,'yesterday'),(2,2,'yesterday'),(2,3,'today'),(2,4,'today');
> 6 rows affected (3.695 seconds)
> {code}
> Data from ORC file (data as expected):
> {code}
> /apps/hive/warehouse/acid_ptn_bucket1/ds=today/delta_001_001_/bucket_0
> {"operation": 0, "originalTransaction": 1, "bucket": 536870912, "rowId": 0,
> "currentTransaction": 1, "row": {"a": 2, "b": 4}}
> {"operation": 0, "originalTransaction": 1, "bucket": 536870912, "rowId": 1,
> "currentTransaction": 1, "row": {"a": 2, "b": 3}}
> /apps/hive/warehouse/acid_ptn_bucket1/ds=today/delta_001_001_/bucket_1
> {"operation": 0, "originalTransaction": 1, "bucket": 536936448, "rowId": 0,
> "currentTransaction": 1, "row": {"a": 1, "b": 3}}
> {"operation": 0, "originalTransaction": 1, "bucket": 536936448, "rowId": 1,
> "currentTransaction": 1, "row": {"a": 1, "b": 2}}
> {code}
> Modifying table schema and inserting new data:
> {code}
> 0: jdbc:hive2://localhost:10010> alter table acid_ptn_bucket1 add columns(c
> int);
> No rows affected (0.541 seconds)
> 0: jdbc:hive2://localhost:10010> insert into acid_ptn_bucket1 partition (ds)
> values(3,2,1000,'yesterday'),(3,3,1001,'today'),(3,4,1002,'yesterday'),(4,2,1003,'today'),
> (4,3,1004,'yesterday'),(4,4,1005,'today');
> 6 rows affected (3.699 seconds)
> {code}
> Data from ORC file (wrong partitioning):
> {code}
> /apps/hive/warehouse/acid_ptn_bucket1/ds=today/delta_003_003_/bucket_0
> {"operation": 0, "originalTransaction": 3, "bucket": 536870912, "rowId": 0,
> "currentTransaction": 3, "row": {"a": 3, "b": 3, "c": 1001}}
> /apps/hive/warehouse/acid_ptn_bucket1/ds=today/delta_003_003_/bucket_1
> {"operation": 0, "originalTransaction": 3, "bucket": 536936448, "rowId": 0,
> "currentTransaction": 3, "row": {"a": 4, "b": 4, "c": 1005}}
> {"operation": 0, "originalTransaction": 3, "bucket": 536936448, "rowId": 1,
> "currentTransaction": 3, "row": {"a": 4, "b": 2, "c": 1003}}
> {code}
> As seen above, the expected behaviour is that new data with column 'a' being
> 3 should go to bucket1 and column 'a' being 4 should go to bucket0, but the
> partitioning is wrong.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21001) Upgrade to calcite-1.18
[ https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21001: Attachment: HIVE-21001.26.patch > Upgrade to calcite-1.18 > --- > > Key: HIVE-21001 > URL: https://issues.apache.org/jira/browse/HIVE-21001 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21001.01.patch, HIVE-21001.01.patch, > HIVE-21001.02.patch, HIVE-21001.03.patch, HIVE-21001.04.patch, > HIVE-21001.05.patch, HIVE-21001.06.patch, HIVE-21001.06.patch, > HIVE-21001.07.patch, HIVE-21001.08.patch, HIVE-21001.08.patch, > HIVE-21001.08.patch, HIVE-21001.09.patch, HIVE-21001.09.patch, > HIVE-21001.09.patch, HIVE-21001.10.patch, HIVE-21001.11.patch, > HIVE-21001.12.patch, HIVE-21001.13.patch, HIVE-21001.15.patch, > HIVE-21001.16.patch, HIVE-21001.17.patch, HIVE-21001.18.patch, > HIVE-21001.18.patch, HIVE-21001.19.patch, HIVE-21001.20.patch, > HIVE-21001.21.patch, HIVE-21001.22.patch, HIVE-21001.22.patch, > HIVE-21001.22.patch, HIVE-21001.23.patch, HIVE-21001.24.patch, > HIVE-21001.26.patch, HIVE-21001.26.patch, HIVE-21001.26.patch, > HIVE-21001.26.patch > > > XLEAR LIBRARY CACHE -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21261) Incremental REPL LOAD adds redundant COPY and MOVE tasks for external table events.
[ https://issues.apache.org/jira/browse/HIVE-21261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-21261: Status: Patch Available (was: Open) > Incremental REPL LOAD adds redundant COPY and MOVE tasks for external table > events. > --- > > Key: HIVE-21261 > URL: https://issues.apache.org/jira/browse/HIVE-21261 > Project: Hive > Issue Type: Improvement > Components: repl >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, replication > Attachments: HIVE-21261.01.patch > > > For external tables replication, the data gets copied as separate task based > on data locations listed in _external_tables_info file in the dump. So, > individual events such as ADD_PARTITION or INSERT on the external tables > should avoid copying data. So, it is enough to create table/add partition DDL > tasks. COPY and MOVE tasks should be skipped. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21261) Incremental REPL LOAD adds redundant COPY and MOVE tasks for external table events.
[ https://issues.apache.org/jira/browse/HIVE-21261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-21261: -- Labels: DR pull-request-available replication (was: DR replication) > Incremental REPL LOAD adds redundant COPY and MOVE tasks for external table > events. > --- > > Key: HIVE-21261 > URL: https://issues.apache.org/jira/browse/HIVE-21261 > Project: Hive > Issue Type: Improvement > Components: repl >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Attachments: HIVE-21261.01.patch > > > For external tables replication, the data gets copied as separate task based > on data locations listed in _external_tables_info file in the dump. So, > individual events such as ADD_PARTITION or INSERT on the external tables > should avoid copying data. So, it is enough to create table/add partition DDL > tasks. COPY and MOVE tasks should be skipped. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21261) Incremental REPL LOAD adds redundant COPY and MOVE tasks for external table events.
[ https://issues.apache.org/jira/browse/HIVE-21261?focusedWorklogId=198502=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-198502 ] ASF GitHub Bot logged work on HIVE-21261: - Author: ASF GitHub Bot Created on: 14/Feb/19 05:39 Start Date: 14/Feb/19 05:39 Worklog Time Spent: 10m Work Description: sankarh commented on pull request #533: HIVE-21261: Incremental REPL LOAD adds redundant COPY and MOVE tasks for external table events. URL: https://github.com/apache/hive/pull/533 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 198502) Time Spent: 10m Remaining Estimate: 0h > Incremental REPL LOAD adds redundant COPY and MOVE tasks for external table > events. > --- > > Key: HIVE-21261 > URL: https://issues.apache.org/jira/browse/HIVE-21261 > Project: Hive > Issue Type: Improvement > Components: repl >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Attachments: HIVE-21261.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > For external tables replication, the data gets copied as separate task based > on data locations listed in _external_tables_info file in the dump. So, > individual events such as ADD_PARTITION or INSERT on the external tables > should avoid copying data. So, it is enough to create table/add partition DDL > tasks. COPY and MOVE tasks should be skipped. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21261) Incremental REPL LOAD adds redundant COPY and MOVE tasks for external table events.
[ https://issues.apache.org/jira/browse/HIVE-21261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-21261: Attachment: HIVE-21261.01.patch > Incremental REPL LOAD adds redundant COPY and MOVE tasks for external table > events. > --- > > Key: HIVE-21261 > URL: https://issues.apache.org/jira/browse/HIVE-21261 > Project: Hive > Issue Type: Improvement > Components: repl >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, replication > Attachments: HIVE-21261.01.patch > > > For external tables replication, the data gets copied as separate task based > on data locations listed in _external_tables_info file in the dump. So, > individual events such as ADD_PARTITION or INSERT on the external tables > should avoid copying data. So, it is enough to create table/add partition DDL > tasks. COPY and MOVE tasks should be skipped. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21262) Running schemeAuthority.q and schemeAuthority2.q in parallel can cause flakyness
[ https://issues.apache.org/jira/browse/HIVE-21262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-21262: -- Attachment: HIVE-21262.2.patch > Running schemeAuthority.q and schemeAuthority2.q in parallel can cause > flakyness > > > Key: HIVE-21262 > URL: https://issues.apache.org/jira/browse/HIVE-21262 > Project: Hive > Issue Type: Bug >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-21262.2.patch, HIVE-21262.patch > > > Found by our internal ptest runs that if the schemeAuthority.q and the > schemeAuthority2.q is ended up in different batches and run parallel by > ptest, then the data files are removed after the first one is finished and > the second one fails because of the missing data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21264) Improvements Around CharTypeInfo
[
https://issues.apache.org/jira/browse/HIVE-21264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21264:
---
Status: Open (was: Patch Available)
> Improvements Around CharTypeInfo
>
>
> Key: HIVE-21264
> URL: https://issues.apache.org/jira/browse/HIVE-21264
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-21264.1.patch, HIVE-21264.2.patch
>
>
> The {{CharTypeInfo}} stores the type name of the data type (char/varchar) and
> the length (1-255). {{CharTypeInfo}} objects are often getting cached once
> they are created.
> The {{hashcode()}} and {{equals()}} of its sub-classes varchar and char are
> inconsistent.
> * Make hashcode and equals consistent (and fast)
> * Simplify the {{getQualifiedName}} implementation and reduce the scope to
> protected
> * Other related nits
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21264) Improvements Around CharTypeInfo
[
https://issues.apache.org/jira/browse/HIVE-21264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21264:
---
Attachment: HIVE-21264.2.patch
> Improvements Around CharTypeInfo
>
>
> Key: HIVE-21264
> URL: https://issues.apache.org/jira/browse/HIVE-21264
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-21264.1.patch, HIVE-21264.2.patch
>
>
> The {{CharTypeInfo}} stores the type name of the data type (char/varchar) and
> the length (1-255). {{CharTypeInfo}} objects are often getting cached once
> they are created.
> The {{hashcode()}} and {{equals()}} of its sub-classes varchar and char are
> inconsistent.
> * Make hashcode and equals consistent (and fast)
> * Simplify the {{getQualifiedName}} implementation and reduce the scope to
> protected
> * Other related nits
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21264) Improvements Around CharTypeInfo
[
https://issues.apache.org/jira/browse/HIVE-21264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21264:
---
Status: Patch Available (was: Open)
> Improvements Around CharTypeInfo
>
>
> Key: HIVE-21264
> URL: https://issues.apache.org/jira/browse/HIVE-21264
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-21264.1.patch, HIVE-21264.2.patch
>
>
> The {{CharTypeInfo}} stores the type name of the data type (char/varchar) and
> the length (1-255). {{CharTypeInfo}} objects are often getting cached once
> they are created.
> The {{hashcode()}} and {{equals()}} of its sub-classes varchar and char are
> inconsistent.
> * Make hashcode and equals consistent (and fast)
> * Simplify the {{getQualifiedName}} implementation and reduce the scope to
> protected
> * Other related nits
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21240) JSON SerDe Deserialize Re-Write
[ https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-21240: --- Status: Open (was: Patch Available) > JSON SerDe Deserialize Re-Write > --- > > Key: HIVE-21240 > URL: https://issues.apache.org/jira/browse/HIVE-21240 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 3.1.1, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch, > HIVE-21240.2.patch, HIVE-21240.3.patch, HIVE-21240.4.patch, HIVE-21240.5.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The JSON SerDe has a few issues, I will link them to this JIRA. > * Use Jackson Tree parser instead of manually parsing > * Added support for base-64 encoded data (the expected format when using JSON) > * Added support to skip blank lines (returns all columns as null values) > * Current JSON parser accepts, but does not apply, custom timestamp formats > in most cases > * Added some unit tests > * Added cache for column-name to column-index searches, currently O\(n\) for > each row processed, for each column in the row -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21240) JSON SerDe Deserialize Re-Write
[
https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21240:
---
Status: Patch Available (was: Open)
Added a {{HiveJsonWriter}} class to match with the existing {{HiveJsonReader}}
class. Replaced a lot of the custom code and uses the Jackson library for
generating the JSON text.
Added a few more unit tests as well.
> JSON SerDe Deserialize Re-Write
> ---
>
> Key: HIVE-21240
> URL: https://issues.apache.org/jira/browse/HIVE-21240
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
>Affects Versions: 3.1.1, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch,
> HIVE-21240.2.patch, HIVE-21240.3.patch, HIVE-21240.4.patch, HIVE-21240.5.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The JSON SerDe has a few issues, I will link them to this JIRA.
> * Use Jackson Tree parser instead of manually parsing
> * Added support for base-64 encoded data (the expected format when using JSON)
> * Added support to skip blank lines (returns all columns as null values)
> * Current JSON parser accepts, but does not apply, custom timestamp formats
> in most cases
> * Added some unit tests
> * Added cache for column-name to column-index searches, currently O\(n\) for
> each row processed, for each column in the row
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21240) JSON SerDe Deserialize Re-Write
[ https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-21240: --- Attachment: HIVE-21240.5.patch > JSON SerDe Deserialize Re-Write > --- > > Key: HIVE-21240 > URL: https://issues.apache.org/jira/browse/HIVE-21240 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 4.0.0, 3.1.1 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch, > HIVE-21240.2.patch, HIVE-21240.3.patch, HIVE-21240.4.patch, HIVE-21240.5.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The JSON SerDe has a few issues, I will link them to this JIRA. > * Use Jackson Tree parser instead of manually parsing > * Added support for base-64 encoded data (the expected format when using JSON) > * Added support to skip blank lines (returns all columns as null values) > * Current JSON parser accepts, but does not apply, custom timestamp formats > in most cases > * Added some unit tests > * Added cache for column-name to column-index searches, currently O\(n\) for > each row processed, for each column in the row -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC
[
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Seung-Hyun Cheong updated HIVE-21194:
-
Status: Open (was: Patch Available)
> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Affects Versions: 3.1.0
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: HIVE-21194.patch, image-2019-02-01-16-31-56-958.png,
> image-2019-02-01-16-32-17-093.png
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0]
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
> type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
> has version[2019-01-31T01:12:32.289+09:00] > task
> version[2019-01-30T16:58:29.992Z]
> at io.druid.indexing.common.task.KillTask.run(KillTask.java:94)
> ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
> at
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444)
> [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
> at
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416)
> [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [?:1.8.0_112]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_112]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_112]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
>
> h1. Reason
> h3. KillTask compares versions
> [KillTask.java#L88|https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillTask.java#L88]
> {code:java}
> if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
> throw new ISE(
> "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
> unusedSegment.getId(),
> unusedSegment.getVersion(),
> myLock.getVersion()
> );
> }
> {code}
>
> h3. KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")
> [TaskLockbox.java#L593|https://github.com/apache/incubator-druid/blob/8eae26fd4e7572060d112864dd3d5f6a865b9c89/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java#L593]
> {code:java}
> version = DateTimes.nowUtc().toString();
> {code}
>
> h3. Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")
> [DruidStorageHandler.java#L755|https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L755]
> {code:java}
> jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new
> DateTime().toString());
> {code}
>
>
> h1. Suggestion
> h3. Because druid uses UTC only, DruidStorageHandler should set a version of
> segment to UTC.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[
https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767841#comment-16767841
]
Hive QA commented on HIVE-21254:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958649/HIVE-21254.5.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16066/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16066/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16066/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-02-14 03:02:04.976
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-16066/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-02-14 03:02:04.980
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 292f6bb HIVE-21038 : Fix checkstyle for standalone-metastore
(Miklos Gergely via Ashutosh Chauhan)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 292f6bb HIVE-21038 : Fix checkstyle for standalone-metastore
(Miklos Gergely via Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-02-14 03:02:06.426
+ rm -rf ../yetus_PreCommit-HIVE-Build-16066
+ mkdir ../yetus_PreCommit-HIVE-Build-16066
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-16066
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-16066/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/PreUpgradeTool.java:
does not exist in index
error:
a/upgrade-acid/pre-upgrade/src/test/java/org/apache/hadoop/hive/upgrade/acid/TestPreUpgradeTool.java:
does not exist in index
Going to apply patch with: git apply -p1
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc8487374082687132198.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc8487374082687132198.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
protoc-jar: executing: [/tmp/protoc2926524034906857864.exe, --version]
libprotoc 2.5.0
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
does not exist: must build
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/parser/Filter.g
org/apache/hadoop/hive/metastore/parser/Filter.g
log4j:WARN No appenders could be found for logger (DataNucleus.Persistence).
log4j:WARN Please initialize the log4j system properly.
DataNucleus Enhancer (version 4.1.17) for API "JDO"
DataNucleus Enhancer completed with success for 41 classes.
ANTLR Parser Generator Version 3.5.2
Output file
[jira] [Commented] (HIVE-20506) HOS times out when cluster is full while Hive-on-MR waits
[
https://issues.apache.org/jira/browse/HIVE-20506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767839#comment-16767839
]
Brock Noland commented on HIVE-20506:
-
Yes it’d still exist without yarn.
> HOS times out when cluster is full while Hive-on-MR waits
> -
>
> Key: HIVE-20506
> URL: https://issues.apache.org/jira/browse/HIVE-20506
> Project: Hive
> Issue Type: Improvement
>Reporter: Brock Noland
>Assignee: Brock Noland
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20506-CDH5.14.2.patch, HIVE-20506.1.patch,
> HIVE-20506.2.patch, HIVE-20506.3.patch, Screen Shot 2018-09-07 at 8.10.37
> AM.png
>
>
> My understanding is as follows:
> Hive-on-MR when the cluster is full will wait for resources to be available
> before submitting a job. This is because the hadoop jar command is the
> primary mechanism Hive uses to know if a job is complete or failed.
>
> Hive-on-Spark will timeout after {{SPARK_RPC_CLIENT_CONNECT_TIMEOUT}} because
> the RPC client in the AppMaster doesn't connect back to the RPC Server in
> HS2.
> This is a behavior difference it'd be great to close.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20921) Oracle backed DbLockManager fail when drop/truncate acid table with large partitions
[
https://issues.apache.org/jira/browse/HIVE-20921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767837#comment-16767837
]
Hive QA commented on HIVE-20921:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958634/HIVE-20921.branch-2.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16065/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16065/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16065/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-02-14 02:59:07.089
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-16065/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z branch-2.1 ]]
+ [[ -d apache-github-branch-2.1-source ]]
+ [[ ! -d apache-github-branch-2.1-source/.git ]]
+ [[ ! -d apache-github-branch-2.1-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-02-14 02:59:07.162
+ cd apache-github-branch-2.1-source
+ git fetch origin
>From https://github.com/apache/hive
0083145..0b8cfa7 branch-2 -> origin/branch-2
4663e50..c791b90 branch-3 -> origin/branch-3
15eca86..008eda8 branch-3.1 -> origin/branch-3.1
6fbdf37..292f6bb master -> origin/master
+ git reset --hard HEAD
HEAD is now at 292a98f HIVE-16480: Empty vector batches of floats or doubles
gets EOFException (Owen O'Malley via Jesus Camacho Rodriguez)
+ git clean -f -d
+ git checkout branch-2.1
Already on 'branch-2.1'
Your branch is up-to-date with 'origin/branch-2.1'.
+ git reset --hard origin/branch-2.1
HEAD is now at 292a98f HIVE-16480: Empty vector batches of floats or doubles
gets EOFException (Owen O'Malley via Jesus Camacho Rodriguez)
+ git merge --ff-only origin/branch-2.1
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-02-14 02:59:29.176
+ rm -rf ../yetus_PreCommit-HIVE-Build-16065
+ mkdir ../yetus_PreCommit-HIVE-Build-16065
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-16065
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-16065/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java: does
not exist in index
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java:
does not exist in index
Going to apply patch with: git apply -p1
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
[ERROR] Failed to execute goal on project hive-shims-common: Could not resolve
dependencies for project
org.apache.hive.shims:hive-shims-common:jar:2.1.2-SNAPSHOT: Could not find
artifact org.apache.hadoop:hadoop-hdfs:jar:2.6.1 in datanucleus
(http://www.datanucleus.org/downloads/maven2) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hive-shims-common
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-16065
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958634 - PreCommit-HIVE-Build
> Oracle backed DbLockManager fail when drop/truncate acid table with large
> partitions
>
>
> Key: HIVE-20921
> URL:
[jira] [Commented] (HIVE-21001) Upgrade to calcite-1.18
[
https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767835#comment-16767835
]
Hive QA commented on HIVE-21001:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958630/HIVE-21001.26.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16064/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16064/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16064/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-02-14 02:56:41.561
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-16064/source-prep.txt
+ [[ true == \t\r\u\e ]]
+ rm -rf ivy maven
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-02-14 02:56:42.223
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 292f6bb HIVE-21038 : Fix checkstyle for standalone-metastore
(Miklos Gergely via Ashutosh Chauhan)
+ git clean -f -d
Removing ${project.basedir}/
Removing itests/${project.basedir}/
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 292f6bb HIVE-21038 : Fix checkstyle for standalone-metastore
(Miklos Gergely via Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-02-14 02:56:42.923
+ rm -rf ../yetus_PreCommit-HIVE-Build-16064
+ mkdir ../yetus_PreCommit-HIVE-Build-16064
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-16064
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-16064/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: git apply -p0
/data/hiveptest/working/scratch/build.patch:847: trailing whitespace.
Map 1
/data/hiveptest/working/scratch/build.patch:868: trailing whitespace.
Reducer 2
/data/hiveptest/working/scratch/build.patch:927: trailing whitespace.
Map 1
/data/hiveptest/working/scratch/build.patch:948: trailing whitespace.
Reducer 2
/data/hiveptest/working/scratch/build.patch:2231: trailing whitespace.
null sort order:
warning: squelched 79 whitespace errors
warning: 84 lines add whitespace errors.
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc6149190882754146828.exe, --version]
protoc-jar: executing: [/tmp/protoc6149190882754146828.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
libprotoc 2.5.0
ANTLR Parser Generator Version 3.5.2
protoc-jar: executing: [/tmp/protoc1503705447598368534.exe, --version]
libprotoc 2.5.0
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
does not exist: must build
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/parser/Filter.g
[jira] [Commented] (HIVE-21217) Optimize range calculation for PTF
[
https://issues.apache.org/jira/browse/HIVE-21217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767833#comment-16767833
]
Hive QA commented on HIVE-21217:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958607/HIVE-21217.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15806 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16063/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16063/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16063/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958607 - PreCommit-HIVE-Build
> Optimize range calculation for PTF
> --
>
> Key: HIVE-21217
> URL: https://issues.apache.org/jira/browse/HIVE-21217
> Project: Hive
> Issue Type: Improvement
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-21217.0.patch, HIVE-21217.1.patch,
> HIVE-21217.2.patch
>
>
> During window function execution Hive has to iterate on neighbouring rows of
> the current row to find the beginning and end of the proper range (on which
> the aggregation will be executed).
> When we're using range based windows and have many rows with a certain key
> value this can take a lot of time. (e.g. partition size of 80M, in which we
> have 2 ranges of 40M rows according to the orderby column: within these 40M
> rowsets we're doing 40M x 40M/2 steps.. which is of n^2 time complexity)
> I propose to introduce a cache that keeps track of already calculated range
> ends so it can be reused in future scans.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21217) Optimize range calculation for PTF
[
https://issues.apache.org/jira/browse/HIVE-21217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767801#comment-16767801
]
Hive QA commented on HIVE-21217:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
1s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
4s{color} | {color:blue} ql in master has 2262 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 1 new + 139 unchanged - 34
fixed = 140 total (was 173) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16063/dev-support/hive-personality.sh
|
| git revision | master / 292f6bb |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16063/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16063/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Optimize range calculation for PTF
> --
>
> Key: HIVE-21217
> URL: https://issues.apache.org/jira/browse/HIVE-21217
> Project: Hive
> Issue Type: Improvement
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-21217.0.patch, HIVE-21217.1.patch,
> HIVE-21217.2.patch
>
>
> During window function execution Hive has to iterate on neighbouring rows of
> the current row to find the beginning and end of the proper range (on which
> the aggregation will be executed).
> When we're using range based windows and have many rows with a certain key
> value this can take a lot of time. (e.g. partition size of 80M, in which we
> have 2 ranges of 40M rows according to the orderby column: within these 40M
> rowsets we're doing 40M x 40M/2 steps.. which is of n^2 time complexity)
> I propose to introduce a cache that keeps track of already calculated range
> ends so it can be reused in future scans.
--
This message was sent by Atlassian JIRA
[jira] [Commented] (HIVE-21232) LLAP: Add a cache-miss friendly split affinity provider
[ https://issues.apache.org/jira/browse/HIVE-21232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767793#comment-16767793 ] slim bouguerra commented on HIVE-21232: --- [~gopalv] please re upload to rerun the test and adjust the Documentations (Java docs) to the new behavior (it will be very helpful). overall +1 > LLAP: Add a cache-miss friendly split affinity provider > --- > > Key: HIVE-21232 > URL: https://issues.apache.org/jira/browse/HIVE-21232 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Attachments: HIVE-21232.1.patch > > > If one of the LLAP nodes have data-locality, preferring that over another > does have advantages for the first query or a more general cache-miss. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21266) Issue with single delta file
[
https://issues.apache.org/jira/browse/HIVE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-21266:
--
Component/s: Transactions
> Issue with single delta file
>
>
> Key: HIVE-21266
> URL: https://issues.apache.org/jira/browse/HIVE-21266
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
>
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java#L353-L357]
>
> {noformat}
> if ((deltaCount + (dir.getBaseDirectory() == null ? 0 : 1)) + origCount <= 1)
> {
> LOG.debug("Not compacting {}; current base is {} and there are {}
> deltas and {} originals", sd.getLocation(), dir
> .getBaseDirectory(), deltaCount, origCount);
> return;
> }
> {noformat}
> Is problematic.
> Suppose you have 1 delta file from streaming ingest: {{delta_11_20}} where
> {{txnid:13}} was aborted. The code above will not rewrite the delta (which
> drops anything that belongs to the aborted txn) and transition the compaction
> to "ready_for_cleaning" state which will drop the metadata about the aborted
> txn in {{markCleaned()}}. Now aborted data will come back as committed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21240) JSON SerDe Deserialize Re-Write
[
https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767750#comment-16767750
]
Hive QA commented on HIVE-21240:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958594/HIVE-21240.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16062/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16062/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16062/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12958594/HIVE-21240.4.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958594 - PreCommit-HIVE-Build
> JSON SerDe Deserialize Re-Write
> ---
>
> Key: HIVE-21240
> URL: https://issues.apache.org/jira/browse/HIVE-21240
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
>Affects Versions: 4.0.0, 3.1.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch,
> HIVE-21240.2.patch, HIVE-21240.3.patch, HIVE-21240.4.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The JSON SerDe has a few issues, I will link them to this JIRA.
> * Use Jackson Tree parser instead of manually parsing
> * Added support for base-64 encoded data (the expected format when using JSON)
> * Added support to skip blank lines (returns all columns as null values)
> * Current JSON parser accepts, but does not apply, custom timestamp formats
> in most cases
> * Added some unit tests
> * Added cache for column-name to column-index searches, currently O\(n\) for
> each row processed, for each column in the row
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767735#comment-16767735 ] Eugene Koifman commented on HIVE-21254: --- +1 patch 5 pending tests > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch, HIVE-21254.5.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767731#comment-16767731 ] Prasanth Jayachandran commented on HIVE-21254: -- Updated patch to fail with workaround suggestion in case access control exception. > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch, HIVE-21254.5.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21254: - Attachment: HIVE-21254.5.patch > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch, HIVE-21254.5.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21267) Extend HiveRelColumnsAlignment to reorder group-by and join keys on decreasing NDV automatically
[
https://issues.apache.org/jira/browse/HIVE-21267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez reassigned HIVE-21267:
--
Assignee: (was: Jesus Camacho Rodriguez)
> Extend HiveRelColumnsAlignment to reorder group-by and join keys on
> decreasing NDV automatically
>
>
> Key: HIVE-21267
> URL: https://issues.apache.org/jira/browse/HIVE-21267
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Priority: Major
> Labels: newbie++
>
> {{HiveRelColumnsAlignment}} was introduced to align the order of columns in
> join, group-by, and order-by operators in the plan pipeline, trying to
> increase the effect of ReduceDeduplication and thus reducing data shuffle.
> The optimization could be extended to reorder group-by and join keys on
> decreasing NDV, which would accelerate comparison runtime.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21267) Extend HiveRelColumnsAlignment to reorder group-by and join keys on decreasing NDV automatically
[
https://issues.apache.org/jira/browse/HIVE-21267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-21267:
---
Labels: newbie++ (was: )
> Extend HiveRelColumnsAlignment to reorder group-by and join keys on
> decreasing NDV automatically
>
>
> Key: HIVE-21267
> URL: https://issues.apache.org/jira/browse/HIVE-21267
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: newbie++
>
> {{HiveRelColumnsAlignment}} was introduced to align the order of columns in
> join, group-by, and order-by operators in the plan pipeline, trying to
> increase the effect of ReduceDeduplication and thus reducing data shuffle.
> The optimization could be extended to reorder group-by and join keys on
> decreasing NDV, which would accelerate comparison runtime.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21267) Extend HiveRelColumnsAlignment to reorder group-by and join keys on decreasing NDV automatically
[
https://issues.apache.org/jira/browse/HIVE-21267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez reassigned HIVE-21267:
--
> Extend HiveRelColumnsAlignment to reorder group-by and join keys on
> decreasing NDV automatically
>
>
> Key: HIVE-21267
> URL: https://issues.apache.org/jira/browse/HIVE-21267
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> {{HiveRelColumnsAlignment}} was introduced to align the order of columns in
> join, group-by, and order-by operators in the plan pipeline, trying to
> increase the effect of ReduceDeduplication and thus reducing data shuffle.
> The optimization could be extended to reorder group-by and join keys on
> decreasing NDV, which would accelerate comparison runtime.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21262) Running schemeAuthority.q and schemeAuthority2.q in parallel can cause flakyness
[
https://issues.apache.org/jira/browse/HIVE-21262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767681#comment-16767681
]
Hive QA commented on HIVE-21262:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958585/HIVE-21262.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15772 tests
executed
*Failed tests:*
{noformat}
TestCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=30)
[autogen_colalias.q,notable_alias3.q,avro_decimal_native.q,vector_acid3.q,nestedvirtual.q,insert_overwrite_local_directory_1.q,correlated_join_keys.q,ppr_pushdown3.q,vector_reduce1.q,groupby2_map.q,tez_vector_dynpart_hashjoin_2.q,rcfile_null_value.q,join24.q,delete_all_partitioned.q,skewjoin_union_remove_2.q,masking_11.q,bucketmapjoin_negative3.q,authorization_cli_createtab.q,cbo_rp_windowing_2.q,insert_non_utf8_encoding_table.q,vector_reduce3.q,fold_when.q,orc_empty_table.q,smb_cache.q,hybridgrace_hashjoin_1.q,union_assertion_type.q,join18_multi_distinct.q,udf_log10.q,exim_19_part_external_location.q,quotedid_smb.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16061/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16061/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16061/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958585 - PreCommit-HIVE-Build
> Running schemeAuthority.q and schemeAuthority2.q in parallel can cause
> flakyness
>
>
> Key: HIVE-21262
> URL: https://issues.apache.org/jira/browse/HIVE-21262
> Project: Hive
> Issue Type: Bug
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-21262.patch
>
>
> Found by our internal ptest runs that if the schemeAuthority.q and the
> schemeAuthority2.q is ended up in different batches and run parallel by
> ptest, then the data files are removed after the first one is finished and
> the second one fails because of the missing data.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20506) HOS times out when cluster is full while Hive-on-MR waits
[
https://issues.apache.org/jira/browse/HIVE-20506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767670#comment-16767670
]
t oo commented on HIVE-20506:
-
Does Hive-on-Spark still have the issue if Spark is setup with standalone
scheduler rather than YARN?
> HOS times out when cluster is full while Hive-on-MR waits
> -
>
> Key: HIVE-20506
> URL: https://issues.apache.org/jira/browse/HIVE-20506
> Project: Hive
> Issue Type: Improvement
>Reporter: Brock Noland
>Assignee: Brock Noland
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20506-CDH5.14.2.patch, HIVE-20506.1.patch,
> HIVE-20506.2.patch, HIVE-20506.3.patch, Screen Shot 2018-09-07 at 8.10.37
> AM.png
>
>
> My understanding is as follows:
> Hive-on-MR when the cluster is full will wait for resources to be available
> before submitting a job. This is because the hadoop jar command is the
> primary mechanism Hive uses to know if a job is complete or failed.
>
> Hive-on-Spark will timeout after {{SPARK_RPC_CLIENT_CONNECT_TIMEOUT}} because
> the RPC client in the AppMaster doesn't connect back to the RPC Server in
> HS2.
> This is a behavior difference it'd be great to close.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20921) Oracle backed DbLockManager fail when drop/truncate acid table with large partitions
[
https://issues.apache.org/jira/browse/HIVE-20921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-20921:
--
Attachment: (was: HIVE-20921.branch-2.1.patch)
> Oracle backed DbLockManager fail when drop/truncate acid table with large
> partitions
>
>
> Key: HIVE-20921
> URL: https://issues.apache.org/jira/browse/HIVE-20921
> Project: Hive
> Issue Type: Bug
> Components: Locking
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-20921.branch-2.1.patch
>
>
> Stack:
> {code}
> org.apache.hive.service.cli.HiveSQLException: Error while processing
> statement: FAILED: Error in acquiring locks: Error communicating with the
> metastore
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:324)
>
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:199)
>
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:76)
>
> at
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:255)
>
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:266)
>
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.lockmgr.LockException: Error
> communicating with the metastore
> at
> org.apache.hadoop.hive.ql.lockmgr.DbLockManager.lock(DbLockManager.java:177)
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocks(DbTxnManager.java:357)
>
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocksWithHeartbeatDelay(DbTxnManager.java:373)
>
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocks(DbTxnManager.java:182)
>
> at org.apache.hadoop.hive.ql.Driver.acquireLocksAndOpenTxn(Driver.java:1082)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1284)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1156)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:197)
>
> ... 11 more
> Caused by: MetaException(message:How did we get here, we heartbeated our lock
> before we started! ( lockid:466073 intLockId:701 txnid:0 db:v5x2442
> table:tbstcnf_load_stg_step
> partition:src_system_cd=MAXIMO/src_hostname_cd=PRD1310/src_table_name=LABTRANS
> state:WAITING type:EXCLUSIVE))
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLock(TxnHandler.java:2642)
>
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLock(TxnHandler.java:1187)
>
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.check_lock(HiveMetaStore.java:6161)
>
> at sun.reflect.GeneratedMethodAccessor135.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
>
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
>
> at com.sun.proxy.$Proxy14.check_lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.checkLock(HiveMetaStoreClient.java:1984)
>
> at sun.reflect.GeneratedMethodAccessor134.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:178)
>
> at com.sun.proxy.$Proxy15.checkLock(Unknown Source)
> at
> org.apache.hadoop.hive.ql.lockmgr.DbLockManager.lock(DbLockManager.java:114)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20921) Oracle backed DbLockManager fail when drop/truncate acid table with large partitions
[
https://issues.apache.org/jira/browse/HIVE-20921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-20921:
--
Attachment: HIVE-20921.branch-2.1.patch
> Oracle backed DbLockManager fail when drop/truncate acid table with large
> partitions
>
>
> Key: HIVE-20921
> URL: https://issues.apache.org/jira/browse/HIVE-20921
> Project: Hive
> Issue Type: Bug
> Components: Locking
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-20921.branch-2.1.patch
>
>
> Stack:
> {code}
> org.apache.hive.service.cli.HiveSQLException: Error while processing
> statement: FAILED: Error in acquiring locks: Error communicating with the
> metastore
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:324)
>
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:199)
>
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:76)
>
> at
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:255)
>
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:266)
>
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.lockmgr.LockException: Error
> communicating with the metastore
> at
> org.apache.hadoop.hive.ql.lockmgr.DbLockManager.lock(DbLockManager.java:177)
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocks(DbTxnManager.java:357)
>
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocksWithHeartbeatDelay(DbTxnManager.java:373)
>
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocks(DbTxnManager.java:182)
>
> at org.apache.hadoop.hive.ql.Driver.acquireLocksAndOpenTxn(Driver.java:1082)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1284)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1156)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:197)
>
> ... 11 more
> Caused by: MetaException(message:How did we get here, we heartbeated our lock
> before we started! ( lockid:466073 intLockId:701 txnid:0 db:v5x2442
> table:tbstcnf_load_stg_step
> partition:src_system_cd=MAXIMO/src_hostname_cd=PRD1310/src_table_name=LABTRANS
> state:WAITING type:EXCLUSIVE))
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLock(TxnHandler.java:2642)
>
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLock(TxnHandler.java:1187)
>
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.check_lock(HiveMetaStore.java:6161)
>
> at sun.reflect.GeneratedMethodAccessor135.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
>
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
>
> at com.sun.proxy.$Proxy14.check_lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.checkLock(HiveMetaStoreClient.java:1984)
>
> at sun.reflect.GeneratedMethodAccessor134.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:178)
>
> at com.sun.proxy.$Proxy15.checkLock(Unknown Source)
> at
> org.apache.hadoop.hive.ql.lockmgr.DbLockManager.lock(DbLockManager.java:114)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21177) Optimize AcidUtils.getLogicalLength()
[
https://issues.apache.org/jira/browse/HIVE-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-21177:
--
Release Note:
I messed up the commit msg for this. The Jira number is correct, but the bug
title is from another issue
{noformat}
commit 07b76f413cb174413f0530a6aae5ae442a301b46
Author: Eugene Koifman
Date: Thu Feb 7 09:49:19 2019 -0800
HIVE-21177: ACID: When there are no delete deltas skip finding min max keys
(Eugene Koifman, reviewed by Prasanth Jayachandran)
{noformat}
was:n/a
> Optimize AcidUtils.getLogicalLength()
> -
>
> Key: HIVE-21177
> URL: https://issues.apache.org/jira/browse/HIVE-21177
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21177.01.patch, HIVE-21177.02.patch,
> HIVE-21177.03.patch
>
>
> {{AcidUtils.getLogicalLength()}} - tries look for the side file
> {{OrcAcidUtils.getSideFile()}} on the file system even when the file couldn't
> possibly be there, e.g. when the path is delta_x_x or base_x. It could only
> be there in delta_x_y, x != y.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21001) Upgrade to calcite-1.18
[ https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21001: Attachment: HIVE-21001.26.patch > Upgrade to calcite-1.18 > --- > > Key: HIVE-21001 > URL: https://issues.apache.org/jira/browse/HIVE-21001 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21001.01.patch, HIVE-21001.01.patch, > HIVE-21001.02.patch, HIVE-21001.03.patch, HIVE-21001.04.patch, > HIVE-21001.05.patch, HIVE-21001.06.patch, HIVE-21001.06.patch, > HIVE-21001.07.patch, HIVE-21001.08.patch, HIVE-21001.08.patch, > HIVE-21001.08.patch, HIVE-21001.09.patch, HIVE-21001.09.patch, > HIVE-21001.09.patch, HIVE-21001.10.patch, HIVE-21001.11.patch, > HIVE-21001.12.patch, HIVE-21001.13.patch, HIVE-21001.15.patch, > HIVE-21001.16.patch, HIVE-21001.17.patch, HIVE-21001.18.patch, > HIVE-21001.18.patch, HIVE-21001.19.patch, HIVE-21001.20.patch, > HIVE-21001.21.patch, HIVE-21001.22.patch, HIVE-21001.22.patch, > HIVE-21001.22.patch, HIVE-21001.23.patch, HIVE-21001.24.patch, > HIVE-21001.26.patch > > > XLEAR LIBRARY CACHE -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21001) Upgrade to calcite-1.18
[ https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21001: Attachment: HIVE-21001.26.patch > Upgrade to calcite-1.18 > --- > > Key: HIVE-21001 > URL: https://issues.apache.org/jira/browse/HIVE-21001 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21001.01.patch, HIVE-21001.01.patch, > HIVE-21001.02.patch, HIVE-21001.03.patch, HIVE-21001.04.patch, > HIVE-21001.05.patch, HIVE-21001.06.patch, HIVE-21001.06.patch, > HIVE-21001.07.patch, HIVE-21001.08.patch, HIVE-21001.08.patch, > HIVE-21001.08.patch, HIVE-21001.09.patch, HIVE-21001.09.patch, > HIVE-21001.09.patch, HIVE-21001.10.patch, HIVE-21001.11.patch, > HIVE-21001.12.patch, HIVE-21001.13.patch, HIVE-21001.15.patch, > HIVE-21001.16.patch, HIVE-21001.17.patch, HIVE-21001.18.patch, > HIVE-21001.18.patch, HIVE-21001.19.patch, HIVE-21001.20.patch, > HIVE-21001.21.patch, HIVE-21001.22.patch, HIVE-21001.22.patch, > HIVE-21001.22.patch, HIVE-21001.23.patch, HIVE-21001.24.patch, > HIVE-21001.26.patch, HIVE-21001.26.patch, HIVE-21001.26.patch > > > XLEAR LIBRARY CACHE -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21001) Upgrade to calcite-1.18
[ https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21001: Attachment: HIVE-21001.26.patch > Upgrade to calcite-1.18 > --- > > Key: HIVE-21001 > URL: https://issues.apache.org/jira/browse/HIVE-21001 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21001.01.patch, HIVE-21001.01.patch, > HIVE-21001.02.patch, HIVE-21001.03.patch, HIVE-21001.04.patch, > HIVE-21001.05.patch, HIVE-21001.06.patch, HIVE-21001.06.patch, > HIVE-21001.07.patch, HIVE-21001.08.patch, HIVE-21001.08.patch, > HIVE-21001.08.patch, HIVE-21001.09.patch, HIVE-21001.09.patch, > HIVE-21001.09.patch, HIVE-21001.10.patch, HIVE-21001.11.patch, > HIVE-21001.12.patch, HIVE-21001.13.patch, HIVE-21001.15.patch, > HIVE-21001.16.patch, HIVE-21001.17.patch, HIVE-21001.18.patch, > HIVE-21001.18.patch, HIVE-21001.19.patch, HIVE-21001.20.patch, > HIVE-21001.21.patch, HIVE-21001.22.patch, HIVE-21001.22.patch, > HIVE-21001.22.patch, HIVE-21001.23.patch, HIVE-21001.24.patch, > HIVE-21001.26.patch, HIVE-21001.26.patch > > > XLEAR LIBRARY CACHE -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21262) Running schemeAuthority.q and schemeAuthority2.q in parallel can cause flakyness
[
https://issues.apache.org/jira/browse/HIVE-21262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767627#comment-16767627
]
Hive QA commented on HIVE-21262:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16061/dev-support/hive-personality.sh
|
| git revision | master / 292f6bb |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16061/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Running schemeAuthority.q and schemeAuthority2.q in parallel can cause
> flakyness
>
>
> Key: HIVE-21262
> URL: https://issues.apache.org/jira/browse/HIVE-21262
> Project: Hive
> Issue Type: Bug
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-21262.patch
>
>
> Found by our internal ptest runs that if the schemeAuthority.q and the
> schemeAuthority2.q is ended up in different batches and run parallel by
> ptest, then the data files are removed after the first one is finished and
> the second one fails because of the missing data.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21240) JSON SerDe Deserialize Re-Write
[
https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767619#comment-16767619
]
Hive QA commented on HIVE-21240:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958594/HIVE-21240.4.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15803 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniHiveKafkaCliDriver.testCliDriver[kafka_storage_handler]
(batchId=275)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16060/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16060/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16060/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958594 - PreCommit-HIVE-Build
> JSON SerDe Deserialize Re-Write
> ---
>
> Key: HIVE-21240
> URL: https://issues.apache.org/jira/browse/HIVE-21240
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
>Affects Versions: 4.0.0, 3.1.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch,
> HIVE-21240.2.patch, HIVE-21240.3.patch, HIVE-21240.4.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The JSON SerDe has a few issues, I will link them to this JIRA.
> * Use Jackson Tree parser instead of manually parsing
> * Added support for base-64 encoded data (the expected format when using JSON)
> * Added support to skip blank lines (returns all columns as null values)
> * Current JSON parser accepts, but does not apply, custom timestamp formats
> in most cases
> * Added some unit tests
> * Added cache for column-name to column-index searches, currently O\(n\) for
> each row processed, for each column in the row
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21240) JSON SerDe Deserialize Re-Write
[
https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767593#comment-16767593
]
Hive QA commented on HIVE-21240:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
51s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} serde in master has 197 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
13s{color} | {color:blue} ql in master has 2262 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} hcatalog/core in master has 29 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
34s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} serde: The patch generated 6 new + 22 unchanged - 7
fixed = 28 total (was 29) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 2 new + 6 unchanged - 5 fixed
= 8 total (was 11) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
55s{color} | {color:red} serde generated 1 new + 193 unchanged - 4 fixed = 194
total (was 197) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 16s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:serde |
| | Boxing/unboxing to parse a primitive
org.apache.hadoop.hive.serde2.json.HiveJsonReader.getColIndex(String) At
HiveJsonReader.java:org.apache.hadoop.hive.serde2.json.HiveJsonReader.getColIndex(String)
At HiveJsonReader.java:[line 488] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16060/dev-support/hive-personality.sh
|
| git revision | master / 292f6bb |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16060/yetus/diff-checkstyle-serde.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16060/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16060/yetus/whitespace-eol.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16060/yetus/new-findbugs-serde.html
|
| modules | C: serde ql hcatalog/core U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16060/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
>
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767581#comment-16767581 ] Prasanth Jayachandran commented on HIVE-21254: -- Yup. Just discussed with [~ashutoshc] about the same. Detect ACL Exception and explicitly throw with property to allow read access and retry. Will update the patch. > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767572#comment-16767572 ] Eugene Koifman commented on HIVE-21254: --- I would think it's a security hole if you can set this from client. Perhaps the utility should fail if it gets an ACL exception and include this prop in the msg... > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21266) Issue with single delta file
[
https://issues.apache.org/jira/browse/HIVE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-21266:
--
Description:
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java#L353-L357]
{noformat}
if ((deltaCount + (dir.getBaseDirectory() == null ? 0 : 1)) + origCount <= 1) {
LOG.debug("Not compacting {}; current base is {} and there are {} deltas
and {} originals", sd.getLocation(), dir
.getBaseDirectory(), deltaCount, origCount);
return;
}
{noformat}
Is problematic.
Suppose you have 1 delta file from streaming ingest: {{delta_11_20}} where
{{txnid:13}} was aborted. The code above will not rewrite the delta (which
drops anything that belongs to the aborted txn) and transition the compaction
to "ready_for_cleaning" state which will drop the metadata about the aborted
txn in {{markCleaned()}}. Now aborted data will come back as committed.
was:
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java#L353-L357]
{noformat}
if ((deltaCount + (dir.getBaseDirectory() == null ? 0 : 1)) + origCount <= 1) {
LOG.debug("Not compacting {}; current base is {} and there are {} deltas
and {} originals", sd.getLocation(), dir
.getBaseDirectory(), deltaCount, origCount);
return;
}
{noformat}
Is problematic.
Suppose you have 1 delta file from streaming ingest: {{delta_11_20}} where
{{txnid:13}} was aborted. The code above will not rewrite the delta (which
drops anything that belongs to the aborted txn) and transition the compaction
to "ready_for_cleaning" which will drop the metadata about the aborted txn.
Now aborted data will come back as committed.
> Issue with single delta file
>
>
> Key: HIVE-21266
> URL: https://issues.apache.org/jira/browse/HIVE-21266
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
>
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java#L353-L357]
>
> {noformat}
> if ((deltaCount + (dir.getBaseDirectory() == null ? 0 : 1)) + origCount <= 1)
> {
> LOG.debug("Not compacting {}; current base is {} and there are {}
> deltas and {} originals", sd.getLocation(), dir
> .getBaseDirectory(), deltaCount, origCount);
> return;
> }
> {noformat}
> Is problematic.
> Suppose you have 1 delta file from streaming ingest: {{delta_11_20}} where
> {{txnid:13}} was aborted. The code above will not rewrite the delta (which
> drops anything that belongs to the aborted txn) and transition the compaction
> to "ready_for_cleaning" state which will drop the metadata about the aborted
> txn in {{markCleaned()}}. Now aborted data will come back as committed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21266) Issue with single delta file
[
https://issues.apache.org/jira/browse/HIVE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman reassigned HIVE-21266:
-
> Issue with single delta file
>
>
> Key: HIVE-21266
> URL: https://issues.apache.org/jira/browse/HIVE-21266
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
>
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java#L353-L357]
>
> {noformat}
> if ((deltaCount + (dir.getBaseDirectory() == null ? 0 : 1)) + origCount <= 1)
> {
> LOG.debug("Not compacting {}; current base is {} and there are {}
> deltas and {} originals", sd.getLocation(), dir
> .getBaseDirectory(), deltaCount, origCount);
> return;
> }
> {noformat}
> Is problematic.
> Suppose you have 1 delta file from streaming ingest: {{delta_11_20}} where
> {{txnid:13}} was aborted. The code above will not rewrite the delta (which
> drops anything that belongs to the aborted txn) and transition the compaction
> to "ready_for_cleaning" which will drop the metadata about the aborted txn.
> Now aborted data will come back as committed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20758) Constraints: Show create table does not show constraints
[
https://issues.apache.org/jira/browse/HIVE-20758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767560#comment-16767560
]
Hive QA commented on HIVE-20758:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958579/HIVE-20758.7.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15786 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16059/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16059/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16059/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958579 - PreCommit-HIVE-Build
> Constraints: Show create table does not show constraints
>
>
> Key: HIVE-20758
> URL: https://issues.apache.org/jira/browse/HIVE-20758
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Barnabas Maidics
>Priority: Major
> Attachments: HIVE-20758.1.patch, HIVE-20758.2.patch,
> HIVE-20758.3.patch, HIVE-20758.4.patch, HIVE-20758.5.patch,
> HIVE-20758.6.patch, HIVE-20758.7.patch, Screen Shot 2019-01-23 at 11.52.04.png
>
>
> Even though the desc formatted shows the constraints, the show create table
> does not
> {code}
> | # Primary Key | NULL
> | NULL
>|
> | Table: |
> tpcds_bin_partitioned_orc_1.inventory | NULL
> |
> | Constraint Name: | pk_in
> | NULL
>|
> | Column Names: | inv_date_sk
> | inv_item_sk
>|
> || NULL
> | NULL
>|
> | # Foreign Keys | NULL
> | NULL
>|
> | Table: |
> tpcds_bin_partitioned_orc_1.inventory | NULL
> |
> | Constraint Name: | inv_d
> | NULL
>|
> | Parent Column Name:tpcds_bin_partitioned_orc_1.date_dim.d_date_sk |
> Column Name:inv_date_sk| Key Sequence:1
> |
> || NULL
> | NULL
>|
> | Constraint Name: | inv_i
> | NULL
>|
> | Parent Column Name:tpcds_bin_partitioned_orc_1.item.i_item_sk | Column
> Name:inv_item_sk| Key Sequence:1
>|
> || NULL
> | NULL
>|
> | Constraint Name: | inv_w
> | NULL
>|
> | Parent Column Name:tpcds_bin_partitioned_orc_1.warehouse.w_warehouse_sk
> | Column Name:inv_warehouse_sk | Key Sequence:1
> |
> || NULL
> | NULL
>|
> {code}
> But
> {code}
> ++
> | createtab_stmt |
> ++
> | CREATE TABLE `inventory`( |
> | `inv_item_sk` bigint,|
> | `inv_warehouse_sk` bigint, |
> |
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767559#comment-16767559 ] Prasanth Jayachandran commented on HIVE-21254: -- Not sure if we can do that since it could be a server side config. > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767555#comment-16767555 ] Prasanth Jayachandran commented on HIVE-21254: -- Seems like there is flag to allow read access on db and tables. hive.security.metastore.authorization.auth.reads Should we set this to false in pre-upgrade tool explicitly to get read access? > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767544#comment-16767544 ] Prasanth Jayachandran commented on HIVE-21254: -- Same is applicable for pre-upgrade tool as well. If getTable() in pre-upgrade tool fails with ACL exception then what are the options? This patch is skipping such cases. > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[
https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767539#comment-16767539
]
Eugene Koifman commented on HIVE-21254:
---
Compactor has to have {{Table}} object to do anything. If it cannot do that,
it will fail the compaction.
> Pre-upgrade tool should handle exceptions and skip db/tables
>
>
> Key: HIVE-21254
> URL: https://issues.apache.org/jira/browse/HIVE-21254
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch,
> HIVE-21254.3.patch, HIVE-21254.4.patch
>
>
> When exceptions like AccessControlException is thrown, pre-upgrade tool
> fails. If hive user does not have read access to database or tables (some
> external tables denies read access to hive), pre-upgrade tool should just
> assume they are external tables and move on without failing pre-upgrade
> process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767537#comment-16767537 ] Prasanth Jayachandran edited comment on HIVE-21254 at 2/13/19 7:57 PM: --- Also how does compactor know the table user, if it cannot do getTable() to get location and find user? (getTable() throws ACL exception if hive user does not have read access to end user location) was (Author: prasanth_j): Also how does compactor know the table user if it cannot do getTable() to get location and find user? getTable() throws ACL exception if hive user does not have read access to end user location. > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767537#comment-16767537 ] Prasanth Jayachandran commented on HIVE-21254: -- Also how does compactor know the table user if it cannot do getTable() to get location and find user? getTable() throws ACL exception if hive user does not have read access to end user location. > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767533#comment-16767533 ] Prasanth Jayachandran commented on HIVE-21254: -- What does compactor do in such cases (ACL exception)? does it switch to table owner user and retry compaction? > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18890) Lower Logging for "Table not found" Error
[
https://issues.apache.org/jira/browse/HIVE-18890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767522#comment-16767522
]
Andrew Sherman commented on HIVE-18890:
---
+1 LGTM
> Lower Logging for "Table not found" Error
> -
>
> Key: HIVE-18890
> URL: https://issues.apache.org/jira/browse/HIVE-18890
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Manoj Narayanan
>Priority: Minor
> Attachments: HIVE-18890.1.patch
>
>
> https://github.com/apache/hive/blob/7cb31c03052b815665b3231f2e513b9e65d3ff8c/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L1105
> {code:java}
> // Get the table from metastore
> org.apache.hadoop.hive.metastore.api.Table tTable = null;
> try {
> tTable = getMSC().getTable(dbName, tableName);
> } catch (NoSuchObjectException e) {
> if (throwException) {
> LOG.error("Table " + tableName + " not found: " + e.getMessage());
> throw new InvalidTableException(tableName);
> }
> return null;
> } catch (Exception e) {
> throw new HiveException("Unable to fetch table " + tableName + ". " +
> e.getMessage(), e);
> }
> {code}
> We should throw an exception or log it, but not both. Right [~mdrob] ? ;)
> And in this case, we are generating scary ERROR level logging in the
> HiveServer2 logs needlessly. This should not be reported as an application
> error. It is a simple user error, indicated by catching the
> _NoSuchObjectException_ Throwable, that can always be ignored by the service.
> It is most likely a simple user typo of the table name. However, the more
> serious general _Exception_ is not logged. This is backwards.
> Please remove the _error_ level logging for the user error... or lower it to
> _debug_ level logging.
> Please include an _error_ level logging to the general Exception case, unless
> this Exception is being captured up the stack, somewhere else, and is being
> logged there at ERROR level logging.
> {code}
> -- Sample log messages found in HS2 logs
> 2018-03-02 10:26:40,363 ERROR hive.ql.metadata.Hive:
> [HiveServer2-Handler-Pool: Thread-4467]: Table default not found:
> default.default table not found
> 2018-03-02 10:26:40,367 ERROR hive.ql.metadata.Hive:
> [HiveServer2-Handler-Pool: Thread-4467]: Table default not found:
> default.default table not found
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767507#comment-16767507 ] Eugene Koifman commented on HIVE-21254: --- this seems dangerous. If the tools misses any tables that were actually Acid and need compacting, all the user sees is a WARN in the log which is easy to miss. And try to use Acid V1 table from Hive 3 will result in data loss (and perhaps corruption) > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20758) Constraints: Show create table does not show constraints
[
https://issues.apache.org/jira/browse/HIVE-20758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767514#comment-16767514
]
Hive QA commented on HIVE-20758:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
25s{color} | {color:blue} ql in master has 2262 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16059/dev-support/hive-personality.sh
|
| git revision | master / fd2be8e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16059/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Constraints: Show create table does not show constraints
>
>
> Key: HIVE-20758
> URL: https://issues.apache.org/jira/browse/HIVE-20758
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Barnabas Maidics
>Priority: Major
> Attachments: HIVE-20758.1.patch, HIVE-20758.2.patch,
> HIVE-20758.3.patch, HIVE-20758.4.patch, HIVE-20758.5.patch,
> HIVE-20758.6.patch, HIVE-20758.7.patch, Screen Shot 2019-01-23 at 11.52.04.png
>
>
> Even though the desc formatted shows the constraints, the show create table
> does not
> {code}
> | # Primary Key | NULL
> | NULL
>|
> | Table: |
> tpcds_bin_partitioned_orc_1.inventory | NULL
> |
> | Constraint Name: | pk_in
> | NULL
>|
> | Column Names: | inv_date_sk
> | inv_item_sk
>|
> || NULL
> | NULL
>|
> | # Foreign Keys | NULL
> | NULL
[jira] [Commented] (HIVE-21058) Make Compactor run in a transaction (Umbrella)
[ https://issues.apache.org/jira/browse/HIVE-21058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767499#comment-16767499 ] Eugene Koifman commented on HIVE-21058: --- [~asomani] - no concrete plans > Make Compactor run in a transaction (Umbrella) > -- > > Key: HIVE-21058 > URL: https://issues.apache.org/jira/browse/HIVE-21058 > Project: Hive > Issue Type: New Feature > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Fix For: 4.0.0 > > > Ensure that files produced by the compactor have their visibility controlled > via Hive transaction commit like any other write to an ACID table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21038) Fix checkstyle for standalone-metastore
[ https://issues.apache.org/jira/browse/HIVE-21038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-21038: Resolution: Fixed Fix Version/s: (was: 3.2.0) 4.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks, Miklos! > Fix checkstyle for standalone-metastore > --- > > Key: HIVE-21038 > URL: https://issues.apache.org/jira/browse/HIVE-21038 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-21038.01.patch > > > Since HIVE-17506 checkstyle is not working for standalone-metastore and it's > sub projects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21149) Refactor LlapServiceDriver
[ https://issues.apache.org/jira/browse/HIVE-21149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-21149: Resolution: Fixed Fix Version/s: (was: 3.1.2) 4.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks, Miklos! > Refactor LlapServiceDriver > -- > > Key: HIVE-21149 > URL: https://issues.apache.org/jira/browse/HIVE-21149 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.2 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21149.01.patch, HIVE-21149.02.patch, > HIVE-21149.03.patch, HIVE-21149.04.patch > > > LlapServiceDriver is one monolith class doing several things, needs to be > refactor in order to make it clearer how it works. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21149) Refactor LlapServiceDriver
[ https://issues.apache.org/jira/browse/HIVE-21149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767481#comment-16767481 ] Ashutosh Chauhan commented on HIVE-21149: - +1 > Refactor LlapServiceDriver > -- > > Key: HIVE-21149 > URL: https://issues.apache.org/jira/browse/HIVE-21149 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.2 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.1.2 > > Attachments: HIVE-21149.01.patch, HIVE-21149.02.patch, > HIVE-21149.03.patch, HIVE-21149.04.patch > > > LlapServiceDriver is one monolith class doing several things, needs to be > refactor in order to make it clearer how it works. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21234) Enforce timestamp range
[
https://issues.apache.org/jira/browse/HIVE-21234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767469#comment-16767469
]
Hive QA commented on HIVE-21234:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958576/HIVE-21234.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 146 failed/errored test(s), 15786 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constprog_type]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[decimal_1] (batchId=96)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[interval_arithmetic]
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[metadata_only_queries]
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_external_time]
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_types_vectorization]
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_10]
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_13]
(batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_14]
(batchId=42)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_15]
(batchId=94)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_16]
(batchId=88)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_17]
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_3]
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_7]
(batchId=92)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_8]
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_9]
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[schema_evol_par_vec_table_dictionary_encoding]
(batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[schema_evol_par_vec_table_non_dictionary_encoding]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[temp_table_windowing_expressions]
(batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[timestamp_3] (batchId=41)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[typechangetest]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_binary_join_groupby]
(batchId=88)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_case_when_2]
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_data_types]
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_expressions]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_interval_arithmetic]
(batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_parquet_types]
(batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_timestamp]
(batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_timestamp_funcs]
(batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[windowing_distinct]
(batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[windowing_expressions]
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[windowing_multipartitioning]
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[windowing_order_null]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[windowing_rank]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[windowing_windowspec]
(batchId=19)
org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_delimited]
(batchId=275)
org.apache.hadoop.hive.cli.TestMiniHiveKafkaCliDriver.testCliDriver[kafka_storage_handler]
(batchId=275)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[parquet_types_vectorization]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_vectorization_original]
(batchId=181)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_text]
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[metadata_only_queries]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[parquet_predicate_pushdown]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[parquet_types]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_nonvec_part_all_primitive]
(batchId=183)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_nonvec_part_all_primitive_llap_io]
(batchId=183)
[jira] [Updated] (HIVE-21234) Enforce timestamp range
[ https://issues.apache.org/jira/browse/HIVE-21234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-21234: - Status: In Progress (was: Patch Available) > Enforce timestamp range > --- > > Key: HIVE-21234 > URL: https://issues.apache.org/jira/browse/HIVE-21234 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: TODOC, backwards-compatibility > Attachments: HIVE-21234.1.patch, HIVE-21234.2.patch > > > Our Wiki specifies a range for DATE, but not for TIMESTAMP (well, there's a > specified format () but no explicitly specified range). [1] > TIMESTAMP used to have inner representation of java.sql.Timestamp which > couldn't handle timestamps outside of the range of years -. ( > converted to 0001) > Since the inner representation was changed to LocalDateTime (HIVE-20007), > negative timestamps overflow because of a formatting error. > I propose simply disabling negative timestamps, and timestamps beyond year > . No data is much better than bad data. > See [2] for more details. > [1] > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-TimestampstimestampTimestamps > [2] > https://docs.google.com/document/d/1y-GcyzzALXM2AJB3bFuyTAEq5fq6p41gu5eH1pF8I7o/edit?usp=sharing -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21234) Enforce timestamp range
[
https://issues.apache.org/jira/browse/HIVE-21234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767421#comment-16767421
]
Hive QA commented on HIVE-21234:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16058/dev-support/hive-personality.sh
|
| git revision | master / fd2be8e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16058/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Enforce timestamp range
> ---
>
> Key: HIVE-21234
> URL: https://issues.apache.org/jira/browse/HIVE-21234
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: TODOC, backwards-compatibility
> Attachments: HIVE-21234.1.patch, HIVE-21234.2.patch
>
>
> Our Wiki specifies a range for DATE, but not for TIMESTAMP (well, there's a
> specified format () but no explicitly specified range). [1]
> TIMESTAMP used to have inner representation of java.sql.Timestamp which
> couldn't handle timestamps outside of the range of years -. (
> converted to 0001)
> Since the inner representation was changed to LocalDateTime (HIVE-20007),
> negative timestamps overflow because of a formatting error.
> I propose simply disabling negative timestamps, and timestamps beyond year
> . No data is much better than bad data.
> See [2] for more details.
> [1]
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-TimestampstimestampTimestamps
> [2]
> https://docs.google.com/document/d/1y-GcyzzALXM2AJB3bFuyTAEq5fq6p41gu5eH1pF8I7o/edit?usp=sharing
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21264) Improvements Around CharTypeInfo
[
https://issues.apache.org/jira/browse/HIVE-21264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767407#comment-16767407
]
Hive QA commented on HIVE-21264:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958566/HIVE-21264.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15786 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16057/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16057/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16057/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958566 - PreCommit-HIVE-Build
> Improvements Around CharTypeInfo
>
>
> Key: HIVE-21264
> URL: https://issues.apache.org/jira/browse/HIVE-21264
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-21264.1.patch
>
>
> The {{CharTypeInfo}} stores the type name of the data type (char/varchar) and
> the length (1-255). {{CharTypeInfo}} objects are often getting cached once
> they are created.
> The {{hashcode()}} and {{equals()}} of its sub-classes varchar and char are
> inconsistent.
> * Make hashcode and equals consistent (and fast)
> * Simplify the {{getQualifiedName}} implementation and reduce the scope to
> protected
> * Other related nits
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21198) Introduce a database object reference class
[
https://issues.apache.org/jira/browse/HIVE-21198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767396#comment-16767396
]
David Lavati commented on HIVE-21198:
-
The groundwork for this was started in HIVE-19769.
> Introduce a database object reference class
> ---
>
> Key: HIVE-21198
> URL: https://issues.apache.org/jira/browse/HIVE-21198
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
>
> There are many places in which "{databasename}.{tablename}" is passed as a
> single string; there are some places where the they travel as 2 separate
> arguments.
> Idea would be to introduce a simple immutable class with 2 fields ; and pass
> these informations together. Making this better is required if we would be
> wanting to enable dot in tablenames
> HIVE-16907, HIVE-21151
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21217) Optimize range calculation for PTF
[ https://issues.apache.org/jira/browse/HIVE-21217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Szita updated HIVE-21217: -- Attachment: HIVE-21217.2.patch > Optimize range calculation for PTF > -- > > Key: HIVE-21217 > URL: https://issues.apache.org/jira/browse/HIVE-21217 > Project: Hive > Issue Type: Improvement >Reporter: Adam Szita >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-21217.0.patch, HIVE-21217.1.patch, > HIVE-21217.2.patch > > > During window function execution Hive has to iterate on neighbouring rows of > the current row to find the beginning and end of the proper range (on which > the aggregation will be executed). > When we're using range based windows and have many rows with a certain key > value this can take a lot of time. (e.g. partition size of 80M, in which we > have 2 ranges of 40M rows according to the orderby column: within these 40M > rowsets we're doing 40M x 40M/2 steps.. which is of n^2 time complexity) > I propose to introduce a cache that keeps track of already calculated range > ends so it can be reused in future scans. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21217) Optimize range calculation for PTF
[ https://issues.apache.org/jira/browse/HIVE-21217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767392#comment-16767392 ] Adam Szita commented on HIVE-21217: --- [~pvary] can you take a look please? > Optimize range calculation for PTF > -- > > Key: HIVE-21217 > URL: https://issues.apache.org/jira/browse/HIVE-21217 > Project: Hive > Issue Type: Improvement >Reporter: Adam Szita >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-21217.0.patch, HIVE-21217.1.patch, > HIVE-21217.2.patch > > > During window function execution Hive has to iterate on neighbouring rows of > the current row to find the beginning and end of the proper range (on which > the aggregation will be executed). > When we're using range based windows and have many rows with a certain key > value this can take a lot of time. (e.g. partition size of 80M, in which we > have 2 ranges of 40M rows according to the orderby column: within these 40M > rowsets we're doing 40M x 40M/2 steps.. which is of n^2 time complexity) > I propose to introduce a cache that keeps track of already calculated range > ends so it can be reused in future scans. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21217) Optimize range calculation for PTF
[ https://issues.apache.org/jira/browse/HIVE-21217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Szita updated HIVE-21217: -- Status: Patch Available (was: In Progress) > Optimize range calculation for PTF > -- > > Key: HIVE-21217 > URL: https://issues.apache.org/jira/browse/HIVE-21217 > Project: Hive > Issue Type: Improvement >Reporter: Adam Szita >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-21217.0.patch, HIVE-21217.1.patch, > HIVE-21217.2.patch > > > During window function execution Hive has to iterate on neighbouring rows of > the current row to find the beginning and end of the proper range (on which > the aggregation will be executed). > When we're using range based windows and have many rows with a certain key > value this can take a lot of time. (e.g. partition size of 80M, in which we > have 2 ranges of 40M rows according to the orderby column: within these 40M > rowsets we're doing 40M x 40M/2 steps.. which is of n^2 time complexity) > I propose to introduce a cache that keeps track of already calculated range > ends so it can be reused in future scans. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21217) Optimize range calculation for PTF
[ https://issues.apache.org/jira/browse/HIVE-21217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Szita updated HIVE-21217: -- Status: In Progress (was: Patch Available) > Optimize range calculation for PTF > -- > > Key: HIVE-21217 > URL: https://issues.apache.org/jira/browse/HIVE-21217 > Project: Hive > Issue Type: Improvement >Reporter: Adam Szita >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-21217.0.patch, HIVE-21217.1.patch, > HIVE-21217.2.patch > > > During window function execution Hive has to iterate on neighbouring rows of > the current row to find the beginning and end of the proper range (on which > the aggregation will be executed). > When we're using range based windows and have many rows with a certain key > value this can take a lot of time. (e.g. partition size of 80M, in which we > have 2 ranges of 40M rows according to the orderby column: within these 40M > rowsets we're doing 40M x 40M/2 steps.. which is of n^2 time complexity) > I propose to introduce a cache that keeps track of already calculated range > ends so it can be reused in future scans. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21264) Improvements Around CharTypeInfo
[
https://issues.apache.org/jira/browse/HIVE-21264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767380#comment-16767380
]
Hive QA commented on HIVE-21264:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} serde in master has 197 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} serde: The patch generated 1 new + 51 unchanged - 2
fixed = 52 total (was 53) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16057/dev-support/hive-personality.sh
|
| git revision | master / fd2be8e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16057/yetus/diff-checkstyle-serde.txt
|
| modules | C: serde U: serde |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16057/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improvements Around CharTypeInfo
>
>
> Key: HIVE-21264
> URL: https://issues.apache.org/jira/browse/HIVE-21264
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-21264.1.patch
>
>
> The {{CharTypeInfo}} stores the type name of the data type (char/varchar) and
> the length (1-255). {{CharTypeInfo}} objects are often getting cached once
> they are created.
> The {{hashcode()}} and {{equals()}} of its sub-classes varchar and char are
> inconsistent.
> * Make hashcode and equals consistent (and fast)
> * Simplify the {{getQualifiedName}} implementation and reduce the scope to
> protected
> * Other related nits
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21265) Hive miss-uses HBase HConnection object and that puts high load on Zookeeper
[
https://issues.apache.org/jira/browse/HIVE-21265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Istvan Fajth updated HIVE-21265:
Issue Type: Improvement (was: Bug)
> Hive miss-uses HBase HConnection object and that puts high load on Zookeeper
>
>
> Key: HIVE-21265
> URL: https://issues.apache.org/jira/browse/HIVE-21265
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler
>Reporter: Istvan Fajth
>Priority: Major
>
> When there is a table in Hive backed by an HBase table, then the following
> access pattern is shown multiple times in Zookeeper even for a simple query
> like "SELECT * FROM table":
> - A client is connecting to Zookeeper
> - Checks whether the /hbase ZNode exists
> - Reads /hbase/hbaseid
> - Client closes the connection.
> The amount of these accesses are depending on the amount of data most likely
> it is correlating to the number of HBase regions.
> The same access pattern one can see in ZK when one runs the following Java
> code:
> {code}import org.apache.hadoop.hbase.client.*;
> public class Test {
> public static void main(String args[]) throws Exception {
> Connection c = ConnectionFactory.createConnection();
> c.close();
> }
> }{code}
> The problem with this is that for large tables this creates an enormous
> amount of session creation which is expensive in ZK, and if the amount of
> queries to this table is high, then the ZK transaction log is heavily
> written, and there are way more snapshots created then otherwise due to the
> amount of createSession closeSession transaction in Zookeeper. In this
> particular case the Zookeeper data directory was filled with about 24GB of
> data and caused the device to almost fill under the Zookeeper data directory.
> ~90% of the data written was createSession and closeSession transactions.
> I am not sure what logs I should provide, but reproducing the behaviour is
> easy enough. In Zookeeper if one enables DEBUG level logging, the logs are
> showing what is being read by sessions. These sessions live for 1-5ms tops.
> I imagine that the solution is to somehow share the connection object between
> the mappers if possible, and use one connection according to the suggestion
> in the API documentation of ConnectionFactory and request table/admin/any
> object from the one connection, or at least use only one connection object
> per map/reduce, and make it a longer living connection that is there for the
> whole map/reduce lifetime.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21217) Optimize range calculation for PTF
[
https://issues.apache.org/jira/browse/HIVE-21217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767363#comment-16767363
]
Hive QA commented on HIVE-21217:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958544/HIVE-21217.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15796 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16055/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16055/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16055/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958544 - PreCommit-HIVE-Build
> Optimize range calculation for PTF
> --
>
> Key: HIVE-21217
> URL: https://issues.apache.org/jira/browse/HIVE-21217
> Project: Hive
> Issue Type: Improvement
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-21217.0.patch, HIVE-21217.1.patch
>
>
> During window function execution Hive has to iterate on neighbouring rows of
> the current row to find the beginning and end of the proper range (on which
> the aggregation will be executed).
> When we're using range based windows and have many rows with a certain key
> value this can take a lot of time. (e.g. partition size of 80M, in which we
> have 2 ranges of 40M rows according to the orderby column: within these 40M
> rowsets we're doing 40M x 40M/2 steps.. which is of n^2 time complexity)
> I propose to introduce a cache that keeps track of already calculated range
> ends so it can be reused in future scans.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21194) DruidStorageHandler should set a version of segment to UTC
[
https://issues.apache.org/jira/browse/HIVE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767367#comment-16767367
]
Hive QA commented on HIVE-21194:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958552/HIVE-21194.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16056/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16056/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16056/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-02-13 16:26:46.784
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-16056/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-02-13 16:26:46.788
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at fd2be8e HIVE-21228: Replace all occurences of new Integer with
Integer.valueOf (Ivan Suller via Zoltan Haindrich)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at fd2be8e HIVE-21228: Replace all occurences of new Integer with
Integer.valueOf (Ivan Suller via Zoltan Haindrich)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-02-13 16:26:47.464
+ rm -rf ../yetus_PreCommit-HIVE-Build-16056
+ mkdir ../yetus_PreCommit-HIVE-Build-16056
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-16056
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-16056/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed:
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java:864
Falling back to three-way merge...
Applied patch to
'druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java'
with conflicts.
Going to apply patch with: git apply -p0
error: patch failed:
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java:864
Falling back to three-way merge...
Applied patch to
'druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java'
with conflicts.
U druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-16056
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958552 - PreCommit-HIVE-Build
> DruidStorageHandler should set a version of segment to UTC
> --
>
> Key: HIVE-21194
> URL: https://issues.apache.org/jira/browse/HIVE-21194
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Affects Versions: 3.1.0
>Reporter: Seung-Hyun Cheong
>Assignee: Seung-Hyun Cheong
>Priority: Minor
> Attachments: HIVE-21194.patch, image-2019-02-01-16-31-56-958.png,
> image-2019-02-01-16-32-17-093.png
>
>
> h1. Exception while running a KillTask
> {code:java}
> 2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0]
> io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running
> task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z,
> type=kill, dataSource=upload}]
> io.druid.java.util.common.ISE: WTF?! Unused
> segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00]
> has version[2019-01-31T01:12:32.289+09:00] > task
>
[jira] [Commented] (HIVE-21252) LazyTimestamp - Use String Equals
[
https://issues.apache.org/jira/browse/HIVE-21252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767357#comment-16767357
]
Naveen Gangam commented on HIVE-21252:
--
Looks good to me. +1
> LazyTimestamp - Use String Equals
> -
>
> Key: HIVE-21252
> URL: https://issues.apache.org/jira/browse/HIVE-21252
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-21252.1.patch
>
>
> {code:java|title=LazyTimestamp.java}
> if (s.compareTo("NULL") == 0) {
> isNull = true;
> logExceptionMessage(bytes, start, length, "TIMESTAMP");
> }
> {code}
> compareTo generates a number to represent the differences between the two
> Strings. It's faster to simply call "equals" which will simply compare the
> two String directly and return a boolean.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21217) Optimize range calculation for PTF
[
https://issues.apache.org/jira/browse/HIVE-21217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767324#comment-16767324
]
Hive QA commented on HIVE-21217:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
6s{color} | {color:blue} ql in master has 2262 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 3 new + 139 unchanged - 34
fixed = 142 total (was 173) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16055/dev-support/hive-personality.sh
|
| git revision | master / fd2be8e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16055/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16055/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Optimize range calculation for PTF
> --
>
> Key: HIVE-21217
> URL: https://issues.apache.org/jira/browse/HIVE-21217
> Project: Hive
> Issue Type: Improvement
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-21217.0.patch, HIVE-21217.1.patch
>
>
> During window function execution Hive has to iterate on neighbouring rows of
> the current row to find the beginning and end of the proper range (on which
> the aggregation will be executed).
> When we're using range based windows and have many rows with a certain key
> value this can take a lot of time. (e.g. partition size of 80M, in which we
> have 2 ranges of 40M rows according to the orderby column: within these 40M
> rowsets we're doing 40M x 40M/2 steps.. which is of n^2 time complexity)
> I propose to introduce a cache that keeps track of already calculated range
> ends so it can be reused in future scans.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21240) JSON SerDe Deserialize Re-Write
[ https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-21240: --- Status: Patch Available (was: Open) I' added a new patch...there was more format-only changes that I missed. PR is updated too. > JSON SerDe Deserialize Re-Write > --- > > Key: HIVE-21240 > URL: https://issues.apache.org/jira/browse/HIVE-21240 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 3.1.1, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch, > HIVE-21240.2.patch, HIVE-21240.3.patch, HIVE-21240.4.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The JSON SerDe has a few issues, I will link them to this JIRA. > * Use Jackson Tree parser instead of manually parsing > * Added support for base-64 encoded data (the expected format when using JSON) > * Added support to skip blank lines (returns all columns as null values) > * Current JSON parser accepts, but does not apply, custom timestamp formats > in most cases > * Added some unit tests > * Added cache for column-name to column-index searches, currently O\(n\) for > each row processed, for each column in the row -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21241) Migrate TimeStamp Parser From Joda Time
[
https://issues.apache.org/jira/browse/HIVE-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767318#comment-16767318
]
Naveen Gangam commented on HIVE-21241:
--
[~belugabehr] I am not too familiar with the joda time parser or the java time
APIs. Let me do some reading to catch up. I will review the patch after that.
Thanks
> Migrate TimeStamp Parser From Joda Time
> ---
>
> Key: HIVE-21241
> URL: https://issues.apache.org/jira/browse/HIVE-21241
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21241.1.patch, HIVE-21241.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive uses Joda time for its TimeStampParser.
> {quote}
> Joda-Time is the de facto standard date and time library for Java prior to
> Java SE 8. Users are now asked to migrate to java.time (JSR-310).
> https://www.joda.org/joda-time/
> {quote}
> Migrate TimeStampParser to {{java.time}}
> I also added a couple new pre-canned timestamp parsers for convenience:
> * ISO 8601
> * RFC 1123
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21240) JSON SerDe Deserialize Re-Write
[ https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-21240: --- Attachment: HIVE-21240.4.patch > JSON SerDe Deserialize Re-Write > --- > > Key: HIVE-21240 > URL: https://issues.apache.org/jira/browse/HIVE-21240 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 4.0.0, 3.1.1 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch, > HIVE-21240.2.patch, HIVE-21240.3.patch, HIVE-21240.4.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The JSON SerDe has a few issues, I will link them to this JIRA. > * Use Jackson Tree parser instead of manually parsing > * Added support for base-64 encoded data (the expected format when using JSON) > * Added support to skip blank lines (returns all columns as null values) > * Current JSON parser accepts, but does not apply, custom timestamp formats > in most cases > * Added some unit tests > * Added cache for column-name to column-index searches, currently O\(n\) for > each row processed, for each column in the row -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21240) JSON SerDe Deserialize Re-Write
[ https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-21240: --- Status: Open (was: Patch Available) > JSON SerDe Deserialize Re-Write > --- > > Key: HIVE-21240 > URL: https://issues.apache.org/jira/browse/HIVE-21240 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 3.1.1, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch, > HIVE-21240.2.patch, HIVE-21240.3.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The JSON SerDe has a few issues, I will link them to this JIRA. > * Use Jackson Tree parser instead of manually parsing > * Added support for base-64 encoded data (the expected format when using JSON) > * Added support to skip blank lines (returns all columns as null values) > * Current JSON parser accepts, but does not apply, custom timestamp formats > in most cases > * Added some unit tests > * Added cache for column-name to column-index searches, currently O\(n\) for > each row processed, for each column in the row -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21234) Enforce timestamp range
[
https://issues.apache.org/jira/browse/HIVE-21234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767289#comment-16767289
]
Hive QA commented on HIVE-21234:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958545/HIVE-21234.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 145 failed/errored test(s), 15786 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constprog_type]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[decimal_1] (batchId=96)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[interval_arithmetic]
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[metadata_only_queries]
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_external_time]
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_types_vectorization]
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_10]
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_13]
(batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_14]
(batchId=42)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_15]
(batchId=94)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_16]
(batchId=88)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_17]
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_3]
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_7]
(batchId=92)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_8]
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_9]
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[schema_evol_par_vec_table_dictionary_encoding]
(batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[schema_evol_par_vec_table_non_dictionary_encoding]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[temp_table_windowing_expressions]
(batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[timestamp_3] (batchId=41)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[typechangetest]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_binary_join_groupby]
(batchId=88)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_case_when_2]
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_data_types]
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_expressions]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_interval_arithmetic]
(batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_parquet_types]
(batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_timestamp]
(batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_timestamp_funcs]
(batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[windowing_distinct]
(batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[windowing_expressions]
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[windowing_multipartitioning]
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[windowing_order_null]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[windowing_rank]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[windowing_windowspec]
(batchId=19)
org.apache.hadoop.hive.cli.TestMiniHiveKafkaCliDriver.testCliDriver[kafka_storage_handler]
(batchId=275)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[parquet_types_vectorization]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_vectorization_original]
(batchId=181)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_text]
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[metadata_only_queries]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[parquet_predicate_pushdown]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[parquet_types]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_nonvec_part_all_primitive]
(batchId=183)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_nonvec_part_all_primitive_llap_io]
(batchId=183)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_vec_part_all_complex]
(batchId=173)
[jira] [Updated] (HIVE-21262) Running schemeAuthority.q and schemeAuthority2.q in parallel can cause flakyness
[ https://issues.apache.org/jira/browse/HIVE-21262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-21262: -- Attachment: HIVE-21262.patch > Running schemeAuthority.q and schemeAuthority2.q in parallel can cause > flakyness > > > Key: HIVE-21262 > URL: https://issues.apache.org/jira/browse/HIVE-21262 > Project: Hive > Issue Type: Bug >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-21262.patch > > > Found by our internal ptest runs that if the schemeAuthority.q and the > schemeAuthority2.q is ended up in different batches and run parallel by > ptest, then the data files are removed after the first one is finished and > the second one fails because of the missing data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21262) Running schemeAuthority.q and schemeAuthority2.q in parallel can cause flakyness
[ https://issues.apache.org/jira/browse/HIVE-21262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-21262: -- Status: Patch Available (was: Open) > Running schemeAuthority.q and schemeAuthority2.q in parallel can cause > flakyness > > > Key: HIVE-21262 > URL: https://issues.apache.org/jira/browse/HIVE-21262 > Project: Hive > Issue Type: Bug >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-21262.patch > > > Found by our internal ptest runs that if the schemeAuthority.q and the > schemeAuthority2.q is ended up in different batches and run parallel by > ptest, then the data files are removed after the first one is finished and > the second one fails because of the missing data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21240) JSON SerDe Deserialize Re-Write
[ https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-21240: --- Attachment: HIVE-21240.3.patch > JSON SerDe Deserialize Re-Write > --- > > Key: HIVE-21240 > URL: https://issues.apache.org/jira/browse/HIVE-21240 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 4.0.0, 3.1.1 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch, > HIVE-21240.2.patch, HIVE-21240.3.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The JSON SerDe has a few issues, I will link them to this JIRA. > * Use Jackson Tree parser instead of manually parsing > * Added support for base-64 encoded data (the expected format when using JSON) > * Added support to skip blank lines (returns all columns as null values) > * Current JSON parser accepts, but does not apply, custom timestamp formats > in most cases > * Added some unit tests > * Added cache for column-name to column-index searches, currently O\(n\) for > each row processed, for each column in the row -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20758) Constraints: Show create table does not show constraints
[
https://issues.apache.org/jira/browse/HIVE-20758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Barnabas Maidics updated HIVE-20758:
Attachment: HIVE-20758.7.patch
Status: Patch Available (was: Open)
> Constraints: Show create table does not show constraints
>
>
> Key: HIVE-20758
> URL: https://issues.apache.org/jira/browse/HIVE-20758
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Barnabas Maidics
>Priority: Major
> Attachments: HIVE-20758.1.patch, HIVE-20758.2.patch,
> HIVE-20758.3.patch, HIVE-20758.4.patch, HIVE-20758.5.patch,
> HIVE-20758.6.patch, HIVE-20758.7.patch, Screen Shot 2019-01-23 at 11.52.04.png
>
>
> Even though the desc formatted shows the constraints, the show create table
> does not
> {code}
> | # Primary Key | NULL
> | NULL
>|
> | Table: |
> tpcds_bin_partitioned_orc_1.inventory | NULL
> |
> | Constraint Name: | pk_in
> | NULL
>|
> | Column Names: | inv_date_sk
> | inv_item_sk
>|
> || NULL
> | NULL
>|
> | # Foreign Keys | NULL
> | NULL
>|
> | Table: |
> tpcds_bin_partitioned_orc_1.inventory | NULL
> |
> | Constraint Name: | inv_d
> | NULL
>|
> | Parent Column Name:tpcds_bin_partitioned_orc_1.date_dim.d_date_sk |
> Column Name:inv_date_sk| Key Sequence:1
> |
> || NULL
> | NULL
>|
> | Constraint Name: | inv_i
> | NULL
>|
> | Parent Column Name:tpcds_bin_partitioned_orc_1.item.i_item_sk | Column
> Name:inv_item_sk| Key Sequence:1
>|
> || NULL
> | NULL
>|
> | Constraint Name: | inv_w
> | NULL
>|
> | Parent Column Name:tpcds_bin_partitioned_orc_1.warehouse.w_warehouse_sk
> | Column Name:inv_warehouse_sk | Key Sequence:1
> |
> || NULL
> | NULL
>|
> {code}
> But
> {code}
> ++
> | createtab_stmt |
> ++
> | CREATE TABLE `inventory`( |
> | `inv_item_sk` bigint,|
> | `inv_warehouse_sk` bigint, |
> | `inv_quantity_on_hand` int, |
> | `inv_date_sk` bigint)|
> | ROW FORMAT SERDE |
> | 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' |
> | STORED AS INPUTFORMAT |
> | 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' |
> | OUTPUTFORMAT |
> | 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' |
> | LOCATION |
> |
> 'hdfs:///warehouse/tablespace/managed/hive/tpcds_bin_partitioned_orc_1.db/inventory'
> |
> | TBLPROPERTIES (|
> | 'bucketing_version'='2', |
> | 'transactional'='true', |
> | 'transactional_properties'='default',|
> |
[jira] [Updated] (HIVE-20758) Constraints: Show create table does not show constraints
[
https://issues.apache.org/jira/browse/HIVE-20758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Barnabas Maidics updated HIVE-20758:
Status: Open (was: Patch Available)
> Constraints: Show create table does not show constraints
>
>
> Key: HIVE-20758
> URL: https://issues.apache.org/jira/browse/HIVE-20758
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Barnabas Maidics
>Priority: Major
> Attachments: HIVE-20758.1.patch, HIVE-20758.2.patch,
> HIVE-20758.3.patch, HIVE-20758.4.patch, HIVE-20758.5.patch,
> HIVE-20758.6.patch, HIVE-20758.7.patch, Screen Shot 2019-01-23 at 11.52.04.png
>
>
> Even though the desc formatted shows the constraints, the show create table
> does not
> {code}
> | # Primary Key | NULL
> | NULL
>|
> | Table: |
> tpcds_bin_partitioned_orc_1.inventory | NULL
> |
> | Constraint Name: | pk_in
> | NULL
>|
> | Column Names: | inv_date_sk
> | inv_item_sk
>|
> || NULL
> | NULL
>|
> | # Foreign Keys | NULL
> | NULL
>|
> | Table: |
> tpcds_bin_partitioned_orc_1.inventory | NULL
> |
> | Constraint Name: | inv_d
> | NULL
>|
> | Parent Column Name:tpcds_bin_partitioned_orc_1.date_dim.d_date_sk |
> Column Name:inv_date_sk| Key Sequence:1
> |
> || NULL
> | NULL
>|
> | Constraint Name: | inv_i
> | NULL
>|
> | Parent Column Name:tpcds_bin_partitioned_orc_1.item.i_item_sk | Column
> Name:inv_item_sk| Key Sequence:1
>|
> || NULL
> | NULL
>|
> | Constraint Name: | inv_w
> | NULL
>|
> | Parent Column Name:tpcds_bin_partitioned_orc_1.warehouse.w_warehouse_sk
> | Column Name:inv_warehouse_sk | Key Sequence:1
> |
> || NULL
> | NULL
>|
> {code}
> But
> {code}
> ++
> | createtab_stmt |
> ++
> | CREATE TABLE `inventory`( |
> | `inv_item_sk` bigint,|
> | `inv_warehouse_sk` bigint, |
> | `inv_quantity_on_hand` int, |
> | `inv_date_sk` bigint)|
> | ROW FORMAT SERDE |
> | 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' |
> | STORED AS INPUTFORMAT |
> | 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' |
> | OUTPUTFORMAT |
> | 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' |
> | LOCATION |
> |
> 'hdfs:///warehouse/tablespace/managed/hive/tpcds_bin_partitioned_orc_1.db/inventory'
> |
> | TBLPROPERTIES (|
> | 'bucketing_version'='2', |
> | 'transactional'='true', |
> | 'transactional_properties'='default',|
> | 'transient_lastDdlTime'='1539710410')
[jira] [Updated] (HIVE-21240) JSON SerDe Deserialize Re-Write
[ https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-21240: --- Status: Patch Available (was: Open) > JSON SerDe Deserialize Re-Write > --- > > Key: HIVE-21240 > URL: https://issues.apache.org/jira/browse/HIVE-21240 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 3.1.1, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch, > HIVE-21240.2.patch, HIVE-21240.3.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The JSON SerDe has a few issues, I will link them to this JIRA. > * Use Jackson Tree parser instead of manually parsing > * Added support for base-64 encoded data (the expected format when using JSON) > * Added support to skip blank lines (returns all columns as null values) > * Current JSON parser accepts, but does not apply, custom timestamp formats > in most cases > * Added some unit tests > * Added cache for column-name to column-index searches, currently O\(n\) for > each row processed, for each column in the row -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21240) JSON SerDe Deserialize Re-Write
[ https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-21240: --- Status: Open (was: Patch Available) > JSON SerDe Deserialize Re-Write > --- > > Key: HIVE-21240 > URL: https://issues.apache.org/jira/browse/HIVE-21240 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 3.1.1, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch, > HIVE-21240.2.patch, HIVE-21240.3.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The JSON SerDe has a few issues, I will link them to this JIRA. > * Use Jackson Tree parser instead of manually parsing > * Added support for base-64 encoded data (the expected format when using JSON) > * Added support to skip blank lines (returns all columns as null values) > * Current JSON parser accepts, but does not apply, custom timestamp formats > in most cases > * Added some unit tests > * Added cache for column-name to column-index searches, currently O\(n\) for > each row processed, for each column in the row -- This message was sent by Atlassian JIRA (v7.6.3#76005)
