[jira] [Created] (HIVE-20720) Add partition column option to JDBC handler

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20720:
-

 Summary: Add partition column option to JDBC handler
 Key: HIVE-20720
 URL: https://issues.apache.org/jira/browse/HIVE-20720
 Project: Hive
  Issue Type: New Feature
  Components: StorageHandler
Reporter: Daniel Dai
Assignee: Daniel Dai


Currently JdbcStorageHandler does not split input in Tez. The reason is 
numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" in 
Tez. And "mapreduce.job.maps" is not a valid param if ranger is in use. User 
ends up always use 1 split.

We need to rely on this new feature if we want to support multi-splits. Here is 
my proposal:
1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound in 
tblproperties if user want to split jdbc data source. In case 
lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min 
query to get this in planner. We can currently limit partitionColumn to only 
numeric/date/timestamp column for simplicity
2. If partitionColumn/numPartitions are not specified, don't split input
3. Splits are equal intervals without respect to data distribution
4. There is also a "hive.sql.query.split" flag vetos the split (can be set 
manually or automatically by calcite)
5. If partitionColumn is not defined, but numPartitions is defined, use 
original limit/offset logic (however, don't rely on numSplit).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68805: HIVE-20538

[Eugene Koifman]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68805/#review209387
---




ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsWithSplitUpdateAndVectorization.java
Line 25 (original), 30 (patched)


What is this change for?  TestTxnCommands is a subclass of 
TxnCommandsBaseForTests.  I think this means none of the TestTxnCommands tests 
run in vectorized mode any more

More generally, what is the point of other changes in this class?


- Eugene Koifman


On Sept. 21, 2018, 3:51 p.m., Jaume Marhuenda wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68805/
> ---
> 
> (Updated Sept. 21, 2018, 3:51 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20538: Allow to store a key value together with a transaction.
> 
> 
> Diffs
> -
> 
>   
> ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsWithSplitUpdateAndVectorization.java
>  a013230025 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CommitTxnKeyValue.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CommitTxnRequest.java
>  db47f9db8b 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  936f7c5a40 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  958f13c18e 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  a3dddf54e4 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  d226db50a5 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>  54e7eda0da 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> ad83162ec3 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  1df1ebce49 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java
>  080cc5284b 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
>  ce590d0f55 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStoreTxns.java
>  db4dd9ec42 
> 
> 
> Diff: https://reviews.apache.org/r/68805/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jaume Marhuenda
> 
>

[jira] [Created] (HIVE-20719) SELECT statement fails after UPDATE with hive.optimize.sort.dynamic.partition optimization on

[Vineet Garg (JIRA)]

Vineet Garg created HIVE-20719:
--

 Summary: SELECT statement fails after UPDATE with 
hive.optimize.sort.dynamic.partition optimization on
 Key: HIVE-20719
 URL: https://issues.apache.org/jira/browse/HIVE-20719
 Project: Hive
  Issue Type: Bug
Affects Versions: 4.0.0
Reporter: Vineet Garg


*Reproducer*
{code:sql}
 set hive.optimize.sort.dynamic.partition=true ;

create table acid_uap(a int, b varchar(128)) partitioned by (ds string) 
clustered by (a) into 2 buckets stored as orc TBLPROPERTIES 
('transactional'='true');

insert into table acid_uap partition (ds='today') select cint, cast(cstring1 as 
varchar(128)) as cs from alltypesorc where cint is not null and cint < 0 order 
by cint, cs limit 10;
insert into table acid_uap partition (ds='tomorrow') select cint, cast(cstring1 
as varchar(128)) as cs from alltypesorc where cint is not null and cint > 10 
order by cint, cs limit 10;

select a,b,ds from acid_uap order by a,b;

update acid_uap set b = 'fred';

select a,b,ds from acid_uap order by a,b;
{code}

*Error*
{code:java}
Status: Failed
Vertex failed, vertexName=Map 1, vertexId=vertex_1539123809352_0001_5_00, 
diagnostics=[Task failed, taskId=task_1539123809352_0001_5_00_00, 
diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( 
failure ) : 
attempt_1539123809352_0001_5_00_00_0:java.lang.RuntimeException: 
java.lang.RuntimeException: java.io.IOException: java.io.IOException: Corrupted 
records with different bucket ids from the containing bucket file found! 
Expected bucket id 0, however found DeleteRecordKey(2,536936448(1.1.0),0).  
(OrcSplit 
[file:/Users/vgarg/hive_temp/vgarg/hive/warehouse/dp_sort.db/acid_uap/ds=today/delta_002_002_/bucket_0,
 start=3, length=361, isOriginal=false, fileLength=798, hasFooter=false, 
hasBase=true, 
deltas=2],file:/Users/vgarg/hive_temp/vgarg/hive/warehouse/dp_sort.db/acid_uap/ds=today/delete_delta_003_003_/bucket_0)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
at 
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at 
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
at 
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
at 
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.io.IOException: 
java.io.IOException: Corrupted records with different bucket ids from the 
containing bucket file found! Expected bucket id 0, however found 
DeleteRecordKey(2,536936448(1.1.0),0).  (OrcSplit 
[file:/Users/vgarg/hive_temp/vgarg/hive/warehouse/dp_sort.db/acid_uap/ds=today/delta_002_002_/bucket_0,
 start=3, length=361, isOriginal=false, fileLength=798, hasFooter=false, 
hasBase=true, 
deltas=2],file:/Users/vgarg/hive_temp/vgarg/hive/warehouse/dp_sort.db/acid_uap/ds=today/delete_delta_003_003_/bucket_0)
at 
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:206)
at 
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:152)
at 
org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:426)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)
... 16 more
Caused 

Re: Review Request 68969: HIVE-20307 : Add support for filterspec to the getPartitions with projection API

[Andrew Sherman via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68969/#review209385
---


Ship it!





standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestGetPartitionsUsingProjection.java
Lines 718 (patched)


This is so much nicer


- Andrew Sherman


On Oct. 9, 2018, 9:23 p.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68969/
> ---
> 
> (Updated Oct. 9, 2018, 9:23 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov and Andrew Sherman.
> 
> 
> Bugs: HIVE-20307
> https://issues.apache.org/jira/browse/HIVE-20307
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20307 : Add support for filterspec to the getPartitions with projection 
> API
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  d59d5d807a26378a430e683533e53d0831cf9514 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  a2b57fb646899c54b63be14a8cde9b8644a973aa 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
>  16f4a50d69f9120d565f61d028b060d7776689fc 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  08614749b7aba54f9eb9b54ac46f79dbac6bc5cd 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  af757932a191675bc8fb9236209a2efba9f3d335 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  66977d79c946f1ac57aacfbe8704d37bfbac3ea3 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  a6d9583364be20758444ebe25c8cf636f0ea740f 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  70490f09e765d4e42391c67eb5cf018e93ad04aa 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  4dd4edccd66f8ea8ea189a2d27f970c8113e3a0f 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  06f4cbce58c16f98257e7f529ffe31c983f2919f 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestGetPartitionsUsingProjection.java
>  dcff606937157e63694657b42392875d50b17be6 
> 
> 
> Diff: https://reviews.apache.org/r/68969/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vihang Karajgaonkar
> 
>

Re: Review Request 68969: HIVE-20307 : Add support for filterspec to the getPartitions with projection API

[Vihang Karajgaonkar via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68969/
---

(Updated Oct. 9, 2018, 9:23 p.m.)


Review request for hive, Alexander Kolbasov and Andrew Sherman.


Changes
---

Added Andrew's suggested changes.


Bugs: HIVE-20307
https://issues.apache.org/jira/browse/HIVE-20307


Repository: hive-git


Description
---

HIVE-20307 : Add support for filterspec to the getPartitions with projection API


Diffs (updated)
-

  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 d59d5d807a26378a430e683533e53d0831cf9514 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
a2b57fb646899c54b63be14a8cde9b8644a973aa 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 16f4a50d69f9120d565f61d028b060d7776689fc 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 08614749b7aba54f9eb9b54ac46f79dbac6bc5cd 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 af757932a191675bc8fb9236209a2efba9f3d335 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 66977d79c946f1ac57aacfbe8704d37bfbac3ea3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 a6d9583364be20758444ebe25c8cf636f0ea740f 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 70490f09e765d4e42391c67eb5cf018e93ad04aa 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 4dd4edccd66f8ea8ea189a2d27f970c8113e3a0f 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 06f4cbce58c16f98257e7f529ffe31c983f2919f 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestGetPartitionsUsingProjection.java
 dcff606937157e63694657b42392875d50b17be6 


Diff: https://reviews.apache.org/r/68969/diff/2/

Changes: https://reviews.apache.org/r/68969/diff/1-2/


Testing
---


Thanks,

Vihang Karajgaonkar

Re: Review Request 68969: HIVE-20307 : Add support for filterspec to the getPartitions with projection API

[Andrew Sherman via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68969/#review209380
---



All looks fine within the bounds of what I understand


standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
Lines 608 (patched)


Add a default to case statement to catch future evil



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 4100 (patched)


This is a total nit but you can do this sort of thing in one line with 
UNTESTED AND UNTRIED stuff like
filters.stream().map(s -> "(" + s + ")").collect(Collectors.joining(" AND 
"));



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
Line 572 (original), 572 (patched)


spelling -> represent



standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestGetPartitionsUsingProjection.java
Lines 645 (patched)


These tests that set a property in client assume stuff about the order of 
tests? Would it be safer for each test to undo the change it made to client at 
the end of the test?



standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestGetPartitionsUsingProjection.java
Lines 738 (patched)


Maybe this boilerplate, which is repeated, should be put in a method? In 
fact the whole methid could use a load of helper methods to make it more 
readable.


- Andrew Sherman


On Oct. 9, 2018, 7:47 p.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68969/
> ---
> 
> (Updated Oct. 9, 2018, 7:47 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov and Andrew Sherman.
> 
> 
> Bugs: HIVE-20307
> https://issues.apache.org/jira/browse/HIVE-20307
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20307 : Add support for filterspec to the getPartitions with projection 
> API
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  d59d5d807a26378a430e683533e53d0831cf9514 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  a2b57fb646899c54b63be14a8cde9b8644a973aa 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
>  16f4a50d69f9120d565f61d028b060d7776689fc 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  08614749b7aba54f9eb9b54ac46f79dbac6bc5cd 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  af757932a191675bc8fb9236209a2efba9f3d335 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  66977d79c946f1ac57aacfbe8704d37bfbac3ea3 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  a6d9583364be20758444ebe25c8cf636f0ea740f 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  70490f09e765d4e42391c67eb5cf018e93ad04aa 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  4dd4edccd66f8ea8ea189a2d27f970c8113e3a0f 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  06f4cbce58c16f98257e7f529ffe31c983f2919f 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestGetPartitionsUsingProjection.java
>  dcff606937157e63694657b42392875d50b17be6 
> 
> 
> Diff: https://reviews.apache.org/r/68969/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vihang Karajgaonkar
> 
>

Review Request 68969: HIVE-20307 : Add support for filterspec to the getPartitions with projection API

[Vihang Karajgaonkar via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68969/
---

Review request for hive, Alexander Kolbasov and Andrew Sherman.


Bugs: HIVE-20307
https://issues.apache.org/jira/browse/HIVE-20307


Repository: hive-git


Description
---

HIVE-20307 : Add support for filterspec to the getPartitions with projection API


Diffs
-

  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 d59d5d807a26378a430e683533e53d0831cf9514 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
a2b57fb646899c54b63be14a8cde9b8644a973aa 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 16f4a50d69f9120d565f61d028b060d7776689fc 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 08614749b7aba54f9eb9b54ac46f79dbac6bc5cd 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 af757932a191675bc8fb9236209a2efba9f3d335 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 66977d79c946f1ac57aacfbe8704d37bfbac3ea3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 a6d9583364be20758444ebe25c8cf636f0ea740f 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 70490f09e765d4e42391c67eb5cf018e93ad04aa 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 4dd4edccd66f8ea8ea189a2d27f970c8113e3a0f 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 06f4cbce58c16f98257e7f529ffe31c983f2919f 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestGetPartitionsUsingProjection.java
 dcff606937157e63694657b42392875d50b17be6 


Diff: https://reviews.apache.org/r/68969/diff/1/


Testing
---


Thanks,

Vihang Karajgaonkar

[jira] [Created] (HIVE-20718) Add perf cli driver with constraints

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-20718:
--

 Summary: Add perf cli driver with constraints
 Key: HIVE-20718
 URL: https://issues.apache.org/jira/browse/HIVE-20718
 Project: Hive
  Issue Type: Bug
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


Now that subtasks in HIVE-17039 will be completed, it will be good to have a 
perf cli driver with constraints declaration.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ANNOUNCE] New committer: Janaki Lahorani

[Deepak Jaiswal]

Congratulations Janaki.

On 10/9/18, 8:52 AM, "Vihang Karajgaonkar"  wrote:

Congratulations Janaki!

On Tue, Oct 9, 2018 at 8:27 AM Andrew Sherman 

wrote:

> Congratulations Janaki!
>
> On Mon, Oct 8, 2018 at 10:05 PM Ashutosh Chauhan 
> wrote:
>
> > Apache Hive's Project Management Committee (PMC) has invited Janaki
> > Lahorani to become a committer, and we are pleased to announce that she
> has
> > accepted.
> > Janaki, welcome, thank you for your contributions, and we look forward 
to
> > your further interactions with the community!
> >
> > Ashutosh Chauhan (on behalf of the Apache Hive PMC)
> >
>

[jira] [Created] (HIVE-20717) Fix test: udaf_histogram_numeric

[Zoltan Haindrich (JIRA)]

Zoltan Haindrich created HIVE-20717:
---

 Summary: Fix test: udaf_histogram_numeric 
 Key: HIVE-20717
 URL: https://issues.apache.org/jira/browse/HIVE-20717
 Project: Hive
  Issue Type: Bug
  Components: Test
Reporter: Zoltan Haindrich


disabled by HIVE-20715 because of high rate of flakiness.

Source of flakiness is probably the fact that the histogram can be a little bit 
different if the values are evaluated in a different order.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20716) Set default value for hive.cbo.stats.correlated.multi.key.joins to true

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-20716:
--

 Summary: Set default value for 
hive.cbo.stats.correlated.multi.key.joins to true
 Key: HIVE-20716
 URL: https://issues.apache.org/jira/browse/HIVE-20716
 Project: Hive
  Issue Type: Bug
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 68967: HIVE-20660: Group by statistics estimation could be improved by bounding the total number of rows to source table

[Vineet Garg]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68967/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-20660
https://issues.apache.org/jira/browse/HIVE-20660


Repository: hive-git


Description
---

See JIRA


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorUtils.java 456786c240 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 1da6d52750 
  ql/src/test/queries/clientpositive/annotate_stats_groupby.q 081f057053 
  ql/src/test/results/clientpositive/annotate_stats_groupby.q.out 932e208097 


Diff: https://reviews.apache.org/r/68967/diff/1/


Testing
---


Thanks,

Vineet Garg

Re: Review Request 68868: HIVE-17043: Remove non unique columns from group by keys if not referenced later

[Vineet Garg]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68868/
---

(Updated Oct. 9, 2018, 4:36 p.m.)


Review request for hive and Jesús Camacho Rodríguez.


Changes
---

Rebased patch


Bugs: HIVE-17043
https://issues.apache.org/jira/browse/HIVE-17043


Repository: hive-git


Description
---

This patch implements/test the following optimizations
* Removal of group by on primary keys
* Reduction of group by keys on primary keys
* is NOT NULL filter removal if NOT NULL constraint is defined


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties 97609cfadd 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveDefaultRelMetadataProvider.java
 635d27e723 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
42e60de6a8 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateJoinTransposeRule.java
 f43ef01293 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelFieldTrimmer.java
 5857f730a8 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/EstimateUniqueKeys.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdColumnUniqueness.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java
 1ca1937ed9 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdUniqueKeys.java
 3bf62c535c 
  ql/src/test/queries/clientpositive/constraints_optimization.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/constraints_optimization.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/materialized_view_rewrite_1.q.out 
735a4db2ee 


Diff: https://reviews.apache.org/r/68868/diff/8/

Changes: https://reviews.apache.org/r/68868/diff/7-8/


Testing
---


Thanks,

Vineet Garg

Re: [ANNOUNCE] New committer: Janaki Lahorani

[Vihang Karajgaonkar]

Congratulations Janaki!

On Tue, Oct 9, 2018 at 8:27 AM Andrew Sherman 
wrote:

> Congratulations Janaki!
>
> On Mon, Oct 8, 2018 at 10:05 PM Ashutosh Chauhan 
> wrote:
>
> > Apache Hive's Project Management Committee (PMC) has invited Janaki
> > Lahorani to become a committer, and we are pleased to announce that she
> has
> > accepted.
> > Janaki, welcome, thank you for your contributions, and we look forward to
> > your further interactions with the community!
> >
> > Ashutosh Chauhan (on behalf of the Apache Hive PMC)
> >
>

Re: [ANNOUNCE] New committer: Janaki Lahorani

[Andrew Sherman]

Congratulations Janaki!

On Mon, Oct 8, 2018 at 10:05 PM Ashutosh Chauhan 
wrote:

> Apache Hive's Project Management Committee (PMC) has invited Janaki
> Lahorani to become a committer, and we are pleased to announce that she has
> accepted.
> Janaki, welcome, thank you for your contributions, and we look forward to
> your further interactions with the community!
>
> Ashutosh Chauhan (on behalf of the Apache Hive PMC)
>

[jira] [Created] (HIVE-20715) Disable test: udaf_histogram_numeric

[Zoltan Haindrich (JIRA)]

Zoltan Haindrich created HIVE-20715:
---

 Summary: Disable test:  udaf_histogram_numeric
 Key: HIVE-20715
 URL: https://issues.apache.org/jira/browse/HIVE-20715
 Project: Hive
  Issue Type: Bug
  Components: Tests
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


this qtest is breaking a lot of testruns lately

I think it should be disabled



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 68960: Race Condition when Multi-Threading in SessionState.createRootHDFSDir

[denys kuzmenko via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68960/
---

Review request for hive, Jason Dere and Peter Vary.


Bugs: HIVE-20711
https://issues.apache.org/jira/browse/HIVE-20711


Repository: hive-git


Description
---

java.util.concurrent.ExecutionException: java.lang.RuntimeException: The root 
scratch dir: 
/home/hiveptest/hive-ptest-cloudera-slaves-17e5-13.gce.cloudera.com-hiveptest-0/cdh-source/itests/hive-unit/target/tmp/scratchdir
 on HDFS should be writable. Current permissions are: rwxr-xr-x at 
org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:714)
 at 
org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:637)
 at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:567) 
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:532) 
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:512) 
at


Diffs
-

  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/exec/spark/TestSparkSessionTimeout.java
 c887297bc2 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 76a30eb912 


Diff: https://reviews.apache.org/r/68960/diff/1/


Testing
---

TestSparkSessionTimeout


Thanks,

denys kuzmenko