Review Request 58914: HIVE-16550

2017-05-01 Thread Deepak Jaiswal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58914/
---

Review request for hive, Jason Dere and Sergey Shelukhin.


Bugs: HIVE-16550
https://issues.apache.org/jira/browse/HIVE-16550


Repository: hive-git


Description
---

Semijoin Hints should be able to skip the optimization if needed.
In addition to that, the patch fixes several issues with hints in general such 
as,
- It now works with subqueries and unions.
- Uses a global data structure instead of per QB.


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
 e1a69526bc 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveOpConverter.java
 d375d1b58d 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 1b054a7e24 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HintParser.g e110fb33df 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 3a1f821bd3 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java 54e37f7c80 
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java 7bf1c599a5 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 654f3b1772 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 5ea7800528 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDynamicListDesc.java 
3143554ec6 
  ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java 032c7bb28d 
  ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java 
f45daa8828 
  ql/src/test/queries/clientpositive/semijoin_hint.q 5de0c8c8c1 
  ql/src/test/results/clientpositive/llap/semijoin_hint.q.out bc248930ec 


Diff: https://reviews.apache.org/r/58914/diff/1/


Testing
---


Thanks,

Deepak Jaiswal

Re: Review Request 58865: HIVE-16552: Limit the number of tasks a Spark job may contain

2017-05-01 Thread Rui Li 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58865/#review173556
---




ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java
Lines 135 (patched)


The log is incorrect because cancelling the job doesn't mean killing the 
application.



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/RemoteSparkJobMonitor.java
Lines 106 (patched)


I think the total task count needs only be computed once. It shouldn't 
change during the execution of the job, assuming we don't count failed/retried 
tasks.


- Rui Li


On May 1, 2017, 5:13 p.m., Xuefu Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58865/
> ---
> 
> (Updated May 1, 2017, 5:13 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-16552
> https://issues.apache.org/jira/browse/HIVE-16552
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See JIRA description
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d3ea824 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java 32a7730 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/RemoteSparkJobMonitor.java
>  dd73f3e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java 
> 0b224f2 
> 
> 
> Diff: https://reviews.apache.org/r/58865/diff/2/
> 
> 
> Testing
> ---
> 
> Test locally
> 
> 
> Thanks,
> 
> Xuefu Zhang
> 
>

Re: Review Request 58777: HIVE-16546: LLAP: Fail map join tasks if hash table memory exceeds threshold

2017-05-01 Thread j . prasanth . j 


> On May 1, 2017, 9:13 p.m., Sergey Shelukhin wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java
> > Lines 225 (patched)
> > 
> >
> > is it possible to handle 0/negative values differently, to have a 
> > complete off switch in case something goes wrong?

Setting value of 0 will not completely turn off the check.


- Prasanth_J


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58777/#review173510
---


On May 2, 2017, 12:22 a.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58777/
> ---
> 
> (Updated May 2, 2017, 12:22 a.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, Sergey Shelukhin, and Siddharth 
> Seth.
> 
> 
> Bugs: HIVE-16546
> https://issues.apache.org/jira/browse/HIVE-16546
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16546: LLAP: Fail map join tasks if hash table memory exceeds threshold
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/MemoryEstimate.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d3ea824 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionError.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionException.java
>  dbe00b6 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionHandler.java
>  7fc3226 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 595d1bd 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
>  04e24bd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 
> a3bccc6 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HybridHashTableContainer.java
>  04e89e8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java
>  c86e5f5 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java
>  6d71fef 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 7b13e90 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java 486d43a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashMap.java
>  6242daf 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashMultiSet.java
>  1a41961 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashSet.java
>  331867c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashTable.java
>  b93e977 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTable.java
>  b6db3bc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTableLoader.java
>  49ecdd1 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastKeyStore.java
>  be51693 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashMap.java
>  6fe98f9 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashMultiSet.java
>  9140aee 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashSet.java
>  d3efb11 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashTable.java
>  8bfa07c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastMultiKeyHashMap.java
>  add4788 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastMultiKeyHashMultiSet.java
>  faefdbb 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastMultiKeyHashSet.java
>  5328910 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashMap.java
>  f13034f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashMultiSet.java
>  53ad7b4 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashSet.java
>  723c729 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastTableContainer.java
>  05f1cf1 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastValueStore.java
>  f9c5b34 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/hashtable/VectorMapJoinHashTable.java
>  c7e585c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMap

Re: Review Request 58777: HIVE-16546: LLAP: Fail map join tasks if hash table memory exceeds threshold

2017-05-01 Thread j . prasanth . j 


> On April 30, 2017, 8:12 a.m., Lefty Leverenz wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> > Lines 3155-3158 (patched)
> > 
> >
> > "after every these many rows" -> "after every interval of this many 
> > rows"
> > 
> > "exceed" -> "exceeds"
> > 
> > Also, please add another \n to the description after "when running" 
> > (line 3157 in diff revision 4).

Thanks Lefty! Fixed in the new patch.


- Prasanth_J


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58777/#review173439
---


On May 2, 2017, 12:22 a.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58777/
> ---
> 
> (Updated May 2, 2017, 12:22 a.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, Sergey Shelukhin, and Siddharth 
> Seth.
> 
> 
> Bugs: HIVE-16546
> https://issues.apache.org/jira/browse/HIVE-16546
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16546: LLAP: Fail map join tasks if hash table memory exceeds threshold
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/MemoryEstimate.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d3ea824 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionError.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionException.java
>  dbe00b6 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionHandler.java
>  7fc3226 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 595d1bd 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
>  04e24bd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 
> a3bccc6 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HybridHashTableContainer.java
>  04e89e8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java
>  c86e5f5 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java
>  6d71fef 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 7b13e90 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java 486d43a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashMap.java
>  6242daf 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashMultiSet.java
>  1a41961 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashSet.java
>  331867c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashTable.java
>  b93e977 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTable.java
>  b6db3bc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTableLoader.java
>  49ecdd1 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastKeyStore.java
>  be51693 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashMap.java
>  6fe98f9 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashMultiSet.java
>  9140aee 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashSet.java
>  d3efb11 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashTable.java
>  8bfa07c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastMultiKeyHashMap.java
>  add4788 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastMultiKeyHashMultiSet.java
>  faefdbb 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastMultiKeyHashSet.java
>  5328910 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashMap.java
>  f13034f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashMultiSet.java
>  53ad7b4 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashSet.java
>  723c729 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastTableContainer.java
>  05f1cf1 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastValueStore.java
>  f9c5b34 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/hashtable/VectorMapJoinHashTable.java

Re: Review Request 58777: HIVE-16546: LLAP: Fail map join tasks if hash table memory exceeds threshold

2017-05-01 Thread j . prasanth . j 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58777/
---

(Updated May 2, 2017, 12:22 a.m.)


Review request for hive, Gunther Hagleitner, Sergey Shelukhin, and Siddharth 
Seth.


Changes
---

Addressed Lefty's and Sergey's review comments.


Bugs: HIVE-16546
https://issues.apache.org/jira/browse/HIVE-16546


Repository: hive-git


Description
---

HIVE-16546: LLAP: Fail map join tasks if hash table memory exceeds threshold


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/MemoryEstimate.java 
PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d3ea824 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionError.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionException.java
 dbe00b6 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionHandler.java
 7fc3226 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 595d1bd 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
 04e24bd 
  ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 
a3bccc6 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HybridHashTableContainer.java
 04e89e8 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java
 c86e5f5 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java
 6d71fef 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 7b13e90 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java 486d43a 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashMap.java
 6242daf 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashMultiSet.java
 1a41961 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashSet.java
 331867c 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashTable.java
 b93e977 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTable.java
 b6db3bc 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTableLoader.java
 49ecdd1 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastKeyStore.java
 be51693 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashMap.java
 6fe98f9 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashMultiSet.java
 9140aee 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashSet.java
 d3efb11 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashTable.java
 8bfa07c 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastMultiKeyHashMap.java
 add4788 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastMultiKeyHashMultiSet.java
 faefdbb 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastMultiKeyHashSet.java
 5328910 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashMap.java
 f13034f 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashMultiSet.java
 53ad7b4 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashSet.java
 723c729 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastTableContainer.java
 05f1cf1 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastValueStore.java
 f9c5b34 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/hashtable/VectorMapJoinHashTable.java
 c7e585c 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedHashSet.java
 93a89d7 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedHashTable.java
 5fe7861 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedStringHashSet.java
 f921b9c 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
ad77e87 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java b2893e7 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveOpConverter.java
 d375d1b 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/GenMRSkewJoinProcessor.java
 93b8a5d 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/GenSparkSkewJoinProcessor.java
 405c3ca 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java d39b8bd 
  ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java 032c7

[jira] [Created] (HIVE-16566) Set column stats default as true when creating new tables/partitions

2017-05-01 Thread Pengcheng Xiong (JIRA) 
Pengcheng Xiong created HIVE-16566:
--

 Summary: Set column stats default as true when creating new 
tables/partitions
 Key: HIVE-16566
 URL: https://issues.apache.org/jira/browse/HIVE-16566
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16565) Improve how the open transactions and aborted transactions are deserialized in ValidReadTxnList.readFromString

2017-05-01 Thread Wei Zheng (JIRA) 
Wei Zheng created HIVE-16565:


 Summary: Improve how the open transactions and aborted 
transactions are deserialized in ValidReadTxnList.readFromString
 Key: HIVE-16565
 URL: https://issues.apache.org/jira/browse/HIVE-16565
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 3.0.0
Reporter: Wei Zheng
Assignee: Wei Zheng


This is a follow-up of HIVE-16534.

In ValidReadTxnList.writeToString, we write out two open and aborted 
transactions as two sorted lists. We can take advantage of that and perform 
merge sort them together when reading them back in readFromString. Note that 
the aborted bits should also be handled properly during the merge sort.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 58777: HIVE-16546: LLAP: Fail map join tasks if hash table memory exceeds threshold

2017-05-01 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58777/#review173510
---




ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java
Lines 225 (patched)


is it possible to handle 0/negative values differently, to have a complete 
off switch in case something goes wrong?


- Sergey Shelukhin


On April 28, 2017, 4:53 a.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58777/
> ---
> 
> (Updated April 28, 2017, 4:53 a.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, Sergey Shelukhin, and Siddharth 
> Seth.
> 
> 
> Bugs: HIVE-16546
> https://issues.apache.org/jira/browse/HIVE-16546
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16546: LLAP: Fail map join tasks if hash table memory exceeds threshold
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/MemoryEstimate.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d3ea824 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionError.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionException.java
>  dbe00b6 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionHandler.java
>  7fc3226 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 595d1bd 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
>  04e24bd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 
> a3bccc6 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HybridHashTableContainer.java
>  04e89e8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java
>  c86e5f5 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java
>  6d71fef 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 7b13e90 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 
> 24d3526 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java 486d43a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashMap.java
>  6242daf 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashMultiSet.java
>  1a41961 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashSet.java
>  331867c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashTable.java
>  b93e977 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTable.java
>  b6db3bc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTableLoader.java
>  49ecdd1 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastKeyStore.java
>  be51693 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashMap.java
>  6fe98f9 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashMultiSet.java
>  9140aee 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashSet.java
>  d3efb11 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashTable.java
>  8bfa07c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastMultiKeyHashMap.java
>  add4788 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastMultiKeyHashMultiSet.java
>  faefdbb 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastMultiKeyHashSet.java
>  5328910 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashMap.java
>  f13034f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashMultiSet.java
>  53ad7b4 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashSet.java
>  723c729 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastTableContainer.java
>  05f1cf1 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastValueStore.java
>  f9c5b34 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/hashtable/VectorMapJoinHashTable.java
>  c7e585c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedHashSet.java
>  93a89d7 
>   
> ql/src/java/org/ap

[jira] [Created] (HIVE-16564) StreamingAPI is locking too much?

2017-05-01 Thread Eugene Koifman (JIRA) 
Eugene Koifman created HIVE-16564:
-

 Summary: StreamingAPI is locking too much?
 Key: HIVE-16564
 URL: https://issues.apache.org/jira/browse/HIVE-16564
 Project: Hive
  Issue Type: Bug
  Components: HCatalog, Transactions
Affects Versions: 1.0.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman


Currently _TransactionBatchImpl.beginNextTransactionImpl()_ acquires Shared 
locks for each Transaction in the batch.  
Especially under high load this creates pressure on the LockManager (i.e. 
Metastore) and degrades performance of Ingest itself.
Because all transactions in a batch write to the same physical file and the 
fact that for Acid tables (which are required for Streaming Ingest) shared 
locks only protect against Exclusive locks (like drop table), 
acquiring/releasing locks doesn't for each txn doesn't achieve much.

One possibility to acquire all locks (i.e. for all txns) at the time the batch 
is created (same as is done for openTxn() for all txns in the batch).  Locks 
for each txn in the batch will be released automatically when commit is called 
for the respective txn.

Alternatively, don't acquire any locks - this means someone may drop a table 
while it's written to but using locks here doesn't buy much.  Say a Drop 
request is issued when a write is in progress.  It will block until the write 
releases it's lock and execute immediately after that.  Thus none of the data 
of that write is visible for any meaningful length of time anyway.

Allow a "meta lock" - a lock not associated with any specific txn, that is held 
for the duration of the TransactionBatch.  This sort of breaks the model 
(especially since HIVE-12636).  Perhaps each batch can open one "extra" txn for 
internal purposes, just to acquire this "meta lock".  No data will ever be 
tagged with this "extra" txn.







--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16563) Alter table partition set location should use fully qualified path for non-default FS

2017-05-01 Thread Chao Sun (JIRA) 
Chao Sun created HIVE-16563:
---

 Summary: Alter table partition set location should use fully 
qualified path for non-default FS
 Key: HIVE-16563
 URL: https://issues.apache.org/jira/browse/HIVE-16563
 Project: Hive
  Issue Type: Bug
Reporter: Chao Sun
Assignee: Chao Sun


Similar to HIVE-6374, for command {{ALTER TABLE .. PARTITION(..) SET LOCATION 
..}}, if location path is not a fully qualified path and Hive is not using 
default namenode, it should use fully qualified path for the partition.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] Should we release hive-storage 2.4.0rc0?

2017-05-01 Thread Owen O'Malley 
-1 I forgot to make sure the license and notice files were there. I'll roll a 
new release candidate. 

.. Owen

> On May 1, 2017, at 09:52, Ashutosh Chauhan  wrote:
> 
> Downloaded tar ball. Shasum matches. Build from src. All looks good.
> +1
> 
>> On Fri, Apr 28, 2017 at 11:39 AM, Pengcheng Xiong  wrote:
>> 
>> I downloaded the tarball, extracted, installed it. It works well with Hive
>> 2.3 (candidate). Thus, +1 on this. Thanks a lot for your work!
>> 
>> On Fri, Apr 28, 2017 at 10:31 AM, Owen O'Malley 
>> wrote:
>> 
>>> All,
>>>   Yes, I know this is confusing, but I wanted to help get the 2.4.0
>>> storage-api released so that hive 2.3.0rc1 can depend on it. So, please
>>> check this release.
>>> 
>>> Artifacts:
>>> tag: https://github.com/apache/hive/releases/tag/storage-
>> release-2.4.0rc0
>>> branch: storage-branch-2.4
>>> tar ball: http://home.apache.org/~omalley/hive-storage-2.4.0/
>>> 
>>> Thanks,
>>>   Owen
>>> 
>>

Re: Review Request 56140: Can't order by an unselected column

2017-05-01 Thread pengcheng xiong 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56140/
---

(Updated May 1, 2017, 5:30 p.m.)


Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

HIVE-15160


Diffs (updated)
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java
 1487ed4f8e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 1b054a7e24 
  ql/src/java/org/apache/hadoop/hive/ql/parse/RowResolver.java 262dafb487 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 654f3b1772 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
8f8eab0d9c 
  ql/src/test/queries/clientpositive/order_by_expr_1.q PRE-CREATION 
  ql/src/test/queries/clientpositive/order_by_expr_2.q PRE-CREATION 
  ql/src/test/results/clientpositive/annotate_stats_select.q.out 873f1abb25 
  ql/src/test/results/clientpositive/cp_sel.q.out 1778ccd6a6 
  ql/src/test/results/clientpositive/druid_basic2.q.out 6177d56987 
  ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out 2abb819558 
  ql/src/test/results/clientpositive/groupby_grouping_sets_grouping.q.out 
473d17a1bd 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out d724131fca 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 584c3b5520 
  ql/src/test/results/clientpositive/llap/limit_pushdown.q.out dd54dd22a6 
  ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out 24645b6426 
  ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
83de1fbea1 
  ql/src/test/results/clientpositive/llap/vector_coalesce.q.out 578f849bdb 
  ql/src/test/results/clientpositive/llap/vector_date_1.q.out a4f1050c89 
  ql/src/test/results/clientpositive/llap/vector_decimal_2.q.out 144356c108 
  ql/src/test/results/clientpositive/llap/vector_decimal_round.q.out 8bd80cf860 
  
ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_grouping.q.out
 5af9e61b0a 
  
ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out
 f731ceecdc 
  ql/src/test/results/clientpositive/llap/vector_interval_1.q.out debf5ab39e 
  ql/src/test/results/clientpositive/llap/vector_interval_arithmetic.q.out 
aadb6e72cd 
  ql/src/test/results/clientpositive/order3.q.out 898f7a8853 
  ql/src/test/results/clientpositive/order_by_expr_1.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/order_by_expr_2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/pcr.q.out a1301fdf79 
  ql/src/test/results/clientpositive/perf/query31.q.out 3ed312d3e3 
  ql/src/test/results/clientpositive/perf/query36.q.out 57ab26acc6 
  ql/src/test/results/clientpositive/perf/query39.q.out 19472c4d5e 
  ql/src/test/results/clientpositive/perf/query42.q.out 3bebac3321 
  ql/src/test/results/clientpositive/perf/query52.q.out 74ecaf28ba 
  ql/src/test/results/clientpositive/perf/query64.q.out 6b42393aad 
  ql/src/test/results/clientpositive/perf/query66.q.out 072bfee92b 
  ql/src/test/results/clientpositive/perf/query70.q.out 8e42fac9c5 
  ql/src/test/results/clientpositive/perf/query75.q.out b1e236d325 
  ql/src/test/results/clientpositive/perf/query81.q.out a09d5c99b5 
  ql/src/test/results/clientpositive/perf/query85.q.out 168bcd2a4a 
  ql/src/test/results/clientpositive/perf/query86.q.out 734e6a480b 
  ql/src/test/results/clientpositive/perf/query89.q.out 66481f710b 
  ql/src/test/results/clientpositive/perf/query91.q.out e592bba8d9 
  ql/src/test/results/clientpositive/pointlookup2.q.out 3438c74608 
  ql/src/test/results/clientpositive/pointlookup3.q.out 2c3e39fd15 
  ql/src/test/results/clientpositive/ppd_udf_case.q.out 7678d03415 
  ql/src/test/results/clientpositive/spark/dynamic_rdd_cache.q.out 6572511967 
  ql/src/test/results/clientpositive/spark/limit_pushdown.q.out ede0096c73 
  ql/src/test/results/clientpositive/spark/pcr.q.out 77ac020d07 
  ql/src/test/results/clientpositive/vector_coalesce.q.out f158236beb 
  ql/src/test/results/clientpositive/vector_date_1.q.out c2389e6b1e 
  ql/src/test/results/clientpositive/vector_decimal_round.q.out de49c170cf 
  ql/src/test/results/clientpositive/vector_interval_1.q.out f53a2c2db5 
  ql/src/test/results/clientpositive/vector_interval_arithmetic.q.out 
75250e30a4 
  ql/src/test/results/clientpositive/view_alias.q.out 90bf28dd9b 


Diff: https://reviews.apache.org/r/56140/diff/8/

Changes: https://reviews.apache.org/r/56140/diff/7-8/


Testing
---


Thanks,

pengcheng xiong

Re: Review Request 58865: HIVE-16552: Limit the number of tasks a Spark job may contain

2017-05-01 Thread Xuefu Zhang 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58865/
---

(Updated May 1, 2017, 5:13 p.m.)


Review request for hive.


Changes
---

Updated patch to reflect Lefty's feedback.


Bugs: HIVE-16552
https://issues.apache.org/jira/browse/HIVE-16552


Repository: hive-git


Description
---

See JIRA description


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d3ea824 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java 32a7730 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/RemoteSparkJobMonitor.java
 dd73f3e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java 
0b224f2 


Diff: https://reviews.apache.org/r/58865/diff/2/

Changes: https://reviews.apache.org/r/58865/diff/1-2/


Testing
---

Test locally


Thanks,

Xuefu Zhang

Re: [VOTE] Should we release hive-storage 2.4.0rc0?

2017-05-01 Thread Ashutosh Chauhan 
Downloaded tar ball. Shasum matches. Build from src. All looks good.
+1

On Fri, Apr 28, 2017 at 11:39 AM, Pengcheng Xiong  wrote:

> I downloaded the tarball, extracted, installed it. It works well with Hive
> 2.3 (candidate). Thus, +1 on this. Thanks a lot for your work!
>
> On Fri, Apr 28, 2017 at 10:31 AM, Owen O'Malley 
> wrote:
>
> > All,
> >Yes, I know this is confusing, but I wanted to help get the 2.4.0
> > storage-api released so that hive 2.3.0rc1 can depend on it. So, please
> > check this release.
> >
> > Artifacts:
> > tag: https://github.com/apache/hive/releases/tag/storage-
> release-2.4.0rc0
> > branch: storage-branch-2.4
> > tar ball: http://home.apache.org/~omalley/hive-storage-2.4.0/
> >
> > Thanks,
> >Owen
> >
>