[jira] [Created] (HIVE-15580) Replace Spark's groupByKey operator with something with bounded memory

2017-01-10 Thread Xuefu Zhang (JIRA) 
Xuefu Zhang created HIVE-15580:
--

 Summary: Replace Spark's groupByKey operator with something with 
bounded memory
 Key: HIVE-15580
 URL: https://issues.apache.org/jira/browse/HIVE-15580
 Project: Hive
  Issue Type: Improvement
  Components: Spark
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15579) Support HADOOP_PROXY_USER for secure impersonation in hive metastore client

2017-01-10 Thread Thejas M Nair (JIRA) 
Thejas M Nair created HIVE-15579:


 Summary: Support HADOOP_PROXY_USER for secure impersonation in 
hive metastore client
 Key: HIVE-15579
 URL: https://issues.apache.org/jira/browse/HIVE-15579
 Project: Hive
  Issue Type: Bug
Reporter: Thejas M Nair


Hadoop clients support HADOOP_PROXY_USER for secure impersonation. It would be 
useful to have similar feature for hive metastore client.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 55156: Min-max runtime filtering

2017-01-10 Thread Deepak Jaiswal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55156/
---

(Updated Jan. 11, 2017, 4:20 a.m.)


Review request for hive, Gopal V, Gunther Hagleitner, Jason Dere, Prasanth_J, 
and Rajesh Balamohan.


Changes
---

Created a new patch after code refresh.
Updated result files.


Bugs: HIVE-15269
https://issues.apache.org/jira/browse/HIVE-15269


Repository: hive-git


Description
---

HIVE-15269 min-max runtime filtering.
The patch also contains the patch for HIVE-15270.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 16f6c1c 
  itests/src/test/resources/testconfiguration.properties be5a747 
  orc/src/test/org/apache/orc/impl/TestRecordReaderImpl.java cdd62ac 
  ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java 
69ba4a2 
  ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java 940f2dd 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DynamicValueRegistry.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeColumnEvaluator.java 
24c8281 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeConstantDefaultEvaluator.java
 89a75eb 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeConstantEvaluator.java 
4fe72a0 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeDynamicValueEvaluator.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluator.java b8d6ab7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorFactory.java 
0d03d8f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorHead.java 42685fb 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorRef.java 0a6b66a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeFieldEvaluator.java 
ff32626 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java 
221abd9 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FilterOperator.java bd0d28c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java 46f0ecd 
  ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java ac5331e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java 9718c48 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCache.java 440e0a1 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheWrapper.java 9768efa 
  ql/src/java/org/apache/hadoop/hive/ql/exec/SelectOperator.java 9049ddd 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ObjectCache.java 008f8a4 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DynamicValueRegistryTez.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapObjectCache.java 0141230 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 
955fa80 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ObjectCache.java 06dca00 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java 
d80f201 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java 
0cb6c8a 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorSMBMapJoinOperator.java 
80b0a14 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
f6b6447 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/DynamicValueVectorExpression.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/sarg/ConvertAstToSearchArg.java 
9d900e4 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
beed6b8 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
 26fcc45 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/FixedBucketPruningOptimizer.java
 9e9beb0 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/RedundantDynamicPruningConditionsRemoval.java
 d9ce017 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 aa1e509 
  ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java e2363eb 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 35f34da 
  ql/src/java/org/apache/hadoop/hive/ql/parse/RuntimeValuesInfo.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java e8b003e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java cdb9e1b 
  ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 13a0811 
  ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicValue.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDynamicValueDesc.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/io/sarg/TestConvertAstToSearchArg.java 
93b50a6 
  ql/src/test/org/apache/hadoop/hive/ql/io/sarg/TestSearchArgumentImpl.java 
8cbc26d 
  ql/src/test/org/apache/hadoop/hive/ql/optimizer/physical/TestVectorizer.java 
3295372 
  ql/src/test/queries/clientpositive/dynamic_semijoin_reduction.q PRE-CREATION 
  ql/src/test/results/clientpositive/case_sensitivity.q.out 1952c8a

[jira] [Created] (HIVE-15578) Simplify IdentifiersParser

2017-01-10 Thread Pengcheng Xiong (JIRA) 
Pengcheng Xiong created HIVE-15578:
--

 Summary: Simplify IdentifiersParser
 Key: HIVE-15578
 URL: https://issues.apache.org/jira/browse/HIVE-15578
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15577) Simplify current parser

2017-01-10 Thread Pengcheng Xiong (JIRA) 
Pengcheng Xiong created HIVE-15577:
--

 Summary: Simplify current parser
 Key: HIVE-15577
 URL: https://issues.apache.org/jira/browse/HIVE-15577
 Project: Hive
  Issue Type: Bug
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong


We encountered "code too large" problem frequently. We need to reduce the code 
size.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15576) Fix bug in QTestUtil where lines after a partial mask will not be masked

2017-01-10 Thread Thomas Poepping (JIRA) 
Thomas Poepping created HIVE-15576:
--

 Summary: Fix bug in QTestUtil where lines after a partial mask 
will not be masked
 Key: HIVE-15576
 URL: https://issues.apache.org/jira/browse/HIVE-15576
 Project: Hive
  Issue Type: Bug
  Components: Testing Infrastructure
Affects Versions: 2.2.0
Reporter: Thomas Poepping
Assignee: Thomas Poepping


If the qfile output of a qtest contains two maskable lines right after one 
another, where the first contains a partial match candidate, the second line 
will not be evaluated for masking. This patch fixes that bug by disregarding 
whether a partial mask was found in the previous line.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15575) ALTER TABLE CONCATENATE and hive.merge.tezfiles seems busted for UNION ALL output

2017-01-10 Thread Mithun Radhakrishnan (JIRA) 
Mithun Radhakrishnan created HIVE-15575:
---

 Summary: ALTER TABLE CONCATENATE and hive.merge.tezfiles seems 
busted for UNION ALL output
 Key: HIVE-15575
 URL: https://issues.apache.org/jira/browse/HIVE-15575
 Project: Hive
  Issue Type: Bug
Reporter: Mithun Radhakrishnan
Priority: Critical


Hive {{UNION ALL}} produces data in sub-directories under the table/partition 
directories. E.g.

{noformat}
hive (mythdb_hadooppf_17544)> create table source ( foo string, bar string, goo 
string ) stored as textfile;
OK
Time taken: 0.322 seconds
hive (mythdb_hadooppf_17544)> create table results_partitioned( foo string, bar 
string, goo string ) partitioned by ( dt string ) stored as orcfile;
OK
Time taken: 0.322 seconds
hive (mythdb_hadooppf_17544)> set hive.merge.tezfiles=false; insert overwrite 
table results_partitioned partition( dt ) select 'goo', 'bar', 'foo', '1' from 
source UNION ALL select 'go', 'far', 'moo', '1' from source;
...
Loading data to table mythdb_hadooppf_17544.results_partitioned partition 
(dt=null)
 Time taken for load dynamic partitions : 311
Loading partition {dt=1}
 Time taken for adding to write entity : 3
OK
Time taken: 27.659 seconds
hive (mythdb_hadooppf_17544)> dfs -ls -R 
/tmp/mythdb_hadooppf_17544/results_partitioned;
drwxrwxrwt   - dfsload hdfs  0 2017-01-10 23:13 
/tmp/mythdb_hadooppf_17544/results_partitioned/dt=1
drwxrwxrwt   - dfsload hdfs  0 2017-01-10 23:13 
/tmp/mythdb_hadooppf_17544/results_partitioned/dt=1/1
-rwxrwxrwt   3 dfsload hdfs349 2017-01-10 23:13 
/tmp/mythdb_hadooppf_17544/results_partitioned/dt=1/1/00_0
drwxrwxrwt   - dfsload hdfs  0 2017-01-10 23:13 
/tmp/mythdb_hadooppf_17544/results_partitioned/dt=1/2
-rwxrwxrwt   3 dfsload hdfs368 2017-01-10 23:13 
/tmp/mythdb_hadooppf_17544/results_partitioned/dt=1/2/00_0
{noformat}

These results can only be read if {{mapred.input.dir.recursive=true}}, as 
{{TezCompiler::init()}} seems to do. But the Hadoop default for this is 
{{false}}. This leads to the following errors:
1. Running {{CONCATENATE}} on the partition on the partition causes data-loss.
{noformat}
hive --database mythdb_hadooppf_17544 -e " set mapred.input.dir.recursive; 
alter table results_partitioned partition ( dt='1' ) concatenate ; set 
mapred.input.dir.recursive; "
...
OK
Time taken: 2.151 seconds
mapred.input.dir.recursive=false


Status: Running (Executing on YARN cluster with App id 
application_1481756273279_5088754)


VERTICES  STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED

File Merge SUCCEEDED  0  000   0   0

VERTICES: 01/01  [>>--] 0%ELAPSED TIME: 0.35 s

Loading data to table mythdb_hadooppf_17544.results_partitioned partition (dt=1)
Moved: 
'hdfs://cluster-nn1.mygrid.myth.net:8020/tmp/mythdb_hadooppf_17544/results_partitioned/dt=1/1'
 to trash at: 
hdfs://cluster-nn1.mygrid.myth.net:8020/user/dfsload/.Trash/Current
Moved: 
'hdfs://cluster-nn1.mygrid.myth.net:8020/tmp/mythdb_hadooppf_17544/results_partitioned/dt=1/2'
 to trash at: 
hdfs://cluster-nn1.mygrid.myth.net:8020/user/dfsload/.Trash/Current
OK
Time taken: 25.873 seconds

$ hdfs dfs -count -h /tmp/mythdb_hadooppf_17544/results_partitioned/dt=1
   10  0 
/tmp/mythdb_hadooppf_17544/results_partitioned/dt=1
{noformat}

2. hive.merge.tezfiles is busted, because the merge-task attempts to merge 
files across {{results_partitioned/dt=1/1}} and {{results_partitioned/dt=1/2}}:
{noformat}
$ hive --database mythdb_hadooppf_17544 -e " set hive.merge.tezfiles=true; 
insert overwrite table results_partitioned partition( dt ) select 'goo', 'bar', 
'foo', '1' from source UNION ALL select 'go', 'far', 'moo', '1' from source; "
...
Query ID = dfsload_20170110233558_51289333-d9da-4851-8671-bfe653d26e45
Total jobs = 3
Launching Job 1 out of 3


Status: Running (Executing on YARN cluster with App id 
application_1481756273279_5089989)


VERTICES  STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED

Map 1 ..   SUCCEEDED  1  100   0   0
Map 3 ..   SUCCEEDED  1  100   0   0

VERTICES: 02/02  [==>>]

[jira] [Created] (HIVE-15574) Beeline '-f ' finishes correctly but with a NullPointerException

2017-01-10 Thread JIRA 
Sergio Peña created HIVE-15574:
--

 Summary: Beeline '-f ' finishes correctly but with a 
NullPointerException
 Key: HIVE-15574
 URL: https://issues.apache.org/jira/browse/HIVE-15574
 Project: Hive
  Issue Type: Bug
  Components: Beeline
Affects Versions: 2.2.0
Reporter: Sergio Peña
Priority: Minor


{noformat}
$ ./execute-hive.sh --service beeline -u jdbc:hive2://localhost:1 -f 
/tmp/a.sql
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/home/ubuntu/hive/packaging/target/apache-hive-2.2.0-SNAPSHOT-bin/apache-hive-2.2.0-SNAPSHOT-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/opt/hadoop/hadoop-2.6.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Connecting to jdbc:hive2://localhost:1
Connected to: Apache Hive (version 2.2.0-SNAPSHOT)
Driver: Hive JDBC (version 2.2.0-SNAPSHOT)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://localhost:1> select * from table1;
++--+
| table1.id  | table1.part  |
++--+
| 1  | 1|
| 2  | 2|
++--+
2 rows selected (0.254 seconds)
0: jdbc:hive2://localhost:1> 
0: jdbc:hive2://localhost:1> 
Closing: 0: jdbc:hive2://localhost:1
17/01/10 16:33:38 [Thread-1]: WARN util.ShutdownHookManager: ShutdownHook '' 
failed, java.lang.NullPointerException
java.lang.NullPointerException
at org.apache.hive.beeline.BeeLine$1.run(BeeLine.java:1247) 
~[hive-beeline-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at 
org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) 
[hadoop-common-2.6.1.jar:?]
{format}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15573) Vectorization: ACID shuffle ReduceSink is not specialized

2017-01-10 Thread Gopal V (JIRA) 
Gopal V created HIVE-15573:
--

 Summary: Vectorization: ACID shuffle ReduceSink is not specialized 
 Key: HIVE-15573
 URL: https://issues.apache.org/jira/browse/HIVE-15573
 Project: Hive
  Issue Type: Improvement
  Components: Transactions, Vectorization
Affects Versions: 2.2.0
Reporter: Gopal V


The ACID shuffle disabled murmur hash for the shuffle, due to the bucketing 
requirements demanding the writable hashcode for the shuffles.

{code}
boolean useUniformHash = desc.getReducerTraits().contains(UNIFORM);
if (!useUniformHash) {
  return false;
}
{code}

This check protects the fast ReduceSink ops from being used in ACID inserts.

A specialized case for the following pattern will make ACID insert much faster.

{code}
Reduce Output Operator
  sort order: 
  Map-reduce partition columns: _col0 (type: bigint)
  value expressions:  
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 55392: HIVE-15469: Fix REPL DUMP/LOAD DROP_PTN so it works on non-string-ptn-key tables

2017-01-10 Thread Vaibhav Gumashta 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55392/
---

Review request for hive, Daniel Dai, Sushanth Sowmyan, and Thejas Nair.


Bugs: HIVE-15469
https://issues.apache.org/jira/browse/HIVE-15469


Repository: hive-git


Description
---

https://issues.apache.org/jira/browse/HIVE-15469


Diffs
-

  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationListener.java
 4eabb24 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestReplicationScenarios.java
 6b86080 
  
metastore/src/java/org/apache/hadoop/hive/metastore/messaging/DropPartitionMessage.java
 26aecb3 
  
metastore/src/java/org/apache/hadoop/hive/metastore/messaging/json/JSONDropPartitionMessage.java
 b8ea224 
  
metastore/src/java/org/apache/hadoop/hive/metastore/messaging/json/JSONMessageFactory.java
 2749371 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ReplicationSemanticAnalyzer.java 
85f8c64 

Diff: https://reviews.apache.org/r/55392/diff/


Testing
---


Thanks,

Vaibhav Gumashta

Re: Review Request 55194: HIVE-15541: Hive OOM when ATSHook enabled and ATS goes down

2017-01-10 Thread Jason Dere 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55194/
---

(Updated Jan. 10, 2017, 9:09 p.m.)


Review request for hive.


Changes
---

Move creation of ATS events to outside of the ATS logging thread, to reduce the 
amount of context being saved by the work queued to the ExecutorService in case 
it's blocked.


Bugs: HIVE-15541
https://issues.apache.org/jira/browse/HIVE-15541


Repository: hive-git


Description
---

Create the ATSHook executor with a bounded queue capacity


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 16f6c1c 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/ATSHook.java 3651c9c 

Diff: https://reviews.apache.org/r/55194/diff/


Testing
---


Thanks,

Jason Dere

[jira] [Created] (HIVE-15572) Improve the response time for query canceling when it happens during acquiring locks

2017-01-10 Thread Yongzhi Chen (JIRA) 
Yongzhi Chen created HIVE-15572:
---

 Summary: Improve the response time for query canceling when it 
happens during acquiring locks
 Key: HIVE-15572
 URL: https://issues.apache.org/jira/browse/HIVE-15572
 Project: Hive
  Issue Type: Improvement
Reporter: Yongzhi Chen
Assignee: Yongzhi Chen


When query canceling command sent during Hive Acquire locks (from zookeeper), 
hive will finish acquiring all the locks and release them. As it is shown in 
the following log:
It took 165 s to finish acquire the lock,then spend 81s to release them.
We can improve the performance by not acquiring any more locks and releasing 
held locks when the query canceling command is received. 

Background-Pool: Thread-224]: 
2017-01-03 10:50:35,413 INFO  org.apache.hadoop.hive.ql.log.PerfLogger: 
[HiveServer2-Background-Pool: Thread-224]: 
2017-01-03 10:51:00,671 INFO  org.apache.hadoop.hive.ql.log.PerfLogger: 
[HiveServer2-Background-Pool: Thread-218]: 
2017-01-03 10:51:00,672 INFO  org.apache.hadoop.hive.ql.log.PerfLogger: 
[HiveServer2-Background-Pool: Thread-218]: 
2017-01-03 10:51:00,672 ERROR org.apache.hadoop.hive.ql.Driver: 
[HiveServer2-Background-Pool: Thread-218]: FAILED: query select count(*) from 
manyparttbl has been cancelled
2017-01-03 10:51:00,673 INFO  org.apache.hadoop.hive.ql.log.PerfLogger: 
[HiveServer2-Background-Pool: Thread-218]: 
2017-01-03 10:51:40,755 INFO  org.apache.hadoop.hive.ql.log.PerfLogger: 
[HiveServer2-Background-Pool: Thread-215]: 




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15571) Support Insert into for druid storage handler

2017-01-10 Thread slim bouguerra (JIRA) 
slim bouguerra created HIVE-15571:
-

 Summary: Support Insert into for druid storage handler
 Key: HIVE-15571
 URL: https://issues.apache.org/jira/browse/HIVE-15571
 Project: Hive
  Issue Type: New Feature
  Components: Druid integration
Reporter: slim bouguerra
Assignee: slim bouguerra






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 55156: Min-max runtime filtering

2017-01-10 Thread Deepak Jaiswal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55156/
---

(Updated Jan. 10, 2017, 7:36 p.m.)


Review request for hive, Gopal V, Gunther Hagleitner, Jason Dere, Prasanth_J, 
and Rajesh Balamohan.


Changes
---

Do not perform semijoin optimization if the synthetic predicate is not pushed 
down to TS.
Some fixes for SMB join not playing well with semijoin filtering.
Results files updated for some tests with explain plan diffs.


Bugs: HIVE-15269
https://issues.apache.org/jira/browse/HIVE-15269


Repository: hive-git


Description
---

HIVE-15269 min-max runtime filtering.
The patch also contains the patch for HIVE-15270.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 47db0c0 
  itests/src/test/resources/testconfiguration.properties 1cebc70 
  orc/src/test/org/apache/orc/impl/TestRecordReaderImpl.java cdd62ac 
  pom.xml 376197e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java 
69ba4a2 
  ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java 940f2dd 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DynamicValueRegistry.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeColumnEvaluator.java 
24c8281 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeConstantDefaultEvaluator.java
 89a75eb 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeConstantEvaluator.java 
4fe72a0 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeDynamicValueEvaluator.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluator.java b8d6ab7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorFactory.java 
0d03d8f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorHead.java 42685fb 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorRef.java 0a6b66a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeFieldEvaluator.java 
ff32626 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java 
221abd9 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FilterOperator.java bd0d28c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java 46f0ecd 
  ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java ac5331e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java 9718c48 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCache.java 440e0a1 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheWrapper.java 9768efa 
  ql/src/java/org/apache/hadoop/hive/ql/exec/SelectOperator.java 9049ddd 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ObjectCache.java 008f8a4 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DynamicValueRegistryTez.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapObjectCache.java 0141230 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 
955fa80 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ObjectCache.java 06dca00 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java 
d80f201 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java 
0cb6c8a 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorSMBMapJoinOperator.java 
80b0a14 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
f6b6447 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/DynamicValueVectorExpression.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/sarg/ConvertAstToSearchArg.java 
9d900e4 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
beed6b8 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
 26fcc45 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/FixedBucketPruningOptimizer.java
 9e9beb0 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/RedundantDynamicPruningConditionsRemoval.java
 d9ce017 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 aa1e509 
  ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java e2363eb 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 35f34da 
  ql/src/java/org/apache/hadoop/hive/ql/parse/RuntimeValuesInfo.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java e8b003e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java cdb9e1b 
  ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 13a0811 
  ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicValue.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDynamicValueDesc.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/io/sarg/TestConvertAstToSearchArg.java 
93b50a6 
  ql/src/test/org/apache/hadoop/hive/ql/io/sarg/TestSearchArgumentImpl.java 
8cbc26d 
  ql/src/test/org/apache/hadoop/hive/ql/optimizer/physical/TestVectorizer.java 
3295372

Review Request 55386: HIVE-15520: Improve the sum performance for Range based window

2017-01-10 Thread Aihua Xu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55386/
---

Review request for hive.


Repository: hive-git


Description
---

HIVE-15520: Improve the sum performance for Range based window


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/PTFOperator.java 
f418a7f26b316505ce17c0c988c25d0d32194ab4 
  ql/src/java/org/apache/hadoop/hive/ql/exec/PTFPartition.java 
edcb8f76c2df74a8a28fcbeb18c3fbb866a18577 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEvaluator.java 
4fc7089598f31298c5c9b73e7c21116dbba9bc5b 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSum.java 
e2cd2139f6b119763c556c3232eafe186a387f0f 
  ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/BasePartitionEvaluator.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/TableFunctionEvaluator.java 
c76118b46e95a3a7030362eeb8db979f40a7a164 
  ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/ValueBoundaryScanner.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/WindowingTableFunction.java 
2fdb492119a754f7cd033e3150d784832b9c0167 
  ql/src/test/queries/clientpositive/cbo_rp_windowing_2.q 
97f113c1e6904b22b498c824c6dba2ceaba5c594 
  ql/src/test/queries/clientpositive/ptf.q 
b5b271b134891bb6ecf462dfd64c806cb3e00fee 
  ql/src/test/queries/clientpositive/vectorized_ptf.q 
64082e99ff89c5ed7ba8bfc5a03ef063627a7aa9 
  ql/src/test/queries/clientpositive/windowing.q 
e60a6ef454868c77a1537afa4e0b58a1a297f041 
  ql/src/test/results/clientpositive/cbo_rp_windowing_2.q.out 
aa34d3dc6ed66a3a3f4a576fabad3397be6790a1 
  ql/src/test/results/clientpositive/leadlag.q.out 
86718aed3b8751a1b4864c9169d6e29c9fff3399 
  ql/src/test/results/clientpositive/llap/cbo_rp_windowing_2.q.out 
34343369a296aa4017943a35285478d0ce7ca68d 
  ql/src/test/results/clientpositive/llap/ptf.q.out 
542347d26b3060582bb7d623fdafbb0bd1b5d308 
  ql/src/test/results/clientpositive/llap/vectorized_ptf.q.out 
7efb7394c80caeed53c8145d788d40f9292f39c5 
  ql/src/test/results/clientpositive/llap/windowing.q.out 
190d13b620dca0c20fb3456a794e1ccc8345d898 
  ql/src/test/results/clientpositive/ptf.q.out 
c4c7f5978f7b9f177088cb04a37518a69590fb70 
  ql/src/test/results/clientpositive/spark/ptf.q.out 
2903003073c0dbd3b308a1f31981ada326160fe7 
  ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out 
9ab7a0b51d32474136a8e2e4c4133719e833a8d2 
  ql/src/test/results/clientpositive/spark/windowing.q.out 
72b22451870609e037ad64015dee005bf01dc02b 
  ql/src/test/results/clientpositive/vectorized_ptf.q.out 
dd6c62c25a362b4dd2c14fc4d3e5ac46de060337 
  ql/src/test/results/clientpositive/windowing.q.out 
b17bfc1ab1087e56bed26d15aa5f0559ff17642c 

Diff: https://reviews.apache.org/r/55386/diff/


Testing
---


Thanks,

Aihua Xu

[jira] [Created] (HIVE-15570) LLAP: Exception in HostAffinitySplitLocationProvider when running in container mode

2017-01-10 Thread Rajesh Balamohan (JIRA) 
Rajesh Balamohan created HIVE-15570:
---

 Summary: LLAP: Exception in HostAffinitySplitLocationProvider when 
running in container mode
 Key: HIVE-15570
 URL: https://issues.apache.org/jira/browse/HIVE-15570
 Project: Hive
  Issue Type: Bug
  Components: llap
Reporter: Rajesh Balamohan
Priority: Minor


Sometimes user might prefer to run with "hive.execution.mode=container" mode 
when LLAP is stopped. If hive config for LLAP had 
"hive.llap.client.consistent.splits=true" in client side, it would end up 
throwing the following exception in {{Utils.java}}.


{noformat}
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:68)
... 25 more
Caused by: java.lang.IllegalStateException: 
org.apache.hadoop.hive.ql.exec.tez.HostAffinitySplitLocationProvider needs at 
least 1 location to function
at 
com.google.common.base.Preconditions.checkState(Preconditions.java:149)
at 
org.apache.hadoop.hive.ql.exec.tez.HostAffinitySplitLocationProvider.(HostAffinitySplitLocationProvider.java:52)
at 
org.apache.hadoop.hive.ql.exec.tez.Utils.getSplitLocationProvider(Utils.java:54)
at 
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.(HiveSplitGenerator.java:121)
... 30 more
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 55276: HIVE-15539

2017-01-10 Thread Jesús Camacho Rodríguez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55276/
---

(Updated Jan. 10, 2017, 10:44 a.m.)


Review request for hive, Ashutosh Chauhan and Vineet Garg.


Bugs: HIVE-15539
https://issues.apache.org/jira/browse/HIVE-15539


Repository: hive-git


Description
---

HIVE-15539


Diffs (updated)
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSemanticException.java
 0038f730e6a5500fda92e795e814d5efce567e2e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
fdb468d6047b68a97cbb6248e38ef0982248ba2e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java 
f549dff3148deed158a149956d5c1ff61db8b00f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
d0131b70239dd32cc82c801f35180d2a6a1cef88 
  ql/src/test/queries/clientpositive/multi_insert_gby4.q PRE-CREATION 
  ql/src/test/queries/clientpositive/multi_insert_with_join2.q 
1529fa2fa3faaf303ac8d2ddac421e86ef0f9ce9 
  ql/src/test/results/clientpositive/auto_sortmerge_join_13.q.out 
b45411cbf638a1bc3e7d720990f6786be9ef445d 
  ql/src/test/results/clientpositive/groupby_sort_1_23.q.out 
6572f6ca577c8647b6023f108bd3cf289d9b0a88 
  ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out 
ce713547c80e519239ce82e680ad07a0ccae7d45 
  ql/src/test/results/clientpositive/llap/auto_smb_mapjoin_14.q.out 
e9990775bd40791b4b5b71716703af48da4a4a4e 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_13.q.out 
a539e03b0892acb4ad6c028d89bf49cb9252152d 
  ql/src/test/results/clientpositive/llap/explainuser_2.q.out 
931f1a2db0dd4f0ecc5f12f859c2853e83fbfbab 
  ql/src/test/results/clientpositive/llap/lineage3.q.out 
72a93445d191e4c465ead011fe6b2e35235d03a4 
  ql/src/test/results/clientpositive/llap/tez_union_multiinsert.q.out 
9b7612d8650d44691c68b6b39eb4646d9dfe0a07 
  ql/src/test/results/clientpositive/llap/unionDistinct_1.q.out 
624d8860aa9cafe667d79cdfea90a45b08cae4cc 
  ql/src/test/results/clientpositive/llap/vector_auto_smb_mapjoin_14.q.out 
735e4f435f6e2e50c9439b2a7aa437c33130b43f 
  ql/src/test/results/clientpositive/multi_insert_gby4.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/multi_insert_union_src.q.out 
2036e6354efbe4a291ef93a37cf877e638e36b4f 
  ql/src/test/results/clientpositive/multi_insert_with_join2.q.out 
70a044de93a61819e1641422914d61b7a52fbd19 
  ql/src/test/results/clientpositive/spark/auto_smb_mapjoin_14.q.out 
0e99972fceab1453d0d349f45d92519d67e27e91 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_13.q.out 
fb077718d0f25e6dbb4bd83380f647a994c2e222 
  ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out 
92ca67bc174a881fd556ec13f997d792c11285ec 
  ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out 
f7f4dbb2fcc73182c0f65e774e8195dad3907273 
  ql/src/test/results/clientpositive/spark/multi_insert_with_join2.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/spark/union17.q.out 
ce23773671859ede11a513a8b43221d746fa73ba 
  ql/src/test/results/clientpositive/spark/union18.q.out 
f9a28bb77f78532516e3ef464afa2da0602ed036 
  ql/src/test/results/clientpositive/spark/union19.q.out 
d81c19b286c517523e9bda7cc97b71058be9091a 
  ql/src/test/results/clientpositive/spark/union31.q.out 
a1f29eb485b5ddcda068c2e97a4515fbedea5a20 
  ql/src/test/results/clientpositive/spark/union_remove_6.q.out 
1bdeb09b5f0030972d937fec2404b9b14361bce3 
  ql/src/test/results/clientpositive/spark/union_remove_6_subq.q.out 
cebea03830c044243a4429ffbeccde474c204792 
  ql/src/test/results/clientpositive/tez/explainanalyze_2.q.out 
c99acc923b19be67e5f2e5eb3fb83b3e6da8b67a 
  ql/src/test/results/clientpositive/union17.q.out 
650aef434cd1941ce87df5f366e09d088ca72c0d 
  ql/src/test/results/clientpositive/union18.q.out 
599328071f2853de5c34ecd9b6408de691e33d78 
  ql/src/test/results/clientpositive/union19.q.out 
89d9c6d546fb35494a3b00792dd81f9c50d2b49e 
  ql/src/test/results/clientpositive/union31.q.out 
bb35d5c78808f2b7bdcafb85844d7304cfc12726 
  ql/src/test/results/clientpositive/union_remove_6.q.out 
0a80e63ae50b1ad3c16bf11d05c18c6ec726d38f 
  ql/src/test/results/clientpositive/union_remove_6_subq.q.out 
7b306edb1c79e61f1d16262f7338f2869193aa0b 

Diff: https://reviews.apache.org/r/55276/diff/


Testing
---


Thanks,

Jesús Camacho Rodríguez

Re: Review Request 55276: HIVE-15539

2017-01-10 Thread Jesús Camacho Rodríguez 


> On Jan. 10, 2017, 12:15 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientpositive/multi_insert_with_join2.q, lines 62-69
> > 
> >
> > Can you add following test?
> > explain
> > FROM T_A a JOIN T_B b ON a.id = b.id
> > INSERT OVERWRITE TABLE join_result_1
> > SELECT a.val, a.id, b.id, b.val
> > WHERE b.id = 'Id_1' AND b.val = 'val_103'
> > INSERT OVERWRITE TABLE join_result_3
> > SELECT a.id, b.val, b.id, a.val
> > WHERE b.val = 'val_104' AND b.id = 'Id_2';

Added and verified plan.


> On Jan. 10, 2017, 12:15 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java, line 517
> > 
> >
> > Is multimap really needed here. Seems like nodes declared above would 
> > have sufficied here?

Actually this is related to the object above. Comment was not in the right 
place. I could remove indeed _nodes_ list. However, this should be a 
_multimap_. The reason is that we need to keep all nodes from all insert 
statements so we can modify them after creating the subquery (loop in L589).


> On Jan. 10, 2017, 12:15 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java, line 608
> > 
> >
> > Will be good to add comment on what this loop is doing?

Updated comment in L599 to reflect we rewrite all references.


- Jesús


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55276/#review160938
---


On Jan. 6, 2017, 7:21 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55276/
> ---
> 
> (Updated Jan. 6, 2017, 7:21 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Vineet Garg.
> 
> 
> Bugs: HIVE-15539
> https://issues.apache.org/jira/browse/HIVE-15539
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15539
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSemanticException.java
>  0038f730e6a5500fda92e795e814d5efce567e2e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> fdb468d6047b68a97cbb6248e38ef0982248ba2e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java 
> f549dff3148deed158a149956d5c1ff61db8b00f 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> d0131b70239dd32cc82c801f35180d2a6a1cef88 
>   ql/src/test/queries/clientpositive/multi_insert_gby4.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/multi_insert_with_join2.q 
> 1529fa2fa3faaf303ac8d2ddac421e86ef0f9ce9 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_13.q.out 
> b45411cbf638a1bc3e7d720990f6786be9ef445d 
>   ql/src/test/results/clientpositive/groupby_sort_1_23.q.out 
> 6572f6ca577c8647b6023f108bd3cf289d9b0a88 
>   ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out 
> ce713547c80e519239ce82e680ad07a0ccae7d45 
>   ql/src/test/results/clientpositive/llap/auto_smb_mapjoin_14.q.out 
> e9990775bd40791b4b5b71716703af48da4a4a4e 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_13.q.out 
> a539e03b0892acb4ad6c028d89bf49cb9252152d 
>   ql/src/test/results/clientpositive/llap/explainuser_2.q.out 
> 931f1a2db0dd4f0ecc5f12f859c2853e83fbfbab 
>   ql/src/test/results/clientpositive/llap/lineage3.q.out 
> 72a93445d191e4c465ead011fe6b2e35235d03a4 
>   ql/src/test/results/clientpositive/llap/tez_union_multiinsert.q.out 
> 9b7612d8650d44691c68b6b39eb4646d9dfe0a07 
>   ql/src/test/results/clientpositive/llap/unionDistinct_1.q.out 
> 624d8860aa9cafe667d79cdfea90a45b08cae4cc 
>   ql/src/test/results/clientpositive/llap/vector_auto_smb_mapjoin_14.q.out 
> 735e4f435f6e2e50c9439b2a7aa437c33130b43f 
>   ql/src/test/results/clientpositive/multi_insert_gby4.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/multi_insert_union_src.q.out 
> 2036e6354efbe4a291ef93a37cf877e638e36b4f 
>   ql/src/test/results/clientpositive/multi_insert_with_join2.q.out 
> 70a044de93a61819e1641422914d61b7a52fbd19 
>   ql/src/test/results/clientpositive/spark/auto_smb_mapjoin_14.q.out 
> 0e99972fceab1453d0d349f45d92519d67e27e91 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_13.q.out 
> fb077718d0f25e6dbb4bd83380f647a994c2e222 
>   ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out 
> 92ca67bc174a881fd556ec13f997d792c11285ec 
>   ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out 
> f7f4dbb2fcc73182c0f65e774e8195dad3907273 
>