[jira] [Created] (HIVE-16423) De-duplicate semijoin branches and add hint to enforce semi join optimization

2017-04-11 Thread Deepak Jaiswal (JIRA) 
Deepak Jaiswal created HIVE-16423:
-

 Summary: De-duplicate semijoin branches and add hint to enforce 
semi join optimization
 Key: HIVE-16423
 URL: https://issues.apache.org/jira/browse/HIVE-16423
 Project: Hive
  Issue Type: Task
Reporter: Deepak Jaiswal
Assignee: Deepak Jaiswal


Currently in an n-way join, a semi join branch is created n times. Instead, it 
should reuse the  same branch.
Add hints in semijoin to enforce particular semi join optimization.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16422) Should kill running Spark Jobs when a query is cancelled.

2017-04-11 Thread zhihai xu (JIRA) 
zhihai xu created HIVE-16422:


 Summary: Should kill running Spark Jobs when a query is cancelled.
 Key: HIVE-16422
 URL: https://issues.apache.org/jira/browse/HIVE-16422
 Project: Hive
  Issue Type: Bug
  Components: Spark
Affects Versions: 2.1.0
Reporter: zhihai xu
Assignee: zhihai xu


Should kill running Spark Jobs when a query is cancelled. When a query is 
cancelled, Driver.releaseDriverContext will be called by Driver.close. 
releaseDriverContext will call DriverContext.shutdown which will call all the 
running tasks' shutdown.
{code}
  public synchronized void shutdown() {
LOG.debug("Shutting down query " + ctx.getCmd());
shutdown = true;
for (TaskRunner runner : running) {
  if (runner.isRunning()) {
Task task = runner.getTask();
LOG.warn("Shutting down task : " + task);
try {
  task.shutdown();
} catch (Exception e) {
  console.printError("Exception on shutting down task " + task.getId() 
+ ": " + e);
}
Thread thread = runner.getRunner();
if (thread != null) {
  thread.interrupt();
}
  }
}
running.clear();
  }
{code}
since SparkTask didn't implement shutdown method to kill the running spark job, 
the spark job may be still running after the query is cancelled. So it will be 
good to kill the spark job in SparkTask.shutdown to save cluster resource.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 58181: HIVE-15986 Support for 'is [NOT] distinct from'

2017-04-11 Thread Vineet Garg 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58181/
---

(Updated April 12, 2017, 12:14 a.m.)


Review request for hive, Ashutosh Chauhan, Jesús Camacho Rodríguez, and 
Pengcheng Xu.


Changes
---

Adding grammar rules to rewrite 'is distinct from' into 'not <=>'  instead of 
adding new tokens


Bugs: HIVE-15986
https://issues.apache.org/jira/browse/HIVE-15986


Repository: hive-git


Description
---

This patch adds support for 'is distinct from' and 'is not distinct from'.


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties fb85b9ece2 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java
 85450c999f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 8c4ee8a38b 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
f979c1432e 
  ql/src/test/queries/clientpositive/is_distinct_from.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/is_distinct_from.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/58181/diff/5/

Changes: https://reviews.apache.org/r/58181/diff/4-5/


Testing
---

Added new tests
Pre-commit testing


Thanks,

Vineet Garg

[jira] [Created] (HIVE-16421) Runtime filtering breaks user-level explain

2017-04-11 Thread Pengcheng Xiong (JIRA) 
Pengcheng Xiong created HIVE-16421:
--

 Summary: Runtime filtering breaks user-level explain
 Key: HIVE-16421
 URL: https://issues.apache.org/jira/browse/HIVE-16421
 Project: Hive
  Issue Type: Bug
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 57893: HIVE-16285: Servlet for dynamically configuring log levels

2017-04-11 Thread j . prasanth . j 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57893/
---

(Updated April 11, 2017, 9:55 p.m.)


Review request for hive, Gopal V and Siddharth Seth.


Changes
---

Minor changes to http headers.


Bugs: HIVE-16285
https://issues.apache.org/jira/browse/HIVE-16285


Repository: hive-git


Description
---

Many long running services like HS2, LLAP etc. will benefit from having an 
endpoint to dynamically change log levels for various loggers. This will help 
greatly with debuggability without requiring a restart of the service.


Diffs (updated)
-

  common/src/java/org/apache/hive/http/HttpServer.java fd3d457 
  common/src/java/org/apache/hive/http/Log4j2ConfiguratorServlet.java 
PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
 7f8c947 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 18ce03c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/AppMasterEventOperator.java 
bf30ef1 
  ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java df1898e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DemuxOperator.java c184742 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 4d727ba 
  ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java 6d6c608 
  ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 3a366f6 
  ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java 0282763 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 07aa2ea 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java 2a46b30 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MuxOperator.java 9849243 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 8b04cd4 
  ql/src/java/org/apache/hadoop/hive/ql/exec/OrcFileMergeOperator.java e3cb765 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java e03f4b7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 7c1e344 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ScriptOperator.java 4767af1 
  ql/src/java/org/apache/hadoop/hive/ql/exec/SelectOperator.java 94af097 
  ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java 68477ca 
  ql/src/java/org/apache/hadoop/hive/ql/exec/UnionOperator.java 3df5533 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecReducer.java 1d2 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ObjectCache.java cfe1750 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ColumnarSplitSizeEstimator.java 
ecd4ddc 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HostAffinitySplitLocationProvider.java
 dcb985f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapObjectCache.java 1ce8ee9 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinCommonOperator.java
 f854132 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinGenerateResultOperator.java
 cb30413 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinInnerBigOnlyLongOperator.java
 43f3951 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinInnerBigOnlyMultiKeyOperator.java
 95fb0c2 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinInnerBigOnlyStringOperator.java
 044e3e6 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinInnerLongOperator.java
 c85e1d8 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinInnerMultiKeyOperator.java
 a108cd0 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinInnerStringOperator.java
 3211d7d 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinLeftSemiLongOperator.java
 b02e6fd 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinLeftSemiMultiKeyOperator.java
 36b8f3f 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinLeftSemiStringOperator.java
 0b3de0a 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinOuterGenerateResultOperator.java
 0e2d65a 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinOuterLongOperator.java
 72309e8 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinOuterMultiKeyOperator.java
 a4fc7d3 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinOuterStringOperator.java
 6e7e5cb 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashTable.java
 b93e977 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashTable.java
 8bfa07c 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java
 fc5aea5 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/ExternalCache.java 9299306 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 8fb7

Re: Review Request 57893: HIVE-16285: Servlet for dynamically configuring log levels

2017-04-11 Thread j . prasanth . j 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57893/
---

(Updated April 11, 2017, 9:10 p.m.)


Review request for hive, Gopal V and Siddharth Seth.


Changes
---

Addressed Gopal's review comments.


Bugs: HIVE-16285
https://issues.apache.org/jira/browse/HIVE-16285


Repository: hive-git


Description
---

Many long running services like HS2, LLAP etc. will benefit from having an 
endpoint to dynamically change log levels for various loggers. This will help 
greatly with debuggability without requiring a restart of the service.


Diffs (updated)
-

  common/src/java/org/apache/hive/http/HttpServer.java fd3d457 
  common/src/java/org/apache/hive/http/Log4j2ConfiguratorServlet.java 
PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
 7f8c947 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 18ce03c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/AppMasterEventOperator.java 
bf30ef1 
  ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java df1898e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DemuxOperator.java c184742 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 4d727ba 
  ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java 6d6c608 
  ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 3a366f6 
  ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java 0282763 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 07aa2ea 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java 2a46b30 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MuxOperator.java 9849243 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 8b04cd4 
  ql/src/java/org/apache/hadoop/hive/ql/exec/OrcFileMergeOperator.java e3cb765 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java e03f4b7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 7c1e344 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ScriptOperator.java 4767af1 
  ql/src/java/org/apache/hadoop/hive/ql/exec/SelectOperator.java 94af097 
  ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java 68477ca 
  ql/src/java/org/apache/hadoop/hive/ql/exec/UnionOperator.java 3df5533 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecReducer.java 1d2 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ObjectCache.java cfe1750 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ColumnarSplitSizeEstimator.java 
ecd4ddc 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HostAffinitySplitLocationProvider.java
 dcb985f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapObjectCache.java 1ce8ee9 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinCommonOperator.java
 f854132 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinGenerateResultOperator.java
 cb30413 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinInnerBigOnlyLongOperator.java
 43f3951 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinInnerBigOnlyMultiKeyOperator.java
 95fb0c2 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinInnerBigOnlyStringOperator.java
 044e3e6 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinInnerLongOperator.java
 c85e1d8 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinInnerMultiKeyOperator.java
 a108cd0 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinInnerStringOperator.java
 3211d7d 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinLeftSemiLongOperator.java
 b02e6fd 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinLeftSemiMultiKeyOperator.java
 36b8f3f 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinLeftSemiStringOperator.java
 0b3de0a 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinOuterGenerateResultOperator.java
 0e2d65a 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinOuterLongOperator.java
 72309e8 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinOuterMultiKeyOperator.java
 a4fc7d3 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinOuterStringOperator.java
 6e7e5cb 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashTable.java
 b93e977 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashTable.java
 8bfa07c 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java
 fc5aea5 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/ExternalCache.java 9299306 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java

Re: Review Request 57944: HIVE-15708 Upgrade Calcite to 1.12 and Avatica to 1.9

2017-04-11 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57944/#review171627
---




ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/ReduceSinkDeDuplication.java
Line 197 (original), 197 (patched)


Is there a reason for it?

Partition cols are used for partitioning during shuffle between map and 
reducers. Key columns are used for sorting on those columns in reducers. In 
general, if child ReduceSink has less restrictive sorting requirement than 
parent that should be ok for merging two RSs, if we chose parent's column as 
key columns in merged RS.



ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
Lines 1346 (patched)


Can delete.



ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
Lines 1787 (patched)


Worth a comment on why basePlan.getCluster().getPlanner() != planner.


- Ashutosh Chauhan


On April 11, 2017, 3:17 p.m., Remus Rusanu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57944/
> ---
> 
> (Updated April 11, 2017, 3:17 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15708 Upgrade Calcite to 1.12 and Avatica to 1.9
> 
> 
> Diffs
> -
> 
>   druid-handler/pom.xml 6afb5b8 
>   
> druid-handler/src/java/org/apache/hadoop/hive/druid/io/DruidQueryBasedInputFormat.java
>  0b35428 
>   pom.xml 400716f 
>   ql/pom.xml df70a7a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java
>  1d78b4c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSemanticException.java
>  0c6996c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
> 6ccd879 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HivePlannerContext.java
>  9a65de3 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelShuttleImpl.java
>  2aadf50 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveExtractDate.java
>  4edc4df 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterJoinRule.java
>  4b8568e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePointLookupOptimizerRule.java
>  4cfe782 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/HiveMaterializedViewFilterScanRule.java
>  38d7906 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdPredicates.java
>  69e157e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
>  0dc0c24 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
>  27990a2 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ExprNodeConverter.java
>  e840938 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
>  a05b89c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/ReduceSinkDeDuplication.java
>  2b075be 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 79662ec 
>   
> ql/src/test/org/apache/hadoop/hive/ql/optimizer/calcite/TestCBORuleFiredOnlyOnce.java
>  7229cc7 
>   ql/src/test/results/clientnegative/subquery_scalar_multi_rows.q.out 92f1365 
>   ql/src/test/results/clientpositive/cbo_rp_auto_join1.q.out fe4bc4f 
>   ql/src/test/results/clientpositive/cbo_rp_outer_join_ppr.q.out 0bb18cc 
>   ql/src/test/results/clientpositive/constprog2.q.out 7bfd0cf 
>   ql/src/test/results/clientpositive/druid_basic2.q.out 495ebbe 
>   ql/src/test/results/clientpositive/druid_intervals.q.out ca3febf 
>   ql/src/test/results/clientpositive/druid_timeseries.q.out 6b2ffe9 
>   ql/src/test/results/clientpositive/druid_topn.q.out 57d6586 
>   ql/src/test/results/clientpositive/filter_cond_pushdown.q.out 8c6f0f1 
>   ql/src/test/results/clientpositive/fouter_join_ppr.q.out 1897aa7 
>   ql/src/test/results/clientpositive/index_auto_unused.q.out 8a270d6 
>   ql/src/test/results/clientpositive/join45.q.out 249fe9c 
>   ql/src/test/results/clientpositive/join_alt_syntax.q.out cef5272 
>   ql/src/test/results/clientpositive/join_cond_pushdown_1.q.out 9461a1b 
>   ql/src/test/results/clientpositive/join_cond_pushdown_3.q.out 2205c4e 
>   ql/src/test/results/clientpositive/join_cond_pushdown_unqual1.q.out 966330d 
>   ql/src/test/results/clientpositive/join_cond_pushdown_unqual3.q.out 54a2d06 
>   ql/src/test/results/clientpositive/join_merging.q.out 79a7023 
>   ql/src/test/results/clien

[jira] [Created] (HIVE-16420) change llap IO memory report to JSON

2017-04-11 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-16420:
---

 Summary: change llap IO memory report to JSON
 Key: HIVE-16420
 URL: https://issues.apache.org/jira/browse/HIVE-16420
 Project: Hive
  Issue Type: Improvement
Reporter: Sergey Shelukhin
Priority: Minor


see HIVE-16037



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 57944: HIVE-15708 Upgrade Calcite to 1.12 and Avatica to 1.9

2017-04-11 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57944/#review171585
---




ql/src/test/results/clientpositive/druid_intervals.q.out
Line 194 (original), 194-195 (patched)


This is pretty basic optimization which should work. We need to find a 
solution for this.
Either cast for nullability or CALCITE-1738 need to be resolved.



ql/src/test/results/clientpositive/druid_topn.q.out
Line 89 (original), 89 (patched)


Is there a follow-up jira for this to enable this optimizaition via config?



ql/src/test/results/clientpositive/perf/query31.q.out
Line 5 (original), 5 (patched)


This is not expected.


- Ashutosh Chauhan


On April 11, 2017, 3:17 p.m., Remus Rusanu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57944/
> ---
> 
> (Updated April 11, 2017, 3:17 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15708 Upgrade Calcite to 1.12 and Avatica to 1.9
> 
> 
> Diffs
> -
> 
>   druid-handler/pom.xml 6afb5b8 
>   
> druid-handler/src/java/org/apache/hadoop/hive/druid/io/DruidQueryBasedInputFormat.java
>  0b35428 
>   pom.xml 400716f 
>   ql/pom.xml df70a7a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java
>  1d78b4c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSemanticException.java
>  0c6996c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
> 6ccd879 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HivePlannerContext.java
>  9a65de3 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelShuttleImpl.java
>  2aadf50 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveExtractDate.java
>  4edc4df 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterJoinRule.java
>  4b8568e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePointLookupOptimizerRule.java
>  4cfe782 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/HiveMaterializedViewFilterScanRule.java
>  38d7906 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdPredicates.java
>  69e157e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
>  0dc0c24 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
>  27990a2 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ExprNodeConverter.java
>  e840938 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
>  a05b89c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/ReduceSinkDeDuplication.java
>  2b075be 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 79662ec 
>   
> ql/src/test/org/apache/hadoop/hive/ql/optimizer/calcite/TestCBORuleFiredOnlyOnce.java
>  7229cc7 
>   ql/src/test/results/clientnegative/subquery_scalar_multi_rows.q.out 92f1365 
>   ql/src/test/results/clientpositive/cbo_rp_auto_join1.q.out fe4bc4f 
>   ql/src/test/results/clientpositive/cbo_rp_outer_join_ppr.q.out 0bb18cc 
>   ql/src/test/results/clientpositive/constprog2.q.out 7bfd0cf 
>   ql/src/test/results/clientpositive/druid_basic2.q.out 495ebbe 
>   ql/src/test/results/clientpositive/druid_intervals.q.out ca3febf 
>   ql/src/test/results/clientpositive/druid_timeseries.q.out 6b2ffe9 
>   ql/src/test/results/clientpositive/druid_topn.q.out 57d6586 
>   ql/src/test/results/clientpositive/filter_cond_pushdown.q.out 8c6f0f1 
>   ql/src/test/results/clientpositive/fouter_join_ppr.q.out 1897aa7 
>   ql/src/test/results/clientpositive/index_auto_unused.q.out 8a270d6 
>   ql/src/test/results/clientpositive/join45.q.out 249fe9c 
>   ql/src/test/results/clientpositive/join_alt_syntax.q.out cef5272 
>   ql/src/test/results/clientpositive/join_cond_pushdown_1.q.out 9461a1b 
>   ql/src/test/results/clientpositive/join_cond_pushdown_3.q.out 2205c4e 
>   ql/src/test/results/clientpositive/join_cond_pushdown_unqual1.q.out 966330d 
>   ql/src/test/results/clientpositive/join_cond_pushdown_unqual3.q.out 54a2d06 
>   ql/src/test/results/clientpositive/join_merging.q.out 79a7023 
>   ql/src/test/results/clientpositive/llap/auto_smb_mapjoin_14.q.out a867bd2 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out c08a534 
>   ql/src/test/results/clientpositive/llap/constprog_semijoin.q.out 98d2328 
>   ql/s

[jira] [Created] (HIVE-16419) Exclude hadoop related classes for JDBC stabdalone jar

2017-04-11 Thread Tao Li (JIRA) 
Tao Li created HIVE-16419:
-

 Summary: Exclude hadoop related classes for JDBC stabdalone jar
 Key: HIVE-16419
 URL: https://issues.apache.org/jira/browse/HIVE-16419
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Tao Li
Priority: Blocker






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 57944: HIVE-15708 Upgrade Calcite to 1.12 and Avatica to 1.9

2017-04-11 Thread Remus Rusanu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57944/
---

(Updated April 11, 2017, 3:17 p.m.)


Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.


Repository: hive-git


Description
---

HIVE-15708 Upgrade Calcite to 1.12 and Avatica to 1.9


Diffs (updated)
-

  druid-handler/pom.xml 6afb5b8 
  
druid-handler/src/java/org/apache/hadoop/hive/druid/io/DruidQueryBasedInputFormat.java
 0b35428 
  pom.xml 400716f 
  ql/pom.xml df70a7a 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java
 1d78b4c 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSemanticException.java
 0c6996c 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
6ccd879 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HivePlannerContext.java 
9a65de3 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelShuttleImpl.java 
2aadf50 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveExtractDate.java
 4edc4df 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterJoinRule.java
 4b8568e 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePointLookupOptimizerRule.java
 4cfe782 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/HiveMaterializedViewFilterScanRule.java
 38d7906 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdPredicates.java
 69e157e 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
 0dc0c24 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
 27990a2 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ExprNodeConverter.java
 e840938 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
 a05b89c 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/ReduceSinkDeDuplication.java
 2b075be 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 79662ec 
  
ql/src/test/org/apache/hadoop/hive/ql/optimizer/calcite/TestCBORuleFiredOnlyOnce.java
 7229cc7 
  ql/src/test/results/clientnegative/subquery_scalar_multi_rows.q.out 92f1365 
  ql/src/test/results/clientpositive/cbo_rp_auto_join1.q.out fe4bc4f 
  ql/src/test/results/clientpositive/cbo_rp_outer_join_ppr.q.out 0bb18cc 
  ql/src/test/results/clientpositive/constprog2.q.out 7bfd0cf 
  ql/src/test/results/clientpositive/druid_basic2.q.out 495ebbe 
  ql/src/test/results/clientpositive/druid_intervals.q.out ca3febf 
  ql/src/test/results/clientpositive/druid_timeseries.q.out 6b2ffe9 
  ql/src/test/results/clientpositive/druid_topn.q.out 57d6586 
  ql/src/test/results/clientpositive/filter_cond_pushdown.q.out 8c6f0f1 
  ql/src/test/results/clientpositive/fouter_join_ppr.q.out 1897aa7 
  ql/src/test/results/clientpositive/index_auto_unused.q.out 8a270d6 
  ql/src/test/results/clientpositive/join45.q.out 249fe9c 
  ql/src/test/results/clientpositive/join_alt_syntax.q.out cef5272 
  ql/src/test/results/clientpositive/join_cond_pushdown_1.q.out 9461a1b 
  ql/src/test/results/clientpositive/join_cond_pushdown_3.q.out 2205c4e 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual1.q.out 966330d 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual3.q.out 54a2d06 
  ql/src/test/results/clientpositive/join_merging.q.out 79a7023 
  ql/src/test/results/clientpositive/llap/auto_smb_mapjoin_14.q.out a867bd2 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out c08a534 
  ql/src/test/results/clientpositive/llap/constprog_semijoin.q.out 98d2328 
  ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction_2.q.out 
d291e7d 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out a69f2ad 
  ql/src/test/results/clientpositive/llap/explainuser_4.q.out 9804569 
  ql/src/test/results/clientpositive/llap/lineage3.q.out 2f53e60 
  ql/src/test/results/clientpositive/llap/mergejoin.q.out 64929cc 
  ql/src/test/results/clientpositive/llap/orc_ppd_decimal.q.out b5e8658 
  ql/src/test/results/clientpositive/llap/orc_predicate_pushdown.q.out 07ef62f 
  ql/src/test/results/clientpositive/llap/parquet_predicate_pushdown.q.out 
6fa478c 
  ql/src/test/results/clientpositive/llap/reduce_deduplicate_extended.q.out 
08a9bd1 
  ql/src/test/results/clientpositive/llap/smb_mapjoin_14.q.out ddb7624 
  ql/src/test/results/clientpositive/llap/subquery_exists.q.out 9e0a1d7 
  ql/src/test/results/clientpositive/llap/subquery_in.q.out bdf34a9 
  ql/src/test/results/clientpositive/llap/subquery_multi.q.out ca99b87 
  ql/src/test/results/clientpositive/llap/subquery_notin.q.out 2c004fb 
  ql/src/test/results/clientpositive/llap/subquery_scalar.q.out 4cecdd6 
  ql/src/test/results/clientpositive/llap/subquery_select.q.out 945603f 
  ql/src/test/res

[jira] [Created] (HIVE-16418) Allow HiveKey to skip some bytes for comparison

2017-04-11 Thread Rui Li (JIRA) 
Rui Li created HIVE-16418:
-

 Summary: Allow HiveKey to skip some bytes for comparison
 Key: HIVE-16418
 URL: https://issues.apache.org/jira/browse/HIVE-16418
 Project: Hive
  Issue Type: New Feature
Reporter: Rui Li
Assignee: Rui Li


The feature is required when we have to serialize some fields and prevent them 
from being used in comparison, e.g. HIVE-14412.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] hive pull request #169: HIVE-15442:Driver.java has a redundancy code

2017-04-11 Thread txhsj 
GitHub user txhsj opened a pull request:

https://github.com/apache/hive/pull/169

HIVE-15442:Driver.java has a redundancy code

Driver.java has a redundancy code about "explain output", i think the if 
statement " if (conf.getBoolVar(ConfVars.HIVE_LOG_EXPLAIN_OUTPUT))" has a 
repeat judge with the above statement.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/txhsj/hive master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/169.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #169


commit 24f38ceb34c2663197abfdadbd22c73a257e4a98
Author: 黄赛金10202350 
Date:   2017-04-11T10:40:51Z

HIVE-15442:Driver.java has a redundancy code




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Do you feel a need for schema when querying JSON files in hive?

2017-04-11 Thread S G 
Hi,

There is a concept of JsonSerDe where you need to specify a structure for
your tables in order to query them.

However, since the schema for an object is prone to change (once every few
months is not unexpected), how do you handle that change in your hive/pig
queries?

Moreover, since JSON files are not demarcated according to schema, it is
possible that a single JSON file has json-data for multiple evolutions of a
schema (Like 10 objects of ClassAnimal1, 20 of ClassAnimal2, 100 of
ClassAnimal3 etc where ClassAnimal1, ClassAnimal2 and ClassAnimal3
represent schema for ClassAnimal at different times).

For such a JSON file, what is the recommended way of querying?

I know that Avro solves this problem by maintaining a single file for a
single-kind of schema. So it will have 3 files for the above case, 1 each
for ClassAnimal1, ClassAnimal2 and ClassAnimal3)

But since Avro is binary, hard to debug and requires a schema-repository
(for non-hive use-cases), we were hoping to solve this problem in JSON.

Related questions:
1) Is it even a problem worth solving?
2) How many people use AvroSerDe as compared to JsonSerDe?

Thanks
SG