[jira] [Created] (HIVE-20301) Enable vectorization for materialized view rewriting tests

2018-08-02 Thread Jesus Camacho Rodriguez (JIRA) 
Jesus Camacho Rodriguez created HIVE-20301:
--

 Summary: Enable vectorization for materialized view rewriting tests
 Key: HIVE-20301
 URL: https://issues.apache.org/jira/browse/HIVE-20301
 Project: Hive
  Issue Type: Test
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20300) VectorFileSinkArrowOperator

2018-08-02 Thread Eric Wohlstadter (JIRA) 
Eric Wohlstadter created HIVE-20300:
---

 Summary: VectorFileSinkArrowOperator
 Key: HIVE-20300
 URL: https://issues.apache.org/jira/browse/HIVE-20300
 Project: Hive
  Issue Type: Improvement
Reporter: Eric Wohlstadter
Assignee: Eric Wohlstadter


Bypass the row-mode FileSinkOperator for pushing Arrow format to the 
LlapOutputFormatService.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20299) potential race in LLAP signer unit test

2018-08-02 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-20299:
---

 Summary: potential race in LLAP signer unit test
 Key: HIVE-20299
 URL: https://issues.apache.org/jira/browse/HIVE-20299
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] hive pull request #407: Hive18799

2018-08-02 Thread alanfgates 
GitHub user alanfgates opened a pull request:

https://github.com/apache/hive/pull/407

Hive18799

This is based off of Harsha's pull request #393 .  I made the following 
changes:

1. I rebased it against master.  Since the metastore is now split into 
common and server, I put all of the client stuff in a new module 
standalone-metastore/registry-client and the server stuff into 
metastore-server.  Harsha, feedback on whether I put the right things in the 
right places would be helpful.
2. I fixed one thrift request that had multiple arguments rather than a 
struct.

There are no tests in this code.  That's a problem, since this is a lot of 
code.  I assume there are tests for this code in the current registry project.  
Did you intend to move them separately?


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alanfgates/hive hive18799

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/407.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #407


commit e4d43f778209dfe5cf67f2ef21361bf3c204471b
Author: Sriharsha Chintalapani 
Date:   2018-07-07T21:33:58Z

HIVE-18799: Add SchemaRegistry APIs to Hive MetaStore.

commit 6fcfceb0c8216943a00528ef56d5a1734c3885d6
Author: Sriharsha Chintalapani 
Date:   2018-07-10T05:50:11Z

Addressed feedback.

commit eebe0afb08a05343800afd569a3dde482ad0112c
Author: Alan Gates 
Date:   2018-08-01T21:20:33Z

Intermediate checkin.  Moved stuff events and model objects from 
standalone-metastore to standalone-meastore/metastore-server.  I haven't moved 
the registry classes yet.  That's next.  They aren't currently building.

commit f178fa6ba6c50f36a87c3a7ad17e2e77945fa8b7
Author: Alan Gates 
Date:   2018-08-02T00:09:13Z

Moved what I'm guessing to be client code into registry-common.

commit 6b286b98af7ab501d0160062187e0b78ba8182fb
Author: Alan Gates 
Date:   2018-08-02T00:26:38Z

Moved server code into metastore-server

commit e8e552c7978a9c99700c15d17be37bb8618b0ab7
Author: Alan Gates 
Date:   2018-08-02T21:31:43Z

Changed the name to registry-client (from registry-common)

commit a1659f58dbb98b0ce79e366cf761174964b36016
Author: Alan Gates 
Date:   2018-08-02T22:27:47Z

Fixed on thrift request that had multiple arguments rather than a request 
structure.  Fixed a couple of compile errors I had missed in earlier commits.




---

[jira] [Created] (HIVE-20298) Illegal null value in column `TBLS`.`WRITE_ID`

2018-08-02 Thread Jason Dere (JIRA) 
Jason Dere created HIVE-20298:
-

 Summary: Illegal null value in column `TBLS`.`WRITE_ID`
 Key: HIVE-20298
 URL: https://issues.apache.org/jira/browse/HIVE-20298
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Reporter: Jason Dere


Manually upgraded my existing local metastore using 
upgrade-3.0.0-to-3.1.0.mysql.sql, upgrade-3.1.0-to-3.2.0.mysql.sql, 
upgrade-3.2.0-to-4.0.0.mysql.sql. When running DESCRIBE EXTENDED of an existing 
table, I was getting the following error in hive.log. It looks like the 
ObjectStore/MTable classes don't seem to be able to support null values in the 
new writeId column that was added to the TBLS table in the metastore.

cc [~sershe] [~ekoifman]

{noformat}
Caused by: javax.jdo.JDODataStoreException: Illegal null value in column 
`TBLS`.`WRITE_ID`
NestedThrowables:
org.datanucleus.store.rdbms.exceptions.NullValueException: Illegal null value 
in column `TBLS`.`WRITE_ID`
at 
org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:553)
at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:255)
at 
org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1802)
at 
org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1838)
at 
org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:1424)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
at com.sun.proxy.$Proxy39.getTable(Unknown Source)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(HiveMetaStore.java:2950)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:2898)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:2882)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
... 36 more
Caused by: org.datanucleus.store.rdbms.exceptions.NullValueException: Illegal 
null value in column `TBLS`.`WRITE_ID`
at 
org.datanucleus.store.rdbms.mapping.datastore.BigIntRDBMSMapping.getLong(BigIntRDBMSMapping.java:140)
at 
org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getLong(SingleFieldMapping.java:155)
at 
org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchLongField(ResultSetGetter.java:124)
at 
org.datanucleus.state.AbstractStateManager.replacingLongField(AbstractStateManager.java:1549)
at 
org.datanucleus.state.StateManagerImpl.replacingLongField(StateManagerImpl.java:120)
at 
org.apache.hadoop.hive.metastore.model.MTable.dnReplaceField(MTable.java)
at 
org.apache.hadoop.hive.metastore.model.MTable.dnReplaceFields(MTable.java)
at 
org.datanucleus.state.StateManagerImpl.replaceFields(StateManagerImpl.java:3109)
at 
org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchFields(PersistentClassROF.java:465)
at 
org.datanucleus.state.StateManagerImpl.loadFieldValues(StateManagerImpl.java:2238)
at 
org.datanucleus.state.StateManagerImpl.initialiseForHollow(StateManagerImpl.java:263)
at 
org.datanucleus.state.ObjectProviderFactoryImpl.newForHollow(ObjectProviderFactoryImpl.java:112)
at 
org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3097)
at 
org.datanucleus.store.rdbms.query.PersistentClassROF.getObjectForDatastoreId(PersistentClassROF.java:460)
at 
org.datanucleus.store.rdbms.query.PersistentClassROF.getObject(PersistentClassROF.java:385)
at 
org.datanucleus.store.rdbms.query.ForwardQueryResult.nextResultSetElement(ForwardQueryResult.java:188)
at 
org.datanucleus.store.rdbms.query.ForwardQueryResult$QueryResultIterator.next(ForwardQueryResult.java:416)
at 
org.datanucleus.store.rdbms.query.ForwardQueryResult.processNumberOfResults(ForwardQueryResult.java:143)
at 
org.datanucleus.store.rdbms.query.ForwardQueryResult.advanceToEndOfResultSet(ForwardQueryResult.java:171)
at

Re: Review Request 68108: HIVE-19097 related equals and in operators may cause inaccurate stats estimations

2018-08-02 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68108/#review206814
---




ql/src/test/results/clientpositive/cbo_rp_simple_select.q.out
Line 918 (original), 918 (patched)


This could be further simplified. c_int = c_int should become c_int is not 
null can be explored in follow-up since Calcite already has this simplification 
logic.



ql/src/test/results/clientpositive/list_bucket_query_multiskew_2.q.out
Lines 382-384 (original)


Loss of this will be regression. We need to fix this before committing it.



ql/src/test/results/clientpositive/perf/tez/query15.q.out
Line 74 (original), 74 (patched)


In this case LHS is column and all elements in IN list are constants, so 
this should have been folded to IN again?



ql/src/test/results/clientpositive/perf/tez/query53.q.out
Line 133 (original), 133 (patched)


Here LHS of in is column ref and RHS are alll constants, so should have 
folded back to IN.


- Ashutosh Chauhan


On Aug. 2, 2018, 2:16 p.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68108/
> ---
> 
> (Updated Aug. 2, 2018, 2:16 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Gopal V.
> 
> 
> Bugs: HIVE-19097
> https://issues.apache.org/jira/browse/HIVE-19097
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> * open in to or - only column can be on left side
> * close ors into in at 2
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 093b4a73f3 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
> 97e405970f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePointLookupOptimizerRule.java
>  01ad41c497 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
>  f544f58632 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
> fa941a1b25 
>   ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java 
> b57b5ddc2c 
>   ql/src/test/queries/clientpositive/pointlookup.q 1b65cec71c 
>   ql/src/test/queries/clientpositive/pointlookup2.q fe19381368 
>   ql/src/test/queries/clientpositive/pointlookup3.q f98feeb164 
>   ql/src/test/results/clientpositive/alter_partition_coltype.q.out 5d033a3c01 
>   ql/src/test/results/clientpositive/annotate_stats_filter.q.out 54395886d2 
>   ql/src/test/results/clientpositive/annotate_stats_part.q.out bafc6de51e 
>   ql/src/test/results/clientpositive/auto_join19.q.out 3e07ec06de 
>   ql/src/test/results/clientpositive/cbo_rp_simple_select.q.out 2e7d79660b 
>   ql/src/test/results/clientpositive/cbo_simple_select.q.out 33f0e71080 
>   ql/src/test/results/clientpositive/druid_intervals.q.out a5203c3182 
>   ql/src/test/results/clientpositive/dynamic_partition_skip_default.q.out 
> 97922c2636 
>   ql/src/test/results/clientpositive/filter_cond_pushdown.q.out b84a2d4b79 
>   ql/src/test/results/clientpositive/fold_eq_with_case_when.q.out d06fb60345 
>   ql/src/test/results/clientpositive/list_bucket_query_multiskew_2.q.out 
> 98ad3656e7 
>   ql/src/test/results/clientpositive/llap/bucketpruning1.q.out cc637db05b 
>   ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_7.q.out 
> 1330a86426 
>   ql/src/test/results/clientpositive/llap/cbo_simple_select.q.out a35edb42a8 
>   ql/src/test/results/clientpositive/llap/check_constraint.q.out 123a3e46fc 
>   ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out 
> 8f06ee58ce 
>   ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out 
> e03cd3437e 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out 4db83c149d 
>   ql/src/test/results/clientpositive/llap/kryo.q.out 234bae89c7 
>   ql/src/test/results/clientpositive/llap/llap_decimal64_reader.q.out 
> 88ddd9c076 
>   ql/src/test/results/clientpositive/llap/materialized_view_rewrite_ssb.q.out 
> 1841f1f4d3 
>   
> ql/src/test/results/clientpositive/llap/materialized_view_rewrite_ssb_2.q.out 
> d7c92d8c59 
>   ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out 65eec521a2 
>   ql/src/test/results/clientpositive/llap/vector_between_in.q.out 801dda315a 
>   ql/src/test/results/clientpositive/llap/vector_string_decimal.q.out 
> 54d9914caa 
>   
> ql/src/test/results/clientpositive/llap/vector_windowing_multipartitioning.q.out
>  725ed34acb 
>   ql/src/test/results/clientpositive/llap/vector_windowing_navfn.q.out 
> 74ac56d1c6 
>

Re: Review Request 68108: HIVE-19097 related equals and in operators may cause inaccurate stats estimations

2018-08-02 Thread Zoltan Haindrich 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68108/
---

(Updated Aug. 2, 2018, 2:16 p.m.)


Review request for hive, Ashutosh Chauhan and Gopal V.


Changes
---

patch#09


Bugs: HIVE-19097
https://issues.apache.org/jira/browse/HIVE-19097


Repository: hive-git


Description
---

* open in to or - only column can be on left side
* close ors into in at 2


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 093b4a73f3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
97e405970f 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePointLookupOptimizerRule.java
 01ad41c497 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
 f544f58632 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
fa941a1b25 
  ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java 
b57b5ddc2c 
  ql/src/test/queries/clientpositive/pointlookup.q 1b65cec71c 
  ql/src/test/queries/clientpositive/pointlookup2.q fe19381368 
  ql/src/test/queries/clientpositive/pointlookup3.q f98feeb164 
  ql/src/test/results/clientpositive/alter_partition_coltype.q.out 5d033a3c01 
  ql/src/test/results/clientpositive/annotate_stats_filter.q.out 54395886d2 
  ql/src/test/results/clientpositive/annotate_stats_part.q.out bafc6de51e 
  ql/src/test/results/clientpositive/auto_join19.q.out 3e07ec06de 
  ql/src/test/results/clientpositive/cbo_rp_simple_select.q.out 2e7d79660b 
  ql/src/test/results/clientpositive/cbo_simple_select.q.out 33f0e71080 
  ql/src/test/results/clientpositive/druid_intervals.q.out a5203c3182 
  ql/src/test/results/clientpositive/dynamic_partition_skip_default.q.out 
97922c2636 
  ql/src/test/results/clientpositive/filter_cond_pushdown.q.out b84a2d4b79 
  ql/src/test/results/clientpositive/fold_eq_with_case_when.q.out d06fb60345 
  ql/src/test/results/clientpositive/list_bucket_query_multiskew_2.q.out 
98ad3656e7 
  ql/src/test/results/clientpositive/llap/bucketpruning1.q.out cc637db05b 
  ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_7.q.out 
1330a86426 
  ql/src/test/results/clientpositive/llap/cbo_simple_select.q.out a35edb42a8 
  ql/src/test/results/clientpositive/llap/check_constraint.q.out 123a3e46fc 
  ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out 
8f06ee58ce 
  ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out 
e03cd3437e 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 4db83c149d 
  ql/src/test/results/clientpositive/llap/kryo.q.out 234bae89c7 
  ql/src/test/results/clientpositive/llap/llap_decimal64_reader.q.out 
88ddd9c076 
  ql/src/test/results/clientpositive/llap/materialized_view_rewrite_ssb.q.out 
1841f1f4d3 
  ql/src/test/results/clientpositive/llap/materialized_view_rewrite_ssb_2.q.out 
d7c92d8c59 
  ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out 65eec521a2 
  ql/src/test/results/clientpositive/llap/vector_between_in.q.out 801dda315a 
  ql/src/test/results/clientpositive/llap/vector_string_decimal.q.out 
54d9914caa 
  
ql/src/test/results/clientpositive/llap/vector_windowing_multipartitioning.q.out
 725ed34acb 
  ql/src/test/results/clientpositive/llap/vector_windowing_navfn.q.out 
74ac56d1c6 
  ql/src/test/results/clientpositive/llap/vectorized_case.q.out d444ae86a1 
  
ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out
 ba004e9716 
  ql/src/test/results/clientpositive/pcr.q.out 919b71234d 
  ql/src/test/results/clientpositive/perf/spark/query13.q.out fb2a061c63 
  ql/src/test/results/clientpositive/perf/spark/query15.q.out 3d6fbdac77 
  ql/src/test/results/clientpositive/perf/spark/query34.q.out b40081e4f0 
  ql/src/test/results/clientpositive/perf/spark/query48.q.out 60a4767a14 
  ql/src/test/results/clientpositive/perf/spark/query53.q.out 2b1cdfea98 
  ql/src/test/results/clientpositive/perf/spark/query63.q.out b506455dbf 
  ql/src/test/results/clientpositive/perf/spark/query71.q.out bf9c06debf 
  ql/src/test/results/clientpositive/perf/spark/query73.q.out 20ec874e88 
  ql/src/test/results/clientpositive/perf/spark/query85.q.out 572ba54f78 
  ql/src/test/results/clientpositive/perf/spark/query89.q.out 1acc577669 
  ql/src/test/results/clientpositive/perf/spark/query91.q.out de8977da51 
  ql/src/test/results/clientpositive/perf/tez/query13.q.out 5cd4e27de3 
  ql/src/test/results/clientpositive/perf/tez/query15.q.out 3c7ae664b1 
  ql/src/test/results/clientpositive/perf/tez/query34.q.out 9b7b482d3b 
  ql/src/test/results/clientpositive/perf/tez/query48.q.out 1cf8d5c0da 
  ql/src/test/results/clientpositive/perf/tez/query53.q.out 3567534ac4 
  ql/src/test/results/clientpositive/perf/tez/query63.q.out a5b7b5a788

Re: Review Request 68108: HIVE-19097 related equals and in operators may cause inaccurate stats estimations

2018-08-02 Thread Zoltan Haindrich 


> On July 30, 2018, 6:11 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/cbo_simple_select.q.out
> > Line 866 (original), 866 (patched)
> > 
> >
> > This didnt get rewritten into IN. Is that expected?

no, this is a different class of comparision; because there are columns on both 
sides ; I think this way the extraction logic is kinda confused...

note: calcite may handle `x=x` to `true` or `x is not null` simplification.


> On July 30, 2018, 6:11 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/druid_basic3.q.out
> > Line 280 (original), 280 (patched)
> > 
> >
> > No folding of OR into IN ? for druid also, IN is more performant.

actually this is a case of: `UDF(x) IN (c1,c2)` and it's not getting refolded 
because of the udf...
probably later could be done; but for opening INs only those are considered 
which have a column on the left side.


> On July 30, 2018, 6:11 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/llap/vectorized_case.q.out
> > Line 54 (original), 54 (patched)
> > 
> >
> > yeah.. i think thats because now constants are of type integer. Note in 
> > OR clause they had S suffix which made them smallint.
> > 
> > This used to happen because of 
> > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java#L1157
> > 
> > This is during parsing of expresions. We need to enhance this logic now 
> > for INs as well.

updated the logic in typecheckprocfactory


> On July 30, 2018, 6:11 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/perf/tez/query15.q.out
> > Line 74 (original), 74 (patched)
> > 
> >
> > No folding back to IN ?

new patch will not `open` INs like this... but that's still not enough to 
refold: `_col3 IN ('CA','WA','GA')`; some work needs to be HIVE-20296


> On July 30, 2018, 6:11 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/perf/tez/query45.q.out
> > Line 81 (original), 81 (patched)
> > 
> >
> > No folding back to IN?

fixed in new patch


> On July 30, 2018, 6:11 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/perf/tez/query63.q.out
> > Line 135 (original), 135 (patched)
> > 
> >
> > No folding back to IN ?

this needed a little tweak in hivepointlookupoptimizer; now it notices some 
redundancies in this condition! :)


> On July 30, 2018, 6:11 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/perf/tez/query8.q.out
> > Line 337 (original), 337 (patched)
> > 
> >
> > No folding back to IN ?

this is not expanded anymore...since it can't be closed back right now;
rule is to only open if left side is a column.


> On July 30, 2018, 6:11 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/vector_non_constant_in_expr.q.out
> > Line 22 (original), 22 (patched)
> > 
> >
> > these ORs didnt get folded in IN, expected?

there are columns on both sides - probably later will be taken care of...


- Zoltan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68108/#review206603
---


On Aug. 2, 2018, 11:13 a.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68108/
> ---
> 
> (Updated Aug. 2, 2018, 11:13 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Gopal V.
> 
> 
> Bugs: HIVE-19097
> https://issues.apache.org/jira/browse/HIVE-19097
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> * open in to or - only column can be on left side
> * close ors into in at 2
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 093b4a73f3 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
> 97e405970f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
>  f544f58632 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
> fa941a1b25 
>   ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java 
> b57b5ddc2c 
>

[jira] [Created] (HIVE-20297) Column Level Stats for Druid Tables

2018-08-02 Thread Nishant Bangarwa (JIRA) 
Nishant Bangarwa created HIVE-20297:
---

 Summary: Column Level Stats for Druid Tables
 Key: HIVE-20297
 URL: https://issues.apache.org/jira/browse/HIVE-20297
 Project: Hive
  Issue Type: Improvement
Reporter: Nishant Bangarwa
Assignee: Nishant Bangarwa


This task is to have correct column level stats for druid in hive metastore. 
- Stats like min/max/cardinality can be gathered using a Druid Segment Metadata 
Query. 
- Druid Query planning we need to ensure that the filters/Aggregations pushed 
inside DruidQuery are accounted for.

Having correct stats would also help optimizer ensure proper join orderings 
when doing federated complex joins between hive/druid. 




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68108: HIVE-19097 related equals and in operators may cause inaccurate stats estimations

2018-08-02 Thread Zoltan Haindrich 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68108/
---

(Updated Aug. 2, 2018, 11:13 a.m.)


Review request for hive, Ashutosh Chauhan and Gopal V.


Changes
---

patch#06


Bugs: HIVE-19097
https://issues.apache.org/jira/browse/HIVE-19097


Repository: hive-git


Description (updated)
---

* open in to or - only column can be on left side
* close ors into in at 2


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 093b4a73f3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
97e405970f 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
 f544f58632 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
fa941a1b25 
  ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java 
b57b5ddc2c 
  ql/src/test/queries/clientpositive/pointlookup.q 1b65cec71c 
  ql/src/test/queries/clientpositive/pointlookup2.q fe19381368 
  ql/src/test/queries/clientpositive/pointlookup3.q f98feeb164 
  ql/src/test/queries/clientpositive/stat_estimate_related_col.q 52da2f759a 
  ql/src/test/results/clientpositive/alter_partition_coltype.q.out 5d033a3c01 
  ql/src/test/results/clientpositive/annotate_stats_filter.q.out 54395886d2 
  ql/src/test/results/clientpositive/annotate_stats_part.q.out bafc6de51e 
  ql/src/test/results/clientpositive/auto_join19.q.out 3e07ec06de 
  ql/src/test/results/clientpositive/cbo_rp_simple_select.q.out 2e7d79660b 
  ql/src/test/results/clientpositive/cbo_simple_select.q.out 33f0e71080 
  ql/src/test/results/clientpositive/druid_intervals.q.out a5203c3182 
  ql/src/test/results/clientpositive/dynamic_partition_skip_default.q.out 
97922c2636 
  ql/src/test/results/clientpositive/filter_cond_pushdown.q.out b84a2d4b79 
  ql/src/test/results/clientpositive/fold_eq_with_case_when.q.out d06fb60345 
  ql/src/test/results/clientpositive/list_bucket_query_multiskew_2.q.out 
98ad3656e7 
  ql/src/test/results/clientpositive/llap/bucketpruning1.q.out cc637db05b 
  ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_7.q.out 
c7f5b887b6 
  ql/src/test/results/clientpositive/llap/cbo_simple_select.q.out a35edb42a8 
  ql/src/test/results/clientpositive/llap/check_constraint.q.out 123a3e46fc 
  ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out 
8f06ee58ce 
  ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out 
e03cd3437e 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 708fa17617 
  ql/src/test/results/clientpositive/llap/kryo.q.out 234bae89c7 
  ql/src/test/results/clientpositive/llap/llap_decimal64_reader.q.out 
88ddd9c076 
  ql/src/test/results/clientpositive/llap/materialized_view_rewrite_ssb.q.out 
1841f1f4d3 
  ql/src/test/results/clientpositive/llap/materialized_view_rewrite_ssb_2.q.out 
d7c92d8c59 
  ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out 65eec521a2 
  ql/src/test/results/clientpositive/llap/vector_between_in.q.out 801dda315a 
  ql/src/test/results/clientpositive/llap/vector_string_decimal.q.out 
54d9914caa 
  
ql/src/test/results/clientpositive/llap/vector_windowing_multipartitioning.q.out
 725ed34acb 
  ql/src/test/results/clientpositive/llap/vector_windowing_navfn.q.out 
74ac56d1c6 
  ql/src/test/results/clientpositive/llap/vectorized_case.q.out d444ae86a1 
  
ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out
 ba004e9716 
  ql/src/test/results/clientpositive/pcr.q.out 919b71234d 
  ql/src/test/results/clientpositive/perf/spark/query13.q.out fb2a061c63 
  ql/src/test/results/clientpositive/perf/spark/query15.q.out 3d6fbdac77 
  ql/src/test/results/clientpositive/perf/spark/query34.q.out b40081e4f0 
  ql/src/test/results/clientpositive/perf/spark/query48.q.out 60a4767a14 
  ql/src/test/results/clientpositive/perf/spark/query53.q.out 2b1cdfea98 
  ql/src/test/results/clientpositive/perf/spark/query63.q.out b506455dbf 
  ql/src/test/results/clientpositive/perf/spark/query71.q.out bf9c06debf 
  ql/src/test/results/clientpositive/perf/spark/query73.q.out 20ec874e88 
  ql/src/test/results/clientpositive/perf/spark/query85.q.out 572ba54f78 
  ql/src/test/results/clientpositive/perf/spark/query89.q.out 1acc577669 
  ql/src/test/results/clientpositive/perf/spark/query91.q.out de8977da51 
  ql/src/test/results/clientpositive/perf/tez/query13.q.out 5cd4e27de3 
  ql/src/test/results/clientpositive/perf/tez/query15.q.out 3c7ae664b1 
  ql/src/test/results/clientpositive/perf/tez/query34.q.out 9b7b482d3b 
  ql/src/test/results/clientpositive/perf/tez/query48.q.out 1cf8d5c0da 
  ql/src/test/results/clientpositive/perf/tez/query53.q.out 3567534ac4 
  ql/src/test/results/clientpositive/perf/tez/query63.q.out a5b7b5a788 
  ql/src/test/results/clientpositive/perf/tez/query71.q.out 4521aabc9f

[jira] [Created] (HIVE-20296) Improve HivePointLookupOptimizerRule to be able to extract from more sophisticated contexts

2018-08-02 Thread Zoltan Haindrich (JIRA) 
Zoltan Haindrich created HIVE-20296:
---

 Summary: Improve HivePointLookupOptimizerRule to be able to 
extract from more sophisticated contexts
 Key: HIVE-20296
 URL: https://issues.apache.org/jira/browse/HIVE-20296
 Project: Hive
  Issue Type: Improvement
  Components: Logical Optimizer
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


Currently it could handle conditions in the form:
{code}
( ( a=1 && b=1 && ... ) || (a=2 && b=2 && ...) || ... )
{code}

If there is some noise somewhere in the or like: 
{code}
( x=18 || ( a=1 && b=1 && ... ) || (a=2 && b=2 && ...) || ... )
{code}
it stops processing.

https://github.com/apache/hive/blob/2cabb8da150b8fb980223fbd6c2c93b842ca3ee5/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePointLookupOptimizerRule.java#L238

HIVE-19097 right now unwinds some ors which can't be packed back; like query15
https://github.com/apache/hive/blob/2cabb8da150b8fb980223fbd6c2c93b842ca3ee5/ql/src/test/queries/clientpositive/perf/query15.q#L14




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20295) Remove !isNumber check after failed constant interpretation

2018-08-02 Thread Zoltan Haindrich (JIRA) 
Zoltan Haindrich created HIVE-20295:
---

 Summary: Remove !isNumber check after failed constant 
interpretation
 Key: HIVE-20295
 URL: https://issues.apache.org/jira/browse/HIVE-20295
 Project: Hive
  Issue Type: Improvement
Reporter: Zoltan Haindrich


During constant interpretation; if the number can't be parsed - it might be 
possible that the comparsion is out of range for the type in question - in 
which case it could be removed.

https://github.com/apache/hive/blob/2cabb8da150b8fb980223fbd6c2c93b842ca3ee5/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java#L1163



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20294) Vectorization: Fix NULL / Wrong Results issues in COALESCE / ELT

2018-08-02 Thread Matt McCline (JIRA) 
Matt McCline created HIVE-20294:
---

 Summary: Vectorization: Fix NULL / Wrong Results issues in 
COALESCE / ELT
 Key: HIVE-20294
 URL: https://issues.apache.org/jira/browse/HIVE-20294
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Matt McCline
Assignee: Matt McCline


Write new UT tests that use random data and intentional isRepeating batches to 
checks for NULL and Wrong Results for vectorized COALESCE and ELT.

Also, add tests for ARRAY and MAP indexing, IS [NOT] NULL and NOT



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20293) Support Replication of ACID table truncate operation

2018-08-02 Thread mahesh kumar behera (JIRA) 
mahesh kumar behera created HIVE-20293:
--

 Summary: Support Replication of ACID table truncate operation
 Key: HIVE-20293
 URL: https://issues.apache.org/jira/browse/HIVE-20293
 Project: Hive
  Issue Type: Task
  Components: repl
Affects Versions: 3.1.0, 4.0.0
Reporter: mahesh kumar behera
Assignee: mahesh kumar behera
 Fix For: 4.0.0, 3.2.0


Add tags in jobconf for distcp related jobs started by replication. This will 
allow hive to kill these jobs in case beacon retries, or hs2 dies and beacon 
issues a kill command.
 * one of the tags should definitely be the query_id that starts the job : With 
this flow beacon before retrying the bootstrap load, will issue a kill command 
to hs2 with the query id of the previous issued command. hs2 will then kill an 
running jobs on yarn tagged with the Query_id.

 * To get around the additional failure point as mentioned above. The jobs can 
be tagged with an additional unique tag_id provided by Beacon in the WITH 
clause in repl load command to be used to tag distcp jobs ). Enhance the kill 
api to take the tag as input and kill jobs associated with that tag. Problem 
here is how do we validate the association of the tag with a hive query id to 
make sure this api is not used to kill jobs run by other components, however we 
can provide this capability to only admins and should be ok in that case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)