[jira] [Created] (HIVE-20864) No running LLAP daemons! Please check LLAP service status and zookeeper configuration

2018-11-02 Thread zhangbutao (JIRA) 
zhangbutao created HIVE-20864:
-

 Summary: No running LLAP daemons! Please check LLAP service status 
and zookeeper configuration
 Key: HIVE-20864
 URL: https://issues.apache.org/jira/browse/HIVE-20864
 Project: Hive
  Issue Type: Bug
  Components: llap
Affects Versions: 3.1.0
Reporter: zhangbutao


I test hive3.1.0 using TPCDS. Jobs would be falied after executing some sql, 
and i have checked that LLAP daemons were still alive as well as LLAP zknodes 
still existed in zookeeper. The error log as follows:
{code:java}
]Vertex failed, vertexName=Map 6, vertexId=vertex_1541034860050_0181_194_04, 
diagnostics=[Vertex vertex_1541034860050_0181_194_04 [Map 6] killed/failed due 
to:INIT_FAILURE, Fail to create InputInitializerManager, 
org.apache.tez.dag.api.TezReflectionException: Unable to instantiate class with 
1 arguments: org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator
at org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:71)
at 
org.apache.tez.common.ReflectionUtils.createClazzInstance(ReflectionUtils.java:89)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$1.run(RootInputInitializerManager.java:152)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$1.run(RootInputInitializerManager.java:148)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager.createInitializer(RootInputInitializerManager.java:148)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager.runInputInitializers(RootInputInitializerManager.java:121)
at 
org.apache.tez.dag.app.dag.impl.VertexImpl.setupInputInitializerManager(VertexImpl.java:4101)
at org.apache.tez.dag.app.dag.impl.VertexImpl.access$3100(VertexImpl.java:205)
at 
org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.handleInitEvent(VertexImpl.java:2912)
at 
org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2859)
at 
org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2841)
at 
org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
at 
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
at 
org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
at 
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
at org.apache.tez.state.StateMachineTez.doTransition(StateMachineTez.java:59)
at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:1939)
at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:204)
at 
org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2317)
at 
org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2303)
at org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:180)
at org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:115)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedConstructorAccessor45.newInstance(Unknown Source)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:68)
... 25 more
Caused by: java.lang.IllegalArgumentException: No running LLAP daemons! Please 
check LLAP service status and zookeeper configuration
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
at 
org.apache.hadoop.hive.ql.exec.tez.Utils.getSplitLocationProvider(Utils.java:57)
at 
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.(HiveSplitGenerator.java:140)
... 29 more
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20861) Pass queryId as the client CallerContext to Spark

2018-11-02 Thread Aihua Xu (JIRA) 
Aihua Xu created HIVE-20861:
---

 Summary: Pass queryId as the client CallerContext to Spark 
 Key: HIVE-20861
 URL: https://issues.apache.org/jira/browse/HIVE-20861
 Project: Hive
  Issue Type: Improvement
Reporter: Aihua Xu
Assignee: Aihua Xu


SPARK-16759 exposes a way for the client to pass the client CallerContext such 
as QueryId. For better debug, hive should pass queryId to spark.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20860) Fix or disable TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit]

2018-11-02 Thread Vihang Karajgaonkar (JIRA) 
Vihang Karajgaonkar created HIVE-20860:
--

 Summary: Fix or disable 
TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit]
 Key: HIVE-20860
 URL: https://issues.apache.org/jira/browse/HIVE-20860
 Project: Hive
  Issue Type: Test
Reporter: Vihang Karajgaonkar


Test failed in one of the precommit job. Looks like there is some case where 
there is additonal space in the diff

{noformat}
Error Message
Client Execution succeeded but contained differences (error code = 1) after 
executing cbo_limit.q 
11c11
<  1  4 2
---
>  1 4 2
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69019: HIVE-20617 Fix type of constants in IN expressions to have correct type

2018-11-02 Thread Jesús Camacho Rodríguez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69019/#review210298
---


Ship it!




Ship It!

- Jesús Camacho Rodríguez


On Nov. 2, 2018, 9:10 p.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69019/
> ---
> 
> (Updated Nov. 2, 2018, 9:10 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Jesús Camacho Rodríguez, and 
> Vineet Garg.
> 
> 
> Bugs: HIVE-20617
> https://issues.apache.org/jira/browse/HIVE-20617
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> For IN expressions the types were never corrected; and pointlookupoptimizer 
> was probably leaving behind fields already which were uncomparable; 
> HIVE-20296 exposed it further by changing the minimal number from  32 to 2.
> 
> This change generalizes the retyping of constants to also run it for the IN 
> operator ; and also for struct-s.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/type/HiveChar.java 
> 29dc06dca1a 
>   ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java e7d71595c7a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePointLookupOptimizerRule.java
>  04800cca91b 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 
> c9df668d4a7 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 82e975a50de 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckCtx.java 1b56ecd0440 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
> 4968d16876c 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactoryUtils.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java 
> c274fd7cc9b 
>   ql/src/test/queries/clientnegative/udf_in_2.q 2288054d583 
>   ql/src/test/queries/clientpositive/groupby_multi_single_reducer3.q 
> 94aea35c422 
>   ql/src/test/queries/clientpositive/in_typecheck_char.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/in_typecheck_mixed.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/in_typecheck_pointlook.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/in_typecheck_varchar.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/pcs.q 2e7eff472b5 
>   ql/src/test/queries/clientpositive/vector_struct_in.q 49a9374bd13 
>   ql/src/test/results/clientnegative/udf_in_2.q.out 83883e3f9e3 
>   ql/src/test/results/clientpositive/alter_partition_coltype.q.out 
> f6c3c5642ea 
>   ql/src/test/results/clientpositive/groupby_multi_single_reducer3.q.out 
> a8b595a6590 
>   ql/src/test/results/clientpositive/in_typecheck_char.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/in_typecheck_mixed.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/in_typecheck_pointlook.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/in_typecheck_varchar.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/infer_const_type.q.out e1d7de54225 
>   ql/src/test/results/clientpositive/join45.q.out 47aaf7d0abc 
>   ql/src/test/results/clientpositive/join45X.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/join47.q.out 4d9e937815e 
>   ql/src/test/results/clientpositive/llap/acid_no_buckets.q.out 5bd3d90249b 
>   ql/src/test/results/clientpositive/llap/bucketpruning1.q.out 9422eb49781 
>   ql/src/test/results/clientpositive/llap/check_constraint.q.out 9b1f8def6eb 
>   ql/src/test/results/clientpositive/llap/dec_str.q.out 554031e952e 
>   
> ql/src/test/results/clientpositive/llap/dynpart_sort_optimization_acid.q.out 
> 40dc5e97e5a 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out f240468558e 
>   ql/src/test/results/clientpositive/llap/is_distinct_from.q.out f5a34750ce3 
>   ql/src/test/results/clientpositive/llap/kryo.q.out 8121c119cc7 
>   ql/src/test/results/clientpositive/llap/lineage3.q.out 9bec309c9c9 
>   ql/src/test/results/clientpositive/llap/multi_column_in.q.out 414be226e6a 
>   ql/src/test/results/clientpositive/llap/multi_column_in_single.q.out 
> db9c676a814 
>   ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out 3b163794b60 
>   ql/src/test/results/clientpositive/llap/orc_predicate_pushdown.q.out 
> 21180a3da8c 
>   ql/src/test/results/clientpositive/llap/parquet_predicate_pushdown.q.out 
> 8330e93115c 
>   ql/src/test/results/clientpositive/llap/vector_between_in.q.out e9ea4611acc 
>   ql/src/test/results/clientpositive/llap/vector_struct_in.q.out 4dfa80e9843 
>   ql/src/test/results/clientpositive/llap/vectorization_13.q.out 4ce654f960b 
>   ql/src/test/results/clientpositive/llap/vectorization_6.q.out a2f730beca2 
>   ql/src/test/results/clientpositive/llap/vectorization_8.q.out 21ce7b8ebd0 
>

[GitHub] hive pull request #484: HIVE-16839: Fix a race condidtion during concurrent ...

2018-11-02 Thread guangyy 
GitHub user guangyy opened a pull request:

https://github.com/apache/hive/pull/484

HIVE-16839: Fix a race condidtion during concurrent partition drops

We have seen a leaked lock on hive metastore DB which caused all
PARTITION insertion failed on timeout waiting for lock until the
metastore service is restarted.

A transaction dump on the DB shows there is a thread that is Sleep which
potentiall holds the the lock, like:
```
  trx_id: 33603171058
 trx_state: RUNNING
   trx_started: 2018-10-23 06:43:22
 trx_requested_lock_id: NULL
  trx_wait_started: NULL
trx_weight: 70298
   trx_mysql_thread_id: 275402202
 trx_query: NULL
   trx_operation_state: NULL
 trx_tables_in_use: 0
 trx_tables_locked: 0
  trx_lock_structs: 21286
 trx_lock_memory_bytes: 2881064
   trx_rows_locked: 98810
 trx_rows_modified: 49012
   trx_concurrency_tickets: 0
   trx_isolation_level: READ COMMITTED
 trx_unique_checks: 1
trx_foreign_key_checks: 1
trx_last_foreign_key_error: NULL
 trx_adaptive_hash_latched: 0
 trx_adaptive_hash_timeout: 0
  trx_is_read_only: 0
trx_autocommit_non_locking: 0
ID: 275402202
  USER: metastore_gold
  HOST: 10.37.182.82:36684
DB: metastoregold
   COMMAND: Sleep
  TIME: 1
 STATE:
  INFO: NULL
  duration: 1316
Given the HOST ip, we trace back to the hive metastore instance and found 
the following exceptions:

No such database row
org.datanucleus.exceptions.NucleusObjectNotFoundException: No such database 
row
at 
org.datanucleus.store.rdbms.request.FetchRequest.execute(FetchRequest.java:357)
at 
org.datanucleus.store.rdbms.RDBMSPersistenceHandler.fetchObject(RDBMSPersistenceHandler.java:324)
at 
org.datanucleus.state.AbstractStateManager.loadFieldsFromDatastore(AbstractStateManager.java:1120)
at 
org.datanucleus.state.JDOStateManager.loadSpecifiedFields(JDOStateManager.java:2916)
at 
org.datanucleus.state.JDOStateManager.isLoaded(JDOStateManager.java:3219)
```
The problem is that the caller expects a NULL if the partition does not 
exist, however, the convertToPart function would throw
an exception which lead to the leak.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/guangyy/hive HIVE-16839

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/484.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #484


commit 5137027ee658990dd1503c09c13a73e2848d8deb
Author: Guang Yang 
Date:   2018-11-02T23:21:35Z

HIVE-16839: Fix a race condidtion during concurrent partition drops




---

[jira] [Created] (HIVE-20862) QueryId no longer shows up in the logs

2018-11-02 Thread Eugene Koifman (JIRA) 
Eugene Koifman created HIVE-20862:
-

 Summary: QueryId no longer shows up in the logs
 Key: HIVE-20862
 URL: https://issues.apache.org/jira/browse/HIVE-20862
 Project: Hive
  Issue Type: Bug
  Components: Logging
Affects Versions: 4.0.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20863) remove dead code

2018-11-02 Thread Eugene Koifman (JIRA) 
Eugene Koifman created HIVE-20863:
-

 Summary: remove dead code
 Key: HIVE-20863
 URL: https://issues.apache.org/jira/browse/HIVE-20863
 Project: Hive
  Issue Type: Improvement
  Components: Transactions
Affects Versions: 3.0.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: External vs. Managed Tables - Current state

2018-11-02 Thread Lars Francke 
Hi again,

I finally got around to creating a Wiki page: <
https://cwiki.apache.org/confluence/display/Hive/Managed+vs.+External+Tables
>

@Sergey: What are "SH tables"?

Any more additions to the content are welcome. I'd be happy to put them in
the Wiki too. All I need are a few cues/keywords.

Thanks!
Lars

On Thu, Aug 23, 2018 at 12:22 AM Lars Francke 
wrote:

> Thanks for the input here and on the issue itself.
>
> @Sergey: That is a good summary. I would try to distill that into a Wiki
> page.
>
>  I still think that it'd be good to list the actual technical differences.
>
> 1) Because this is not actually written down in a concise way so lots of
> our users don't know this distinction. I for example have heard all kinds
> of "rumors"
>
> 2) I've been using Hive for 8+ years or so and I use it almost weekly and
> follow the dev list and Jira issues and I still missed about 80% of the
> features that have been implemented in the last few years affecting this.
> And these things can change a decision. Back when the only difference was
> whether a DROP would delete data it was much simpler. Now you might opt for
> an internal table even if it does not match your ETL flows etc. because it
> gives you better performance or some other feature.
>
> 3) It's a pity to have all these features not being used because no one
> outside of a small circle knows about them :)
>
> So I'd still appreciate any and all input you might have for me.
>
> Cheers,
> Lars
>
>
> On Wed, Aug 22, 2018 at 9:45 PM, Sergey Shelukhin 
> wrote:
>
>> I think it’s very simple conceptually (well the way I’ve been thinking
>> about it), and any limitations can be derived from this using logic. If
>> someone has specific questions we can clarify; if some limitations are
>> missing (e.g. when Hive was using stats for hbase tables) it is a bug.
>>
>> For managed tables, Hive owns and manages the data:
>>
>> * Data, properties, etc. can only be changed via Hive commands or APIs;
>> Hive assumes it’s true and any other modification on the underlying
>> storage is undefined behavior (not just results - basically all bets are
>> off ;)).
>> * Data is attached to the Hive entity (partition, table). Whatever happens
>> to the entity happens to the data (e.g. the behavior mentioned in the
>> wiki, where  dropping the table/partition deletes the data).
>>
>>
>> For external tables (also SH tables before Hive made those explicitly
>> external), Hive doesn’t manage or own the data:
>>
>> * Hive assumes the data can be modified externally at any time. Any
>> feature that requires assumptions about data being unchanged (queries from
>> stats, ACID, some MV, query cache) won’t work. If it works, it’s a bug.
>> * Data is detached from Hive entities, e.g. dropping the table by default
>> doesn’t delete the data - we assume someone else owns it and we are just
>> pointing at it.
>> * Additionally, since Hive cannot track modifications, external
>> products/users/etc. are fully responsible for any concurrency control. In
>> particular, modifying data under the running query may produce
>> inconsistent results (no atomicity between tasks), undefined results due
>> to retries, etc.
>>
>>
>>
>>
>>
>> On 18/8/21, 10:31, "Mithun RK"  wrote:
>>
>> >+1. I'm out of the loop as well. I'd be keen on grokking what's not
>> >available with EXTERNAL tables, given that these are popular on our
>> >clusters as well. It would be good to have that documented in one place.
>> >
>> >On Tue, Aug 21, 2018 at 8:19 AM Lars Francke 
>> >wrote:
>> >
>> >> As this came up with yet another customer and I've heard more things
>> >>that I
>> >> believe to be false:
>> >>
>> >> Please, anyone have any kind of feedback for me here? Is my list
>> >>correct,
>> >> can you add stuff to it, have more details to certain points etc.?
>> >>
>> >> Thank you!
>> >>
>> >> On Wed, Aug 1, 2018 at 12:17 PM, Lars Francke 
>> >> wrote:
>> >>
>> >> > I have opened https://issues.apache.org/jira/browse/HIVE-20287 and
>> >>would
>> >> > love to get feedback
>> >> >
>> >> > On Wed, Aug 1, 2018 at 11:14 AM, Lars Francke <
>> lars.fran...@gmail.com>
>> >> > wrote:
>> >> >
>> >> >> Thanks Lefty! I haven't created a JIRA but it's on my list.
>> >> >>
>> >> >> Still interested in any input anyone might have?
>> >> >>
>> >> >> Some of you guys _must_ have some knowledge on this stuff.
>> >> >>
>> >> >> You're correct that the DROP stuff is documented. What I meant is
>> >>that I
>> >> >> couldn't find the relevant parts in the code.
>> >> >>
>> >> >> On Sat, Jul 21, 2018 at 6:15 AM, Lefty Leverenz <
>> >> leftylever...@gmail.com>
>> >> >> wrote:
>> >> >>
>> >> >>> Agreed, the Hive wiki is woefully incomplete.  But it does mention
>> >>DROP
>> >> >>> for
>> >> >>> external tables in two sections of the DDL doc:
>> >> >>>
>> >> >>>- External Tables
>> >> >>>> >> >>> anual+DDL#LanguageManualDDL-ExternalTables>:
>> >> >>> "When dropping an

Re: Review Request 69019: HIVE-20617 Fix type of constants in IN expressions to have correct type

2018-11-02 Thread Zoltan Haindrich 


> On Nov. 1, 2018, 10:37 p.m., Jesús Camacho Rodríguez wrote:
> > common/src/java/org/apache/hadoop/hive/common/type/HiveChar.java
> > Lines 37 (patched)
> > 
> >
> > Could we avoid adding this new constructor and let other logic in Hive 
> > handle this case?

I've inlined it


> On Nov. 1, 2018, 10:37 p.m., Jesús Camacho Rodríguez wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePointLookupOptimizerRule.java
> > Lines 142 (patched)
> > 
> >
> > You can move the project copy into the _copyNode_ method so we do not 
> > have (ugly) return null over there.

I was considering that; but copyNode accepts only `RexNode newCondition`; but 
projections have an array of them...

I went the other way around...since the "copyNode" is not really the needed 
serivce of the pointlookup rule writing - more like a benefit gained from 
similarity between the earlier 2 rules.
I've inlined that and removed the `analyzeCondition` method; since it was 
already unused for the projection path...

I think these things look better now.


> On Nov. 1, 2018, 10:37 p.m., Jesús Camacho Rodríguez wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePointLookupOptimizerRule.java
> > Line 148 (original), 174 (patched)
> > 
> >
> > Can change the numbering, i.e., 3.

I've removed the numbering.

I think merge IN expression is right now a no-op...so we might just remove that 
some day


> On Nov. 1, 2018, 10:37 p.m., Jesús Camacho Rodríguez wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePointLookupOptimizerRule.java
> > Line 258 (original), 345 (patched)
> > 
> >
> > Lambda?

tried it; looks good at first - but the compiler keeps complaining about that 
it's not initialized


> On Nov. 1, 2018, 10:37 p.m., Jesús Camacho Rodríguez wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePointLookupOptimizerRule.java
> > Lines 459 (patched)
> > 
> >
> > This is called only once, you could just make the transformation 
> > in-place instead of creating new method.

yes; this was left here by a refactoring step; inlined/fixed/renamed


> On Nov. 1, 2018, 10:37 p.m., Jesús Camacho Rodríguez wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java
> > Lines 1206 (patched)
> > 
> >
> > Since we transform IN into OR when we are not in CBO, do we still need 
> > this special handling for nulls?

yes; there are some cases in which we are not opening up the IN; right now I 
have to add to not open it in case the left hand side is an array or a list


> On Nov. 1, 2018, 10:37 p.m., Jesús Camacho Rodríguez wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java
> > Lines 1372 (patched)
> > 
> >
> > Is there a _isConstantOrColumn1_? What is the difference?

not anymore :)


> On Nov. 1, 2018, 10:37 p.m., Jesús Camacho Rodríguez wrote:
> > ql/src/test/results/clientpositive/join47.q.out
> > Line 709 (original), 709 (patched)
> > 
> >
> > Shouldn't we be able to close this one into an IN clause after latest 
> > in HivePointLookupOptimizer?

its closed; the q.out was probably outdated


> On Nov. 1, 2018, 10:37 p.m., Jesús Camacho Rodríguez wrote:
> > ql/src/test/results/clientpositive/mapjoin47.q.out
> > Line 751 (original), 751 (patched)
> > 
> >
> > same as with join47.q

same here; and now it's not comparing string to ints anymore...

as a second taught: ...I'm afraid that the previous expression was never 
returning correct results(col0 and col2 are string columns) - but the new one 
looks like this:

```
{(struct(UDFToDouble(_col0),UDFToDouble(_col2))) IN (const 
struct(100.0D,100.0D), const struct(101.0D,101.0D), const 
struct(102.0D,102.0D))}
```


- Zoltan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69019/#review210259
---


On Oct. 30, 2018, 11:32 p.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69019/
> ---
> 
> (Updated Oct. 30, 2018,

Re: Review Request 69019: HIVE-20617 Fix type of constants in IN expressions to have correct type

2018-11-02 Thread Zoltan Haindrich 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69019/
---

(Updated Nov. 2, 2018, 10:10 p.m.)


Review request for hive, Ashutosh Chauhan, Jesús Camacho Rodríguez, and Vineet 
Garg.


Changes
---

patch#18

* be more safe; and not open INs when comparision of list/array would have 
happen
* address review comments
* actualize q.out-s


Bugs: HIVE-20617
https://issues.apache.org/jira/browse/HIVE-20617


Repository: hive-git


Description
---

For IN expressions the types were never corrected; and pointlookupoptimizer was 
probably leaving behind fields already which were uncomparable; HIVE-20296 
exposed it further by changing the minimal number from  32 to 2.

This change generalizes the retyping of constants to also run it for the IN 
operator ; and also for struct-s.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/type/HiveChar.java 29dc06dca1a 
  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java e7d71595c7a 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePointLookupOptimizerRule.java
 04800cca91b 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 
c9df668d4a7 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 82e975a50de 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckCtx.java 1b56ecd0440 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
4968d16876c 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactoryUtils.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java c274fd7cc9b 
  ql/src/test/queries/clientnegative/udf_in_2.q 2288054d583 
  ql/src/test/queries/clientpositive/groupby_multi_single_reducer3.q 
94aea35c422 
  ql/src/test/queries/clientpositive/in_typecheck_char.q PRE-CREATION 
  ql/src/test/queries/clientpositive/in_typecheck_mixed.q PRE-CREATION 
  ql/src/test/queries/clientpositive/in_typecheck_pointlook.q PRE-CREATION 
  ql/src/test/queries/clientpositive/in_typecheck_varchar.q PRE-CREATION 
  ql/src/test/queries/clientpositive/pcs.q 2e7eff472b5 
  ql/src/test/queries/clientpositive/vector_struct_in.q 49a9374bd13 
  ql/src/test/results/clientnegative/udf_in_2.q.out 83883e3f9e3 
  ql/src/test/results/clientpositive/alter_partition_coltype.q.out f6c3c5642ea 
  ql/src/test/results/clientpositive/groupby_multi_single_reducer3.q.out 
a8b595a6590 
  ql/src/test/results/clientpositive/in_typecheck_char.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/in_typecheck_mixed.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/in_typecheck_pointlook.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/in_typecheck_varchar.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/infer_const_type.q.out e1d7de54225 
  ql/src/test/results/clientpositive/join45.q.out 47aaf7d0abc 
  ql/src/test/results/clientpositive/join45X.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/join47.q.out 4d9e937815e 
  ql/src/test/results/clientpositive/llap/acid_no_buckets.q.out 5bd3d90249b 
  ql/src/test/results/clientpositive/llap/bucketpruning1.q.out 9422eb49781 
  ql/src/test/results/clientpositive/llap/check_constraint.q.out 9b1f8def6eb 
  ql/src/test/results/clientpositive/llap/dec_str.q.out 554031e952e 
  ql/src/test/results/clientpositive/llap/dynpart_sort_optimization_acid.q.out 
40dc5e97e5a 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out f240468558e 
  ql/src/test/results/clientpositive/llap/is_distinct_from.q.out f5a34750ce3 
  ql/src/test/results/clientpositive/llap/kryo.q.out 8121c119cc7 
  ql/src/test/results/clientpositive/llap/lineage3.q.out 9bec309c9c9 
  ql/src/test/results/clientpositive/llap/multi_column_in.q.out 414be226e6a 
  ql/src/test/results/clientpositive/llap/multi_column_in_single.q.out 
db9c676a814 
  ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out 3b163794b60 
  ql/src/test/results/clientpositive/llap/orc_predicate_pushdown.q.out 
21180a3da8c 
  ql/src/test/results/clientpositive/llap/parquet_predicate_pushdown.q.out 
8330e93115c 
  ql/src/test/results/clientpositive/llap/vector_between_in.q.out e9ea4611acc 
  ql/src/test/results/clientpositive/llap/vector_struct_in.q.out 4dfa80e9843 
  ql/src/test/results/clientpositive/llap/vectorization_13.q.out 4ce654f960b 
  ql/src/test/results/clientpositive/llap/vectorization_6.q.out a2f730beca2 
  ql/src/test/results/clientpositive/llap/vectorization_8.q.out 21ce7b8ebd0 
  ql/src/test/results/clientpositive/llap/vectorization_short_regress.q.out 
7f1c6a295e1 
  ql/src/test/results/clientpositive/mapjoin47.q.out 294dd69de5c 
  ql/src/test/results/clientpositive/parquet_vectorization_13.q.out 0efce98b555 
  ql/src/test/results/clientpositive/parquet_vectorization_6.q.out 0bb68883640 
  ql/src/test/results/clientpositive/parquet_vectorization_8.q.out 957bd7b264c