[jira] [Created] (HIVE-15569) failures in RetryingHMSHandler. do not get retried

2017-01-09 Thread Thejas M Nair (JIRA) 
Thejas M Nair created HIVE-15569:


 Summary: failures in RetryingHMSHandler. do not get retried
 Key: HIVE-15569
 URL: https://issues.apache.org/jira/browse/HIVE-15569
 Project: Hive
  Issue Type: Bug
Reporter: Thejas M Nair


RetryingHMSHandler.  is called during Hive metastore startup, and any 
transient db failures during that call are not retried. This can result in 
failure for HiveMetastore startup.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 55148: HIVE-15481: Support for multiple and nested subqueries

2017-01-09 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55148/#review161009
---




ql/src/java/org/apache/hadoop/hive/ql/parse/QBSubQuery.java (lines 882 - 887)


Can this be removed now?


- Ashutosh Chauhan


On Jan. 6, 2017, 11:54 p.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55148/
> ---
> 
> (Updated Jan. 6, 2017, 11:54 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.
> 
> 
> Bugs: HIVE-15481
> https://issues.apache.org/jira/browse/HIVE-15481
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch lifts the following restrictions:
> Restriction.8.m We allow only 1 SubQuery expression per Query.
> Restriction.9.m We will not support nested SubQuery expressions.
> Check.4.h For Exists and Not Exists, the Sub Query must have 1 or more 
> correlated predicates
> Check.12.h SubQuery predicates cannot only refer to Parent Query columns.
> 
> 
> Diffs
> -
> 
>   data/files/part_tiny_nulls.txt PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 70e7197 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveSubQRemoveRelBuilder.java
>  e028a99 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveFilter.java
>  ce207da 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java
>  5f58aae 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java fdb468d 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBSubQuery.java cfac6c0 
>   ql/src/test/queries/clientnegative/subquery_restrictions.q 80870d7 
>   ql/src/test/queries/clientnegative/subquery_subquery_chain_exists.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query16.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query23.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query33.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query45.q 0b34bfd 
>   ql/src/test/queries/clientpositive/perf/query56.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query60.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query69.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query83.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/subquery_exists.q be80721 
>   ql/src/test/queries/clientpositive/subquery_in.q 5b22dce 
>   ql/src/test/queries/clientpositive/subquery_in_having.q 2433556 
>   ql/src/test/queries/clientpositive/subquery_multi.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/subquery_notexists.q 2d7e9ce 
>   ql/src/test/queries/clientpositive/subquery_notin.q c29e63f 
>   ql/src/test/results/clientnegative/subquery_corr_grandparent.q.out fb72270 
>   ql/src/test/results/clientnegative/subquery_restrictions.q.out a546d49 
>   ql/src/test/results/clientnegative/subquery_subquery_chain_exists.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out fa54bb7 
>   ql/src/test/results/clientpositive/llap/lineage3.q.out 72a9344 
>   ql/src/test/results/clientpositive/llap/subquery_exists.q.out b132cb6 
>   ql/src/test/results/clientpositive/llap/subquery_in.q.out e71add5 
>   ql/src/test/results/clientpositive/llap/subquery_multi.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/subquery_notin.q.out 252b058 
>   ql/src/test/results/clientpositive/llap/vector_groupby_mapjoin.q.out 
> 27f32db 
>   ql/src/test/results/clientpositive/perf/query16.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query23.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query33.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query56.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query60.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query69.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query83.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/subquery_exists.q.out c28a218 
>   ql/src/test/results/clientpositive/spark/subquery_in.q.out 36e3e6e 
>   ql/src/test/results/clientpositive/subquery_exists.q.out 1019e7a 
>   ql/src/test/results/clientpositive/subquery_in_having.q.out 1ab7e7f 
>   ql/src/test/results/clientpositive/subquery_notexists.q.out 6ec3b46 
>   ql/src/test/results/clientpositive/subquery_notin_having.q.out 9f72cc9 
>   ql/src/test/results/clientpositive/vector_groupby_mapjoin.q.out af42e41 
> 
> Diff: https://reviews.apache.org/r/55148/diff/
> 
> 
> Testing
> ---
> 
> *Added new tests
> *Verfied result correctness with postgress (manually)
> *Pre-commit testing
> 
> 
>

Re: Invitation for Hive committers to become ORC committers

2017-01-09 Thread Owen O'Malley 
Ok, I added Sid as well.

.. Owen

On Wed, Jan 4, 2017 at 10:01 PM, Siddharth Seth  wrote:

> Hi Owen,
> I'd be interested as well, if not too late.
> Thanks,
> Sid
>
> On Wed, Jan 4, 2017 at 10:34 AM, Owen O'Malley  wrote:
>
> > Ferd, I've added you.
> >
> > Suneel, I'm sorry, but the offer is limited to current Hive committers.
> > http://people.apache.org/phonebook.html?unix=hive
> >
> > .. Owen
> >
> > On Mon, Jan 2, 2017 at 6:39 PM, Suneel Jakka 
> > wrote:
> >
> > > Hi Owen,
> > >
> > > Am also interested.
> > >
> > > Regards,
> > > Suneel Jakka
> > >
> > >
> > > On Mon, Jan 2, 2017 at 8:19 PM, Xu, Cheng A 
> > wrote:
> > >
> > > > Hi Owen,
> > > > Sorry for my late response. I'm also interested.
> > > >
> > > > Thanks,
> > > > Ferd
> > > >
> > > > -Original Message-
> > > > From: Owen O'Malley [mailto:omal...@apache.org]
> > > > Sent: Friday, December 23, 2016 11:55 AM
> > > > To: dev@hive.apache.org
> > > > Subject: Re: Invitation for Hive committers to become ORC committers
> > > >
> > > > Ok, I believe that I have got everyone. If you don't have karma as
> > shown
> > > > here: http://people.apache.org/phonebook.html?unix=orc
> > > >
> > > > Please, let me know. I believe I have also updated the ORC website
> with
> > > > everyone.
> > > >
> > > > Thanks,
> > > >Owen
> > > >
> > > > On Sat, Dec 17, 2016 at 5:16 AM, Lars Francke <
> lars.fran...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Owen,
> > > > >
> > > > > I'm also interested.
> > > > >
> > > > > Thanks,
> > > > > Lars
> > > > >
> > > > > On Fri, Dec 16, 2016 at 10:20 PM, Sergio Pena
> > > > > 
> > > > > wrote:
> > > > >
> > > > > > Hi Ownen,
> > > > > >
> > > > > > I'm also interested.
> > > > > > - Sergio
> > > > > >
> > > > > > On Fri, Dec 16, 2016 at 11:39 AM, Daniel Dai <
> > da...@hortonworks.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I am interested.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Daniel
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On 12/15/16, 1:12 PM, "Owen O'Malley" 
> > wrote:
> > > > > > >
> > > > > > > >All,
> > > > > > > >   As you are aware, we are in the last stages of removing the
> > > > > > > >forked
> > > > > > ORC
> > > > > > > >code out of Hive. The goal of moving ORC out of Hive was to
> > > > > > > >increase
> > > > > its
> > > > > > > >community and we want to be very deliberately inclusive of the
> > > > > > > >Hive development community. Towards that end, the ORC PMC
> wants
> > > > > > > >to welcome anyone who is already a Hive committer to become a
> > > > committer on ORC.
> > > > > > > >
> > > > > > > >  Please respond on this thread to let us know if you are
> > > > interested.
> > > > > > > >
> > > > > > > >Thanks,
> > > > > > > >   Owen on behalf of the ORC PMC
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Review Request 55276: HIVE-15539

2017-01-09 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55276/#review160938
---




ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java (line 467)


Better comment:
// Tries to optimize FROM clause of multi-insert. No attempt to optimize 
insert clauses of the query.



ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java (line 512)


This is not used anywhere.



ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java (line 514)


Is multimap really needed here. Seems like nodes declared above would have 
sufficied here?



ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java (line 605)


Will be good to add comment on what this loop is doing?



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java (line 1488)


good to add comment on why == 2?



ql/src/test/queries/clientpositive/multi_insert_with_join2.q (lines 62 - 69)


Can you add following test?
explain
FROM T_A a JOIN T_B b ON a.id = b.id
INSERT OVERWRITE TABLE join_result_1
SELECT a.val, a.id, b.id, b.val
WHERE b.id = 'Id_1' AND b.val = 'val_103'
INSERT OVERWRITE TABLE join_result_3
SELECT a.id, b.val, b.id, a.val
WHERE b.val = 'val_104' AND b.id = 'Id_2';


- Ashutosh Chauhan


On Jan. 6, 2017, 7:21 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55276/
> ---
> 
> (Updated Jan. 6, 2017, 7:21 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Vineet Garg.
> 
> 
> Bugs: HIVE-15539
> https://issues.apache.org/jira/browse/HIVE-15539
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15539
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSemanticException.java
>  0038f730e6a5500fda92e795e814d5efce567e2e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> fdb468d6047b68a97cbb6248e38ef0982248ba2e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java 
> f549dff3148deed158a149956d5c1ff61db8b00f 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> d0131b70239dd32cc82c801f35180d2a6a1cef88 
>   ql/src/test/queries/clientpositive/multi_insert_gby4.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/multi_insert_with_join2.q 
> 1529fa2fa3faaf303ac8d2ddac421e86ef0f9ce9 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_13.q.out 
> b45411cbf638a1bc3e7d720990f6786be9ef445d 
>   ql/src/test/results/clientpositive/groupby_sort_1_23.q.out 
> 6572f6ca577c8647b6023f108bd3cf289d9b0a88 
>   ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out 
> ce713547c80e519239ce82e680ad07a0ccae7d45 
>   ql/src/test/results/clientpositive/llap/auto_smb_mapjoin_14.q.out 
> e9990775bd40791b4b5b71716703af48da4a4a4e 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_13.q.out 
> a539e03b0892acb4ad6c028d89bf49cb9252152d 
>   ql/src/test/results/clientpositive/llap/explainuser_2.q.out 
> 931f1a2db0dd4f0ecc5f12f859c2853e83fbfbab 
>   ql/src/test/results/clientpositive/llap/lineage3.q.out 
> 72a93445d191e4c465ead011fe6b2e35235d03a4 
>   ql/src/test/results/clientpositive/llap/tez_union_multiinsert.q.out 
> 9b7612d8650d44691c68b6b39eb4646d9dfe0a07 
>   ql/src/test/results/clientpositive/llap/unionDistinct_1.q.out 
> 624d8860aa9cafe667d79cdfea90a45b08cae4cc 
>   ql/src/test/results/clientpositive/llap/vector_auto_smb_mapjoin_14.q.out 
> 735e4f435f6e2e50c9439b2a7aa437c33130b43f 
>   ql/src/test/results/clientpositive/multi_insert_gby4.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/multi_insert_union_src.q.out 
> 2036e6354efbe4a291ef93a37cf877e638e36b4f 
>   ql/src/test/results/clientpositive/multi_insert_with_join2.q.out 
> 70a044de93a61819e1641422914d61b7a52fbd19 
>   ql/src/test/results/clientpositive/spark/auto_smb_mapjoin_14.q.out 
> 0e99972fceab1453d0d349f45d92519d67e27e91 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_13.q.out 
> fb077718d0f25e6dbb4bd83380f647a994c2e222 
>   ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out 
> 92ca67bc174a881fd556ec13f997d792c11285ec 
>   ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out 
> f7f4dbb2fcc73182c0f65e774e8195dad3907273 
>   ql/src/test/results/clientpositive/spark/union17.q.out 
> ce23773671859ede11a513a8b43221d746fa73ba 
>

[jira] [Created] (HIVE-15568) Nested column pruning: more optimization on lateral view

2017-01-09 Thread Chao Sun (JIRA) 
Chao Sun created HIVE-15568:
---

 Summary: Nested column pruning: more optimization on lateral view
 Key: HIVE-15568
 URL: https://issues.apache.org/jira/browse/HIVE-15568
 Project: Hive
  Issue Type: Sub-task
  Components: Logical Optimizer
Affects Versions: 2.2.0
Reporter: Chao Sun


For queries with lateral view such as:
{code}
SELECT k FROM tbl
LATERAL VIEW EXPLODE(some_map_col) tbl2 AS k, v
{code}
where {{some_map_col}} is some column of map type, we should only need to scan 
all the keys for the map. Currently we need to scan both key and values.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15567) Nested column pruning: further prune the case for SELECT array.field

2017-01-09 Thread Chao Sun (JIRA) 
Chao Sun created HIVE-15567:
---

 Summary: Nested column pruning: further prune the case for SELECT 
array.field 
 Key: HIVE-15567
 URL: https://issues.apache.org/jira/browse/HIVE-15567
 Project: Hive
  Issue Type: Sub-task
Reporter: Chao Sun






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 55148: HIVE-15481: Support for multiple and nested subqueries

2017-01-09 Thread Vineet Garg 


> On Jan. 9, 2017, 2:52 p.m., Jesús Camacho Rodríguez wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java,
> >  line 78
> > 
> >
> > Why do we have this HiveSubQRemoveRelBuilder? If there was a bug in the 
> > original RelBuilder, would not be sufficient to override that method in the 
> > HiveRelBuilder? That would avoid bringing the whole class.
> > 
> > Maybe this is not part of this issue and it is a follow-up; I am just 
> > trying to keep things tidy.

We tried overriding RelBuilder but we ran into various issues which prevented 
us to override it and we had to replicate it. We plan to remove this and 
replace with RelBuilder once calcite 1.11 is released.


- Vineet


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55148/#review160893
---


On Jan. 6, 2017, 11:54 p.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55148/
> ---
> 
> (Updated Jan. 6, 2017, 11:54 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.
> 
> 
> Bugs: HIVE-15481
> https://issues.apache.org/jira/browse/HIVE-15481
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch lifts the following restrictions:
> Restriction.8.m We allow only 1 SubQuery expression per Query.
> Restriction.9.m We will not support nested SubQuery expressions.
> Check.4.h For Exists and Not Exists, the Sub Query must have 1 or more 
> correlated predicates
> Check.12.h SubQuery predicates cannot only refer to Parent Query columns.
> 
> 
> Diffs
> -
> 
>   data/files/part_tiny_nulls.txt PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 70e7197 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveSubQRemoveRelBuilder.java
>  e028a99 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveFilter.java
>  ce207da 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java
>  5f58aae 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java fdb468d 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBSubQuery.java cfac6c0 
>   ql/src/test/queries/clientnegative/subquery_restrictions.q 80870d7 
>   ql/src/test/queries/clientnegative/subquery_subquery_chain_exists.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query16.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query23.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query33.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query45.q 0b34bfd 
>   ql/src/test/queries/clientpositive/perf/query56.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query60.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query69.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query83.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/subquery_exists.q be80721 
>   ql/src/test/queries/clientpositive/subquery_in.q 5b22dce 
>   ql/src/test/queries/clientpositive/subquery_in_having.q 2433556 
>   ql/src/test/queries/clientpositive/subquery_multi.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/subquery_notexists.q 2d7e9ce 
>   ql/src/test/queries/clientpositive/subquery_notin.q c29e63f 
>   ql/src/test/results/clientnegative/subquery_corr_grandparent.q.out fb72270 
>   ql/src/test/results/clientnegative/subquery_restrictions.q.out a546d49 
>   ql/src/test/results/clientnegative/subquery_subquery_chain_exists.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out fa54bb7 
>   ql/src/test/results/clientpositive/llap/lineage3.q.out 72a9344 
>   ql/src/test/results/clientpositive/llap/subquery_exists.q.out b132cb6 
>   ql/src/test/results/clientpositive/llap/subquery_in.q.out e71add5 
>   ql/src/test/results/clientpositive/llap/subquery_multi.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/subquery_notin.q.out 252b058 
>   ql/src/test/results/clientpositive/llap/vector_groupby_mapjoin.q.out 
> 27f32db 
>   ql/src/test/results/clientpositive/perf/query16.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query23.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query33.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query56.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query60.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query69.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query83.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/subquery_exists.q.out

Re: Review Request 55154: HIVE-15366: REPL LOAD & DUMP support for incremental INSERT events

2017-01-09 Thread Vaibhav Gumashta 


> On Jan. 9, 2017, 7:07 p.m., Sushanth Sowmyan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java, line 2352
> > 
> >
> > We probably don't have a problem here, in that all entries in the list 
> > of newFiles are all probably in the same filesystem, but if ever that 
> > changes, we can have off-by-one issues here wherein we cannot line up the 
> > file to its checksum, if some files have checksums and others in the middle 
> > don't. Would it make sense to put in a  "" or something like that to 
> > indicate that there was no checksum for this file?
> > 
> > Note - this is not a blocker issue, and the patch can continue as-is. I 
> > mention more because this is something that might change in the future.

Fixing in this jira itself since it's a tiny change.


- Vaibhav


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55154/#review160929
---


On Jan. 6, 2017, 6:43 a.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55154/
> ---
> 
> (Updated Jan. 6, 2017, 6:43 a.m.)
> 
> 
> Review request for hive, Daniel Dai, Sushanth Sowmyan, and Thejas Nair.
> 
> 
> Bugs: HIVE-15366
> https://issues.apache.org/jira/browse/HIVE-15366
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-15366
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationListener.java
>  39356ae 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestReplicationScenarios.java
>  e29aa22 
>   metastore/if/hive_metastore.thrift 79592ea 
>   metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp 1311b20 
>   
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/InsertEventRequestData.java
>  39a607d 
>   metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb ebed504 
>   metastore/src/java/org/apache/hadoop/hive/metastore/events/InsertEvent.java 
> d9a42a7 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/messaging/InsertMessage.java
>  fe747df 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/messaging/MessageFactory.java
>  fdb8e80 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/messaging/json/JSONInsertMessage.java
>  bd9f9ec 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/messaging/json/JSONMessageFactory.java
>  9954902 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java 4c0f817 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java be5a6a9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/EximUtil.java 6e9602f 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ExportSemanticAnalyzer.java 
> f61274b 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
> 5561e06 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/ReplicationSemanticAnalyzer.java 
> 9b83407 
> 
> Diff: https://reviews.apache.org/r/55154/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

Re: Review Request 55154: HIVE-15366: REPL LOAD & DUMP support for incremental INSERT events

2017-01-09 Thread Sushanth Sowmyan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55154/#review160929
---


Fix it, then Ship it!




Looks good to me. I have one potential issue marked, but that can be solved in 
a future patch. Thanks, Vaibhav!


ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java (line 2352)


We probably don't have a problem here, in that all entries in the list of 
newFiles are all probably in the same filesystem, but if ever that changes, we 
can have off-by-one issues here wherein we cannot line up the file to its 
checksum, if some files have checksums and others in the middle don't. Would it 
make sense to put in a  "" or something like that to indicate that there was no 
checksum for this file?

Note - this is not a blocker issue, and the patch can continue as-is. I 
mention more because this is something that might change in the future.


- Sushanth Sowmyan


On Jan. 6, 2017, 6:43 a.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55154/
> ---
> 
> (Updated Jan. 6, 2017, 6:43 a.m.)
> 
> 
> Review request for hive, Daniel Dai, Sushanth Sowmyan, and Thejas Nair.
> 
> 
> Bugs: HIVE-15366
> https://issues.apache.org/jira/browse/HIVE-15366
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-15366
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationListener.java
>  39356ae 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestReplicationScenarios.java
>  e29aa22 
>   metastore/if/hive_metastore.thrift 79592ea 
>   metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp 1311b20 
>   
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/InsertEventRequestData.java
>  39a607d 
>   metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb ebed504 
>   metastore/src/java/org/apache/hadoop/hive/metastore/events/InsertEvent.java 
> d9a42a7 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/messaging/InsertMessage.java
>  fe747df 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/messaging/MessageFactory.java
>  fdb8e80 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/messaging/json/JSONInsertMessage.java
>  bd9f9ec 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/messaging/json/JSONMessageFactory.java
>  9954902 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java 4c0f817 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java be5a6a9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/EximUtil.java 6e9602f 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ExportSemanticAnalyzer.java 
> f61274b 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
> 5561e06 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/ReplicationSemanticAnalyzer.java 
> 9b83407 
> 
> Diff: https://reviews.apache.org/r/55154/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

[jira] [Created] (HIVE-15566) Schema tool upgrade schema fails from 1.2.1 to 2.1.1 because COMPACTION_QUEUE does not exist

2017-01-09 Thread Vihang Karajgaonkar (JIRA) 
Vihang Karajgaonkar created HIVE-15566:
--

 Summary: Schema tool upgrade schema fails from 1.2.1 to 2.1.1 
because COMPACTION_QUEUE does not exist
 Key: HIVE-15566
 URL: https://issues.apache.org/jira/browse/HIVE-15566
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Vihang Karajgaonkar
Assignee: Vihang Karajgaonkar


When we use schematool to upgrade metastore schema from 1.2.1 to 2.x* it fails 
with the error 

"ALTER TABLE' cannot be performed on 'COMPACTION_QUEUE' because it does not 
exist"

The table COMPACTION_QUEUE is created hive-txn-schema-2.1.0.derby.sql but the 
upgrade script does not seem to call it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 55148: HIVE-15481: Support for multiple and nested subqueries

2017-01-09 Thread Jesús Camacho Rodríguez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55148/#review160893
---




ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveSubQRemoveRelBuilder.java
 (line 151)


Why do we have this HiveSubQRemoveRelBuilder? If there was a bug in the 
original RelBuilder, would not be sufficient to override that method in the 
HiveRelBuilder?

Maybe this is not part of this issue and it is a follow-up; I am just 
trying to keep things tidy.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java
 (line 78)


Why do we have this HiveSubQRemoveRelBuilder? If there was a bug in the 
original RelBuilder, would not be sufficient to override that method in the 
HiveRelBuilder? That would avoid bringing the whole class.

Maybe this is not part of this issue and it is a follow-up; I am just 
trying to keep things tidy.



ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java (line 2084)


Comment should be removed?


- Jesús Camacho Rodríguez


On Jan. 6, 2017, 11:54 p.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55148/
> ---
> 
> (Updated Jan. 6, 2017, 11:54 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.
> 
> 
> Bugs: HIVE-15481
> https://issues.apache.org/jira/browse/HIVE-15481
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch lifts the following restrictions:
> Restriction.8.m We allow only 1 SubQuery expression per Query.
> Restriction.9.m We will not support nested SubQuery expressions.
> Check.4.h For Exists and Not Exists, the Sub Query must have 1 or more 
> correlated predicates
> Check.12.h SubQuery predicates cannot only refer to Parent Query columns.
> 
> 
> Diffs
> -
> 
>   data/files/part_tiny_nulls.txt PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 70e7197 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveSubQRemoveRelBuilder.java
>  e028a99 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveFilter.java
>  ce207da 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java
>  5f58aae 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java fdb468d 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBSubQuery.java cfac6c0 
>   ql/src/test/queries/clientnegative/subquery_restrictions.q 80870d7 
>   ql/src/test/queries/clientnegative/subquery_subquery_chain_exists.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query16.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query23.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query33.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query45.q 0b34bfd 
>   ql/src/test/queries/clientpositive/perf/query56.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query60.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query69.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query83.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/subquery_exists.q be80721 
>   ql/src/test/queries/clientpositive/subquery_in.q 5b22dce 
>   ql/src/test/queries/clientpositive/subquery_in_having.q 2433556 
>   ql/src/test/queries/clientpositive/subquery_multi.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/subquery_notexists.q 2d7e9ce 
>   ql/src/test/queries/clientpositive/subquery_notin.q c29e63f 
>   ql/src/test/results/clientnegative/subquery_corr_grandparent.q.out fb72270 
>   ql/src/test/results/clientnegative/subquery_restrictions.q.out a546d49 
>   ql/src/test/results/clientnegative/subquery_subquery_chain_exists.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out fa54bb7 
>   ql/src/test/results/clientpositive/llap/lineage3.q.out 72a9344 
>   ql/src/test/results/clientpositive/llap/subquery_exists.q.out b132cb6 
>   ql/src/test/results/clientpositive/llap/subquery_in.q.out e71add5 
>   ql/src/test/results/clientpositive/llap/subquery_multi.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/subquery_notin.q.out 252b058 
>   ql/src/test/results/clientpositive/llap/vector_groupby_mapjoin.q.out 
> 27f32db 
>   ql/src/test/results/clientpositive/perf/query16.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query23.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query33.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query56.q.out PRE-CREATION 
>