[jira] [Created] (HIVE-16737) LLAP: Shuffle handler TCP listen queue overflows

2017-05-22 Thread Gopal V (JIRA) 
Gopal V created HIVE-16737:
--

 Summary: LLAP: Shuffle handler TCP listen queue overflows
 Key: HIVE-16737
 URL: https://issues.apache.org/jira/browse/HIVE-16737
 Project: Hive
  Issue Type: Bug
  Components: llap
Affects Versions: 3.0.0
Reporter: Gopal V


{code}
$ netstat -s | grep "listen queue of a socket"
localhost: 297070 times the listen queue of a socket overflowed
{code}

{code}
$ ss -tl
LISTEN 0  50 *:15551*:*
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 56140: Can't order by an unselected column

2017-05-22 Thread pengcheng xiong 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56140/#review175753
---




ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
Lines 3093 (patched)


We can see 
Stage: Stage-0
Fetch Operator
  limit: 10
  Processor Tree:
ListSink

if we run explain rather than explain vectorization. This means, we may 
need to modify explain vectorization to show more info. for fetch work.

And, if there is no limit 10, it will return 43 rows. 
This means the result is correct.


- pengcheng xiong


On May 1, 2017, 5:30 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56140/
> ---
> 
> (Updated May 1, 2017, 5:30 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15160
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java
>  1487ed4f8e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 1b054a7e24 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/RowResolver.java 262dafb487 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 654f3b1772 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
> 8f8eab0d9c 
>   ql/src/test/queries/clientpositive/order_by_expr_1.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/order_by_expr_2.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out 873f1abb25 
>   ql/src/test/results/clientpositive/cp_sel.q.out 1778ccd6a6 
>   ql/src/test/results/clientpositive/druid_basic2.q.out 6177d56987 
>   ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out 2abb819558 
>   ql/src/test/results/clientpositive/groupby_grouping_sets_grouping.q.out 
> 473d17a1bd 
>   ql/src/test/results/clientpositive/llap/bucket_groupby.q.out d724131fca 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out 584c3b5520 
>   ql/src/test/results/clientpositive/llap/limit_pushdown.q.out dd54dd22a6 
>   ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out 24645b6426 
>   ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
> 83de1fbea1 
>   ql/src/test/results/clientpositive/llap/vector_coalesce.q.out 578f849bdb 
>   ql/src/test/results/clientpositive/llap/vector_date_1.q.out a4f1050c89 
>   ql/src/test/results/clientpositive/llap/vector_decimal_2.q.out 144356c108 
>   ql/src/test/results/clientpositive/llap/vector_decimal_round.q.out 
> 8bd80cf860 
>   
> ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_grouping.q.out
>  5af9e61b0a 
>   
> ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out
>  f731ceecdc 
>   ql/src/test/results/clientpositive/llap/vector_interval_1.q.out debf5ab39e 
>   ql/src/test/results/clientpositive/llap/vector_interval_arithmetic.q.out 
> aadb6e72cd 
>   ql/src/test/results/clientpositive/order3.q.out 898f7a8853 
>   ql/src/test/results/clientpositive/order_by_expr_1.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/order_by_expr_2.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/pcr.q.out a1301fdf79 
>   ql/src/test/results/clientpositive/perf/query31.q.out 3ed312d3e3 
>   ql/src/test/results/clientpositive/perf/query36.q.out 57ab26acc6 
>   ql/src/test/results/clientpositive/perf/query39.q.out 19472c4d5e 
>   ql/src/test/results/clientpositive/perf/query42.q.out 3bebac3321 
>   ql/src/test/results/clientpositive/perf/query52.q.out 74ecaf28ba 
>   ql/src/test/results/clientpositive/perf/query64.q.out 6b42393aad 
>   ql/src/test/results/clientpositive/perf/query66.q.out 072bfee92b 
>   ql/src/test/results/clientpositive/perf/query70.q.out 8e42fac9c5 
>   ql/src/test/results/clientpositive/perf/query75.q.out b1e236d325 
>   ql/src/test/results/clientpositive/perf/query81.q.out a09d5c99b5 
>   ql/src/test/results/clientpositive/perf/query85.q.out 168bcd2a4a 
>   ql/src/test/results/clientpositive/perf/query86.q.out 734e6a480b 
>   ql/src/test/results/clientpositive/perf/query89.q.out 66481f710b 
>   ql/src/test/results/clientpositive/perf/query91.q.out e592bba8d9 
>   ql/src/test/results/clientpositive/pointlookup2.q.out 3438c74608 
>   ql/src/test/results/clientpositive/pointlookup3.q.out 2c3e39fd15 
>   ql/src/test/results/clientpositive/ppd_udf_case.q.out 7678d03415 
>   ql/src/test/results/clientpositive/spark/dynamic_rdd_cache.q.out 6572511967 
>   ql/src/test/results/clientpositive/spark/limit_pushdown.q.out ede0096c73 
>   ql/src/test/results/clientpositive/spark/pcr.q.out

[jira] [Created] (HIVE-16736) General Improvements to BufferedRows

2017-05-22 Thread BELUGA BEHR (JIRA) 
BELUGA BEHR created HIVE-16736:
--

 Summary: General Improvements to BufferedRows
 Key: HIVE-16736
 URL: https://issues.apache.org/jira/browse/HIVE-16736
 Project: Hive
  Issue Type: Improvement
Affects Versions: 3.0.0
Reporter: BELUGA BEHR
Priority: Minor


General improvements {{BufferedRows.java}}.  Use {{ArrayList}} instead of 
{{LinkedList}}, prevent having to loop through the entire data set twice in 
{{normalizeWidths}} method, some simplifications.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 59468: Optimize a combination of avg(), sum(), count(distinct) etc

2017-05-22 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59468/#review175736
---




ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java
Line 206 (original), 210 (patched)


I think we should call this optimizer here. This way a) its called after 
ReduceSinkDedup so that there is no chance that extra RS introduced by this 
optimization does not get removed and b) It won't violate any assumptions made 
by GroupbyOptimizer.



ql/src/test/queries/clientpositive/count_dist_rewrite.q
Lines 24 (patched)


Can you add few tests when group by is on some key e.g. select max(key), 
count(distinct key), min(key), avg(key) from src group by value;



ql/src/test/results/clientpositive/llap/metadataonly1.q.out
Line 256 (original), 256-257 (patched)


Earlier plan was better. When metadataonly optimizer can kick in that 
should.


- Ashutosh Chauhan


On May 22, 2017, 10:31 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59468/
> ---
> 
> (Updated May 22, 2017, 10:31 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Gopal V.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16654
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7dedd23591 
>   itests/src/test/resources/testconfiguration.properties e23ef6317f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 8b04cd44fa 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/CountDistinctRewriteProc.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GroupByOptimizer.java 
> 3233157d8d 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 7dace9076f 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java 38a9ef2af1 
>   ql/src/test/queries/clientpositive/count_dist_rewrite.q PRE-CREATION 
>   ql/src/test/results/clientpositive/count_dist_rewrite.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/groupby_sort_11.q.out 2b3bf4a07a 
>   ql/src/test/results/clientpositive/groupby_sort_8.q.out 4faa0757cc 
>   ql/src/test/results/clientpositive/llap/count_dist_rewrite.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/metadataonly1.q.out 27218cf599 
>   ql/src/test/results/clientpositive/nullgroup4.q.out e5a814 
>   ql/src/test/results/clientpositive/perf/query16.q.out cf90c0c162 
>   ql/src/test/results/clientpositive/perf/query28.q.out 78129cf68b 
>   ql/src/test/results/clientpositive/perf/query94.q.out 836b16bf9f 
>   ql/src/test/results/clientpositive/perf/query95.q.out fa94d0842b 
>   ql/src/test/results/clientpositive/spark/nullgroup4.q.out 24f0291dec 
>   ql/src/test/results/clientpositive/udf_count.q.out f60ad0485e 
>   ql/src/test/results/clientpositive/vector_empty_where.q.out b2dec6d7f6 
> 
> 
> Diff: https://reviews.apache.org/r/59468/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

[jira] [Created] (HIVE-16735) Please add that hive.mapred.local.mem must be set in MBs in the documentation

2017-05-22 Thread Laurel Hale (JIRA) 
Laurel Hale created HIVE-16735:
--

 Summary: Please add that hive.mapred.local.mem must be set in MBs 
in the documentation
 Key: HIVE-16735
 URL: https://issues.apache.org/jira/browse/HIVE-16735
 Project: Hive
  Issue Type: Task
  Components: Documentation
Reporter: Laurel Hale
Assignee: Lefty Leverenz
Priority: Minor


Hi Lefty,
One of my developers noticed this and we do not document the local mode. 
Thought I'd let you know.

Here's a good place where you could put this:

https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-QueryandDDLExecution

There's already a small subsection where hive.mapred.local.mem is documented, 
but there isn't any information about how to set the value. My developer 
discovered that when you set it in bytes, it throws the following error:
2017-03-12 09:24:31,835 ERROR [HiveServer2-Background-Pool: Thread-22163()]: 
mr.MapredLocalTask (MapredLocalTask.java:executeInChildVM(341)) - Exception: 
java.lang.NumberFormatException: For input string: "4294967296"
java.lang.NumberFormatException: For input string: "4294967296"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:495)
at java.lang.Integer.parseInt(Integer.java:527)
at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1259)
at org.apache.hadoop.hive.conf.HiveConf.getIntVar(HiveConf.java:2415)
at org.apache.hadoop.hive.conf.HiveConf.getIntVar(HiveConf.java:2424)
at 
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInChildVM(MapredLocalTask.java:228)

Instead, this value should be set in MBs for success.

Thanks,
Laurel



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16734) Support original tpcds queries in perfclidriver after order by unselect column feature is done

2017-05-22 Thread Pengcheng Xiong (JIRA) 
Pengcheng Xiong created HIVE-16734:
--

 Summary: Support original tpcds queries in perfclidriver after 
order by unselect column feature is done
 Key: HIVE-16734
 URL: https://issues.apache.org/jira/browse/HIVE-16734
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 56140: Can't order by an unselected column

2017-05-22 Thread pengcheng xiong 


> On May 18, 2017, 12:29 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
> > Lines 4049-4059 (original)
> > 
> >
> > Aren't all these restrictions still valid?

They are not valid any more as we generate a select before OB and a select 
after OB for all the cases. There is no need to fix the schema after OB anymore.


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56140/#review175299
---


On May 1, 2017, 5:30 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56140/
> ---
> 
> (Updated May 1, 2017, 5:30 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15160
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java
>  1487ed4f8e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 1b054a7e24 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/RowResolver.java 262dafb487 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 654f3b1772 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
> 8f8eab0d9c 
>   ql/src/test/queries/clientpositive/order_by_expr_1.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/order_by_expr_2.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out 873f1abb25 
>   ql/src/test/results/clientpositive/cp_sel.q.out 1778ccd6a6 
>   ql/src/test/results/clientpositive/druid_basic2.q.out 6177d56987 
>   ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out 2abb819558 
>   ql/src/test/results/clientpositive/groupby_grouping_sets_grouping.q.out 
> 473d17a1bd 
>   ql/src/test/results/clientpositive/llap/bucket_groupby.q.out d724131fca 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out 584c3b5520 
>   ql/src/test/results/clientpositive/llap/limit_pushdown.q.out dd54dd22a6 
>   ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out 24645b6426 
>   ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
> 83de1fbea1 
>   ql/src/test/results/clientpositive/llap/vector_coalesce.q.out 578f849bdb 
>   ql/src/test/results/clientpositive/llap/vector_date_1.q.out a4f1050c89 
>   ql/src/test/results/clientpositive/llap/vector_decimal_2.q.out 144356c108 
>   ql/src/test/results/clientpositive/llap/vector_decimal_round.q.out 
> 8bd80cf860 
>   
> ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_grouping.q.out
>  5af9e61b0a 
>   
> ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out
>  f731ceecdc 
>   ql/src/test/results/clientpositive/llap/vector_interval_1.q.out debf5ab39e 
>   ql/src/test/results/clientpositive/llap/vector_interval_arithmetic.q.out 
> aadb6e72cd 
>   ql/src/test/results/clientpositive/order3.q.out 898f7a8853 
>   ql/src/test/results/clientpositive/order_by_expr_1.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/order_by_expr_2.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/pcr.q.out a1301fdf79 
>   ql/src/test/results/clientpositive/perf/query31.q.out 3ed312d3e3 
>   ql/src/test/results/clientpositive/perf/query36.q.out 57ab26acc6 
>   ql/src/test/results/clientpositive/perf/query39.q.out 19472c4d5e 
>   ql/src/test/results/clientpositive/perf/query42.q.out 3bebac3321 
>   ql/src/test/results/clientpositive/perf/query52.q.out 74ecaf28ba 
>   ql/src/test/results/clientpositive/perf/query64.q.out 6b42393aad 
>   ql/src/test/results/clientpositive/perf/query66.q.out 072bfee92b 
>   ql/src/test/results/clientpositive/perf/query70.q.out 8e42fac9c5 
>   ql/src/test/results/clientpositive/perf/query75.q.out b1e236d325 
>   ql/src/test/results/clientpositive/perf/query81.q.out a09d5c99b5 
>   ql/src/test/results/clientpositive/perf/query85.q.out 168bcd2a4a 
>   ql/src/test/results/clientpositive/perf/query86.q.out 734e6a480b 
>   ql/src/test/results/clientpositive/perf/query89.q.out 66481f710b 
>   ql/src/test/results/clientpositive/perf/query91.q.out e592bba8d9 
>   ql/src/test/results/clientpositive/pointlookup2.q.out 3438c74608 
>   ql/src/test/results/clientpositive/pointlookup3.q.out 2c3e39fd15 
>   ql/src/test/results/clientpositive/ppd_udf_case.q.out 7678d03415 
>   ql/src/test/results/clientpositive/spark/dynamic_rdd_cache.q.out 6572511967 
>   ql/src/test/results/clientpositive/spark/limit_pushdown.q.out ede0096c73 
>   ql/src/test/results/clientpositive/spark/pcr.q.out 77ac020d07 
>   ql/src/test/results/clientpositive/vector_coalesce.q.out

Re: Review Request 56140: Can't order by an unselected column

2017-05-22 Thread pengcheng xiong 


> On May 18, 2017, 6:59 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientpositive/order_by_expr_1.q
> > Lines 42 (patched)
> > 
> >
> > Can you also modify perfclidriver queries to their original form which 
> > contained such examples? They are currently rewritten to avoid this missing 
> > feature.

Open a new jira to track this.


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56140/#review175406
---


On May 1, 2017, 5:30 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56140/
> ---
> 
> (Updated May 1, 2017, 5:30 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15160
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java
>  1487ed4f8e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 1b054a7e24 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/RowResolver.java 262dafb487 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 654f3b1772 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
> 8f8eab0d9c 
>   ql/src/test/queries/clientpositive/order_by_expr_1.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/order_by_expr_2.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out 873f1abb25 
>   ql/src/test/results/clientpositive/cp_sel.q.out 1778ccd6a6 
>   ql/src/test/results/clientpositive/druid_basic2.q.out 6177d56987 
>   ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out 2abb819558 
>   ql/src/test/results/clientpositive/groupby_grouping_sets_grouping.q.out 
> 473d17a1bd 
>   ql/src/test/results/clientpositive/llap/bucket_groupby.q.out d724131fca 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out 584c3b5520 
>   ql/src/test/results/clientpositive/llap/limit_pushdown.q.out dd54dd22a6 
>   ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out 24645b6426 
>   ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
> 83de1fbea1 
>   ql/src/test/results/clientpositive/llap/vector_coalesce.q.out 578f849bdb 
>   ql/src/test/results/clientpositive/llap/vector_date_1.q.out a4f1050c89 
>   ql/src/test/results/clientpositive/llap/vector_decimal_2.q.out 144356c108 
>   ql/src/test/results/clientpositive/llap/vector_decimal_round.q.out 
> 8bd80cf860 
>   
> ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_grouping.q.out
>  5af9e61b0a 
>   
> ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out
>  f731ceecdc 
>   ql/src/test/results/clientpositive/llap/vector_interval_1.q.out debf5ab39e 
>   ql/src/test/results/clientpositive/llap/vector_interval_arithmetic.q.out 
> aadb6e72cd 
>   ql/src/test/results/clientpositive/order3.q.out 898f7a8853 
>   ql/src/test/results/clientpositive/order_by_expr_1.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/order_by_expr_2.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/pcr.q.out a1301fdf79 
>   ql/src/test/results/clientpositive/perf/query31.q.out 3ed312d3e3 
>   ql/src/test/results/clientpositive/perf/query36.q.out 57ab26acc6 
>   ql/src/test/results/clientpositive/perf/query39.q.out 19472c4d5e 
>   ql/src/test/results/clientpositive/perf/query42.q.out 3bebac3321 
>   ql/src/test/results/clientpositive/perf/query52.q.out 74ecaf28ba 
>   ql/src/test/results/clientpositive/perf/query64.q.out 6b42393aad 
>   ql/src/test/results/clientpositive/perf/query66.q.out 072bfee92b 
>   ql/src/test/results/clientpositive/perf/query70.q.out 8e42fac9c5 
>   ql/src/test/results/clientpositive/perf/query75.q.out b1e236d325 
>   ql/src/test/results/clientpositive/perf/query81.q.out a09d5c99b5 
>   ql/src/test/results/clientpositive/perf/query85.q.out 168bcd2a4a 
>   ql/src/test/results/clientpositive/perf/query86.q.out 734e6a480b 
>   ql/src/test/results/clientpositive/perf/query89.q.out 66481f710b 
>   ql/src/test/results/clientpositive/perf/query91.q.out e592bba8d9 
>   ql/src/test/results/clientpositive/pointlookup2.q.out 3438c74608 
>   ql/src/test/results/clientpositive/pointlookup3.q.out 2c3e39fd15 
>   ql/src/test/results/clientpositive/ppd_udf_case.q.out 7678d03415 
>   ql/src/test/results/clientpositive/spark/dynamic_rdd_cache.q.out 6572511967 
>   ql/src/test/results/clientpositive/spark/limit_pushdown.q.out ede0096c73 
>   ql/src/test/results/clientpositive/spark/pcr.q.out 77ac020d07 
>   ql/src/test/results/clientpositive/vector_coalesce.q.out f158236beb 
>

[jira] [Created] (HIVE-16733) Support conflict column name in order by

2017-05-22 Thread Pengcheng Xiong (JIRA) 
Pengcheng Xiong created HIVE-16733:
--

 Summary: Support conflict column name in order by
 Key: HIVE-16733
 URL: https://issues.apache.org/jira/browse/HIVE-16733
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong


There is a bug in RR which is exposed in HIVE-15160. After resolving the bug, 
we can support both:

select key as value from src order by src.value
select key as value from src order by value





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Review Request 59468: Optimize a combination of avg(), sum(), count(distinct) etc

2017-05-22 Thread pengcheng xiong 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59468/
---

Review request for hive, Ashutosh Chauhan and Gopal V.


Repository: hive-git


Description
---

HIVE-16654


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7dedd23591 
  itests/src/test/resources/testconfiguration.properties e23ef6317f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 8b04cd44fa 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/CountDistinctRewriteProc.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GroupByOptimizer.java 
3233157d8d 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 7dace9076f 
  ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java 38a9ef2af1 
  ql/src/test/queries/clientpositive/count_dist_rewrite.q PRE-CREATION 
  ql/src/test/results/clientpositive/count_dist_rewrite.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/groupby_sort_11.q.out 2b3bf4a07a 
  ql/src/test/results/clientpositive/groupby_sort_8.q.out 4faa0757cc 
  ql/src/test/results/clientpositive/llap/count_dist_rewrite.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/metadataonly1.q.out 27218cf599 
  ql/src/test/results/clientpositive/nullgroup4.q.out e5a814 
  ql/src/test/results/clientpositive/perf/query16.q.out cf90c0c162 
  ql/src/test/results/clientpositive/perf/query28.q.out 78129cf68b 
  ql/src/test/results/clientpositive/perf/query94.q.out 836b16bf9f 
  ql/src/test/results/clientpositive/perf/query95.q.out fa94d0842b 
  ql/src/test/results/clientpositive/spark/nullgroup4.q.out 24f0291dec 
  ql/src/test/results/clientpositive/udf_count.q.out f60ad0485e 
  ql/src/test/results/clientpositive/vector_empty_where.q.out b2dec6d7f6 


Diff: https://reviews.apache.org/r/59468/diff/1/


Testing
---


Thanks,

pengcheng xiong

Re: Review Request 59429: HIVE-16723 : Enable configurable MetaStoreSchemaInfo

2017-05-22 Thread Vihang Karajgaonkar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59429/
---

(Updated May 22, 2017, 10:13 p.m.)


Review request for hive, Naveen Gangam, Sergio Pena, and Sahil Takiar.


Changes
---

addressed review comments


Bugs: HIVE-16723
https://issues.apache.org/jira/browse/HIVE-16723


Repository: hive-git


Description
---

HIVE-16723 : Enable configurable MetaStoreSchemaInfo


Diffs (updated)
-

  beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java 
a45358077a07f09f07dd14eb028fb54b812e8a69 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
1c37b6e091cb02f5bf1a6a795712807b50d61424 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java
 7188af62acb3cb0305e2cdaf9727576d61418013 
  itests/hive-unit/src/test/java/org/apache/hive/beeline/TestSchemaTool.java 
438a7d6dc86cb291d7d0a9b4501a9df07f1934d7 
  metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreSchemaInfo.java 
PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java 
320902bed5fc49323d2b372354fe28d38358459e 
  
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfoFactory.java
 PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
b28983f7a9e36832d643c8a2293e19665dd79528 
  
metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreSchemaFactory.java
 PRE-CREATION 
  
metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreSchemaInfo.java
 71420012756ca26bf399228efdf011597eb70b17 


Diff: https://reviews.apache.org/r/59429/diff/3/

Changes: https://reviews.apache.org/r/59429/diff/2-3/


Testing
---


Thanks,

Vihang Karajgaonkar

[jira] [Created] (HIVE-16732) Transactional tables should block LOAD DATA

2017-05-22 Thread Eugene Koifman (JIRA) 
Eugene Koifman created HIVE-16732:
-

 Summary: Transactional tables should block LOAD DATA 
 Key: HIVE-16732
 URL: https://issues.apache.org/jira/browse/HIVE-16732
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 1.0.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman


This has always been the design.

see LoadSemanticAnalyzer.analyzeInternal()
StrictChecks.checkBucketing(conf);

Some examples (this is exposed by HIVE-16177)
insert_values_orig_table.q
 insert_orig_table.q
 insert_values_orig_table_use_metadata.q



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 59429: HIVE-16723 : Enable configurable MetaStoreSchemaInfo

2017-05-22 Thread Naveen Gangam 


> On May 22, 2017, 8:30 p.m., Sahil Takiar wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreSchemaInfo.java
> > Lines 27 (patched)
> > 
> >
> > Can you add javadocs for the rest of these methods.
> > 
> > Since this is an interface can you all add target audience annotations.

+1


> On May 22, 2017, 8:30 p.m., Sahil Takiar wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfoFactory.java
> > Lines 59 (patched)
> > 
> >
> > Should also document that this constructor is necessary

good point. Can we add a testcase to ensure that the class has such constructor?


- Naveen


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59429/#review175690
---


On May 22, 2017, 4:03 p.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59429/
> ---
> 
> (Updated May 22, 2017, 4:03 p.m.)
> 
> 
> Review request for hive, Naveen Gangam, Sergio Pena, and Sahil Takiar.
> 
> 
> Bugs: HIVE-16723
> https://issues.apache.org/jira/browse/HIVE-16723
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16723 : Enable configurable MetaStoreSchemaInfo
> 
> 
> Diffs
> -
> 
>   beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java 
> a45358077a07f09f07dd14eb028fb54b812e8a69 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 1c37b6e091cb02f5bf1a6a795712807b50d61424 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java
>  7188af62acb3cb0305e2cdaf9727576d61418013 
>   itests/hive-unit/src/test/java/org/apache/hive/beeline/TestSchemaTool.java 
> 438a7d6dc86cb291d7d0a9b4501a9df07f1934d7 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreSchemaInfo.java 
> PRE-CREATION 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java 
> 320902bed5fc49323d2b372354fe28d38358459e 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfoFactory.java
>  PRE-CREATION 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> b28983f7a9e36832d643c8a2293e19665dd79528 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreSchemaFactory.java
>  PRE-CREATION 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreSchemaInfo.java
>  71420012756ca26bf399228efdf011597eb70b17 
> 
> 
> Diff: https://reviews.apache.org/r/59429/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vihang Karajgaonkar
> 
>

Re: Review Request 59429: HIVE-16723 : Enable configurable MetaStoreSchemaInfo

2017-05-22 Thread Sahil Takiar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59429/#review175690
---


Fix it, then Ship it!





metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreSchemaInfo.java
Lines 27 (patched)


Can you add javadocs for the rest of these methods.

Since this is an interface can you all add target audience annotations.



metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfoFactory.java
Lines 59 (patched)


Should also document that this constructor is necessary


- Sahil Takiar


On May 22, 2017, 4:03 p.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59429/
> ---
> 
> (Updated May 22, 2017, 4:03 p.m.)
> 
> 
> Review request for hive, Naveen Gangam, Sergio Pena, and Sahil Takiar.
> 
> 
> Bugs: HIVE-16723
> https://issues.apache.org/jira/browse/HIVE-16723
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16723 : Enable configurable MetaStoreSchemaInfo
> 
> 
> Diffs
> -
> 
>   beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java 
> a45358077a07f09f07dd14eb028fb54b812e8a69 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 1c37b6e091cb02f5bf1a6a795712807b50d61424 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java
>  7188af62acb3cb0305e2cdaf9727576d61418013 
>   itests/hive-unit/src/test/java/org/apache/hive/beeline/TestSchemaTool.java 
> 438a7d6dc86cb291d7d0a9b4501a9df07f1934d7 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreSchemaInfo.java 
> PRE-CREATION 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java 
> 320902bed5fc49323d2b372354fe28d38358459e 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfoFactory.java
>  PRE-CREATION 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> b28983f7a9e36832d643c8a2293e19665dd79528 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreSchemaFactory.java
>  PRE-CREATION 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreSchemaInfo.java
>  71420012756ca26bf399228efdf011597eb70b17 
> 
> 
> Diff: https://reviews.apache.org/r/59429/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vihang Karajgaonkar
> 
>

Re: Review Request 59205: HIVE-16579: CachedStore: improvements to partition col stats caching and cache column stats for unpartitioned table

2017-05-22 Thread Daniel Dai 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/#review175684
---


Ship it!




Ship It!

- Daniel Dai


On May 22, 2017, 7:06 p.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59205/
> ---
> 
> (Updated May 22, 2017, 7:06 p.m.)
> 
> 
> Review request for hive, Daniel Dai and Thejas Nair.
> 
> 
> Bugs: HIVE-16579
> https://issues.apache.org/jira/browse/HIVE-16579
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-16579
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  91a3a38 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java d296851 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
> b96c27e 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> 870896c 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> b28983f 
>   metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java 
> fcf6f27 
>   metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 
> 668499b 
>   metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 
> 1cc838f 
>   metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 
> 7beee42 
>   metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 
> 0c7d8bb 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java
>  da6cd46 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DateColumnStatsMerger.java
>  PRE-CREATION 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  f613c30 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  1720e37 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
>  0ab20d6 
> 
> 
> Diff: https://reviews.apache.org/r/59205/diff/19/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

[jira] [Created] (HIVE-16731) Vectorization: Make "CASE WHEN (day_name='Sunday') THEN column1 ELSE null end" that involves a column name or expression THEN or ELSE vectorize

2017-05-22 Thread Matt McCline (JIRA) 
Matt McCline created HIVE-16731:
---

 Summary: Vectorization: Make "CASE WHEN (day_name='Sunday') THEN 
column1 ELSE null end" that involves a column name or expression THEN or ELSE 
vectorize
 Key: HIVE-16731
 URL: https://issues.apache.org/jira/browse/HIVE-16731
 Project: Hive
  Issue Type: Bug
Reporter: Matt McCline
Assignee: Teddy Choi
Priority: Critical


Currently, CASE WHEN statements like that become VectorUDFAdaptor expressions.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16730) Vectorization: Schema Evolution for Text Vectorization / Complex Types

2017-05-22 Thread Matt McCline (JIRA) 
Matt McCline created HIVE-16730:
---

 Summary: Vectorization: Schema Evolution for Text Vectorization / 
Complex Types
 Key: HIVE-16730
 URL: https://issues.apache.org/jira/browse/HIVE-16730
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Matt McCline
Assignee: Teddy Choi
Priority: Critical


With HIVE-16589: "Vectorization: Support Complex Types and GroupBy modes 
PARTIAL2, FINAL, and COMPLETE  for AVG" change, the tests 
schema_evol_text_vec_part_all_complex.q and 
schema_evol_text_vecrow_part_all_complex.q fail.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 59205: HIVE-16579: CachedStore: improvements to partition col stats caching and cache column stats for unpartitioned table

2017-05-22 Thread Vaibhav Gumashta 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/
---

(Updated May 22, 2017, 7:06 p.m.)


Review request for hive, Daniel Dai and Thejas Nair.


Bugs: HIVE-16579
https://issues.apache.org/jira/browse/HIVE-16579


Repository: hive-git


Description
---

https://issues.apache.org/jira/browse/HIVE-16579


Diffs (updated)
-

  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 91a3a38 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java d296851 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
b96c27e 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
870896c 
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java b28983f 
  metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690 
  metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java 
fcf6f27 
  metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 
668499b 
  metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 
1cc838f 
  metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 
7beee42 
  metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 
0c7d8bb 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java
 da6cd46 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DateColumnStatsMerger.java
 PRE-CREATION 
  
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 f613c30 
  
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 1720e37 
  
metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 
0ab20d6 


Diff: https://reviews.apache.org/r/59205/diff/19/

Changes: https://reviews.apache.org/r/59205/diff/18-19/


Testing
---


Thanks,

Vaibhav Gumashta

[jira] [Created] (HIVE-16729) Improve location validator to check for blank paths.

2017-05-22 Thread Naveen Gangam (JIRA) 
Naveen Gangam created HIVE-16729:


 Summary: Improve location validator to check for blank paths.
 Key: HIVE-16729
 URL: https://issues.apache.org/jira/browse/HIVE-16729
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 3.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam
Priority: Minor


Currently, the schema tool location validator succeeds even when the location 
for hive table/partitions have paths like
hdfs://myhost.com:8020/
hdfs://myhost.com:8020

where there is actually no "real" path. Having the validator report such path 
would be beneficial in preventing runtime errors.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16728) Fix some regression caused by HIVE-14879

2017-05-22 Thread Wei Zheng (JIRA) 
Wei Zheng created HIVE-16728:


 Summary: Fix some regression caused by HIVE-14879
 Key: HIVE-16728
 URL: https://issues.apache.org/jira/browse/HIVE-16728
 Project: Hive
  Issue Type: Sub-task
Affects Versions: hive-14535
Reporter: Wei Zheng
Assignee: Wei Zheng


HIVE-14879 integrates ACID logic with MM table. But it broke some existing ACID 
tests.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 59429: HIVE-16723 : Enable configurable MetaStoreSchemaInfo

2017-05-22 Thread Vihang Karajgaonkar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59429/
---

(Updated May 22, 2017, 4:03 p.m.)


Review request for hive, Naveen Gangam, Sergio Pena, and Sahil Takiar.


Changes
---

Added suggestions by Peter


Bugs: HIVE-16723
https://issues.apache.org/jira/browse/HIVE-16723


Repository: hive-git


Description
---

HIVE-16723 : Enable configurable MetaStoreSchemaInfo


Diffs (updated)
-

  beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java 
a45358077a07f09f07dd14eb028fb54b812e8a69 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
1c37b6e091cb02f5bf1a6a795712807b50d61424 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java
 7188af62acb3cb0305e2cdaf9727576d61418013 
  itests/hive-unit/src/test/java/org/apache/hive/beeline/TestSchemaTool.java 
438a7d6dc86cb291d7d0a9b4501a9df07f1934d7 
  metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreSchemaInfo.java 
PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java 
320902bed5fc49323d2b372354fe28d38358459e 
  
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfoFactory.java
 PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
b28983f7a9e36832d643c8a2293e19665dd79528 
  
metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreSchemaFactory.java
 PRE-CREATION 
  
metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreSchemaInfo.java
 71420012756ca26bf399228efdf011597eb70b17 


Diff: https://reviews.apache.org/r/59429/diff/2/

Changes: https://reviews.apache.org/r/59429/diff/1-2/


Testing
---


Thanks,

Vihang Karajgaonkar

Review Request 59446: HIVE-16559: Parquet schema evolution for partitioned tables may break if table and partition serdes differ

2017-05-22 Thread Barna Zsombor Klara 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59446/
---

Review request for hive and Sergio Pena.


Bugs: HIVE-16559
https://issues.apache.org/jira/browse/HIVE-16559


Repository: hive-git


Description
---

HIVE-16559: Parquet schema evolution for partitioned tables may break if table 
and partition serdes differ


Diffs
-

  ql/src/test/queries/clientpositive/parquet_schema_evolution.q 
e767b8186ead8e005ef4ee40f583a8afc48fa175 
  ql/src/test/results/clientpositive/parquet_schema_evolution.q.out 
43d75dc105ff104946d093d7c5e3c9de1cdcb196 
  serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 
a2a85b3dc44d7d7d676ad17eb476edbb5560f1cc 
  serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 
b9d23dd72580a684788747bfdff4c35b08726fba 
  serde/src/test/org/apache/hadoop/hive/serde2/TestSerDeUtils.java PRE-CREATION 


Diff: https://reviews.apache.org/r/59446/diff/1/


Testing
---

Added unit tests and a qtest. Manually tested that no regression is caused for 
avro and textfile SerDes when columns are added or replaced in a partitioned 
table.


Thanks,

Barna Zsombor Klara

Re: Review Request 59408: HIVE-16719 HiveMetaTool fails when the data does not fit in memory

2017-05-22 Thread Peter Vary 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59408/#review175622
---



Thanks for the patch Zsolt!

It is important to be able to check the data even if does not fit into memory! 
Thanks for the catch!

As for the patch, few comments from YETUS:
- Checkstyle: I will not put all of them here, but there is a problem with the 
indentation. Could you please check the checkstyle output of 
 
 mvn checkstyle:checkstyle -Dcheckstyle.consoleOutput=true -DskipTests 
-Pitests -Ptest-patch 
 
- ASF headers are missing:
 !? 
/Users/petervary/dev/upstream/hive/metastore/src/java/org/apache/hadoop/hive/metastore/metatool/BlockRetrieverIterable.java
 !? 
/Users/petervary/dev/upstream/hive/metastore/src/java/org/apache/hadoop/hive/metastore/metatool/BlockRetrieverIterator.java
 !? 
/Users/petervary/dev/upstream/hive/metastore/src/java/org/apache/hadoop/hive/metastore/metatool/EntityUpdater.java
 !? 
/Users/petervary/dev/upstream/hive/metastore/src/java/org/apache/hadoop/hive/metastore/metatool/IDataProvider.java
 !? 
/Users/petervary/dev/upstream/hive/metastore/src/java/org/apache/hadoop/hive/metastore/metatool/LocationEntity.java
 !? 
/Users/petervary/dev/upstream/hive/metastore/src/java/org/apache/hadoop/hive/metastore/metatool/LocationEntityImplementations.java
 !? 
/Users/petervary/dev/upstream/hive/metastore/src/java/org/apache/hadoop/hive/metastore/metatool/LocationUpdater.java
 !? 
/Users/petervary/dev/upstream/hive/metastore/src/java/org/apache/hadoop/hive/metastore/metatool/RetrieverIterable.java
 !? 
/Users/petervary/dev/upstream/hive/metastore/src/java/org/apache/hadoop/hive/metastore/metatool/ReturnValue.java
 !? 
/Users/petervary/dev/upstream/hive/metastore/src/java/org/apache/hadoop/hive/metastore/metatool/UpdateParams.java
 !? 
/Users/petervary/dev/upstream/hive/metastore/src/java/org/apache/hadoop/hive/metastore/metatool/UriUpdateChecker.java
 !? 
/Users/petervary/dev/upstream/hive/metastore/src/test/org/apache/hadoop/hive/metastore/metatool/BlockRetrieverIterableTest.java
 !? 
/Users/petervary/dev/upstream/hive/metastore/src/test/org/apache/hadoop/hive/metastore/metatool/DataProviderStub.java
 !? 
/Users/petervary/dev/upstream/hive/metastore/src/test/org/apache/hadoop/hive/metastore/metatool/EntityUpdaterTest.java
 !? 
/Users/petervary/dev/upstream/hive/metastore/src/test/org/apache/hadoop/hive/metastore/metatool/LocationEntityImplementationsTest.java
 !? 
/Users/petervary/dev/upstream/hive/metastore/src/test/org/apache/hadoop/hive/metastore/metatool/LocationUpdaterTest.java
 !? 
/Users/petervary/dev/upstream/hive/metastore/src/test/org/apache/hadoop/hive/metastore/metatool/ReturnValueTest.java
 !? 
/Users/petervary/dev/upstream/hive/metastore/src/test/org/apache/hadoop/hive/metastore/metatool/UriUpdateCheckerTest.java
 !? 
/Users/petervary/dev/upstream/hive/metastore/src/test/org/apache/hadoop/hive/metastore/tools/HiveMetaToolTest.java
 
Thanks,
Peter

- Peter Vary


On May 19, 2017, 5:05 p.m., Zsolt Fekete wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59408/
> ---
> 
> (Updated May 19, 2017, 5:05 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Currently HiveMetaTool reads full tables (as DataNucleus entities) into 
> memory by calling PersistenceManager's retrieveAll().
> 
> See these methods of ObjectStore: updateMDatabaseURI, updateTblPropURI, 
> updateMStorageDescriptorTblPropURI, updateMStorageDescriptorTblURI, 
> updateSerdeURI.
> 
> This might cause failure when the affected tables (SDS, DBS, TABLE_PARAMS, 
> SD_PARAMS, SERDES, SERDE_PARAMS) are too big.
> 
> 
> Diffs
> -
> 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> b28983f 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/metatool/BlockRetrieverIterable.java
>  PRE-CREATION 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/metatool/BlockRetrieverIterator.java
>  PRE-CREATION 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/metatool/EntityUpdater.java
>  PRE-CREATION 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/metatool/IDataProvider.java
>  PRE-CREATION 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/metatool/LocationEntity.java
>  PRE-CREATION 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/metatool/LocationEntityImplementations.java
>  PRE-CREATION 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/metatool/LocationUpdater.java
>  PRE-CREATION 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/metatool/RetrieverIterable.java
>  PRE-CREATION 
>   
>

Re: Review Request 59205: HIVE-16579: CachedStore: improvements to partition col stats caching and cache column stats for unpartitioned table

2017-05-22 Thread Vaibhav Gumashta 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/
---

(Updated May 22, 2017, 9:04 a.m.)


Review request for hive, Daniel Dai and Thejas Nair.


Bugs: HIVE-16579
https://issues.apache.org/jira/browse/HIVE-16579


Repository: hive-git


Description
---

https://issues.apache.org/jira/browse/HIVE-16579


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7dedd23 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 91a3a38 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java d296851 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
b96c27e 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
870896c 
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java b28983f 
  metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690 
  metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java 
fcf6f27 
  metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 
668499b 
  metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 
1cc838f 
  metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 
7beee42 
  metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 
0c7d8bb 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java
 da6cd46 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DateColumnStatsMerger.java
 PRE-CREATION 
  
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 f613c30 
  
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 1720e37 
  
metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 
0ab20d6 


Diff: https://reviews.apache.org/r/59205/diff/18/

Changes: https://reviews.apache.org/r/59205/diff/17-18/


Testing
---


Thanks,

Vaibhav Gumashta

Re: Review Request 59429: HIVE-16723 : Enable configurable MetaStoreSchemaInfo

2017-05-22 Thread Peter Vary 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59429/#review175608
---



Seems like a good idea to me!
Thanks for the patch!

Nits from YETUS:
- Checkstyle:
./common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:780:
METASTORE_SCHEMA_INFO_CLASS("hive.metastore.schema.info.class", 
"org.apache.hadoop.hive.metastore.MetaStoreSchemaInfo",: warning: Line is 
longer than 100 characters (found 123).
./common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:782:+ 
"schematool to fetch the schema information. This class should implement the 
IMetaStoreSchemaInfo interface"),: warning: Line is longer than 100 characters 
(found 120).

./metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreSchemaInfo.java:0::
 warning: File does not end with a newline.
./metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreSchemaInfo.java:22:public
 interface IMetaStoreSchemaInfo {: warning: Missing a Javadoc comment.
./metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreSchemaInfo.java:23:
  static String SQL_FILE_EXTENSION = ".sql";:3: warning: Redundant 'static' 
modifier.
./metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreSchemaInfo.java:25:
  /***: warning: First sentence should end with a period.
./metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreSchemaInfo.java:34:
  /***: warning: First sentence should end with a period.
./metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreSchemaInfo.java:43:
  /**: warning: First sentence should end with a period.
./metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java:34:public
 class MetaStoreSchemaInfo implements IMetaStoreSchemaInfo {: warning: Missing 
a Javadoc comment.
./metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java:39:
  protected final String dbType;:26: warning: Variable 'dbType' must be private 
and have accessor methods.
./metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java:40:
  private String hiveSchemaVersions[];:36: warning: Array brackets at illegal 
position.
./metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfoFactory.java:28:public
 class MetaStoreSchemaInfoFactory {: warning: Missing a Javadoc comment.
./metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfoFactory.java:28:public
 class MetaStoreSchemaInfoFactory {:1: warning: Utility classes should not have 
a public or default constructor.

./beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java:38:import 
org.apache.hadoop.hive.metastore.MetaStoreSchemaInfo;:8: warning: Unused import 
- org.apache.hadoop.hive.metastore.MetaStoreSchemaInfo.

- Findbugs:
Found reliance on default encoding in 
org.apache.hadoop.hive.metastore.MetaStoreSchemaInfo.loadAllUpgradeScripts(String):
 new java.io.FileReader(String)
Bug type DM_DEFAULT_ENCODING (click for details) 
In class org.apache.hadoop.hive.metastore.MetaStoreSchemaInfo
In method 
org.apache.hadoop.hive.metastore.MetaStoreSchemaInfo.loadAllUpgradeScripts(String)
Called method new java.io.FileReader(String)
At MetaStoreSchemaInfo.java:[line 64]


Do as you see fit with them (I prefer fix them all, but we can have a 
discussion about changing the rules on the dev list)

Thanks,
Peter


metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreSchemaInfo.java
Lines 27 (patched)


nit: Could you please remove extra white spaces?



metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreSchemaInfo.java
Lines 36 (patched)


nit: Could you please remove extra white spaces?



metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreSchemaInfo.java
Lines 45 (patched)


nit: Could you please remove extra white spaces?



metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java
Lines 93 (patched)


Could you please explain why it is changed so? I do not really understand 
this change.



metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreSchemaInfo.java
Lines 21 (patched)


nit: Do we need this extra import?


- Peter Vary


On May 21, 2017, 4:51 a.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59429/
> ---
> 
> (Updated May 21, 2017, 4:51 a.m.)
> 
> 
> Review request for hive, Naveen Gangam, Sergio Pena, and Sahil Takiar.
> 
> 
> Bugs: HIVE-16723
> https://issues.apache.org/jira/browse/HIVE-16723
> 
> 
>

[jira] [Created] (HIVE-16727) REPL DUMP for insert event should't fail if the table is already dropped.

2017-05-22 Thread Sankar Hariappan (JIRA) 
Sankar Hariappan created HIVE-16727:
---

 Summary: REPL DUMP for insert event should't fail if the table is 
already dropped.
 Key: HIVE-16727
 URL: https://issues.apache.org/jira/browse/HIVE-16727
 Project: Hive
  Issue Type: Sub-task
  Components: Hive, repl
Affects Versions: 2.1.0
Reporter: Sankar Hariappan
Assignee: Sankar Hariappan


Currently, insert event doesn't log the table object as part of event 
notification message. During dump, the table object is obtained from metastore 
which can be null if the table is already dropped and hence REPL DUMP fails.
Steps:
1. Bootstrap dump/load with a table.
2. Insert into the table.
3. Drop the table.
4. REPL DUMP (incremental).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16726) beeline+kerberos connect faild by domestic consumer from ldap

2017-05-22 Thread Saijin Huang (JIRA) 
Saijin Huang created HIVE-16726:
---

 Summary: beeline+kerberos connect faild by domestic consumer from 
ldap
 Key: HIVE-16726
 URL: https://issues.apache.org/jira/browse/HIVE-16726
 Project: Hive
  Issue Type: Bug
Reporter: Saijin Huang






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)