Re: Review Request 58934: HIVE-16568: Support complex types in external LLAP InputFormat

[j . prasanth . j]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58934/#review173697
---




llap-client/src/java/org/apache/hadoop/hive/llap/LlapRowRecordReader.java
Lines 154 (patched)


IIRC, there are utilities already to do this in ObjectInspectorUtils.. 
copyToXXX() methods.. can that be reused?



llap-common/src/java/org/apache/hadoop/hive/llap/TypeDesc.java
Lines 154 (patched)


This also looks repetitive. TypeInfoUtils already has something like this I 
guess. We need to make sure TypeInfo parser can parse the string generated by 
this method. It's easier to reuse TypeInfoUtils or have a TypeDesc converted to 
TypeInfo.

My point there is duplicacy in
TypeInfo
TypeDesc
TypeDescriptor (ORC has this)

Wondering if TypeInfo or TypeDescriptor from ORC can be reused here. 
Thoughts?


- Prasanth_J


On May 2, 2017, 9:52 p.m., Jason Dere wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58934/
> ---
> 
> (Updated May 2, 2017, 9:52 p.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, Prasanth_J, and Siddharth Seth.
> 
> 
> Bugs: HIVE-16568
> https://issues.apache.org/jira/browse/HIVE-16568
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> - Support list/map/struct types in the LLAPRowInputFormat Schema/TypeDesc
> - Support list/map/struct types in the LLAPRowInputFormat Row. Changes in the 
> Row getters/setters needed (no longer using Writable).
> 
> 
> Diffs
> -
> 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniLlap.java 
> de47412 
>   llap-client/src/java/org/apache/hadoop/hive/llap/LlapRowRecordReader.java 
> ee92f3e 
>   llap-common/src/java/org/apache/hadoop/hive/llap/Row.java a84fadc 
>   llap-common/src/java/org/apache/hadoop/hive/llap/TypeDesc.java dda5928 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
> 9ddbd7e 
> 
> 
> Diff: https://reviews.apache.org/r/58934/diff/1/
> 
> 
> Testing
> ---
> 
> Added test to TestJdbcWithMiniLlap
> 
> 
> Thanks,
> 
> Jason Dere
> 
>

Re: Review Request 58865: HIVE-16552: Limit the number of tasks a Spark job may contain

[Xuefu Zhang]

> On May 3, 2017, 3:35 a.m., Rui Li wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java
> > Lines 132 (patched)
> > 
> >
> > I think the log is unnecessary because the failure should already be 
> > logged in the monitor

This is not new code.


> On May 3, 2017, 3:35 a.m., Rui Li wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java
> > Lines 135 (patched)
> > 
> >
> > Same as above. Can we consolidate the logs a bit?

Jobmonitor prints it on console, while the log here is written to hive.log.


> On May 3, 2017, 3:35 a.m., Rui Li wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/RemoteSparkJobMonitor.java
> > Lines 104 (patched)
> > 
> >
> > Maybe I was being misleading. I mean we can compute the total task only 
> > once when the job first reaches RUNNING state, i.e. in the "if (!running)". 
> > At this point, the total count is determined and won't change.

Yeah. However, I'd like to keep the state transition to running first before 
breaking up and returning rc=4. In fact, if we lose the transition, Hive 
actually goes into an instable state. What you said was what I tried in first 
place.


- Xuefu


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58865/#review173689
---


On May 2, 2017, 6:49 p.m., Xuefu Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58865/
> ---
> 
> (Updated May 2, 2017, 6:49 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-16552
> https://issues.apache.org/jira/browse/HIVE-16552
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See JIRA description
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 84398c6 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java 32a7730 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/RemoteSparkJobMonitor.java
>  dd73f3e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java 
> 0b224f2 
> 
> 
> Diff: https://reviews.apache.org/r/58865/diff/3/
> 
> 
> Testing
> ---
> 
> Test locally
> 
> 
> Thanks,
> 
> Xuefu Zhang
> 
>

Re: Review Request 56140: Can't order by an unselected column

[Ashutosh Chauhan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56140/#review173690
---




ql/src/test/results/clientpositive/cp_sel.q.out
Line 46 (original), 50 (patched)


Is this expected? 
Seems like this may generate wrong results since there might be multiple 
tasks for Reducers each of which emit 1 row. Limit in fetch operator is needed.



ql/src/test/results/clientpositive/llap/vector_coalesce.q.out
Line 447 (original)


No RS for order by.



ql/src/test/results/clientpositive/vector_date_1.q.out
Lines 598-607 (original)


This plan looks incorrect. For an order by there should necessarily be a RS 
in plan, otherwise we can get sorting in map only plan.



ql/src/test/results/clientpositive/vector_interval_arithmetic.q.out
Lines 580-588 (original)


Order by must have RS.


- Ashutosh Chauhan


On May 1, 2017, 5:30 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56140/
> ---
> 
> (Updated May 1, 2017, 5:30 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15160
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java
>  1487ed4f8e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 1b054a7e24 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/RowResolver.java 262dafb487 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 654f3b1772 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
> 8f8eab0d9c 
>   ql/src/test/queries/clientpositive/order_by_expr_1.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/order_by_expr_2.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out 873f1abb25 
>   ql/src/test/results/clientpositive/cp_sel.q.out 1778ccd6a6 
>   ql/src/test/results/clientpositive/druid_basic2.q.out 6177d56987 
>   ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out 2abb819558 
>   ql/src/test/results/clientpositive/groupby_grouping_sets_grouping.q.out 
> 473d17a1bd 
>   ql/src/test/results/clientpositive/llap/bucket_groupby.q.out d724131fca 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out 584c3b5520 
>   ql/src/test/results/clientpositive/llap/limit_pushdown.q.out dd54dd22a6 
>   ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out 24645b6426 
>   ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
> 83de1fbea1 
>   ql/src/test/results/clientpositive/llap/vector_coalesce.q.out 578f849bdb 
>   ql/src/test/results/clientpositive/llap/vector_date_1.q.out a4f1050c89 
>   ql/src/test/results/clientpositive/llap/vector_decimal_2.q.out 144356c108 
>   ql/src/test/results/clientpositive/llap/vector_decimal_round.q.out 
> 8bd80cf860 
>   
> ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_grouping.q.out
>  5af9e61b0a 
>   
> ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out
>  f731ceecdc 
>   ql/src/test/results/clientpositive/llap/vector_interval_1.q.out debf5ab39e 
>   ql/src/test/results/clientpositive/llap/vector_interval_arithmetic.q.out 
> aadb6e72cd 
>   ql/src/test/results/clientpositive/order3.q.out 898f7a8853 
>   ql/src/test/results/clientpositive/order_by_expr_1.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/order_by_expr_2.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/pcr.q.out a1301fdf79 
>   ql/src/test/results/clientpositive/perf/query31.q.out 3ed312d3e3 
>   ql/src/test/results/clientpositive/perf/query36.q.out 57ab26acc6 
>   ql/src/test/results/clientpositive/perf/query39.q.out 19472c4d5e 
>   ql/src/test/results/clientpositive/perf/query42.q.out 3bebac3321 
>   ql/src/test/results/clientpositive/perf/query52.q.out 74ecaf28ba 
>   ql/src/test/results/clientpositive/perf/query64.q.out 6b42393aad 
>   ql/src/test/results/clientpositive/perf/query66.q.out 072bfee92b 
>   ql/src/test/results/clientpositive/perf/query70.q.out 8e42fac9c5 
>   ql/src/test/results/clientpositive/perf/query75.q.out b1e236d325 
>   ql/src/test/results/clientpositive/perf/query81.q.out a09d5c99b5 
>   ql/src/test/results/clientpositive/perf/query85.q.out 168bcd2a4a 
>   ql/src/test/results/clientpositive/perf/query86.q.out 734e6a480b 
>   ql/src/test/results/clientpositive/perf/query89.q.out 66481f710b 
>   ql/src/test/results/clientpositive/perf/query91.q.out e592bba8d9 
>   

Re: [VOTE] Apache Hive 2.3.0 Release Candidate 0

[Rui Li]

The patch has been reverted in master and branch-2.3

On Wed, May 3, 2017 at 3:01 AM, Sergio Pena 
wrote:

> Hi Pengcheng,
>
> There is a request from the HDFS team to revert the patch committed on
> HIVE-16047 from
> our code because it might cause problems when future Hadoop versions are
> released due to being a
> private API on Hadoop. This API method signature has been changed between
> releases, and
> we don't want to have additional shims to support future Hadoop versions
> just for this method.
>
> I'd like to revert it from 2.3.0 release before doing the release. It is
> marked as being fixed on 2.2 but it is not cherry-picked on branch-2.2 but
> branch-2.3.
>
> Do you agree?
>
> - Sergio
>
> On Fri, Apr 28, 2017 at 1:40 PM, Pengcheng Xiong 
> wrote:
>
> > Withdraw the VOTE on candidate 0. Will propose candidate 1 soon. Thanks.
> >
> > On Thu, Apr 27, 2017 at 8:10 PM, Owen O'Malley 
> > wrote:
> >
> > > -1 you need a release of storage-API first.
> > >
> > > .. Owen
> > >
> > > > On Apr 27, 2017, at 17:43, Pengcheng Xiong 
> wrote:
> > > >
> > > > Apache Hive 2.3.0 Release Candidate 0 is available here:
> > > > http://home.apache.org/~pxiong/apache-hive-2.3.0-rc0/
> > > >
> > > >
> > > > Maven artifacts are available here:
> > > > https://repository.apache.org/content/repositories/
> orgapachehive-1073/
> > > >
> > > >
> > > > Source tag for RC0 is at:
> > > >
> > > > https://github.com/apache/hive/releases/tag/release-2.3.0-rc0
> > > >
> > > > Voting will conclude in 72 hours.
> > > >
> > > > Hive PMC Members: Please test and vote.
> > > >
> > > > Thanks.
> > >
> >
>



-- 
Best regards!
Rui Li
Cell: (+86) 13564950210

Re: Review Request 58865: HIVE-16552: Limit the number of tasks a Spark job may contain

[Rui Li]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58865/#review173689
---




ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java
Lines 132 (patched)


I think the log is unnecessary because the failure should already be logged 
in the monitor



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java
Lines 135 (patched)


Same as above. Can we consolidate the logs a bit?



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/RemoteSparkJobMonitor.java
Lines 104 (patched)


Maybe I was being misleading. I mean we can compute the total task only 
once when the job first reaches RUNNING state, i.e. in the "if (!running)". At 
this point, the total count is determined and won't change.


- Rui Li


On May 2, 2017, 6:49 p.m., Xuefu Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58865/
> ---
> 
> (Updated May 2, 2017, 6:49 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-16552
> https://issues.apache.org/jira/browse/HIVE-16552
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See JIRA description
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 84398c6 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java 32a7730 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/RemoteSparkJobMonitor.java
>  dd73f3e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java 
> 0b224f2 
> 
> 
> Diff: https://reviews.apache.org/r/58865/diff/3/
> 
> 
> Testing
> ---
> 
> Test locally
> 
> 
> Thanks,
> 
> Xuefu Zhang
> 
>

[jira] [Created] (HIVE-16572) Rename a partition should not drop its column stats

[Chaoyu Tang (JIRA)]

Chaoyu Tang created HIVE-16572:
--

 Summary: Rename a partition should not drop its column stats
 Key: HIVE-16572
 URL: https://issues.apache.org/jira/browse/HIVE-16572
 Project: Hive
  Issue Type: Bug
  Components: Statistics
Reporter: Chaoyu Tang
Assignee: Chaoyu Tang


The column stats for the table sample_pt partition (dummy=1) is as following:
{code}
hive> describe formatted sample_pt partition (dummy=1) code;
OK
# col_name  data_type   min max 
num_nulls   distinct_count  avg_col_len 
max_col_len num_trues   num_falses  
comment 

 
codestring  
0   303 6.985   
7   
from deserializer   
Time taken: 0.259 seconds, Fetched: 3 row(s)
{code}
But when this partition is renamed, say
alter table sample_pt partition (dummy=1) rename to partition (dummy=11);
The COLUMN_STATS in partition description are true, but column stats are 
actually all deleted.
{code}
hive> describe formatted sample_pt partition (dummy=11);
OK
# col_name  data_type   comment 
 
codestring  
description string  
salary  int 
total_emp   int 
 
# Partition Information  
# col_name  data_type   comment 
 
dummy   int 
 
# Detailed Partition Information 
Partition Value:[11] 
Database:   default  
Table:  sample_pt
CreateTime: Thu Mar 30 23:03:59 EDT 2017 
LastAccessTime: UNKNOWN  
Location:   file:/user/hive/warehouse/apache/sample_pt/dummy=11 
 
Partition Parameters:
COLUMN_STATS_ACCURATE   
{\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"code\":\"true\",\"description\":\"true\",\"salary\":\"true\",\"total_emp\":\"true\"}}
numFiles1   
numRows 200 
rawDataSize 10228   
totalSize   10428   
transient_lastDdlTime   1490929439  
 
# Storage Information
SerDe Library:  org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe  
 
InputFormat:org.apache.hadoop.mapred.TextInputFormat 
OutputFormat:   
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat   
Compressed: No   
Num Buckets:-1   
Bucket Columns: []   
Sort Columns:   []   
Storage Desc Params: 
serialization.format1   
Time taken: 6.783 seconds, Fetched: 37 row(s)

===
hive> describe formatted sample_pt partition (dummy=11) code;
OK
# col_name  data_type   comment 
 

 
codestring  from deserializer   
 
Time taken: 9.429 seconds, Fetched: 3 row(s)
{code}
The column stats should not be drop when a partition is renamed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 58914: HIVE-16550

[Deepak Jaiswal]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58914/
---

(Updated May 3, 2017, 1:16 a.m.)


Review request for hive, Jason Dere and Sergey Shelukhin.


Changes
---

Worked on Jason's comments and added some more tests.


Bugs: HIVE-16550
https://issues.apache.org/jira/browse/HIVE-16550


Repository: hive-git


Description
---

Semijoin Hints should be able to skip the optimization if needed.
In addition to that, the patch fixes several issues with hints in general such 
as,
- It now works with subqueries and unions.
- Uses a global data structure instead of per QB.


Diffs (updated)
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
 e1a69526bc 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveOpConverter.java
 40c0f3ba2a 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 1b054a7e24 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HintParser.g e110fb33df 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 3a1f821bd3 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java 54e37f7c80 
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java 7bf1c599a5 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java cbbb7d0c94 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 5ea7800528 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDynamicListDesc.java 
3143554ec6 
  ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java 7d4267d6a8 
  ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java 
f45daa8828 
  ql/src/test/queries/clientpositive/semijoin_hint.q 5de0c8c8c1 
  ql/src/test/results/clientpositive/llap/semijoin_hint.q.out bc248930ec 


Diff: https://reviews.apache.org/r/58914/diff/3/

Changes: https://reviews.apache.org/r/58914/diff/2-3/


Testing
---


Thanks,

Deepak Jaiswal

Re: [VOTE] Should we release hive-storage-2.3.0rc0?

[Owen O'Malley]

With three +1 votes and no -1's the vote passes.

Thanks,
   Owen

On Tue, May 2, 2017 at 2:55 PM, Alan Gates  wrote:

> +1.  Did a build with a fresh maven repo, checked the signatures, ran rat.
>
> Alan.
>
> On Apr 27, 2017, 16:08 -0700, Owen O'Malley , wrote:
> > All,
> > I'd like to make a new release of the storage-api for the upcoming 2.2
> > release. Since storage-api rolls releases faster than Hive, this will be
> > the storage-api 2.3.0.
> >
> > Artifacts:
> > tag: https://github.com/apache/hive/releases/tag/storage-
> release-2.3.0rc0
> > tar ball: http://home.apache.org/~omalley/hive-storage-2.3.0/
> >
> > Thanks,
> > Owen
>

[jira] [Created] (HIVE-16571) HiveServer2: Prefer LIFO over round-robin for Tez session reuse

[Gopal V (JIRA)]

Gopal V created HIVE-16571:
--

 Summary: HiveServer2: Prefer LIFO over round-robin for Tez session 
reuse
 Key: HIVE-16571
 URL: https://issues.apache.org/jira/browse/HIVE-16571
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2, Tez
Affects Versions: 3.0.0
Reporter: Gopal V


Currently Tez session reuse is entirely round-robin, which means a single user 
might have to run upto 32 queries before reusing a warm session on a 
HiveServer2.

This is not the case when session reuse is disabled, with a user warming up 
their session on the 1st query.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Review Request 58936: HIVE-16143 : Improve msck repair batching

[Vihang Karajgaonkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58936/
---

Review request for hive, Sergio Pena and Sahil Takiar.


Bugs: HIVE-16143
https://issues.apache.org/jira/browse/HIVE-16143


Repository: hive-git


Description
---

HIVE-16143 : Improve msck repair batching


Diffs
-

  common/src/java/org/apache/hive/common/util/RetryUtilities.java PRE-CREATION 
  common/src/test/org/apache/hive/common/util/TestRetryUtilities.java 
PRE-CREATION 
  itests/hive-blobstore/src/test/queries/clientpositive/create_like.q 
38f384e4c547d3c93d510b89fccfbc2b8e2cba09 
  itests/hive-blobstore/src/test/results/clientpositive/create_like.q.out 
0d362a716291637404a3859fe81068594d82c9e0 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
2ae1eacb68cef6990ae3f2050af0bed7c8e9843f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
917e565f28b2c9aaea18033ea3b6b20fa41fcd0a 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java
 PRE-CREATION 
  ql/src/test/queries/clientpositive/msck_repair_0.q 
22542331621ca4ce5277c2f46a4264b7540a4d1e 
  ql/src/test/queries/clientpositive/msck_repair_1.q 
ea596cbbd2d4c230f2b5afbe379fc1e8836b6fbd 
  ql/src/test/queries/clientpositive/msck_repair_2.q 
d8338211e970ebac68a7471ee0960ccf2d51cba3 
  ql/src/test/queries/clientpositive/msck_repair_3.q 
fdefca121a2de361dbd19e7ef34fb220e1733ed2 
  ql/src/test/queries/clientpositive/msck_repair_batchsize.q 
e56e97ac36a6544f3e20478fdb0e8fa783a857ef 
  ql/src/test/results/clientpositive/msck_repair_0.q.out 
2e0d9dc423071ebbd9a55606f196cf7752e27b1a 
  ql/src/test/results/clientpositive/msck_repair_1.q.out 
3f2fe75b194f1248bd5c073dd7db6b71b2ffc2ba 
  ql/src/test/results/clientpositive/msck_repair_2.q.out 
3f2fe75b194f1248bd5c073dd7db6b71b2ffc2ba 
  ql/src/test/results/clientpositive/msck_repair_3.q.out 
3f2fe75b194f1248bd5c073dd7db6b71b2ffc2ba 
  ql/src/test/results/clientpositive/msck_repair_batchsize.q.out 
ba99024163a1f2c59d59e9ed7ea276c154c99d24 
  ql/src/test/results/clientpositive/repair.q.out 
c1834640a35500c521a904a115a718c94546df10 


Diff: https://reviews.apache.org/r/58936/diff/1/


Testing
---


Thanks,

Vihang Karajgaonkar

Review Request 58935: Set column stats default as true when creating new tables/partitions

[pengcheng xiong]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58935/
---

Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

HIVE-16566


Diffs
-

  common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java a9e17c240d 
  hbase-handler/src/test/results/positive/external_table_ppd.q.out f09b880c06 
  hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out 
3d3e82d5da 
  hbase-handler/src/test/results/positive/hbasestats.q.out bf902e49b5 
  
itests/hive-blobstore/src/test/results/clientpositive/write_final_output_blobstore.q.out
 8a90a9edc1 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
d67e03f3f5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 757b7fc0ea 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java dec73a783f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
0cf92052bc 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 4f614a899b 
  ql/src/test/queries/clientpositive/alterColumnStats.q PRE-CREATION 
  ql/src/test/queries/clientpositive/alterColumnStatsPart.q PRE-CREATION 
  ql/src/test/results/clientnegative/alter_file_format.q.out e3f3b4cc62 
  ql/src/test/results/clientnegative/unset_table_property.q.out 4aedfc5803 
  ql/src/test/results/clientpositive/alterColumnStats.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/alterColumnStatsPart.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/alter_file_format.q.out ca569be13f 
  ql/src/test/results/clientpositive/alter_partition_clusterby_sortby.q.out 
8bd952514d 
  ql/src/test/results/clientpositive/alter_skewed_table.q.out fefef4c3a3 
  ql/src/test/results/clientpositive/alter_table_add_partition.q.out ae6624f6de 
  ql/src/test/results/clientpositive/alter_table_not_sorted.q.out 5afb7fadee 
  ql/src/test/results/clientpositive/autoColumnStats_3.q.out b916eba9fc 
  ql/src/test/results/clientpositive/autoColumnStats_4.q.out 101cc63ed8 
  ql/src/test/results/clientpositive/binary_output_format.q.out 96afeb679a 
  ql/src/test/results/clientpositive/bucket1.q.out 78571aa765 
  ql/src/test/results/clientpositive/bucket2.q.out a8e9f900a9 
  ql/src/test/results/clientpositive/bucket_map_join_spark1.q.out 5478f1218c 
  ql/src/test/results/clientpositive/bucket_map_join_spark2.q.out e335683604 
  ql/src/test/results/clientpositive/bucket_map_join_spark3.q.out bd5a438917 
  ql/src/test/results/clientpositive/bucketmapjoin5.q.out febc2b62b5 
  ql/src/test/results/clientpositive/bucketmapjoin_negative.q.out ee075c170f 
  ql/src/test/results/clientpositive/bucketmapjoin_negative2.q.out 2728bb7afb 
  
ql/src/test/results/clientpositive/column_names_with_leading_and_trailing_spaces.q.out
 bc9258f11f 
  ql/src/test/results/clientpositive/compustat_avro.q.out 016b8e784d 
  ql/src/test/results/clientpositive/create_alter_list_bucketing_table1.q.out 
7d8e5c1353 
  ql/src/test/results/clientpositive/create_like.q.out ff2e752a95 
  ql/src/test/results/clientpositive/create_like2.q.out 38bd499d28 
  ql/src/test/results/clientpositive/create_like_tbl_props.q.out 19459fbcd1 
  ql/src/test/results/clientpositive/create_like_view.q.out 1e84b461b7 
  ql/src/test/results/clientpositive/create_skewed_table1.q.out e4080b0750 
  ql/src/test/results/clientpositive/create_table_like_stats.q.out f5ec72c030 
  ql/src/test/results/clientpositive/create_with_constraints.q.out 64c3ec6c68 
  ql/src/test/results/clientpositive/database_location.q.out 926db3a490 
  ql/src/test/results/clientpositive/default_file_format.q.out ef0ca52dd2 
  ql/src/test/results/clientpositive/describe_comment_indent.q.out 7b109b3026 
  ql/src/test/results/clientpositive/describe_comment_nonascii.q.out 70f234e7e1 
  ql/src/test/results/clientpositive/describe_formatted_view_partitioned.q.out 
c55239b322 
  ql/src/test/results/clientpositive/describe_syntax.q.out fe05b903d3 
  ql/src/test/results/clientpositive/describe_table.q.out 93b2030839 
  ql/src/test/results/clientpositive/display_colstats_tbllvl.q.out 7b49e4adc0 
  ql/src/test/results/clientpositive/druid_basic1.q.out c16f81e798 
  ql/src/test/results/clientpositive/druid_basic2.q.out 6177d56987 
  ql/src/test/results/clientpositive/druid_intervals.q.out db8633e818 
  ql/src/test/results/clientpositive/druid_timeseries.q.out ef3f8dd9ac 
  ql/src/test/results/clientpositive/druid_topn.q.out 323eda237d 
  ql/src/test/results/clientpositive/groupby_map_ppr.q.out 9be41df8a7 
  ql/src/test/results/clientpositive/groupby_map_ppr_multi_distinct.q.out 
606ab4ed0e 
  ql/src/test/results/clientpositive/groupby_ppr.q.out 8a75c2e44f 
  ql/src/test/results/clientpositive/groupby_ppr_multi_distinct.q.out 
68d91eb8c1 
  ql/src/test/results/clientpositive/groupby_sort_1_23.q.out 92babd79df 
  ql/src/test/results/clientpositive/groupby_sort_6.q.out 80baa664e3 

Re: [VOTE] Should we release hive-storage-2.3.0rc0?

[Alan Gates]

+1.  Did a build with a fresh maven repo, checked the signatures, ran rat.

Alan.

On Apr 27, 2017, 16:08 -0700, Owen O'Malley , wrote:
> All,
> I'd like to make a new release of the storage-api for the upcoming 2.2
> release. Since storage-api rolls releases faster than Hive, this will be
> the storage-api 2.3.0.
>
> Artifacts:
> tag: https://github.com/apache/hive/releases/tag/storage-release-2.3.0rc0
> tar ball: http://home.apache.org/~omalley/hive-storage-2.3.0/
>
> Thanks,
> Owen

Review Request 58934: HIVE-16568: Support complex types in external LLAP InputFormat

[Jason Dere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58934/
---

Review request for hive, Gunther Hagleitner, Prasanth_J, and Siddharth Seth.


Bugs: HIVE-16568
https://issues.apache.org/jira/browse/HIVE-16568


Repository: hive-git


Description
---

- Support list/map/struct types in the LLAPRowInputFormat Schema/TypeDesc
- Support list/map/struct types in the LLAPRowInputFormat Row. Changes in the 
Row getters/setters needed (no longer using Writable).


Diffs
-

  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniLlap.java 
de47412 
  llap-client/src/java/org/apache/hadoop/hive/llap/LlapRowRecordReader.java 
ee92f3e 
  llap-common/src/java/org/apache/hadoop/hive/llap/Row.java a84fadc 
  llap-common/src/java/org/apache/hadoop/hive/llap/TypeDesc.java dda5928 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
9ddbd7e 


Diff: https://reviews.apache.org/r/58934/diff/1/


Testing
---

Added test to TestJdbcWithMiniLlap


Thanks,

Jason Dere

[jira] [Created] (HIVE-16570) improve upon HIVE-16523

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-16570:
---

 Summary: improve upon HIVE-16523
 Key: HIVE-16570
 URL: https://issues.apache.org/jira/browse/HIVE-16570
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


Some things could be faster



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16569) Flaky test: TestAccumuloCliDriver.testCliDriver[accumulo_index]

[Sahil Takiar (JIRA)]

Sahil Takiar created HIVE-16569:
---

 Summary: Flaky test: 
TestAccumuloCliDriver.testCliDriver[accumulo_index]
 Key: HIVE-16569
 URL: https://issues.apache.org/jira/browse/HIVE-16569
 Project: Hive
  Issue Type: Sub-task
  Components: Tests
Reporter: Sahil Takiar


Has been failing pretty consistently:

https://builds.apache.org/job/PreCommit-HIVE-Build/5002/testReport/org.apache.hadoop.hive.cli/TestAccumuloCliDriver/testCliDriver_accumulo_index_/

Exception:

{code}
Error Message

Client execution failed with error code = 1 running "
select * from accumulo_index_test where eff_rating <= 4.5" 
fname=accumulo_index.q 
Stacktrace

java.lang.AssertionError: Client execution failed with error code = 1 running "
select * from accumulo_index_test where eff_rating <= 4.5" 
fname=accumulo_index.q 
at org.junit.Assert.fail(Assert.java:88)
at org.apache.hadoop.hive.ql.QTestUtil.failed(QTestUtil.java:2216)
at 
org.apache.hadoop.hive.cli.control.CoreAccumuloCliDriver.runTest(CoreAccumuloCliDriver.java:94)
at 
org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
at 
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver(TestAccumuloCliDriver.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.apache.hadoop.hive.cli.control.CliAdapter$2$1.evaluate(CliAdapter.java:92)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runners.Suite.runChild(Suite.java:127)
at org.junit.runners.Suite.runChild(Suite.java:26)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at 
org.apache.hadoop.hive.cli.control.CliAdapter$1$1.evaluate(CliAdapter.java:73)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
{code}

Digging through the logs, the real exception is:

{code}
java.io.IOException: java.lang.NumberFormatException: For input string: "4.5"
at 
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:525)
at 
org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:432)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:147)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2171)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:253)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
at 

[jira] [Created] (HIVE-16568) Support complex types in external LLAP InputFormat

[Jason Dere (JIRA)]

Jason Dere created HIVE-16568:
-

 Summary: Support complex types in external LLAP InputFormat
 Key: HIVE-16568
 URL: https://issues.apache.org/jira/browse/HIVE-16568
 Project: Hive
  Issue Type: Sub-task
  Components: llap
Reporter: Jason Dere
Assignee: Jason Dere


Currently just supports primitive types



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 58914: HIVE-16550

[Deepak Jaiswal]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58914/
---

(Updated May 2, 2017, 9:01 p.m.)


Review request for hive, Jason Dere and Sergey Shelukhin.


Changes
---

Implemented review comments.
Added the special case for "None" to skip the runtime filtering.


Bugs: HIVE-16550
https://issues.apache.org/jira/browse/HIVE-16550


Repository: hive-git


Description
---

Semijoin Hints should be able to skip the optimization if needed.
In addition to that, the patch fixes several issues with hints in general such 
as,
- It now works with subqueries and unions.
- Uses a global data structure instead of per QB.


Diffs (updated)
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
 e1a69526bc 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveOpConverter.java
 40c0f3ba2a 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 1b054a7e24 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HintParser.g e110fb33df 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 3a1f821bd3 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java 54e37f7c80 
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java 7bf1c599a5 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java cbbb7d0c94 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 5ea7800528 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDynamicListDesc.java 
3143554ec6 
  ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java 7d4267d6a8 
  ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java 
f45daa8828 
  ql/src/test/queries/clientpositive/semijoin_hint.q 5de0c8c8c1 
  ql/src/test/results/clientpositive/llap/semijoin_hint.q.out bc248930ec 


Diff: https://reviews.apache.org/r/58914/diff/2/

Changes: https://reviews.apache.org/r/58914/diff/1-2/


Testing
---


Thanks,

Deepak Jaiswal

[jira] [Created] (HIVE-16567) NPE when reading Parquet file when getting old timestamp configuration

[Matt McCline (JIRA)]

Matt McCline created HIVE-16567:
---

 Summary: NPE when reading Parquet file when getting old timestamp 
configuration
 Key: HIVE-16567
 URL: https://issues.apache.org/jira/browse/HIVE-16567
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 1.2.0
Reporter: Matt McCline
Priority: Blocker


In branch-1.2, the file 
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ETypeConverter.java 
throws an NPE on line 148:

{code}
 boolean skipConversion = 
Boolean.valueOf(metadata.get(HiveConf.ConfVars.HIVE_PARQUET_TIMESTAMP_SKIP_CONVERSION.varname));
{code}

when the metadata reference is null.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 58914: HIVE-16550

[Deepak Jaiswal]

> On May 2, 2017, 8:15 p.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
> > Lines 11192 (patched)
> > 
> >
> > Can you give the method name something different than OldHints - how 
> > about getHintsFromQB?

sure.


> On May 2, 2017, 8:15 p.m., Jason Dere wrote:
> > ql/src/test/queries/clientpositive/semijoin_hint.q
> > Lines 46 (patched)
> > 
> >
> > Does this query, with no hints, have any runtime filtering? Otherwise 
> > you wouldn't be able to distinguish whether the None option works or not.

Yes it does.


> On May 2, 2017, 8:15 p.m., Jason Dere wrote:
> > ql/src/test/results/clientpositive/llap/semijoin_hint.q.out
> > Lines 233 (patched)
> > 
> >
> > Should the expectedEntries be 5000 per the hint?

Thanks for brining this up. It is happening incase of return path true. 
Investigating.


> On May 2, 2017, 8:15 p.m., Jason Dere wrote:
> > ql/src/test/results/clientpositive/llap/semijoin_hint.q.out
> > Lines 266 (patched)
> > 
> >
> > Should the expectedEntries be 5000 per the hint?

Yes, looking into it.


> On May 2, 2017, 8:15 p.m., Jason Dere wrote:
> > ql/src/test/results/clientpositive/llap/semijoin_hint.q.out
> > Lines 300 (patched)
> > 
> >
> > Should the expectedEntries be 5000 per the hint?

ditto


- Deepak


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58914/#review173628
---


On May 2, 2017, 5:47 a.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58914/
> ---
> 
> (Updated May 2, 2017, 5:47 a.m.)
> 
> 
> Review request for hive, Jason Dere and Sergey Shelukhin.
> 
> 
> Bugs: HIVE-16550
> https://issues.apache.org/jira/browse/HIVE-16550
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Semijoin Hints should be able to skip the optimization if needed.
> In addition to that, the patch fixes several issues with hints in general 
> such as,
> - It now works with subqueries and unions.
> - Uses a global data structure instead of per QB.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
>  e1a69526bc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveOpConverter.java
>  d375d1b58d 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 1b054a7e24 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HintParser.g e110fb33df 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 3a1f821bd3 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java 54e37f7c80 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java 7bf1c599a5 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 654f3b1772 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 5ea7800528 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDynamicListDesc.java 
> 3143554ec6 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java 032c7bb28d 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java 
> f45daa8828 
>   ql/src/test/queries/clientpositive/semijoin_hint.q 5de0c8c8c1 
>   ql/src/test/results/clientpositive/llap/semijoin_hint.q.out bc248930ec 
> 
> 
> Diff: https://reviews.apache.org/r/58914/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

Re: Review Request 58914: HIVE-16550

[Jason Dere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58914/#review173628
---




ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
Lines 11192 (patched)


Can you give the method name something different than OldHints - how about 
getHintsFromQB?



ql/src/test/queries/clientpositive/semijoin_hint.q
Lines 46 (patched)


Does this query, with no hints, have any runtime filtering? Otherwise you 
wouldn't be able to distinguish whether the None option works or not.



ql/src/test/results/clientpositive/llap/semijoin_hint.q.out
Lines 233 (patched)


Should the expectedEntries be 5000 per the hint?



ql/src/test/results/clientpositive/llap/semijoin_hint.q.out
Lines 266 (patched)


Should the expectedEntries be 5000 per the hint?



ql/src/test/results/clientpositive/llap/semijoin_hint.q.out
Lines 300 (patched)


Should the expectedEntries be 5000 per the hint?


- Jason Dere


On May 2, 2017, 5:47 a.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58914/
> ---
> 
> (Updated May 2, 2017, 5:47 a.m.)
> 
> 
> Review request for hive, Jason Dere and Sergey Shelukhin.
> 
> 
> Bugs: HIVE-16550
> https://issues.apache.org/jira/browse/HIVE-16550
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Semijoin Hints should be able to skip the optimization if needed.
> In addition to that, the patch fixes several issues with hints in general 
> such as,
> - It now works with subqueries and unions.
> - Uses a global data structure instead of per QB.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
>  e1a69526bc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveOpConverter.java
>  d375d1b58d 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 1b054a7e24 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HintParser.g e110fb33df 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 3a1f821bd3 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java 54e37f7c80 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java 7bf1c599a5 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 654f3b1772 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 5ea7800528 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDynamicListDesc.java 
> 3143554ec6 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java 032c7bb28d 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java 
> f45daa8828 
>   ql/src/test/queries/clientpositive/semijoin_hint.q 5de0c8c8c1 
>   ql/src/test/results/clientpositive/llap/semijoin_hint.q.out bc248930ec 
> 
> 
> Diff: https://reviews.apache.org/r/58914/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

Re: [VOTE] Apache Hive 2.3.0 Release Candidate 0

[Sergio Pena]

Hi Pengcheng,

There is a request from the HDFS team to revert the patch committed on
HIVE-16047 from
our code because it might cause problems when future Hadoop versions are
released due to being a
private API on Hadoop. This API method signature has been changed between
releases, and
we don't want to have additional shims to support future Hadoop versions
just for this method.

I'd like to revert it from 2.3.0 release before doing the release. It is
marked as being fixed on 2.2 but it is not cherry-picked on branch-2.2 but
branch-2.3.

Do you agree?

- Sergio

On Fri, Apr 28, 2017 at 1:40 PM, Pengcheng Xiong  wrote:

> Withdraw the VOTE on candidate 0. Will propose candidate 1 soon. Thanks.
>
> On Thu, Apr 27, 2017 at 8:10 PM, Owen O'Malley 
> wrote:
>
> > -1 you need a release of storage-API first.
> >
> > .. Owen
> >
> > > On Apr 27, 2017, at 17:43, Pengcheng Xiong  wrote:
> > >
> > > Apache Hive 2.3.0 Release Candidate 0 is available here:
> > > http://home.apache.org/~pxiong/apache-hive-2.3.0-rc0/
> > >
> > >
> > > Maven artifacts are available here:
> > > https://repository.apache.org/content/repositories/orgapachehive-1073/
> > >
> > >
> > > Source tag for RC0 is at:
> > >
> > > https://github.com/apache/hive/releases/tag/release-2.3.0-rc0
> > >
> > > Voting will conclude in 72 hours.
> > >
> > > Hive PMC Members: Please test and vote.
> > >
> > > Thanks.
> >
>

Re: Review Request 58914: HIVE-16550

[Deepak Jaiswal]

> On May 2, 2017, 6:35 p.m., Vineet Garg wrote:
> > ql/src/test/queries/clientpositive/semijoin_hint.q
> > Line 38 (original), 38 (patched)
> > 
> >
> > Why is returnpath on?
> > This feature is not yet fully developed/supported and is off by 
> > default. 
> > You might want to try the same queries with this feature off.
> 
> Deepak Jaiswal wrote:
> We have to make sure we dont break it.
> There are tests below with the feature turned off.
> 
> Vineet Garg wrote:
> Ok in that case can you add the same tests with returnpath off? I don't 
> see union all query with returnpath off

sure.


- Deepak


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58914/#review173558
---


On May 2, 2017, 5:47 a.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58914/
> ---
> 
> (Updated May 2, 2017, 5:47 a.m.)
> 
> 
> Review request for hive, Jason Dere and Sergey Shelukhin.
> 
> 
> Bugs: HIVE-16550
> https://issues.apache.org/jira/browse/HIVE-16550
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Semijoin Hints should be able to skip the optimization if needed.
> In addition to that, the patch fixes several issues with hints in general 
> such as,
> - It now works with subqueries and unions.
> - Uses a global data structure instead of per QB.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
>  e1a69526bc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveOpConverter.java
>  d375d1b58d 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 1b054a7e24 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HintParser.g e110fb33df 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 3a1f821bd3 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java 54e37f7c80 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java 7bf1c599a5 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 654f3b1772 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 5ea7800528 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDynamicListDesc.java 
> 3143554ec6 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java 032c7bb28d 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java 
> f45daa8828 
>   ql/src/test/queries/clientpositive/semijoin_hint.q 5de0c8c8c1 
>   ql/src/test/results/clientpositive/llap/semijoin_hint.q.out bc248930ec 
> 
> 
> Diff: https://reviews.apache.org/r/58914/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

Re: Review Request 58914: HIVE-16550

[Vineet Garg]

> On May 2, 2017, 6:35 p.m., Vineet Garg wrote:
> > ql/src/test/queries/clientpositive/semijoin_hint.q
> > Line 38 (original), 38 (patched)
> > 
> >
> > Why is returnpath on?
> > This feature is not yet fully developed/supported and is off by 
> > default. 
> > You might want to try the same queries with this feature off.
> 
> Deepak Jaiswal wrote:
> We have to make sure we dont break it.
> There are tests below with the feature turned off.

Ok in that case can you add the same tests with returnpath off? I don't see 
union all query with returnpath off


- Vineet


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58914/#review173558
---


On May 2, 2017, 5:47 a.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58914/
> ---
> 
> (Updated May 2, 2017, 5:47 a.m.)
> 
> 
> Review request for hive, Jason Dere and Sergey Shelukhin.
> 
> 
> Bugs: HIVE-16550
> https://issues.apache.org/jira/browse/HIVE-16550
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Semijoin Hints should be able to skip the optimization if needed.
> In addition to that, the patch fixes several issues with hints in general 
> such as,
> - It now works with subqueries and unions.
> - Uses a global data structure instead of per QB.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
>  e1a69526bc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveOpConverter.java
>  d375d1b58d 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 1b054a7e24 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HintParser.g e110fb33df 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 3a1f821bd3 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java 54e37f7c80 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java 7bf1c599a5 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 654f3b1772 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 5ea7800528 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDynamicListDesc.java 
> 3143554ec6 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java 032c7bb28d 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java 
> f45daa8828 
>   ql/src/test/queries/clientpositive/semijoin_hint.q 5de0c8c8c1 
>   ql/src/test/results/clientpositive/llap/semijoin_hint.q.out bc248930ec 
> 
> 
> Diff: https://reviews.apache.org/r/58914/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

Re: Review Request 58865: HIVE-16552: Limit the number of tasks a Spark job may contain

[Xuefu Zhang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58865/
---

(Updated May 2, 2017, 6:49 p.m.)


Review request for hive.


Bugs: HIVE-16552
https://issues.apache.org/jira/browse/HIVE-16552


Repository: hive-git


Description
---

See JIRA description


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 84398c6 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java 32a7730 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/RemoteSparkJobMonitor.java
 dd73f3e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java 
0b224f2 


Diff: https://reviews.apache.org/r/58865/diff/3/

Changes: https://reviews.apache.org/r/58865/diff/2-3/


Testing
---

Test locally


Thanks,

Xuefu Zhang

Re: Review Request 58914: HIVE-16550

[Deepak Jaiswal]

> On May 2, 2017, 6:35 p.m., Vineet Garg wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
> > Line 338 (original), 338 (patched)
> > 
> >
> > Can you add comments to explain why are we doing this i.e. why are we 
> > propagating hints?

Will do it thanks.


> On May 2, 2017, 6:35 p.m., Vineet Garg wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
> > Lines 368 (patched)
> > 
> >
> > If I understand it correctly before generating calcite plan (i.e. 
> > calling getOptimizedAST) QB has hints but after generating optimized AST QB 
> > is reset and lose hints? That is why you are propagating hints?

Yes.


> On May 2, 2017, 6:35 p.m., Vineet Garg wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
> > Line 408 (original), 413 (patched)
> > 
> >
> > What happens in this case? Why is it necessary to log warning here?

I think just paranoia. It shouldn't happen, however, if it does, we log it.


> On May 2, 2017, 6:35 p.m., Vineet Garg wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java
> > Line 444 (original), 444 (patched)
> > 
> >
> > Is this because we can have hint now? If so can you rather check for 
> > hint and continue instead of continuing for all nodes

Makes sense. Will do.


> On May 2, 2017, 6:35 p.m., Vineet Garg wrote:
> > ql/src/test/queries/clientpositive/semijoin_hint.q
> > Line 38 (original), 38 (patched)
> > 
> >
> > Why is returnpath on?
> > This feature is not yet fully developed/supported and is off by 
> > default. 
> > You might want to try the same queries with this feature off.

We have to make sure we dont break it.
There are tests below with the feature turned off.


- Deepak


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58914/#review173558
---


On May 2, 2017, 5:47 a.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58914/
> ---
> 
> (Updated May 2, 2017, 5:47 a.m.)
> 
> 
> Review request for hive, Jason Dere and Sergey Shelukhin.
> 
> 
> Bugs: HIVE-16550
> https://issues.apache.org/jira/browse/HIVE-16550
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Semijoin Hints should be able to skip the optimization if needed.
> In addition to that, the patch fixes several issues with hints in general 
> such as,
> - It now works with subqueries and unions.
> - Uses a global data structure instead of per QB.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
>  e1a69526bc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveOpConverter.java
>  d375d1b58d 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 1b054a7e24 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HintParser.g e110fb33df 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 3a1f821bd3 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java 54e37f7c80 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java 7bf1c599a5 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 654f3b1772 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 5ea7800528 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDynamicListDesc.java 
> 3143554ec6 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java 032c7bb28d 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java 
> f45daa8828 
>   ql/src/test/queries/clientpositive/semijoin_hint.q 5de0c8c8c1 
>   ql/src/test/results/clientpositive/llap/semijoin_hint.q.out bc248930ec 
> 
> 
> Diff: https://reviews.apache.org/r/58914/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

Re: Review Request 58914: HIVE-16550

[Vineet Garg]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58914/#review173558
---




ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
Line 338 (original), 338 (patched)


Can you add comments to explain why are we doing this i.e. why are we 
propagating hints?



ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
Lines 368 (patched)


If I understand it correctly before generating calcite plan (i.e. calling 
getOptimizedAST) QB has hints but after generating optimized AST QB is reset 
and lose hints? That is why you are propagating hints?



ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
Line 408 (original), 413 (patched)


What happens in this case? Why is it necessary to log warning here?



ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java
Line 444 (original), 444 (patched)


Is this because we can have hint now? If so can you rather check for hint 
and continue instead of continuing for all nodes



ql/src/test/queries/clientpositive/semijoin_hint.q
Line 38 (original), 38 (patched)


Why is returnpath on?
This feature is not yet fully developed/supported and is off by default. 
You might want to try the same queries with this feature off.


- Vineet Garg


On May 2, 2017, 5:47 a.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58914/
> ---
> 
> (Updated May 2, 2017, 5:47 a.m.)
> 
> 
> Review request for hive, Jason Dere and Sergey Shelukhin.
> 
> 
> Bugs: HIVE-16550
> https://issues.apache.org/jira/browse/HIVE-16550
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Semijoin Hints should be able to skip the optimization if needed.
> In addition to that, the patch fixes several issues with hints in general 
> such as,
> - It now works with subqueries and unions.
> - Uses a global data structure instead of per QB.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
>  e1a69526bc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveOpConverter.java
>  d375d1b58d 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 1b054a7e24 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HintParser.g e110fb33df 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 3a1f821bd3 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java 54e37f7c80 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java 7bf1c599a5 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 654f3b1772 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 5ea7800528 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDynamicListDesc.java 
> 3143554ec6 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java 032c7bb28d 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java 
> f45daa8828 
>   ql/src/test/queries/clientpositive/semijoin_hint.q 5de0c8c8c1 
>   ql/src/test/results/clientpositive/llap/semijoin_hint.q.out bc248930ec 
> 
> 
> Diff: https://reviews.apache.org/r/58914/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

Re: Review Request 58501: HIVE-16469: Parquet timestamp table property is not always taken into account

[Sergio Pena]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58501/#review173610
---




ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java
Lines 333 (patched)


If getNextSplits() already sets the table property if it is not set, why 
are we doing it again here?



ql/src/java/org/apache/hadoop/hive/ql/exec/StatsNoJobTask.java
Lines 262 (patched)


I've seen this check a few times on the code. Shouldn't be good to create a 
static method that wraps this check? like 
ParquetHiveSerDe.isParquetTable(table)?



ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java
Line 72 (original), 72 (patched)


How does this work? I don't understand this change.



ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java
Lines 72 (patched)


Can this new code be wrapped in another method?



ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ParquetRecordReaderBase.java
Line 181 (original), 181 (patched)


Is this compatible with old parquet tables? if the property is not set, 
then the validateTimeZonemight fail, right? If so, do we want to fail 
reading tables that do not have a property set?

Something else to consider, if a user sets a timezone improperly in a 
different tool or something  happened that we got an invalid timezone, then 
do we want to fail when reading those files? Just  wondering this scenario, 
no need to fix it right away.



ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetTableUtils.java
Lines 32 (patched)


Could you write the information about parameters?



ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetTableUtils.java
Lines 35 (patched)


Why is Map used instead of Map? Aren't all table 
properties key, value string pairs?

Also, the ensureTablePropertySet() name seems not related to what we want 
to do. I thought it was going to throw an exception if the property was not 
set, but it is setting the value on the JobConf. Should we use a different 
name, such as setParquetTimeZoneIfNotSet(),  setParquetTimeZoneIfAbsent() 
or something like that helps us understand quickly without looking at the 
javadoc.



ql/src/java/org/apache/hadoop/hive/ql/io/parquet/timestamp/NanoTimeUtils.java
Lines 168 (patched)


Shouldn't we throw an IllegalArgumentException here? The same for line 174?

I think it makes more sense to use the above exception when arguments are 
not valid.



ql/src/test/org/apache/hadoop/hive/ql/io/parquet/AbstractTestParquetDirect.java
Line 24 (original), 24 (patched)


Keep the standard here. Let's import each class instead of all.


- Sergio Pena


On April 20, 2017, 2:11 p.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58501/
> ---
> 
> (Updated April 20, 2017, 2:11 p.m.)
> 
> 
> Review request for hive, Sergio Pena and Zoltan Ivanfi.
> 
> 
> Bugs: HIVE-16469
> https://issues.apache.org/jira/browse/HIVE-16469
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16469: Parquet timestamp table property is not always taken into account
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
> 917e565f28b2c9aaea18033ea3b6b20fa41fcd0a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java 
> 004bb2f60299a0635b8f9ca7649ead00b8e16d08 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/StatsNoJobTask.java 
> 9c3a664b9aea2d6e050ffe2d7626127827dbc52a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java 
> 1bd4db7805689ae1f91921ffbb5ff7da59f4bf60 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java
>  f4fadbb61bf45f62945700284c0b050f0984b696 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ParquetRecordReaderBase.java 
> 2954601ce5bb25905cdb29ca0ca4551c2ca12b95 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetTableUtils.java 
> b339cc4347eea143dca2f6d98f9aaafdc427 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/timestamp/NanoTimeUtils.java 
> dbd6fb3d0bc8c753abf86e99b52377617f248b5a 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/AbstractTestParquetDirect.java
>  

Re: Review Request 58865: HIVE-16552: Limit the number of tasks a Spark job may contain

[Chao Sun]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58865/#review173407
---




common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 3341 (patched)


nit: add new line after "may have."
nit: maximu -> maximum



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java
Lines 135 (patched)


Maybe also log the actual limit number?



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/RemoteSparkJobMonitor.java
Lines 106 (patched)


Maybe we should skip counting this if `sparkJobMaxTaskCount` is -1?



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/RemoteSparkJobMonitor.java
Lines 106 (patched)


Also, we don't need to compute this if `sparkJobMaxTaskCount` is -1.


- Chao Sun


On May 1, 2017, 5:13 p.m., Xuefu Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58865/
> ---
> 
> (Updated May 1, 2017, 5:13 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-16552
> https://issues.apache.org/jira/browse/HIVE-16552
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See JIRA description
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d3ea824 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java 32a7730 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/RemoteSparkJobMonitor.java
>  dd73f3e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java 
> 0b224f2 
> 
> 
> Diff: https://reviews.apache.org/r/58865/diff/2/
> 
> 
> Testing
> ---
> 
> Test locally
> 
> 
> Thanks,
> 
> Xuefu Zhang
> 
>

Re: Review Request 58777: HIVE-16546: LLAP: Fail map join tasks if hash table memory exceeds threshold

[j . prasanth . j]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58777/
---

(Updated May 2, 2017, 8:22 a.m.)


Review request for hive, Gunther Hagleitner, Sergey Shelukhin, and Siddharth 
Seth.


Changes
---

Fix for integer overflows.


Bugs: HIVE-16546
https://issues.apache.org/jira/browse/HIVE-16546


Repository: hive-git


Description
---

HIVE-16546: LLAP: Fail map join tasks if hash table memory exceeds threshold


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/MemoryEstimate.java 
PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 3400560 
  
llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java
 ff6e7ce 
  ql/src/gen/vectorization/UDAFTemplates/VectorUDAFAvg.txt 4393c3b 
  ql/src/gen/vectorization/UDAFTemplates/VectorUDAFMinMax.txt 7468c2f 
  ql/src/gen/vectorization/UDAFTemplates/VectorUDAFMinMaxDecimal.txt 57b7ea5 
  ql/src/gen/vectorization/UDAFTemplates/VectorUDAFMinMaxIntervalDayTime.txt 
749e97e 
  ql/src/gen/vectorization/UDAFTemplates/VectorUDAFMinMaxString.txt 9dfc147 
  ql/src/gen/vectorization/UDAFTemplates/VectorUDAFMinMaxTimestamp.txt 32ecb34 
  ql/src/gen/vectorization/UDAFTemplates/VectorUDAFSum.txt bd0f14d 
  ql/src/gen/vectorization/UDAFTemplates/VectorUDAFVar.txt dc9d4b1 
  ql/src/gen/vectorization/UDAFTemplates/VectorUDAFVarDecimal.txt 01062a9 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionError.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionException.java
 dbe00b6 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionHandler.java
 7fc3226 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 595d1bd 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
 04e24bd 
  ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 
a3bccc6 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HybridHashTableContainer.java
 04e89e8 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java
 c86e5f5 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java
 6d71fef 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 7b13e90 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java 486d43a 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorAggregationBufferBatch.java
 630046d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorGroupByOperator.java 
5b4c7c3 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorAggregateExpression.java
 0866f63 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFAvgDecimal.java
 74e25ae 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFAvgTimestamp.java
 483d9dc 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFBloomFilter.java
 2139eae 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFBloomFilterMerge.java
 d2446d5 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFCount.java
 494febc 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFCountMerge.java
 dec88cb 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFCountStar.java
 337ba0a 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFStdPopTimestamp.java
 8cd3506 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFStdSampTimestamp.java
 61d6977 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFSumDecimal.java
 b10f66f 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFVarPopTimestamp.java
 2709b07 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFVarSampTimestamp.java
 03dce1e 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashMap.java
 6242daf 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashMultiSet.java
 1a41961 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashSet.java
 331867c 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashTable.java
 b93e977 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTable.java
 b6db3bc 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTableLoader.java
 49ecdd1 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastKeyStore.java
 be51693