Re: [ANNOUNCE] New committer: Nishant Bangarwa

[Sankar Hariappan]

Congrats Nishant!

Best regards
Sankar









On 15/10/18, 12:45 PM, "Ashutosh Chauhan"  wrote:

>Apache Hive's Project Management Committee (PMC) has invited Nishant
>Bangarwa
>to become a committer, and we are pleased to announce that he has accepted.
>
>Nishant, welcome, thank you for your contributions, and we look forward your
>further interactions with the community!
>
>Ashutosh Chauhan (on behalf of the Apache Hive PMC)

Re: [ANNOUNCE] New committer: Nishant Bangarwa

[Lefty Leverenz]

Congratulations Nishant!

-- Lefty


On Mon, Oct 15, 2018 at 3:15 AM Ashutosh Chauhan 
wrote:

> Apache Hive's Project Management Committee (PMC) has invited Nishant
> Bangarwa
> to become a committer, and we are pleased to announce that he has accepted.
>
> Nishant, welcome, thank you for your contributions, and we look forward
> your
> further interactions with the community!
>
> Ashutosh Chauhan (on behalf of the Apache Hive PMC)
>

Review Request 69078: HIVE-20767

[Jesús Camacho Rodríguez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69078/
---

Review request for hive, Ashutosh Chauhan and Vineet Garg.


Bugs: HIVE-20767
https://issues.apache.org/jira/browse/HIVE-20767


Repository: hive-git


Description
---

HIVE-20767


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinProjectTransposeRule.java
 e6844326068210e7ab7364ec9f3ec60908b36e88 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
22f3266c87f1d42c254893b424b68e757fb2953b 
  ql/src/test/results/clientnegative/subquery_scalar_multi_rows.q.out 
0a780db7ef98ddf14fc18a90f2b628a13337bcda 
  ql/src/test/results/clientpositive/bool_unknown.q.out 
8e9b48ccafce2093e91d7035fdf581018bab1979 
  ql/src/test/results/clientpositive/llap/lineage2.q.out 
d32f490a704e1ba6bdc54f4d54dff028c9ca974c 
  ql/src/test/results/clientpositive/llap/mapjoin_hint.q.out 
ac505a5c1e47211260a795390a3ca7c45ee30c01 
  ql/src/test/results/clientpositive/llap/materialized_view_rewrite_7.q.out 
6f00a5c82b2c2784d2576222d6eb4bd9faf71bea 
  ql/src/test/results/clientpositive/llap/multiMapJoin1.q.out 
ae821f600b3e9bf3128e93c0a4097b438f26f35b 
  ql/src/test/results/clientpositive/llap/subquery_scalar.q.out 
166bd60ce25712021c06d3506246ff87d35058c4 
  ql/src/test/results/clientpositive/llap/subquery_select.q.out 
854d215a2382e5b09714c7b8b9916cf048458f66 
  ql/src/test/results/clientpositive/perf/spark/query54.q.out 
f10250f307cc58486dbb202c95b2c8af7943e528 
  ql/src/test/results/clientpositive/perf/tez/query54.q.out 
1c17d2a53a1be39b28d01412fe6b13878b1680f3 
  ql/src/test/results/clientpositive/spark/subquery_scalar.q.out 
b3252f54158d5401c3d10b0978af69b9dbba1290 
  ql/src/test/results/clientpositive/spark/subquery_select.q.out 
ad7c8a3fc795004ea5dbd2bc4355de24b73352be 


Diff: https://reviews.apache.org/r/69078/diff/1/


Testing
---


Thanks,

Jesús Camacho Rodríguez

Re: Review Request 69077: HIVE-20748

[Jesús Camacho Rodríguez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69077/
---

(Updated Oct. 19, 2018, 12:17 a.m.)


Review request for hive and Ashutosh Chauhan.


Summary (updated)
-

HIVE-20748


Bugs: HIVE-20748
https://issues.apache.org/jira/browse/HIVE-20748


Repository: hive-git


Description (updated)
---

HIVE-20748


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
807f159daa98d40e667914adc6c53fb8ecabf998 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOpMaterializationValidator.java
 df216e7555bff4756130f5e097bdb6b0e5e7eef5 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOptAutomaticRewritingMaterializationValidator.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
22f3266c87f1d42c254893b424b68e757fb2953b 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java 
be1c59f93272352705731c8c7a02433c7ac3d6dc 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
eed875e7a4475f207727d5d536521fdba0c329fb 
  ql/src/test/queries/clientnegative/materialized_view_no_cbo_rewrite.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/materialized_view_no_cbo_rewrite_2.q 
PRE-CREATION 
  
ql/src/test/queries/clientnegative/materialized_view_no_supported_op_rewrite.q 
PRE-CREATION 
  
ql/src/test/queries/clientnegative/materialized_view_no_supported_op_rewrite_2.q
 PRE-CREATION 
  ql/src/test/results/clientnegative/materialized_view_no_cbo_rewrite.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/materialized_view_no_cbo_rewrite_2.q.out 
PRE-CREATION 
  
ql/src/test/results/clientnegative/materialized_view_no_supported_op_rewrite.q.out
 PRE-CREATION 
  
ql/src/test/results/clientnegative/materialized_view_no_supported_op_rewrite_2.q.out
 PRE-CREATION 


Diff: https://reviews.apache.org/r/69077/diff/1/


Testing
---


Thanks,

Jesús Camacho Rodríguez

Review Request 69077: HIVE-20748 (still pending tests)

[Jesús Camacho Rodríguez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69077/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-20748
https://issues.apache.org/jira/browse/HIVE-20748


Repository: hive-git


Description
---

HIVE-20748 (still pending tests)


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
807f159daa98d40e667914adc6c53fb8ecabf998 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOpMaterializationValidator.java
 df216e7555bff4756130f5e097bdb6b0e5e7eef5 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOptAutomaticRewritingMaterializationValidator.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
22f3266c87f1d42c254893b424b68e757fb2953b 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java 
be1c59f93272352705731c8c7a02433c7ac3d6dc 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
eed875e7a4475f207727d5d536521fdba0c329fb 
  ql/src/test/queries/clientnegative/materialized_view_no_cbo_rewrite.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/materialized_view_no_cbo_rewrite_2.q 
PRE-CREATION 
  
ql/src/test/queries/clientnegative/materialized_view_no_supported_op_rewrite.q 
PRE-CREATION 
  
ql/src/test/queries/clientnegative/materialized_view_no_supported_op_rewrite_2.q
 PRE-CREATION 
  ql/src/test/results/clientnegative/materialized_view_no_cbo_rewrite.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/materialized_view_no_cbo_rewrite_2.q.out 
PRE-CREATION 
  
ql/src/test/results/clientnegative/materialized_view_no_supported_op_rewrite.q.out
 PRE-CREATION 
  
ql/src/test/results/clientnegative/materialized_view_no_supported_op_rewrite_2.q.out
 PRE-CREATION 


Diff: https://reviews.apache.org/r/69077/diff/1/


Testing
---


Thanks,

Jesús Camacho Rodríguez

Re: Review Request 69019: HIVE-20617 Fix type of constants in IN expressions to have correct type

[Vineet Garg]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69019/#review209764
---




ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java
Line 1171 (original), 1177 (patched)


wrong comparisions spelling



ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java
Lines 1296 (patched)


Should be this be more appropriate as assert? If this is not the case it 
sounds like an invalid IN



ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java
Lines 1323 (patched)


assert?



ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java
Lines 1329 (patched)


Can you add test case e.g. select 'expected 3',count(*) from ax where (s,t) 
in (('a','a'),(null, 'b'));

I don't think returning null for whole struct is correct, null could be 
representing valid value and need to be typed appropriately.



ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java
Lines 1340 (patched)


These utility methods would be more suited in ExprNodeDescUtils



ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java
Line 1287 (original), 1400 (patched)


Currently we don't handle case such as (s,t) IN ('c', 'b'). Here right side 
is not struct but has two expr node constant but left side is struct


- Vineet Garg


On Oct. 15, 2018, 6:05 a.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69019/
> ---
> 
> (Updated Oct. 15, 2018, 6:05 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-20617
> https://issues.apache.org/jira/browse/HIVE-20617
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> For IN expressions the types were never corrected; and pointlookupoptimizer 
> was probably leaving behind fields already which were uncomparable; 
> HIVE-20296 exposed it further by changing the minimal number from  32 to 2.
> 
> This change generalizes the retyping of constants to also run it for the IN 
> operator ; and also for struct-s.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
> 4968d16876c5c9cc36ec9a3ec48c2740c2c67dcd 
>   ql/src/test/queries/clientpositive/in_typecheck1.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/in_typecheck2.q PRE-CREATION 
>   ql/src/test/results/clientpositive/alter_partition_coltype.q.out 
> 5727f0a65c6e4736f41017e4e962d932dedbd6bd 
>   ql/src/test/results/clientpositive/cbo_rp_simple_select.q.out 
> 43cb5ab89fdebde8be168d7837d8e54a38f4d10b 
>   ql/src/test/results/clientpositive/cbo_simple_select.q.out 
> 2073c6b802a1ae0ff4228a86f18ec366ff92ab02 
>   ql/src/test/results/clientpositive/in_typecheck1.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/in_typecheck2.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/infer_const_type.q.out 
> 4129bd0c715635eb83c0c0d248eb43a5779c7be9 
>   ql/src/test/results/clientpositive/join45.q.out 
> 77dbaa2cd8b5be7158545c696b30dc1068238f91 
>   ql/src/test/results/clientpositive/join47.q.out 
> 2536f7f4b6e9295d1177632b7f32f0b66974e3a4 
>   ql/src/test/results/clientpositive/llap/cbo_simple_select.q.out 
> e61300b5c853eb733d4443c047344e3fc6fe0ff3 
>   ql/src/test/results/clientpositive/llap/dec_str.q.out 
> 3b7f92d735db79f7c4c0d96debe7fff8e3c05f11 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out 
> bc1f97dd49ccc905d5e32d5a02a62bb692d444a6 
>   ql/src/test/results/clientpositive/llap/lineage3.q.out 
> cf388161272002ad6097839ceaea2bcfbcf9b7ef 
>   ql/src/test/results/clientpositive/llap/mapjoin_hint.q.out 
> 3c6270a05240097e9645e664ed30e0568052d98e 
>   ql/src/test/results/clientpositive/llap/subquery_scalar.q.out 
> feadbcd874818a78e2cc30b86cdebc1e4cb6a04f 
>   ql/src/test/results/clientpositive/llap/vectorization_13.q.out 
> 398cb56915f1b24b7c4dc325b60cb114d7ff2b8c 
>   ql/src/test/results/clientpositive/llap/vectorization_6.q.out 
> 70542ac7bd69e46098cf8158cae347e6c896c5b2 
>   ql/src/test/results/clientpositive/llap/vectorization_8.q.out 
> 662409d4f148cc3da5c4f788ddf59c6f40ede572 
>   ql/src/test/results/clientpositive/llap/vectorization_short_regress.q.out 
> a59a586144fc9dc14ac2fa87177c189feae47402 
>   ql/src/test/results/clientpositive/mapjoin47.q.out 
> c42094d7858fa70626a5184485a13fdacd45be7c 
>   ql/src/test/results/clientpositive/parquet_vectorization_13.q.out 
> 

[jira] [Created] (HIVE-20775) Factor cost of each SJ reduction when costing a follow-up reduction

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-20775:
--

 Summary: Factor cost of each SJ reduction when costing a follow-up 
reduction
 Key: HIVE-20775
 URL: https://issues.apache.org/jira/browse/HIVE-20775
 Project: Hive
  Issue Type: Bug
  Components: Physical Optimizer
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


Currently, while costing the SJ in a plan, the stats of the a TS that is 
reduced by a SJ are not adjusted after we have decided to keep a SJ in the 
tree. Ideally, we could adjust the stats to take into account decisions that 
have already been made.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20774) use less calls for GetTables operation

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20774:
---

 Summary: use less calls for GetTables operation
 Key: HIVE-20774
 URL: https://issues.apache.org/jira/browse/HIVE-20774
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20773) Query result cache might contain stale MV data

[Oliver Draese (JIRA)]

Oliver Draese created HIVE-20773:


 Summary: Query result cache might contain stale MV data
 Key: HIVE-20773
 URL: https://issues.apache.org/jira/browse/HIVE-20773
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 4.0.0
Reporter: Oliver Draese
Assignee: Jesus Camacho Rodriguez






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 69073: HIVE-20772 record per-task CPU counters in LLAP

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69073/
---

Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

.


Diffs
-

  llap-common/src/java/org/apache/hadoop/hive/llap/LlapUtil.java 50c0e22837 
  llap-common/src/java/org/apache/hadoop/hive/llap/counters/LlapIOCounters.java 
059d5b9ae3 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheCounters.java
 91df036bfd 
  
llap-server/src/java/org/apache/hadoop/hive/llap/counters/QueryFragmentCounters.java
 be4dfad95c 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/StatsRecordingThreadPool.java
 27462e1bcb 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
 27a5b0f3e4 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/EncodedDataConsumer.java
 d5c2d48db1 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
 40248a37f8 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 4f5b0a9e65 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
 658bc7d621 


Diff: https://reviews.apache.org/r/69073/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-20772) record per-task CPU counters in LLAP

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20772:
---

 Summary: record per-task CPU counters in LLAP
 Key: HIVE-20772
 URL: https://issues.apache.org/jira/browse/HIVE-20772
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69054: HIVE-20740 : Remove global lock in ObjectStore.setConf method

[Vihang Karajgaonkar via Review Board]

> On Oct. 18, 2018, 2:33 p.m., Karthik Manamcheri wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> > Line 244 (original), 341 (patched)
> > 
> >
> > You are ignoring the return value? Should you have 
> > pmf=getUpdatedPmfIfNeeded(..)?

The pmf is updated by the method if needed, so we don't need to use the return 
value. Will rename the method to updatePmfIfNeeded to make it more readable.


> On Oct. 18, 2018, 2:33 p.m., Karthik Manamcheri wrote:
> > standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
> > Lines 927 (patched)
> > 
> >
> > Is this testing the new read/write lock behavior? As in, would this 
> > test have failed if you had the previous behavior of a global lock?

This is performance fix. There was nothing broken in the previous 
implementation so its hard to come up with a test which break on the previous 
code. The purpose of this test to cover the newly added code so that if there 
are any obvious bugs they will be seen. If you have more ideas on how can we 
add more tests please suggest and I would be happy to add them.


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69054/#review209742
---


On Oct. 16, 2018, 10:47 p.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69054/
> ---
> 
> (Updated Oct. 16, 2018, 10:47 p.m.)
> 
> 
> Review request for hive, Andrew Sherman, Alan Gates, and Peter Vary.
> 
> 
> Bugs: HIVE-20740
> https://issues.apache.org/jira/browse/HIVE-20740
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20740 : Remove global lock in ObjectStore.setConf method
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  66977d79c946f1ac57aacfbe8704d37bfbac3ea3 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
>  b74c3048fa2e18adc7f0d7cc813a180d4466fa36 
> 
> 
> Diff: https://reviews.apache.org/r/69054/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vihang Karajgaonkar
> 
>

Re: Review Request 69022: HIVE-20737: Local SparkContext is shared between user sessions and should be closed only when there is no active

[denys kuzmenko via Review Board]

> On Oct. 16, 2018, 1:47 p.m., Sahil Takiar wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/session/SparkSessionImpl.java
> > Line 125 (original), 120 (patched)
> > 
> >
> > we might need to re-think how we are synchronizing this method a bit. I 
> > think we want to support the use case where we call `close()` while 
> > `open()` is being run. The offers a way for the user to cancel a session 
> > while it is being opened, which can be useful if opening a session takes a 
> > long time, which can happen on a busy cluster where there aren't enough 
> > resources to open a session.
> > 
> > fixing that might be out of the scope of this JIRA, so I would 
> > recommend using a separate lock to guard against multiple users calling 
> > open on the same session.
> 
> Sahil Takiar wrote:
> Tracking the aformentioned fix in HIVE-20519, unless you want to fix it 
> in this patch.
> 
> denys kuzmenko wrote:
> i think it should be addressed in another JIRA, right now we need to have 
> working at least basic use-case
> 
> Sahil Takiar wrote:
> okay, still recommend using a separate lock
> 
> denys kuzmenko wrote:
> open() and close() both manipulate with the shared variable (isOpen), so 
> they have to be synchronized on a same monitor (at least in current approach).
> I am not sure whether SparkContext supports instant interruption 
> (Thread.interrupt or sc.stop()). However when closing session that is in 
> progress, you have to take care of SparkContext.
> 
> Sahil Takiar wrote:
> yes, but we need to support calling `close()` while `open()` is being 
> run, as I described in my first comment. the (2) bullet point in the RB 
> description states that you want to guard against multiple callers invoking 
> the `open()` method, so logically you should just have a single lock to 
> handle this behavior. i suggest you use a separate lock for this scenario 
> because the `closeLock` is meant to handle synchronization of the `close()` 
> method, it was not meant to handle synchronization of the `open()` method by 
> multiple callers. IMO by re-using the lock we make the code harder to 
> understand because we are using the `closeLock` for functionality that should 
> be outside its scope.
> 
> I don't feel strongly about this though given we will need to re-factor 
> this code later anyway to support calling `close()` while `open()` is 
> running, so I'll leave it up to you.

Let's have a seperate JIRA for this use-case.


- denys


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69022/#review209617
---


On Oct. 16, 2018, 4:49 p.m., denys kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69022/
> ---
> 
> (Updated Oct. 16, 2018, 4:49 p.m.)
> 
> 
> Review request for hive, Sahil Takiar and Adam Szita.
> 
> 
> Bugs: HIVE-20737
> https://issues.apache.org/jira/browse/HIVE-20737
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> 1. Local SparkContext is shared between user sessions and should be closed 
> only when there is no active. 
> 2. Possible race condition in SparkSession.open() in case when user queries 
> run in parallel within the same session.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/LocalHiveSparkClient.java 
> 72ff53e3bd 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/session/SparkSessionImpl.java
>  bb50129518 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestLocalHiveSparkClient.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/69022/diff/3/
> 
> 
> Testing
> ---
> 
> Added TestLocalHiveSparkClient test
> 
> 
> File Attachments
> 
> 
> HIVE-20737.7.patch
>   
> https://reviews.apache.org/media/uploaded/files/2018/10/15/9cf8a2b3-9ec1-4316-81d0-3cd124b1a9fd__HIVE-20737.7.patch
> 
> 
> Thanks,
> 
> denys kuzmenko
> 
>

Re: Review Request 69022: HIVE-20737: Local SparkContext is shared between user sessions and should be closed only when there is no active

[denys kuzmenko via Review Board]

> On Oct. 16, 2018, 1:47 p.m., Sahil Takiar wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/session/SparkSessionImpl.java
> > Line 352 (original)
> > 
> >
> > why remove this?
> 
> denys kuzmenko wrote:
> it's not required. close() method is covered with the lock, and 
> activeJobs is a concurrent collection
> 
> Sahil Takiar wrote:
> what happens if a job is submitted after `hasTimedOut` returns true?
> 
> denys kuzmenko wrote:
> I see. However existing lock won't help as it doesn't prevent other 
> threads from adding new queries. 
> 
> public void onQuerySubmission(String queryId) {
> activeJobs.add(queryId);
> }
> 
> we might need to cover this with separate lock (onQueryCompletion, 
> onQuerySubmission, triggerTimeout)
> What do you think?
> 
> denys kuzmenko wrote:
> what happens if a job is submitted after hasTimedOut returns true? 
> current Session will be closed and a new one will be opened.
> 
> denys kuzmenko wrote:
> there might be an issue when 2nd session checkes state and get isOpen and 
> before it's reaching the submit phase, 1st one calls the close.
> I think we need synchronization for active sessions manipulation.
> 
> denys kuzmenko wrote:
> fixed. reverted back to the original locking. Above tricky case will be 
> handled by preventing new queries to execute open() before close() is 
> complete.
> 
> denys kuzmenko wrote:
> @before triggerTimeout() is complete
> 
> Sahil Takiar wrote:
> A little confused by all the comments here. Looks like the change has 
> been reverted. Do we agree that the current code is sufficient, or are their 
> other edge cases you think need to be covered?

Sorry for that, that was just my thoughts. 
I think, current code is sufficient for existing use case, however it's 
definetely not designed to support case where close() is called and could be 
executed while open() is being run


- denys


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69022/#review209617
---


On Oct. 16, 2018, 4:49 p.m., denys kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69022/
> ---
> 
> (Updated Oct. 16, 2018, 4:49 p.m.)
> 
> 
> Review request for hive, Sahil Takiar and Adam Szita.
> 
> 
> Bugs: HIVE-20737
> https://issues.apache.org/jira/browse/HIVE-20737
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> 1. Local SparkContext is shared between user sessions and should be closed 
> only when there is no active. 
> 2. Possible race condition in SparkSession.open() in case when user queries 
> run in parallel within the same session.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/LocalHiveSparkClient.java 
> 72ff53e3bd 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/session/SparkSessionImpl.java
>  bb50129518 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestLocalHiveSparkClient.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/69022/diff/3/
> 
> 
> Testing
> ---
> 
> Added TestLocalHiveSparkClient test
> 
> 
> File Attachments
> 
> 
> HIVE-20737.7.patch
>   
> https://reviews.apache.org/media/uploaded/files/2018/10/15/9cf8a2b3-9ec1-4316-81d0-3cd124b1a9fd__HIVE-20737.7.patch
> 
> 
> Thanks,
> 
> denys kuzmenko
> 
>

Re: Review Request 69022: HIVE-20737: Local SparkContext is shared between user sessions and should be closed only when there is no active

[Sahil Takiar]

> On Oct. 16, 2018, 1:47 p.m., Sahil Takiar wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/LocalHiveSparkClient.java
> > Line 73 (original), 73 (patched)
> > 
> >
> > if we expect multiple sessions to access this, should we make this 
> > `volatile`?
> 
> denys kuzmenko wrote:
> it's being accesed only inside of the critical section (within the lock 
> boundaries)
> 
> Sahil Takiar wrote:
> does java guarantee that non-volatile variables accessed inside a 
> critical section are not cached locally by a CPU?
> 
> denys kuzmenko wrote:
> In short - yes.
> 
> JSR 133 (Java Memory Model)
> 
> Synchronization ensures that memory writes by a thread before or during a 
> synchronized block are made visible in a predictable manner to other threads 
> which synchronize on the same monitor. After we exit a synchronized block, we 
> release the monitor, which has the effect of flushing the cache to main 
> memory, so that writes made by this thread can be visible to other threads. 
> Before we can enter a synchronized block, we acquire the monitor, which has 
> the effect of invalidating the local processor cache so that variables will 
> be reloaded from main memory. We will then be able to see all of the writes 
> made visible by the previous release.

makes sense


> On Oct. 16, 2018, 1:47 p.m., Sahil Takiar wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/session/SparkSessionImpl.java
> > Line 125 (original), 120 (patched)
> > 
> >
> > we might need to re-think how we are synchronizing this method a bit. I 
> > think we want to support the use case where we call `close()` while 
> > `open()` is being run. The offers a way for the user to cancel a session 
> > while it is being opened, which can be useful if opening a session takes a 
> > long time, which can happen on a busy cluster where there aren't enough 
> > resources to open a session.
> > 
> > fixing that might be out of the scope of this JIRA, so I would 
> > recommend using a separate lock to guard against multiple users calling 
> > open on the same session.
> 
> Sahil Takiar wrote:
> Tracking the aformentioned fix in HIVE-20519, unless you want to fix it 
> in this patch.
> 
> denys kuzmenko wrote:
> i think it should be addressed in another JIRA, right now we need to have 
> working at least basic use-case
> 
> Sahil Takiar wrote:
> okay, still recommend using a separate lock
> 
> denys kuzmenko wrote:
> open() and close() both manipulate with the shared variable (isOpen), so 
> they have to be synchronized on a same monitor (at least in current approach).
> I am not sure whether SparkContext supports instant interruption 
> (Thread.interrupt or sc.stop()). However when closing session that is in 
> progress, you have to take care of SparkContext.

yes, but we need to support calling `close()` while `open()` is being run, as I 
described in my first comment. the (2) bullet point in the RB description 
states that you want to guard against multiple callers invoking the `open()` 
method, so logically you should just have a single lock to handle this 
behavior. i suggest you use a separate lock for this scenario because the 
`closeLock` is meant to handle synchronization of the `close()` method, it was 
not meant to handle synchronization of the `open()` method by multiple callers. 
IMO by re-using the lock we make the code harder to understand because we are 
using the `closeLock` for functionality that should be outside its scope.

I don't feel strongly about this though given we will need to re-factor this 
code later anyway to support calling `close()` while `open()` is running, so 
I'll leave it up to you.


> On Oct. 16, 2018, 1:47 p.m., Sahil Takiar wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/session/SparkSessionImpl.java
> > Line 352 (original)
> > 
> >
> > why remove this?
> 
> denys kuzmenko wrote:
> it's not required. close() method is covered with the lock, and 
> activeJobs is a concurrent collection
> 
> Sahil Takiar wrote:
> what happens if a job is submitted after `hasTimedOut` returns true?
> 
> denys kuzmenko wrote:
> I see. However existing lock won't help as it doesn't prevent other 
> threads from adding new queries. 
> 
> public void onQuerySubmission(String queryId) {
> activeJobs.add(queryId);
> }
> 
> we might need to cover this with separate lock (onQueryCompletion, 
> onQuerySubmission, triggerTimeout)
> What do you think?
> 
> denys kuzmenko wrote:
> what happens if a job is submitted after hasTimedOut returns true? 
> current Session will be closed and a new one will be opened.
> 
> denys kuzmenko wrote:
> there might be an issue when 2nd 

Re: Review Request 69054: HIVE-20740 : Remove global lock in ObjectStore.setConf method

[Karthik Manamcheri via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69054/#review209742
---




standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Line 244 (original), 341 (patched)


You are ignoring the return value? Should you have 
pmf=getUpdatedPmfIfNeeded(..)?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Line 287 (original), 374 (patched)


What is dss.log?



standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
Lines 927 (patched)


Is this testing the new read/write lock behavior? As in, would this test 
have failed if you had the previous behavior of a global lock?


- Karthik Manamcheri


On Oct. 16, 2018, 10:47 p.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69054/
> ---
> 
> (Updated Oct. 16, 2018, 10:47 p.m.)
> 
> 
> Review request for hive, Andrew Sherman, Alan Gates, and Peter Vary.
> 
> 
> Bugs: HIVE-20740
> https://issues.apache.org/jira/browse/HIVE-20740
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20740 : Remove global lock in ObjectStore.setConf method
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  66977d79c946f1ac57aacfbe8704d37bfbac3ea3 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
>  b74c3048fa2e18adc7f0d7cc813a180d4466fa36 
> 
> 
> Diff: https://reviews.apache.org/r/69054/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vihang Karajgaonkar
> 
>

Re: Review Request 69022: HIVE-20737: Local SparkContext is shared between user sessions and should be closed only when there is no active

[Antal Sinkovits via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69022/#review209739
---


Ship it!




Ship It!

- Antal Sinkovits


On okt. 16, 2018, 4:49 du, denys kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69022/
> ---
> 
> (Updated okt. 16, 2018, 4:49 du)
> 
> 
> Review request for hive, Sahil Takiar and Adam Szita.
> 
> 
> Bugs: HIVE-20737
> https://issues.apache.org/jira/browse/HIVE-20737
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> 1. Local SparkContext is shared between user sessions and should be closed 
> only when there is no active. 
> 2. Possible race condition in SparkSession.open() in case when user queries 
> run in parallel within the same session.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/LocalHiveSparkClient.java 
> 72ff53e3bd 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/session/SparkSessionImpl.java
>  bb50129518 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestLocalHiveSparkClient.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/69022/diff/3/
> 
> 
> Testing
> ---
> 
> Added TestLocalHiveSparkClient test
> 
> 
> File Attachments
> 
> 
> HIVE-20737.7.patch
>   
> https://reviews.apache.org/media/uploaded/files/2018/10/15/9cf8a2b3-9ec1-4316-81d0-3cd124b1a9fd__HIVE-20737.7.patch
> 
> 
> Thanks,
> 
> denys kuzmenko
> 
>

Re: Review Request 68975: HIVE-20661: Dynamic partitions loading calls add partition for every partition 1-by-1

[Laszlo Pinter via Review Board]

> On Oct. 18, 2018, 9:13 a.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Lines 1921 (patched)
> > 
> >
> > nit: one extra space

done.


> On Oct. 18, 2018, 9:13 a.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Line 1876 (original)
> > 
> >
> > This perf logging is removed... Let's think through if we need other 
> > places to add some instead considering the new calling structure

It was not removed, but moved to another place. Now the tracking is started and 
finished in loadPartition().


> On Oct. 18, 2018, 9:13 a.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Lines 2171 (patched)
> > 
> >
> > What happens when only 1 partiton is already exists from multiple ones?
> > 
> > Also, do we need SynchronizedMSC?

Yes, we need this one in every case. The SynchronizedMetaStoreClient is just a 
facede for the SessionHiveMetaStoreClient, which in the other hand, does some 
pre-processing (eg. adding the partition to the temp table), before calling the 
HiveMetaStoreClient.


> On Oct. 18, 2018, 9:13 a.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Lines 2570 (patched)
> > 
> >
> > Do we need to call this every time in the for loop? I kinda remember 
> > that we allow only partitions for a single table only... Or I might be 
> > mistaken, but still might be a good idea to not generate this every time...

I think we do need. The fetching of the snapshot is done inside a callable task 
which is executed on thread pool in parallel. Altough we use only one table, 
the snapshot of it can change from time to time, depending on how many tasks 
were already finished. It's better to be safe, and get a new snapshot every 
time, than have an outdated one.


- Laszlo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68975/#review209735
---


On Oct. 18, 2018, 7:01 a.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68975/
> ---
> 
> (Updated Oct. 18, 2018, 7:01 a.m.)
> 
> 
> Review request for hive and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20661: Dynamic partitions loading calls add partition for every 
> partition 1-by-1
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/metastore/SynchronizedMetaStoreClient.java 
> 0ab77e84c6222d35bcec9ce95ed02014911ef144 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
> 4de038913a5c9a2c199f71702b8f70ca84d0856b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  dd23d7db3e70c9540e48c42eb7b9a33ed775cea6 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  aba63f050b5b98a2aeeb0df6ff2de5e6e06761f2 
> 
> 
> Diff: https://reviews.apache.org/r/68975/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

Question about HMS filter hooks

[Karthik Manamcheri]

In HMS, I noticed that all the filter hooks are applied on the client side
(in HiveMetaStoreClient.java). Is there any reason why we can't apply the
filters on the server-side?

Motivation: Some newer apache projects such as Kudu
 use HMS for metadata storage. Kudu is not
completely Java-based and there are interaction points where they have C++
clients. In such cases, we want to have consistent behavior from HMS side
as far as filters, etc are concerned.

I wanted to implement the filter hooks on the server side, but before I do
that, I wanted to understand if there were any concerning reasons why we
shouldn't do so.

Thanks,
Karthik

-- 
*Karthik Manamcheri*
Software Engineer

Re: Review Request 68975: HIVE-20661: Dynamic partitions loading calls add partition for every partition 1-by-1

[Peter Vary via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68975/#review209735
---



Thanks Laci!
1 serions question
1 mild one
and several annoying nits :)
Thanks,
Peter


ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
Lines 1921 (patched)


nit: one extra space



ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
Line 1876 (original)


This perf logging is removed... Let's think through if we need other places 
to add some instead considering the new calling structure



ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
Lines 2171 (patched)


What happens when only 1 partiton is already exists from multiple ones?

Also, do we need SynchronizedMSC?



ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
Lines 2570 (patched)


Do we need to call this every time in the for loop? I kinda remember that 
we allow only partitions for a single table only... Or I might be mistaken, but 
still might be a good idea to not generate this every time...



ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
Lines 2687 (patched)


I am not really a big fun of printing a stacktrace and rethrowing an error, 
unless I am quite sure that the exception rethrown is not printed later. 
Otherwise this could be quite confusing.


- Peter Vary


On okt. 18, 2018, 7:01 de, Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68975/
> ---
> 
> (Updated okt. 18, 2018, 7:01 de)
> 
> 
> Review request for hive and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20661: Dynamic partitions loading calls add partition for every 
> partition 1-by-1
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/metastore/SynchronizedMetaStoreClient.java 
> 0ab77e84c6222d35bcec9ce95ed02014911ef144 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
> 4de038913a5c9a2c199f71702b8f70ca84d0856b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  dd23d7db3e70c9540e48c42eb7b9a33ed775cea6 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  aba63f050b5b98a2aeeb0df6ff2de5e6e06761f2 
> 
> 
> Diff: https://reviews.apache.org/r/68975/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

[GitHub] hive pull request #450: [HIVE-20771] LazyBinarySerDe fails on empty structs.

[cvaliente]

GitHub user cvaliente opened a pull request:

https://github.com/apache/hive/pull/450

[HIVE-20771] LazyBinarySerDe fails on empty structs.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cvaliente/hive HIVE-20771

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/450.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #450


commit c4b839f610e4a57c497cfc108823d8da1b466fa7
Author: Clemens Valiente 
Date:   2018-10-18T08:08:08Z

HIVE-20771 enable LazyBinarySerDe to read fields with empty structs




---

[jira] [Created] (HIVE-20771) LazyBinarySerDe fails on empty structs.

[Clemens Valiente (JIRA)]

Clemens Valiente created HIVE-20771:
---

 Summary: LazyBinarySerDe fails on empty structs.
 Key: HIVE-20771
 URL: https://issues.apache.org/jira/browse/HIVE-20771
 Project: Hive
  Issue Type: Bug
  Components: Serializers/Deserializers
Reporter: Clemens Valiente
Assignee: Clemens Valiente


{code:java}
CREATE TABLE cvaliente.structtest AS
SELECT named_struct();

SHOW CREATE TABLE cvaliente.structtest;

SELECT * FROM cvaliente.structtest ORDER BY rand();
{code}

The resulting schema is:
{code:sql}
CREATE TABLE `cvaliente.structtest`(
  `_c0` struct<>)
ROW FORMAT SERDE 
  'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' 
STORED AS INPUTFORMAT 
  'org.apache.hadoop.mapred.TextInputFormat' 
OUTPUTFORMAT 
  'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
  'hdfs://nameservice1/user/cvaliente/cvaliente/structtest2'
TBLPROPERTIES (
  'COLUMN_STATS_ACCURATE'='true', 
  'numFiles'='1', 
  'numRows'='1', 
  'rawDataSize'='0', 
  'totalSize'='1',
  'transient_lastDdlTime'='1539781607');
{code}
Between the MAP and REDUCE phase hive serializes to LazyBinaryStruct and when 
trying to read the same object back the {{SELECT}} query above fails:

{code}
2018-10-17 14:32:02,298 [FATAL] [TezChild] |tez.ReduceRecordSource|: 
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
processing row (tag=0) 
{"key":{"reducesinkkey0":0.13508293503238622},"value":{"_col0":{}}}
at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:338)
at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:259)
at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:169)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:164)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
at 
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating 
VALUE._col0
at 
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:82)
at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:329)
... 17 more
Caused by: java.lang.RuntimeException: length should be positive!
at 
org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryNonPrimitive.init(LazyBinaryNonPrimitive.java:54)
at 
org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.init(LazyBinaryStruct.java:95)
at 
org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.uncheckedGetField(LazyBinaryStruct.java:264)
at 
org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getField(LazyBinaryStruct.java:201)
at 
org.apache.hadoop.hive.serde2.lazybinary.objectinspector.LazyBinaryStructObjectInspector.getStructFieldData(LazyBinaryStructObjectInspector.java:64)
at 
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator._evaluate(ExprNodeColumnEvaluator.java:98)
at 
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
at 
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
at 
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:77)
... 18 more
{code}

this is because the LazyBinaryNonPrimitive doesn't allow for empty structs in 
https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryNonPrimitive.java#L53





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68975: HIVE-20661: Dynamic partitions loading calls add partition for every partition 1-by-1

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68975/
---

(Updated Oct. 18, 2018, 7:01 a.m.)


Review request for hive and Peter Vary.


Repository: hive-git


Description
---

HIVE-20661: Dynamic partitions loading calls add partition for every partition 
1-by-1


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/metastore/SynchronizedMetaStoreClient.java 
0ab77e84c6222d35bcec9ce95ed02014911ef144 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
4de038913a5c9a2c199f71702b8f70ca84d0856b 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
dd23d7db3e70c9540e48c42eb7b9a33ed775cea6 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 aba63f050b5b98a2aeeb0df6ff2de5e6e06761f2 


Diff: https://reviews.apache.org/r/68975/diff/5/

Changes: https://reviews.apache.org/r/68975/diff/4-5/


Testing
---


Thanks,

Laszlo Pinter