Re: Review Request 53593: HIVE-14089 complex type support in LLAP IO is broken

2016-11-14 Thread j . prasanth . j 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53593/#review155870
---


Ship it!




Ship It!

- Prasanth_J


On Nov. 14, 2016, 8:45 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53593/
> ---
> 
> (Updated Nov. 14, 2016, 8:45 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
>  a27266f 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
>  2fa68a9 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  72f733a 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java
>  c9b0a4d 
>   orc/src/java/org/apache/orc/impl/ConvertTreeReaderFactory.java c347181 
>   orc/src/java/org/apache/orc/impl/TreeReaderFactory.java 245e3ce 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java a9dbc3e 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ccb39da 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 
> 4d09dcd 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
> a4925b9 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
>  ebbdf8d 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 4405232 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 5cc3663 
>   ql/src/test/queries/clientpositive/vector_complex_all.q 91a7368 
>   ql/src/test/results/clientpositive/llap/vector_complex_all.q.out 08d49bc 
>   ql/src/test/results/clientpositive/llap/vector_complex_join.q.out 97d5642 
>   ql/src/test/results/clientpositive/vector_complex_all.q.out a54a371 
>   
> storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java
>  907181e 
> 
> Diff: https://reviews.apache.org/r/53593/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 53593: HIVE-14089 complex type support in LLAP IO is broken

2016-11-14 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53593/
---

(Updated Nov. 14, 2016, 8:45 p.m.)


Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
 a27266f 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
 2fa68a9 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 72f733a 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java
 c9b0a4d 
  orc/src/java/org/apache/orc/impl/ConvertTreeReaderFactory.java c347181 
  orc/src/java/org/apache/orc/impl/TreeReaderFactory.java 245e3ce 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java a9dbc3e 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ccb39da 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 
4d09dcd 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
a4925b9 
  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
 ebbdf8d 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 4405232 
  ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 5cc3663 
  ql/src/test/queries/clientpositive/vector_complex_all.q 91a7368 
  ql/src/test/results/clientpositive/llap/vector_complex_all.q.out 08d49bc 
  ql/src/test/results/clientpositive/llap/vector_complex_join.q.out 97d5642 
  ql/src/test/results/clientpositive/vector_complex_all.q.out a54a371 
  
storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java
 907181e 

Diff: https://reviews.apache.org/r/53593/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 53593: HIVE-14089 complex type support in LLAP IO is broken

2016-11-14 Thread Sergey Shelukhin 


> On Nov. 11, 2016, 12:28 a.m., Prasanth_J wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java,
> >  line 329
> > 
> >
> > nit: Can the batchkey be null?

no


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53593/#review155661
---


On Nov. 9, 2016, 2:34 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53593/
> ---
> 
> (Updated Nov. 9, 2016, 2:34 a.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
>  a27266f 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
>  2fa68a9 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  eb8ee6c 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java
>  c9b0a4d 
>   orc/src/java/org/apache/orc/impl/ConvertTreeReaderFactory.java c347181 
>   orc/src/java/org/apache/orc/impl/TreeReaderFactory.java 245e3ce 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java a9dbc3e 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ccb39da 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 
> 4d09dcd 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
> bcb54d6 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
>  ebbdf8d 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 4405232 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 5cc3663 
>   ql/src/test/queries/clientpositive/vector_complex_all.q 91a7368 
>   ql/src/test/results/clientpositive/llap/vector_complex_all.q.out 08d49bc 
>   ql/src/test/results/clientpositive/vector_complex_all.q.out a54a371 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> 76a1286 
>   
> storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java
>  907181e 
> 
> Diff: https://reviews.apache.org/r/53593/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 53593: HIVE-14089 complex type support in LLAP IO is broken

2016-11-10 Thread j . prasanth . j 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53593/#review155661
---




llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
 (line 119)


Remove it altogether?



llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
 (line 259)


nit: Can the batchkey be null?



orc/src/java/org/apache/orc/impl/TreeReaderFactory.java (line 1831)


RLE reader for tags stream (tags stored in DATA stream) should be created.



ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
(line 219)


All error changes seem unnecessary? Can we log this at DEBUG or TRACE 
instead?



ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
 (line 2021)


We should do seek into lengths when the stream is not empty.



ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
 (line 2141)


We should seek into lengths too. No sure how easy it would be to add test 
case for this. Since we don't do PPD on complext types.


- Prasanth_J


On Nov. 9, 2016, 2:34 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53593/
> ---
> 
> (Updated Nov. 9, 2016, 2:34 a.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
>  a27266f 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
>  2fa68a9 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  eb8ee6c 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java
>  c9b0a4d 
>   orc/src/java/org/apache/orc/impl/ConvertTreeReaderFactory.java c347181 
>   orc/src/java/org/apache/orc/impl/TreeReaderFactory.java 245e3ce 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java a9dbc3e 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ccb39da 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 
> 4d09dcd 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
> bcb54d6 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
>  ebbdf8d 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 4405232 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 5cc3663 
>   ql/src/test/queries/clientpositive/vector_complex_all.q 91a7368 
>   ql/src/test/results/clientpositive/llap/vector_complex_all.q.out 08d49bc 
>   ql/src/test/results/clientpositive/vector_complex_all.q.out a54a371 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> 76a1286 
>   
> storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java
>  907181e 
> 
> Diff: https://reviews.apache.org/r/53593/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 53593: HIVE-14089 complex type support in LLAP IO is broken

2016-11-09 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53593/#review155539
---




llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 


this should not be removed



ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
(line 210)


should be removed



storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java
 (line 84)


stale comment


- Sergey Shelukhin


On Nov. 9, 2016, 2:34 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53593/
> ---
> 
> (Updated Nov. 9, 2016, 2:34 a.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
>  a27266f 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
>  2fa68a9 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  eb8ee6c 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java
>  c9b0a4d 
>   orc/src/java/org/apache/orc/impl/ConvertTreeReaderFactory.java c347181 
>   orc/src/java/org/apache/orc/impl/TreeReaderFactory.java 245e3ce 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java a9dbc3e 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ccb39da 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 
> 4d09dcd 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
> bcb54d6 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
>  ebbdf8d 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 4405232 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 5cc3663 
>   ql/src/test/queries/clientpositive/vector_complex_all.q 91a7368 
>   ql/src/test/results/clientpositive/llap/vector_complex_all.q.out 08d49bc 
>   ql/src/test/results/clientpositive/vector_complex_all.q.out a54a371 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> 76a1286 
>   
> storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java
>  907181e 
> 
> Diff: https://reviews.apache.org/r/53593/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Review Request 53593: HIVE-14089 complex type support in LLAP IO is broken

2016-11-08 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53593/
---

Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs
-

  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
 a27266f 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
 2fa68a9 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 eb8ee6c 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java
 c9b0a4d 
  orc/src/java/org/apache/orc/impl/ConvertTreeReaderFactory.java c347181 
  orc/src/java/org/apache/orc/impl/TreeReaderFactory.java 245e3ce 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java a9dbc3e 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ccb39da 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 
4d09dcd 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
bcb54d6 
  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
 ebbdf8d 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 4405232 
  ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 5cc3663 
  ql/src/test/queries/clientpositive/vector_complex_all.q 91a7368 
  ql/src/test/results/clientpositive/llap/vector_complex_all.q.out 08d49bc 
  ql/src/test/results/clientpositive/vector_complex_all.q.out a54a371 
  service/src/java/org/apache/hive/service/cli/operation/Operation.java 76a1286 
  
storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java
 907181e 

Diff: https://reviews.apache.org/r/53593/diff/


Testing
---


Thanks,

Sergey Shelukhin