Re: Review Request 61164: HIVE-17006 LLAP: Parquet caching

2017-08-30 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61164/
---

(Updated Aug. 30, 2017, 10:21 p.m.)


Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/FileUtils.java e784797f62 
  itests/src/test/resources/testconfiguration.properties fa6a2aaea0 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIo.java 
42129b7511 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java 
0cbc8f6f4c 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LlapAllocatorBuffer.java 
52144c2432 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
35b9d1f942 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/llap/LlapCacheAwareFs.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java fa7f59d7dc 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 442c921a84 
  
ql/src/java/org/apache/hadoop/hive/ql/io/LlapCacheOnlyInputFormatInterface.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java 
f4fadbb61b 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/VectorizedParquetInputFormat.java
 322178a2f7 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetFooterInputFromCache.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java
 6a7a219dfe 
  ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 8b99ae07ac 
  ql/src/test/queries/clientpositive/parquet_ppd_decimal.q dfca486241 
  ql/src/test/queries/clientpositive/parquet_predicate_pushdown.q a38cdbe007 
  ql/src/test/queries/clientpositive/parquet_types.q db37d2e1b2 
  ql/src/test/queries/clientpositive/parquet_types_vectorization.q bb0e5b258f 
  ql/src/test/queries/clientpositive/vectorized_parquet.q db02ec04a5 
  ql/src/test/queries/clientpositive/vectorized_parquet_types.q 7467cb3cf6 
  ql/src/test/results/clientpositive/llap/parquet_predicate_pushdown.q.out 
aecbcfdfe4 
  ql/src/test/results/clientpositive/llap/parquet_types_vectorization.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/vectorized_parquet.q.out 8a84d3dc22 
  ql/src/test/results/clientpositive/llap/vectorized_parquet_types.q.out 
cdf6b3d225 
  storage-api/src/java/org/apache/hadoop/hive/common/io/FileMetadataCache.java 
PRE-CREATION 
  
storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/MemoryBufferOrBuffers.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/61164/diff/3/

Changes: https://reviews.apache.org/r/61164/diff/2-3/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 61164: HIVE-17006 LLAP: Parquet caching

2017-08-30 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61164/#review184177
---




ql/src/test/results/clientpositive/llap/parquet_types_vectorization.q.out
Lines 1 (patched)


verify that this runs thru the cache if possible


- Sergey Shelukhin


On July 27, 2017, 10:27 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61164/
> ---
> 
> (Updated July 27, 2017, 10:27 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/FileUtils.java e8a3a7a49e 
>   itests/src/test/resources/testconfiguration.properties f66e19be3e 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIo.java 
> 42129b7511 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java
>  0cbc8f6f4c 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
> 35b9d1f942 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/llap/LlapCacheAwareFs.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java 9b8b76102a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 21394c6aab 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/LlapCacheOnlyInputFormatInterface.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java
>  f4fadbb61b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/VectorizedParquetInputFormat.java
>  322178a2f7 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetFooterInputFromCache.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java
>  6a7a219dfe 
>   ql/src/test/queries/clientpositive/parquet_ppd_decimal.q dfca486241 
>   ql/src/test/queries/clientpositive/parquet_predicate_pushdown.q a38cdbe007 
>   ql/src/test/queries/clientpositive/parquet_types.q db37d2e1b2 
>   ql/src/test/queries/clientpositive/parquet_types_vectorization.q bb0e5b258f 
>   ql/src/test/queries/clientpositive/vectorized_parquet.q e6ebdaac62 
>   ql/src/test/queries/clientpositive/vectorized_parquet_types.q 7467cb3cf6 
>   ql/src/test/results/clientpositive/llap/parquet_types_vectorization.q.out 
> PRE-CREATION 
>   
> storage-api/src/java/org/apache/hadoop/hive/common/io/FileMetadataCache.java 
> PRE-CREATION 
>   
> storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/MemoryBufferOrBuffers.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/61164/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 61164: HIVE-17006 LLAP: Parquet caching

2017-08-30 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61164/#review184175
---




llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
Lines 75 (patched)


comment about not needing to discard



ql/src/java/org/apache/hadoop/hive/llap/LlapCacheAwareFs.java
Lines 60 (patched)


revise comment



ql/src/java/org/apache/hadoop/hive/llap/LlapCacheAwareFs.java
Lines 223 (patched)


should this use "off"?



ql/src/java/org/apache/hadoop/hive/llap/LlapCacheAwareFs.java
Lines 265 (patched)


.



ql/src/java/org/apache/hadoop/hive/llap/LlapCacheAwareFs.java
Lines 307 (patched)


needs to be done?



ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java
Lines 198 (patched)


follow up



ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java
Lines 327 (patched)


consider reading into cache, then doing the same thing as above


- Sergey Shelukhin


On July 27, 2017, 10:27 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61164/
> ---
> 
> (Updated July 27, 2017, 10:27 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/FileUtils.java e8a3a7a49e 
>   itests/src/test/resources/testconfiguration.properties f66e19be3e 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIo.java 
> 42129b7511 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java
>  0cbc8f6f4c 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
> 35b9d1f942 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/llap/LlapCacheAwareFs.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java 9b8b76102a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 21394c6aab 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/LlapCacheOnlyInputFormatInterface.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java
>  f4fadbb61b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/VectorizedParquetInputFormat.java
>  322178a2f7 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetFooterInputFromCache.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java
>  6a7a219dfe 
>   ql/src/test/queries/clientpositive/parquet_ppd_decimal.q dfca486241 
>   ql/src/test/queries/clientpositive/parquet_predicate_pushdown.q a38cdbe007 
>   ql/src/test/queries/clientpositive/parquet_types.q db37d2e1b2 
>   ql/src/test/queries/clientpositive/parquet_types_vectorization.q bb0e5b258f 
>   ql/src/test/queries/clientpositive/vectorized_parquet.q e6ebdaac62 
>   ql/src/test/queries/clientpositive/vectorized_parquet_types.q 7467cb3cf6 
>   ql/src/test/results/clientpositive/llap/parquet_types_vectorization.q.out 
> PRE-CREATION 
>   
> storage-api/src/java/org/apache/hadoop/hive/common/io/FileMetadataCache.java 
> PRE-CREATION 
>   
> storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/MemoryBufferOrBuffers.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/61164/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 61164: HIVE-17006 LLAP: Parquet caching

2017-07-27 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61164/
---

(Updated July 27, 2017, 10:27 p.m.)


Review request for hive and Prasanth_J.


Repository: hive-git


Description (updated)
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/FileUtils.java e8a3a7a49e 
  itests/src/test/resources/testconfiguration.properties f66e19be3e 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIo.java 
42129b7511 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java 
0cbc8f6f4c 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
35b9d1f942 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/llap/LlapCacheAwareFs.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java 9b8b76102a 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 21394c6aab 
  
ql/src/java/org/apache/hadoop/hive/ql/io/LlapCacheOnlyInputFormatInterface.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java 
f4fadbb61b 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/VectorizedParquetInputFormat.java
 322178a2f7 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetFooterInputFromCache.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java
 6a7a219dfe 
  ql/src/test/queries/clientpositive/parquet_ppd_decimal.q dfca486241 
  ql/src/test/queries/clientpositive/parquet_predicate_pushdown.q a38cdbe007 
  ql/src/test/queries/clientpositive/parquet_types.q db37d2e1b2 
  ql/src/test/queries/clientpositive/parquet_types_vectorization.q bb0e5b258f 
  ql/src/test/queries/clientpositive/vectorized_parquet.q e6ebdaac62 
  ql/src/test/queries/clientpositive/vectorized_parquet_types.q 7467cb3cf6 
  ql/src/test/results/clientpositive/llap/parquet_types_vectorization.q.out 
PRE-CREATION 
  storage-api/src/java/org/apache/hadoop/hive/common/io/FileMetadataCache.java 
PRE-CREATION 
  
storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/MemoryBufferOrBuffers.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/61164/diff/2/

Changes: https://reviews.apache.org/r/61164/diff/1-2/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 61164: HIVE-17006 LLAP: Parquet caching

2017-07-26 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61164/#review181494
---




ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java
Lines 17 (patched)


lots of spurious imports, will remove in the next iter


- Sergey Shelukhin


On July 26, 2017, 9:53 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61164/
> ---
> 
> (Updated July 26, 2017, 9:53 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/FileUtils.java e8a3a7a49e 
>   itests/src/test/resources/testconfiguration.properties f66e19be3e 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIo.java 
> 42129b7511 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java
>  0cbc8f6f4c 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
> 35b9d1f942 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/llap/LlapCacheAwareFs.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java 9b8b76102a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 21394c6aab 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/LlapCacheOnlyInputFormatInterface.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java
>  f4fadbb61b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/VectorizedParquetInputFormat.java
>  322178a2f7 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetFooterInputFromCache.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java
>  6a7a219dfe 
>   ql/src/test/queries/clientpositive/parquet_types_vectorization.q bb0e5b258f 
>   ql/src/test/results/clientpositive/llap/parquet_types_vectorization.q.out 
> PRE-CREATION 
>   
> storage-api/src/java/org/apache/hadoop/hive/common/io/FileMetadataCache.java 
> PRE-CREATION 
>   
> storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/MemoryBufferOrBuffers.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/61164/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>