[
https://issues.apache.org/jira/browse/HIVE-26013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503414#comment-17503414
]
Rajesh Balamohan commented on HIVE-26013:
-----------------------------------------
This could be a false alarm as blocks are already getting filtered out for the
split.
Ref:
[https://github.com/apache/hive/blob/master/ql%2Fsrc%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhive%2Fql%2Fio%2Fparquet%2FParquetRecordReaderBase.java#L123]
> Parquet predicate filters are not properly propogated to task configs at
> runtime
> --------------------------------------------------------------------------------
>
> Key: HIVE-26013
> URL: https://issues.apache.org/jira/browse/HIVE-26013
> Project: Hive
> Issue Type: Bug
> Reporter: Rajesh Balamohan
> Priority: Major
> Labels: performance
>
> Hive ParquetRecordReader sets the predicate filter in the config for parquet
> libs to read.
> Ref:
> [https://github.com/apache/hive/blob/master/ql%2Fsrc%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhive%2Fql%2Fio%2Fparquet%2FParquetRecordReaderBase.java#L188]
> {code:java}
> ParquetInputFormat.setFilterPredicate(conf, p);
> {code}
> This internally sets {color:#FF0000}"parquet.private.read.filter.predicate"
> {color}variable in config.
> Ref:
> [https://github.com/apache/parquet-mr/blob/master/parquet-hadoop%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fparquet%2Fhadoop%2FParquetInputFormat.java#L231]
> Config set in compilation phase isn't visible at runtime for the tasks. This
> causes filters to be lost and tasks run with excessive IO.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
