[
https://issues.apache.org/jira/browse/DRILL-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16216117#comment-16216117
]
ASF GitHub Bot commented on DRILL-5795:
---------------------------------------
Github user cchang738 commented on the issue:
https://github.com/apache/drill/pull/949
There is a plan verification failure due to plan change. The plan baseline
needs to be changed after this PR is merged.
Plan Verification Failures:
/root/drillAutomation/mapr/framework/resources/Functional/int96/q28.q
Query:
explain plan for select voter_id, name from `hive1_parquet_part` where
date_part('year', create_timestamp1)=2018
Expected and actual text plans are different.
Expected:
.*numFiles=2, usedMetadataFile=true.*
Actual:
00-00 Screen
00-01 Project(voter_id=[$0], name=[$1])
00-02 Project(voter_id=[$1], name=[$2])
00-03 Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
[path=/drill/testdata/subqueries/hive1_parquet_part/0_0_10.parquet],
ReadEntryWithPath
[path=/drill/testdata/subqueries/hive1_parquet_part/0_0_9.parquet]],
selectionRoot=/drill/testdata/subqueries/hive1_parquet_part, numFiles=2,
numRowGroups=2, usedMetadataFile=true,
cacheFileRoot=/drill/testdata/subqueries/hive1_parquet_part,
columns=[`create_timestamp1`, `voter_id`, `name`]]])
> Filter pushdown for parquet handles multi rowgroup file
> -------------------------------------------------------
>
> Key: DRILL-5795
> URL: https://issues.apache.org/jira/browse/DRILL-5795
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - Parquet
> Affects Versions: 1.11.0
> Reporter: Damien Profeta
> Assignee: Damien Profeta
> Labels: doc-impacting
> Fix For: 1.12.0
>
> Attachments: multirowgroup_overlap.parquet
>
>
> DRILL-1950 implemented the filter pushdown for parquet file but only in the
> case of one rowgroup per parquet file. In the case of multiple rowgroups per
> files, it detects that the rowgroup can be pruned but then tell to the
> drillbit to read the whole file which leads to performance issue.
> Having multiple rowgroup per file helps to handle partitioned dataset and
> still read only the relevant subset of data without ending with more file
> than really needed.
> Let's say for instance you have a Parquet file composed of RG1 and RG2 with
> only one column a. Min/max in RG1 are 1-2 and min/max in RG2 are 2-3.
> If I do "select a from file where a=3", today it will read the whole file,
> with the patch it will only read RG2.
> *For documentation*
> Support / Other section in
> https://drill.apache.org/docs/parquet-filter-pushdown/ should be updated.
> After the fix files with multiple row groups will be supported.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
