[
https://issues.apache.org/jira/browse/DRILL-5846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16420653#comment-16420653
]
ASF GitHub Bot commented on DRILL-5846:
---------------------------------------
Github user sachouche commented on the issue:
https://github.com/apache/drill/pull/1060
Parth,
- I have attached, within the DRILL-5846, two profiles with latest Apache
code and this PR request (bounds checks are off):
o Used one thread in each run
o I observe ~3x performance difference when the new logic is turned on
o The difference is 4x if I include the implicit column optimization
(which is not part of this PR)
o The impact of the new optimizations can be felt when there are many
variable length columns
- The rational of trying to approve this PR
o The optimizations that I have included are local to the Flat Parquet
Reader (incapsulated)
o The logic is backward compatible and turned off by default
o I have added the new Batch Sizing functionality on top of this PR
(columnar processing pattern)
o The result of DRILL-6301 would only result in a local refactoring step
o Not being able to add the new code results in a substantial
maintenance overhead
> Improve Parquet Reader Performance for Flat Data types
> -------------------------------------------------------
>
> Key: DRILL-5846
> URL: https://issues.apache.org/jira/browse/DRILL-5846
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - Parquet
> Affects Versions: 1.11.0
> Reporter: salim achouche
> Assignee: salim achouche
> Priority: Major
> Labels: performance
> Fix For: 1.14.0
>
> Attachments: 2542d447-9837-3924-dd12-f759108461e5.sys.drill,
> 2542d49b-88ef-38e3-a02b-b441c1295817.sys.drill
>
>
> The Parquet Reader is a key use-case for Drill. This JIRA is an attempt to
> further improve the Parquet Reader performance as several users reported that
> Parquet parsing represents the lion share of the overall query execution. It
> tracks Flat Data types only as Nested DTs might involve functional and
> processing enhancements (e.g., a nested column can be seen as a Document;
> user might want to perform operations scoped at the document level that is no
> need to span all rows). Another JIRA will be created to handle the nested
> columns use-case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
