[
https://issues.apache.org/jira/browse/DRILL-7069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807286#comment-16807286
]
Kunal Khatua commented on DRILL-7069:
-------------------------------------
[~ben-zvi] does this require any additional Documentation beyond a mention in
the release notes? (cc: [~bbevens])
> Poor performance of transformBinaryInMetadataCache
> --------------------------------------------------
>
> Key: DRILL-7069
> URL: https://issues.apache.org/jira/browse/DRILL-7069
> Project: Apache Drill
> Issue Type: Improvement
> Components: Metadata
> Affects Versions: 1.15.0
> Reporter: Boaz Ben-Zvi
> Assignee: Boaz Ben-Zvi
> Priority: Major
> Labels: ready-to-commit
> Fix For: 1.16.0
>
>
> The performance of the method *transformBinaryInMetadataCache* scales poorly
> as the table's numbers of underlying files, row-groups and columns grow. This
> method is invoked during planning of every query using this table.
> A test on a table using 219 directories (each with 20 files), 1
> row-group in each file, and 94 columns, measured about *1340 milliseconds*.
> The main culprit are the version checks, which take place in *every
> iteration* (i.e., about 400k times in the previous example) and involve
> construction of 6 MetadataVersion objects (and possibly garbage collections).
> Removing the version checks from the loops improved this method's
> performance on the above test down to about *250 milliseconds*.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
