[
https://issues.apache.org/jira/browse/DRILL-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16580555#comment-16580555
]
Sorabh Hamirwasia edited comment on DRILL-6678 at 8/14/18 11:46 PM:
--------------------------------------------------------------------
The idea is the 3rd proposal using Batch Sizing. Updated the description as well
{quote}Or, is the idea to append rows from multiple incoming batches until the
output batch reaches the target size? That is, if A, B and C are incoming
batches, the output batch may have all selected rows from A and B, and, say,
have the selected rows from C.
{quote}
was (Author: shamirwasia):
The idea is the 3rd proposal using Batch Sizing.
{quote}Or, is the idea to append rows from multiple incoming batches until the
output batch reaches the target size? That is, if A, B and C are incoming
batches, the output batch may have all selected rows from A and B, and, say,
have the selected rows from C.
{quote}
> Improve SelectionVectorRemover to pack output batch based on BatchSizing
> ------------------------------------------------------------------------
>
> Key: DRILL-6678
> URL: https://issues.apache.org/jira/browse/DRILL-6678
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Relational Operators
> Affects Versions: 1.14.0
> Reporter: Sorabh Hamirwasia
> Assignee: Sorabh Hamirwasia
> Priority: Major
>
> SelectionVectorRemover in most of the cases is downstream to Filter which
> reduces the number of records to be copied in output container. In those
> cases if SelectionVectorRemover can pack the output batch to approximate
> maximum utilization based on RecordBatchSizer target record count, that will
> reduce the number of output batches from it and will help to improve
> performance. During Lateral & Unnest Performance evaluation we have noticed a
> significant decrease in performance as number of batches increases for same
> number of rows (i.e. Batch is not fully packed)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
