[
https://issues.apache.org/jira/browse/DRILL-6123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349514#comment-16349514
]
ASF GitHub Bot commented on DRILL-6123:
---------------------------------------
GitHub user ppadma opened a pull request:
https://github.com/apache/drill/pull/1107
DRILL-6123: Limit batch size for Merge Join based on memory
Merge join limits output batch size to 32K rows irrespective of row size.
This can create large batches (in terms of memory), depending upon average row
width. Changed the logic to figure out output row count based on memory
specified with the new outputBatchSize option and average outgoing row width.
Average outgoing row width will be sum of left and right batch row widths.
Output row count will be minimum of 1 and max of 64k.
Added AbstractRecordBatchMemoryManager class to be used across all
operators. Restructured the code a little bit for that.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ppadma/drill DRILL-6123
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/1107.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1107
----
commit e01da78116730afbcd9b5062e316678cebc4848f
Author: Padma Penumarthy <ppenumar97@...>
Date: 2018-01-31T00:58:57Z
DRILL-6123: Limit batch size for Merge Join based on memory
----
> Limit batch size for Merge Join based on memory
> -----------------------------------------------
>
> Key: DRILL-6123
> URL: https://issues.apache.org/jira/browse/DRILL-6123
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
> Affects Versions: 1.12.0
> Reporter: Padma Penumarthy
> Assignee: Padma Penumarthy
> Priority: Major
> Fix For: 1.13.0
>
>
> Merge join limits output batch size to 32K rows irrespective of row size.
> This can create very large or very small batches (in terms of memory),
> depending upon average row width. Change this to figure out output row count
> based on memory specified with the new outputBatchSize option and average row
> width of incoming left and right batches. Output row count will be minimum of
> 1 and max of 64k.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
