Padma Penumarthy created DRILL-6113:
---------------------------------------
Summary: Limit batch size for Merge Receiver
Key: DRILL-6113
URL: https://issues.apache.org/jira/browse/DRILL-6113
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.12.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
Fix For: 1.13.0
Merge receiver has hard coded limit of 32K rows as batch. Since rows can be of
varying width, it is difficult to predict the output batch size (in terms of
memory) for this operator. Change this to derive row count based on actual
memory available. We are introducing a new option called outputBatchSize to
limit the batch size of each operator. Use the memory configured from that.
Figure out the average row width of outgoing batch based on averages of batches
coming from incoming streams. Limit the row count based on memory available to
use and average row width.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
