[
https://issues.apache.org/jira/browse/DRILL-274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13812605#comment-13812605
]
Jacques Nadeau commented on DRILL-274:
--------------------------------------
Looks good. a few thoughts:
- Ways and whether to effectively cascade down a IterOutcome.NOT_YET on the
first fail.
- ConnectionThrottle was irrelevant when we're just packing unlimited in
memory. If/when we should actually throttle incoming RPCs through this buffer.
- I wonder if might be good to have a buffer set. For example, if we have a
set of separate incoming buffers and we're going to be merging them, it would
be best to balance memory consumption rather than letting the first stream fill
up most of the available buffering memory and spooling nearly all of the other
streams.
- You don't note it hear but we've talked previously about providing a HDFS URI
for buffering purposes. I wonder whether we should also allow multiple paths
instead of only one to do pseudo disk striping like MR does.
> Spooling batch buffer
> ---------------------
>
> Key: DRILL-274
> URL: https://issues.apache.org/jira/browse/DRILL-274
> Project: Apache Drill
> Issue Type: Bug
> Reporter: Steven Phillips
> Assignee: Steven Phillips
>
> incoming batches are currently queued up and held in memory without limit. If
> execution on a node is delayed, or is not able keep up with the rate of
> incoming batches, this could result in out of memory errors. We want to allow
> spooling incoming batches to disk once the total size of the batches in the
> queue reaches a threshold.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
