[
https://issues.apache.org/jira/browse/DRILL-274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13823258#comment-13823258
]
Steven Phillips commented on DRILL-274:
---------------------------------------
After discussion with Jacques and Ben, I decided to take a slightly different
approach. I still have the RawFragmentBatchWrap, but I no longer have a
spooling manager.
Now, if we start spooling, the enqueue() method call does not return until the
batch has been written to disk. And if the getNext() method is called and the
next batch has been spooled to disk, the batch is retrieved at that point.
There are no longer any queues of batch to be spooled or unspooled.
I created a new FileSystem implementation that allows syncing the data to the
local filesystem. This is to avoid having to close the output stream in order
to read the data back.
> Spooling batch buffer
> ---------------------
>
> Key: DRILL-274
> URL: https://issues.apache.org/jira/browse/DRILL-274
> Project: Apache Drill
> Issue Type: Bug
> Reporter: Steven Phillips
> Assignee: Steven Phillips
> Attachments: DRILL-274.patch, DRILL-274.patch
>
>
> incoming batches are currently queued up and held in memory without limit. If
> execution on a node is delayed, or is not able keep up with the rate of
> incoming batches, this could result in out of memory errors. We want to allow
> spooling incoming batches to disk once the total size of the batches in the
> queue reaches a threshold.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
