[
https://issues.apache.org/jira/browse/HAMA-764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13683078#comment-13683078
]
MaoYuan Xian commented on HAMA-764:
-----------------------------------
Anyway, attach my patch. spillThreadService_ will be null before thread is
invoked, so this.spillThreadService_.shutdownNow() can not be put into last
finally code block.
Also, another problem:
{code}
SpillingStream(int numBuffers, int bufferSize, int threshold,
boolean direct, SpilledDataProcessor processor, int interBufferSize) {
assert (threshold >= bufferSize);
assert (threshold < numBuffers * bufferSize);
if (interBufferSize > bufferSize) {
interBufferSize = bufferSize / 2;
}
defaultBufferSize_ = interBufferSize;
this.b = new byte[1];
this.buf = new byte[defaultBufferSize_];
count = 0;
direct_ = direct;
numberBuffers_ = numBuffers;
bufferSize_ = bufferSize;
bufferList_ = new ArrayList<SpilledByteBuffer>(numberBuffers_);
bufferState_ = new BitSet(numBuffers);
for (int i = 0; i < numBuffers / 2; ++i) {
bufferList_.add(new SpilledByteBuffer(direct_, bufferSize_));
}
currentBuffer_ = bufferList_.get(0);
...
{code}
When numBuffers is 1, bufferList_.get(0) will trigger an exception.
> In SpillingQueue, spilling thread dos not start when messages number less
> than threshold
> ----------------------------------------------------------------------------------------
>
> Key: HAMA-764
> URL: https://issues.apache.org/jira/browse/HAMA-764
> Project: Hama
> Issue Type: Bug
> Components: bsp core
> Affects Versions: 0.7.0
> Reporter: MaoYuan Xian
> Attachments: HAMA-764.patch
>
>
> In the SpillingDataOutputBuffer.SpillingStream, we can find the following
> implementations to starting the spilling thread.
> {code}
> private void checkSpillStart() throws IOException {
> if (bytesWritten_ >= thresholdSize_) {
> try {
> startSpilling();
> } catch (InterruptedException e) {
> throw new IOException("Internal error occured writing to buffer.",
> e);
> }
> }
> }
> {code}
> But I encountered the message loss, when total bytes written number is less
> than thresholdSize. Because the spilling thread is not invoked, and
> handleSpilledBuffer method of SpilledDataProcessor can not get the chance to
> run.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
