[
https://issues.apache.org/jira/browse/HAMA-766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688809#comment-13688809
]
Suraj Menon commented on HAMA-766:
----------------------------------
We may need the messages to be still written to disk for a complete fault
tolerance solution. The spilling queue was designed with async network
communication in mind. The WriteSpilledDataProcessor can have a handle to say
MessageDispatcher class that dispatches the SpilledByteBuffer as a
ByteBufferMessageBundle to the peer. The trick is whether the spilling thread
would block the peer thread. We can make the message dispatcher to read from
the files written by spilling queues and send them asynchronously. We can
evaluate both approaches. Let me know what your take is. The spilled data
processor gets pipelined as Combiner spilled data processor -> Write to file
processor + Send on n/w processor. Thus we can get the combiner working on
spilled data.
> Implements network based send queue
> -----------------------------------
>
> Key: HAMA-766
> URL: https://issues.apache.org/jira/browse/HAMA-766
> Project: Hama
> Issue Type: Improvement
> Components: bsp core
> Reporter: MaoYuan Xian
> Priority: Minor
>
> I would like to implement a so call directly send queue for performance
> enhancement. Currently, we have the memory queue, DiskQueue and
> SpillingQueue. The memory queue need to hold all of the message before send
> them out, DiskQueue and SpillingQueue need to operate Disk and cause some
> performance reduction.
> Can we implement a send queue? Something like SpillingQueue, the difference
> is, it will spill message directly to network instead of to disk.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
