[
https://issues.apache.org/jira/browse/HAMA-766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689101#comment-13689101
]
MaoYuan Xian commented on HAMA-766:
-----------------------------------
Yes, I agree with Suraj, we need spilled data processor -> Write to file + send
to network. In current mechanism, message dispatcher (send to network) does not
start until all data prepared (message sending starts in BSPPeerImpl.sync()).
Maybe spilling queue will implement the network sending concurrently with local
file writing (message dispatch does not need to wait for disk operations), mark
this issue as duplicated with SpillingQueue maybe? Sorry for unclear about the
spillingQueue developing road map.
I did a very beginning implementations (no combine and file saving) in
https://github.com/kennethxian/hama/commit/0effb82ae19f3bde059d2d94169824a2c5e1b046.
But, some message lost when handle big data. Because of the trick
implementations in DirectByteBufferInputStream, I still can not find out the
root cause (Problem maybe in the sepearted one messeg into two buffer or
de-serialize them).
However, I think, for distributing message to network, we can skip the
serialize/deserialize process between Queue add and message bundle sending,
right?
> Implements network based send queue
> -----------------------------------
>
> Key: HAMA-766
> URL: https://issues.apache.org/jira/browse/HAMA-766
> Project: Hama
> Issue Type: Improvement
> Components: bsp core
> Reporter: MaoYuan Xian
> Priority: Minor
>
> I would like to implement a so call directly send queue for performance
> enhancement. Currently, we have the memory queue, DiskQueue and
> SpillingQueue. The memory queue need to hold all of the message before send
> them out, DiskQueue and SpillingQueue need to operate Disk and cause some
> performance reduction.
> Can we implement a send queue? Something like SpillingQueue, the difference
> is, it will spill message directly to network instead of to disk.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
