Re: [DISCUSS/VOTE] Refactor of message queue .
OKay, I'll go forward. On Fri, Aug 29, 2014 at 8:12 PM, Edward J. Yoon edwardy...@apache.org wrote: Thanks for the notice. I have no detail plan yet. On Friday, August 29, 2014, Chia-Hung Lin cli...@googlemail.com wrote: Is incoming bundles manager equals to localQueueForNextIteration/ localQueue in 0.6.4 version? My understanding for localQueueForNextIteration/ localQueue is that their role serves as input for the coming superstep; for instance, in the N-th superstep its incoming messages are obtained from localQueue (AbstractMessageManager.getCurrentMessage). In this case it look like safe to save messages to local disk because once a node fails, steps to recover from the previous superstep should be the same without change by placing the messages saved previous to the localQueue again. If not, then probably we need to consider about this issue. On 29 August 2014 08:09, Edward J. Yoon edwardy...@apache.org wrote: First of all, Our main problem is that current system requires a lot of memory space, especially graph module. As you already might know, the main memory consumer is the message queue. To solve this problem, we considered the use of local disk space e.g., DiskQueue and SpillingQueue. However, those queues are basically not able to bundle and group the messages by destination server, in memory-efficient way. So, I don't think this approach is right way. My solution for saving the memory usage and the performance degradation, is storing serializable message objects as a byte array in queue. In graph case, 3X ~ 6X memory efficiency is expected than before (GraphJobMessage consists of destination vertex ID and message value multi-objects). In 0.6.4, Outgoing queue is replaced with outgoing bundles manager, and it showed nice memory improvement. Now I wanna start refactoring of incoming queue. My plan is that adding incoming bundles manager. Bundles can also simply be written to local disk if when memory space is not enough. So, incoming bundles manager can be performed a similar role of DiskQueue and SpillingQueue in the future. If you have any other opinion, Please let me know. If there are no objections, I'll do it. -- Best Regards, Edward J. Yoon CEO at DataSayer Co., Ltd. -- Best Regards, Edward J. Yoon CEO at DataSayer Co., Ltd. -- Best Regards, Edward J. Yoon CEO at DataSayer Co., Ltd.
Re: [DISCUSS/VOTE] Refactor of message queue .
Is incoming bundles manager equals to localQueueForNextIteration/ localQueue in 0.6.4 version? My understanding for localQueueForNextIteration/ localQueue is that their role serves as input for the coming superstep; for instance, in the N-th superstep its incoming messages are obtained from localQueue (AbstractMessageManager.getCurrentMessage). In this case it look like safe to save messages to local disk because once a node fails, steps to recover from the previous superstep should be the same without change by placing the messages saved previous to the localQueue again. If not, then probably we need to consider about this issue. On 29 August 2014 08:09, Edward J. Yoon edwardy...@apache.org wrote: First of all, Our main problem is that current system requires a lot of memory space, especially graph module. As you already might know, the main memory consumer is the message queue. To solve this problem, we considered the use of local disk space e.g., DiskQueue and SpillingQueue. However, those queues are basically not able to bundle and group the messages by destination server, in memory-efficient way. So, I don't think this approach is right way. My solution for saving the memory usage and the performance degradation, is storing serializable message objects as a byte array in queue. In graph case, 3X ~ 6X memory efficiency is expected than before (GraphJobMessage consists of destination vertex ID and message value multi-objects). In 0.6.4, Outgoing queue is replaced with outgoing bundles manager, and it showed nice memory improvement. Now I wanna start refactoring of incoming queue. My plan is that adding incoming bundles manager. Bundles can also simply be written to local disk if when memory space is not enough. So, incoming bundles manager can be performed a similar role of DiskQueue and SpillingQueue in the future. If you have any other opinion, Please let me know. If there are no objections, I'll do it. -- Best Regards, Edward J. Yoon CEO at DataSayer Co., Ltd.
Re: [DISCUSS/VOTE] Refactor of message queue .
Thanks for the notice. I have no detail plan yet. On Friday, August 29, 2014, Chia-Hung Lin cli...@googlemail.com wrote: Is incoming bundles manager equals to localQueueForNextIteration/ localQueue in 0.6.4 version? My understanding for localQueueForNextIteration/ localQueue is that their role serves as input for the coming superstep; for instance, in the N-th superstep its incoming messages are obtained from localQueue (AbstractMessageManager.getCurrentMessage). In this case it look like safe to save messages to local disk because once a node fails, steps to recover from the previous superstep should be the same without change by placing the messages saved previous to the localQueue again. If not, then probably we need to consider about this issue. On 29 August 2014 08:09, Edward J. Yoon edwardy...@apache.org javascript:; wrote: First of all, Our main problem is that current system requires a lot of memory space, especially graph module. As you already might know, the main memory consumer is the message queue. To solve this problem, we considered the use of local disk space e.g., DiskQueue and SpillingQueue. However, those queues are basically not able to bundle and group the messages by destination server, in memory-efficient way. So, I don't think this approach is right way. My solution for saving the memory usage and the performance degradation, is storing serializable message objects as a byte array in queue. In graph case, 3X ~ 6X memory efficiency is expected than before (GraphJobMessage consists of destination vertex ID and message value multi-objects). In 0.6.4, Outgoing queue is replaced with outgoing bundles manager, and it showed nice memory improvement. Now I wanna start refactoring of incoming queue. My plan is that adding incoming bundles manager. Bundles can also simply be written to local disk if when memory space is not enough. So, incoming bundles manager can be performed a similar role of DiskQueue and SpillingQueue in the future. If you have any other opinion, Please let me know. If there are no objections, I'll do it. -- Best Regards, Edward J. Yoon CEO at DataSayer Co., Ltd. -- Best Regards, Edward J. Yoon CEO at DataSayer Co., Ltd.
