Thanks for the notice. I have no detail plan yet. On Friday, August 29, 2014, Chia-Hung Lin <cli...@googlemail.com> wrote:
> Is incoming bundles manager equals to localQueueForNextIteration/ > localQueue in 0.6.4 version? > > My understanding for localQueueForNextIteration/ localQueue is that > their role serves as input for the coming superstep; for instance, in > the N-th superstep its incoming messages are obtained from localQueue > (AbstractMessageManager.getCurrentMessage). In this case it look like > safe to save messages to local disk because once a node fails, steps > to recover from the previous superstep should be the same without > change by placing the messages saved previous to the localQueue again. > If not, then probably we need to consider about this issue. > > > > > > > > On 29 August 2014 08:09, Edward J. Yoon <edwardy...@apache.org > <javascript:;>> wrote: > > First of all, Our main problem is that current system requires a lot > > of memory space, especially graph module. As you already might know, > > the main memory consumer is the message queue. > > > > To solve this problem, we considered the use of local disk space e.g., > > DiskQueue and SpillingQueue. However, those queues are basically not > > able to bundle and group the messages by destination server, in > > memory-efficient way. So, I don't think this approach is right way. > > > > My solution for saving the memory usage and the performance > > degradation, is storing serializable message objects as a byte array > > in queue. In graph case, 3X ~ 6X memory efficiency is expected than > > before (GraphJobMessage consists of destination vertex ID and message > > value multi-objects). > > > > In 0.6.4, Outgoing queue is replaced with outgoing bundles manager, > > and it showed nice memory improvement. Now I wanna start refactoring > > of incoming queue. > > > > My plan is that adding incoming bundles manager. Bundles can also > > simply be written to local disk if when memory space is not enough. > > So, incoming bundles manager can be performed a similar role of > > DiskQueue and SpillingQueue in the future. > > > > If you have any other opinion, Please let me know. If there are no > > objections, I'll do it. > > > > -- > > Best Regards, Edward J. Yoon > > CEO at DataSayer Co., Ltd. > -- Best Regards, Edward J. Yoon CEO at DataSayer Co., Ltd.
