OKay, I'll go forward. On Fri, Aug 29, 2014 at 8:12 PM, Edward J. Yoon <edwardy...@apache.org> wrote: > Thanks for the notice. I have no detail plan yet. > > > On Friday, August 29, 2014, Chia-Hung Lin <cli...@googlemail.com> wrote: >> >> Is incoming bundles manager equals to localQueueForNextIteration/ >> localQueue in 0.6.4 version? >> >> My understanding for localQueueForNextIteration/ localQueue is that >> their role serves as input for the coming superstep; for instance, in >> the N-th superstep its incoming messages are obtained from localQueue >> (AbstractMessageManager.getCurrentMessage). In this case it look like >> safe to save messages to local disk because once a node fails, steps >> to recover from the previous superstep should be the same without >> change by placing the messages saved previous to the localQueue again. >> If not, then probably we need to consider about this issue. >> >> >> >> >> >> >> >> On 29 August 2014 08:09, Edward J. Yoon <edwardy...@apache.org> wrote: >> > First of all, Our main problem is that current system requires a lot >> > of memory space, especially graph module. As you already might know, >> > the main memory consumer is the message queue. >> > >> > To solve this problem, we considered the use of local disk space e.g., >> > DiskQueue and SpillingQueue. However, those queues are basically not >> > able to bundle and group the messages by destination server, in >> > memory-efficient way. So, I don't think this approach is right way. >> > >> > My solution for saving the memory usage and the performance >> > degradation, is storing serializable message objects as a byte array >> > in queue. In graph case, 3X ~ 6X memory efficiency is expected than >> > before (GraphJobMessage consists of destination vertex ID and message >> > value multi-objects). >> > >> > In 0.6.4, Outgoing queue is replaced with outgoing bundles manager, >> > and it showed nice memory improvement. Now I wanna start refactoring >> > of incoming queue. >> > >> > My plan is that adding incoming bundles manager. Bundles can also >> > simply be written to local disk if when memory space is not enough. >> > So, incoming bundles manager can be performed a similar role of >> > DiskQueue and SpillingQueue in the future. >> > >> > If you have any other opinion, Please let me know. If there are no >> > objections, I'll do it. >> > >> > -- >> > Best Regards, Edward J. Yoon >> > CEO at DataSayer Co., Ltd. > > > > -- > Best Regards, Edward J. Yoon > CEO at DataSayer Co., Ltd.
-- Best Regards, Edward J. Yoon CEO at DataSayer Co., Ltd.
