Just opened https://issues.apache.org/jira/browse/HAMA-903
On Tue, May 13, 2014 at 10:51 AM, Edward J. Yoon <[email protected]> wrote: > The AbstractMessageManager.loopBackMessages() method unbundle messages > into localQueueForNextIteration. Finally, in clearOutgoingMessages() > method, localQueue is switched by localQueueForNextIteration. > > My goal is simplify this procedure. This is huge overhead. > > Basically, BSPMessageBundle provides iterator access to the messages > contained in the bundle. So, we can skip whole conversion from bundle > to queue. > >> Is it going to use rpc? > > I won't touch RPC mechanism. Only queue implementations will be changed. > > On Mon, May 12, 2014 at 11:59 PM, Chia-Hung Lin <[email protected]> wrote: >> Is it going to use rpc? >> >> Will it still use the interface, for instance, MessageManager.java? >> >> Just to check if any point for integrating with the current ongoing >> refactoring process. >> >> If possible, perhaps decoupling io part and rpc from interface would >> somehow simplify the integration progress. >> >> >> On 12 May 2014 09:01, Edward J. Yoon <[email protected]> wrote: >>> The old design of outgoing/incoming message queues is readable but it >>> has some problems, and the most performance and memory issues are >>> dependent upon this part. >>> >>> 1) To send a messages to destination Peer, we serialize, compress, and >>> bundle the messages. So, using disk or spilling queue for the outgoing >>> messages is pointless and cause of degradation. This issue SOLVED by >>> HAMA-853. We'll need to add disk-based bundle in the future. >>> >>> 2) Receive-side queue is also the same. Instead of unbundling (and >>> deserializing, decompressing) bundles into {memory, disk, or spilling} >>> queue, we should use bundles in efficient and asynchronous way. >>> >>> If you agree with this, I'll start to refactor the whole queue system. >>> >>> If you have any other ideas e.g., asynchronous message >>> synchronization, Pls let me know. >>> >>> Thanks. >>> >>> -- >>> Best Regards, Edward J. Yoon >>> CEO at DataSayer Co., Ltd. > > > > -- > Best Regards, Edward J. Yoon > CEO at DataSayer Co., Ltd. -- Best Regards, Edward J. Yoon CEO at DataSayer Co., Ltd.
