Re: [DISCUSS] Disk Queue and Spilling Queue
Even though we use the disk or spilling queue, we need to add whole
messages into the bundle object (memory). So, I said, they are
pointless. The bundle should be spillable. Receive-side queue is also
the same.
If there's no objections, I'll remove the spilling queue.
On Tue, May 13, 2014 at 4:55 PM, Andronidis Anastasios
andronat_...@hotmail.com wrote:
I am +1. That plan is going to make things much more straight to other
developers too.
Anastasis
On 13 Μαϊ 2014, at 7:27 π.μ., Edward J. Yoon edwardy...@apache.org wrote:
Just opened https://issues.apache.org/jira/browse/HAMA-903
On Tue, May 13, 2014 at 10:51 AM, Edward J. Yoon edwardy...@apache.org
wrote:
The AbstractMessageManager.loopBackMessages() method unbundle messages
into localQueueForNextIteration. Finally, in clearOutgoingMessages()
method, localQueue is switched by localQueueForNextIteration.
My goal is simplify this procedure. This is huge overhead.
Basically, BSPMessageBundle provides iterator access to the messages
contained in the bundle. So, we can skip whole conversion from bundle
to queue.
Is it going to use rpc?
I won't touch RPC mechanism. Only queue implementations will be changed.
On Mon, May 12, 2014 at 11:59 PM, Chia-Hung Lin cli...@googlemail.com
wrote:
Is it going to use rpc?
Will it still use the interface, for instance, MessageManager.java?
Just to check if any point for integrating with the current ongoing
refactoring process.
If possible, perhaps decoupling io part and rpc from interface would
somehow simplify the integration progress.
On 12 May 2014 09:01, Edward J. Yoon edwardy...@apache.org wrote:
The old design of outgoing/incoming message queues is readable but it
has some problems, and the most performance and memory issues are
dependent upon this part.
1) To send a messages to destination Peer, we serialize, compress, and
bundle the messages. So, using disk or spilling queue for the outgoing
messages is pointless and cause of degradation. This issue SOLVED by
HAMA-853. We'll need to add disk-based bundle in the future.
2) Receive-side queue is also the same. Instead of unbundling (and
deserializing, decompressing) bundles into {memory, disk, or spilling}
queue, we should use bundles in efficient and asynchronous way.
If you agree with this, I'll start to refactor the whole queue system.
If you have any other ideas e.g., asynchronous message
synchronization, Pls let me know.
Thanks.
--
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.
--
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.
--
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.
--
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.
Re: [DISCUSS] Disk Queue and Spilling Queue
2014-05-28 12:21 GMT+02:00 Edward J. Yoon edwardy...@apache.org:
Even though we use the disk or spilling queue, we need to add whole
messages into the bundle object (memory). So, I said, they are
pointless. The bundle should be spillable. Receive-side queue is also
the same.
+1
If there's no objections, I'll remove the spilling queue.
I'm not sure that's the right move, in the latest months we've been adding
and removing things back and forth in this area.
For example beforehand we didn't bundle everything together and so the
spilling queue made sense (if I recall correctly), again I would prefer to
have everything to be configurable, so by default for example we have no
spilling queue and bundle message but we may configure things in order to
use the spilling queue and to eventually send single messages rather than
the whole bundle.
Regards,
Tommaso
On Tue, May 13, 2014 at 4:55 PM, Andronidis Anastasios
andronat_...@hotmail.com wrote:
I am +1. That plan is going to make things much more straight to other
developers too.
Anastasis
On 13 Μαϊ 2014, at 7:27 π.μ., Edward J. Yoon edwardy...@apache.org
wrote:
Just opened https://issues.apache.org/jira/browse/HAMA-903
On Tue, May 13, 2014 at 10:51 AM, Edward J. Yoon edwardy...@apache.org
wrote:
The AbstractMessageManager.loopBackMessages() method unbundle messages
into localQueueForNextIteration. Finally, in clearOutgoingMessages()
method, localQueue is switched by localQueueForNextIteration.
My goal is simplify this procedure. This is huge overhead.
Basically, BSPMessageBundle provides iterator access to the messages
contained in the bundle. So, we can skip whole conversion from bundle
to queue.
Is it going to use rpc?
I won't touch RPC mechanism. Only queue implementations will be
changed.
On Mon, May 12, 2014 at 11:59 PM, Chia-Hung Lin cli...@googlemail.com
wrote:
Is it going to use rpc?
Will it still use the interface, for instance, MessageManager.java?
Just to check if any point for integrating with the current ongoing
refactoring process.
If possible, perhaps decoupling io part and rpc from interface would
somehow simplify the integration progress.
On 12 May 2014 09:01, Edward J. Yoon edwardy...@apache.org wrote:
The old design of outgoing/incoming message queues is readable but it
has some problems, and the most performance and memory issues are
dependent upon this part.
1) To send a messages to destination Peer, we serialize, compress,
and
bundle the messages. So, using disk or spilling queue for the
outgoing
messages is pointless and cause of degradation. This issue SOLVED by
HAMA-853. We'll need to add disk-based bundle in the future.
2) Receive-side queue is also the same. Instead of unbundling (and
deserializing, decompressing) bundles into {memory, disk, or
spilling}
queue, we should use bundles in efficient and asynchronous way.
If you agree with this, I'll start to refactor the whole queue
system.
If you have any other ideas e.g., asynchronous message
synchronization, Pls let me know.
Thanks.
--
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.
--
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.
--
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.
--
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.
Re: [DISCUSS] Disk Queue and Spilling Queue
Say, that's an idea. However, I think the complex option is a recipe
for confusions.
IMO, we have to finish the basics clearly. Otherwise, everything can
be fizzled out, like Fault Tolerance and Superstep API.
On Wed, May 28, 2014 at 7:42 PM, Tommaso Teofili
tommaso.teof...@gmail.com wrote:
2014-05-28 12:21 GMT+02:00 Edward J. Yoon edwardy...@apache.org:
Even though we use the disk or spilling queue, we need to add whole
messages into the bundle object (memory). So, I said, they are
pointless. The bundle should be spillable. Receive-side queue is also
the same.
+1
If there's no objections, I'll remove the spilling queue.
I'm not sure that's the right move, in the latest months we've been adding
and removing things back and forth in this area.
For example beforehand we didn't bundle everything together and so the
spilling queue made sense (if I recall correctly), again I would prefer to
have everything to be configurable, so by default for example we have no
spilling queue and bundle message but we may configure things in order to
use the spilling queue and to eventually send single messages rather than
the whole bundle.
Regards,
Tommaso
On Tue, May 13, 2014 at 4:55 PM, Andronidis Anastasios
andronat_...@hotmail.com wrote:
I am +1. That plan is going to make things much more straight to other
developers too.
Anastasis
On 13 Μαϊ 2014, at 7:27 π.μ., Edward J. Yoon edwardy...@apache.org
wrote:
Just opened https://issues.apache.org/jira/browse/HAMA-903
On Tue, May 13, 2014 at 10:51 AM, Edward J. Yoon edwardy...@apache.org
wrote:
The AbstractMessageManager.loopBackMessages() method unbundle messages
into localQueueForNextIteration. Finally, in clearOutgoingMessages()
method, localQueue is switched by localQueueForNextIteration.
My goal is simplify this procedure. This is huge overhead.
Basically, BSPMessageBundle provides iterator access to the messages
contained in the bundle. So, we can skip whole conversion from bundle
to queue.
Is it going to use rpc?
I won't touch RPC mechanism. Only queue implementations will be
changed.
On Mon, May 12, 2014 at 11:59 PM, Chia-Hung Lin cli...@googlemail.com
wrote:
Is it going to use rpc?
Will it still use the interface, for instance, MessageManager.java?
Just to check if any point for integrating with the current ongoing
refactoring process.
If possible, perhaps decoupling io part and rpc from interface would
somehow simplify the integration progress.
On 12 May 2014 09:01, Edward J. Yoon edwardy...@apache.org wrote:
The old design of outgoing/incoming message queues is readable but it
has some problems, and the most performance and memory issues are
dependent upon this part.
1) To send a messages to destination Peer, we serialize, compress,
and
bundle the messages. So, using disk or spilling queue for the
outgoing
messages is pointless and cause of degradation. This issue SOLVED by
HAMA-853. We'll need to add disk-based bundle in the future.
2) Receive-side queue is also the same. Instead of unbundling (and
deserializing, decompressing) bundles into {memory, disk, or
spilling}
queue, we should use bundles in efficient and asynchronous way.
If you agree with this, I'll start to refactor the whole queue
system.
If you have any other ideas e.g., asynchronous message
synchronization, Pls let me know.
Thanks.
--
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.
--
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.
--
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.
--
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.
--
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.
Re: [DISCUSS] Disk Queue and Spilling Queue
I am +1. That plan is going to make things much more straight to other
developers too.
Anastasis
On 13 Μαϊ 2014, at 7:27 π.μ., Edward J. Yoon edwardy...@apache.org wrote:
Just opened https://issues.apache.org/jira/browse/HAMA-903
On Tue, May 13, 2014 at 10:51 AM, Edward J. Yoon edwardy...@apache.org
wrote:
The AbstractMessageManager.loopBackMessages() method unbundle messages
into localQueueForNextIteration. Finally, in clearOutgoingMessages()
method, localQueue is switched by localQueueForNextIteration.
My goal is simplify this procedure. This is huge overhead.
Basically, BSPMessageBundle provides iterator access to the messages
contained in the bundle. So, we can skip whole conversion from bundle
to queue.
Is it going to use rpc?
I won't touch RPC mechanism. Only queue implementations will be changed.
On Mon, May 12, 2014 at 11:59 PM, Chia-Hung Lin cli...@googlemail.com
wrote:
Is it going to use rpc?
Will it still use the interface, for instance, MessageManager.java?
Just to check if any point for integrating with the current ongoing
refactoring process.
If possible, perhaps decoupling io part and rpc from interface would
somehow simplify the integration progress.
On 12 May 2014 09:01, Edward J. Yoon edwardy...@apache.org wrote:
The old design of outgoing/incoming message queues is readable but it
has some problems, and the most performance and memory issues are
dependent upon this part.
1) To send a messages to destination Peer, we serialize, compress, and
bundle the messages. So, using disk or spilling queue for the outgoing
messages is pointless and cause of degradation. This issue SOLVED by
HAMA-853. We'll need to add disk-based bundle in the future.
2) Receive-side queue is also the same. Instead of unbundling (and
deserializing, decompressing) bundles into {memory, disk, or spilling}
queue, we should use bundles in efficient and asynchronous way.
If you agree with this, I'll start to refactor the whole queue system.
If you have any other ideas e.g., asynchronous message
synchronization, Pls let me know.
Thanks.
--
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.
--
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.
--
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.
Re: [DISCUSS] Disk Queue and Spilling Queue
Is it going to use rpc?
Will it still use the interface, for instance, MessageManager.java?
Just to check if any point for integrating with the current ongoing
refactoring process.
If possible, perhaps decoupling io part and rpc from interface would
somehow simplify the integration progress.
On 12 May 2014 09:01, Edward J. Yoon edwardy...@apache.org wrote:
The old design of outgoing/incoming message queues is readable but it
has some problems, and the most performance and memory issues are
dependent upon this part.
1) To send a messages to destination Peer, we serialize, compress, and
bundle the messages. So, using disk or spilling queue for the outgoing
messages is pointless and cause of degradation. This issue SOLVED by
HAMA-853. We'll need to add disk-based bundle in the future.
2) Receive-side queue is also the same. Instead of unbundling (and
deserializing, decompressing) bundles into {memory, disk, or spilling}
queue, we should use bundles in efficient and asynchronous way.
If you agree with this, I'll start to refactor the whole queue system.
If you have any other ideas e.g., asynchronous message
synchronization, Pls let me know.
Thanks.
--
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.
Re: [DISCUSS] Disk Queue and Spilling Queue
The AbstractMessageManager.loopBackMessages() method unbundle messages
into localQueueForNextIteration. Finally, in clearOutgoingMessages()
method, localQueue is switched by localQueueForNextIteration.
My goal is simplify this procedure. This is huge overhead.
Basically, BSPMessageBundle provides iterator access to the messages
contained in the bundle. So, we can skip whole conversion from bundle
to queue.
Is it going to use rpc?
I won't touch RPC mechanism. Only queue implementations will be changed.
On Mon, May 12, 2014 at 11:59 PM, Chia-Hung Lin cli...@googlemail.com wrote:
Is it going to use rpc?
Will it still use the interface, for instance, MessageManager.java?
Just to check if any point for integrating with the current ongoing
refactoring process.
If possible, perhaps decoupling io part and rpc from interface would
somehow simplify the integration progress.
On 12 May 2014 09:01, Edward J. Yoon edwardy...@apache.org wrote:
The old design of outgoing/incoming message queues is readable but it
has some problems, and the most performance and memory issues are
dependent upon this part.
1) To send a messages to destination Peer, we serialize, compress, and
bundle the messages. So, using disk or spilling queue for the outgoing
messages is pointless and cause of degradation. This issue SOLVED by
HAMA-853. We'll need to add disk-based bundle in the future.
2) Receive-side queue is also the same. Instead of unbundling (and
deserializing, decompressing) bundles into {memory, disk, or spilling}
queue, we should use bundles in efficient and asynchronous way.
If you agree with this, I'll start to refactor the whole queue system.
If you have any other ideas e.g., asynchronous message
synchronization, Pls let me know.
Thanks.
--
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.
--
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.
Re: [DISCUSS] Disk Queue and Spilling Queue
Just opened https://issues.apache.org/jira/browse/HAMA-903
On Tue, May 13, 2014 at 10:51 AM, Edward J. Yoon edwardy...@apache.org wrote:
The AbstractMessageManager.loopBackMessages() method unbundle messages
into localQueueForNextIteration. Finally, in clearOutgoingMessages()
method, localQueue is switched by localQueueForNextIteration.
My goal is simplify this procedure. This is huge overhead.
Basically, BSPMessageBundle provides iterator access to the messages
contained in the bundle. So, we can skip whole conversion from bundle
to queue.
Is it going to use rpc?
I won't touch RPC mechanism. Only queue implementations will be changed.
On Mon, May 12, 2014 at 11:59 PM, Chia-Hung Lin cli...@googlemail.com wrote:
Is it going to use rpc?
Will it still use the interface, for instance, MessageManager.java?
Just to check if any point for integrating with the current ongoing
refactoring process.
If possible, perhaps decoupling io part and rpc from interface would
somehow simplify the integration progress.
On 12 May 2014 09:01, Edward J. Yoon edwardy...@apache.org wrote:
The old design of outgoing/incoming message queues is readable but it
has some problems, and the most performance and memory issues are
dependent upon this part.
1) To send a messages to destination Peer, we serialize, compress, and
bundle the messages. So, using disk or spilling queue for the outgoing
messages is pointless and cause of degradation. This issue SOLVED by
HAMA-853. We'll need to add disk-based bundle in the future.
2) Receive-side queue is also the same. Instead of unbundling (and
deserializing, decompressing) bundles into {memory, disk, or spilling}
queue, we should use bundles in efficient and asynchronous way.
If you agree with this, I'll start to refactor the whole queue system.
If you have any other ideas e.g., asynchronous message
synchronization, Pls let me know.
Thanks.
--
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.
--
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.
--
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.
[DISCUSS] Disk Queue and Spilling Queue
The old design of outgoing/incoming message queues is readable but it
has some problems, and the most performance and memory issues are
dependent upon this part.
1) To send a messages to destination Peer, we serialize, compress, and
bundle the messages. So, using disk or spilling queue for the outgoing
messages is pointless and cause of degradation. This issue SOLVED by
HAMA-853. We'll need to add disk-based bundle in the future.
2) Receive-side queue is also the same. Instead of unbundling (and
deserializing, decompressing) bundles into {memory, disk, or spilling}
queue, we should use bundles in efficient and asynchronous way.
If you agree with this, I'll start to refactor the whole queue system.
If you have any other ideas e.g., asynchronous message
synchronization, Pls let me know.
Thanks.
--
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.
