Re: [zeromq-dev] Question about reliability in ZeroMQ
I see. Thank you Joshua.
Yuya
On Tue, Oct 18, 2016 at 3:52 PM, Joshua Foster wrote:
> Yes, the old data still in the queue would be sent once the dealer starts
> receiving messages again.
>
> Joshua
>
>
>
> On 10/18/2016 6:08 PM, 白木 勇矢 wrote:
>
>> Thank you for your response Joshua. So I cannot avoid queue drop when
>> sending packet from Pub to Sub or from Dealer to Router unless I
>> create some logic to solve this issue.
>>
>> According to what you explained, when I call zmq_send continuously on
>> Router to send messages to specific Dealer, if Dealer is in mute
>> state, because the Router socket send is blocked and cannot dequeue a
>> message, it's going to reach high water mark in Router. After it
>> reached high water mark in Router, rest of the message I tried to
>> enqueue using zmq_send in Router is dropped. But do the previous data
>> enqueued already in ZMQ queue of Router would be able to reach Dealer
>> for sure if Dealer move out from mute state?
>>
>> Thanks,
>> Yuya
>>
>> On Tue, Oct 18, 2016 at 10:48 AM, Joshua Foster
>> wrote:
>>
>>> Lets see if I can explain it. Each socket has its own queue/buffer. This
>>> is
>>> what is used to track how many messages. PUB/SUB uses a drop approach
>>> when
>>> its queue gets full. If a subscriber is slow (has a full queue), the
>>> publisher continues publishing and does not block. The slow subscriber
>>> would
>>> then miss all messages until it was able to clear some of its queue.
>>>
>>> I don't know much about PGM, but I believe that it just provides more
>>> reliability at the packet level so the behavior of PUB/SUB would be the
>>> same.
>>>
>>> PUSH/PULL block when their queues get full. This puts back-pressure on
>>> the
>>> producers. Router and Dealers are sockets with a pair of Push/Pull
>>> sockets
>>> so they behave the same. So if a Dealer is blocked, the router socket
>>> would
>>> block on send until the dealer can accept.
>>>
>>> The correct answer to avoiding queue drops is that you don't. The correct
>>> approach is to create you solution that allows messages to drop. There
>>> are
>>> MANY reasons for messages to drop and it is really just a tradeoff as to
>>> what can drop and what performance is required. One simple way to handle
>>> dropped messages is to use idempotent messages. The basic idea is that
>>> you
>>> create messages that contain the most recent state of whatever you need.
>>> That way if all previous messages get dropped, you still have all the
>>> data
>>> needed. In contrast, transactional messages are not able to be dropped.
>>>
>>> Idempotent Example:
>>> {1}
>>> {5}
>>> {10}
>>>
>>> Transactional Example:
>>> {+1}
>>> {+4}
>>> {+5}
>>>
>>> Joshua
>>>
>>>
>>> On 10/18/2016 1:21 PM, 白木勇矢 wrote:
>>>
Hello everyone,
I'm trying to use ZeroMQ Pub/Sub and Router/Dealer for my school
project. I have a question about reliability of these sockets.
According to the API Page(http://api.zeromq.org/), when queue reached
a high water mark and move to mute state, Dealer is going to block,
Router is going to drop, Sub I'm not sure, and Pub is going to drop
the message. But I cannot clearly see the behavior of each socket.
When Dealer send message to Router via TCP and Router is in mute
state, Dealer would be able to know this and resend message again
after some interval? If Router sends the message and Dealer is in mute
state and blocked, how Router is going to act?
Also when Pub sends the message to Sub via TCP and if Sub is in mute
state, how Pub is going to behave? If Pub wants to know whether the
message is correctly enqueued in Sub, PGM is going to solve this?
If someone would be able to answer these questions and tell me how to
avoid queue drops, I appreciate it so much.
Thanks,
Yuya
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>> ___
>>> zeromq-dev mailing list
>>> zeromq-dev@lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>> ___
>> zeromq-dev mailing list
>> zeromq-dev@lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Question about reliability in ZeroMQ
Yes, the old data still in the queue would be sent once the dealer
starts receiving messages again.
Joshua
On 10/18/2016 6:08 PM, 白木 勇矢 wrote:
Thank you for your response Joshua. So I cannot avoid queue drop when
sending packet from Pub to Sub or from Dealer to Router unless I
create some logic to solve this issue.
According to what you explained, when I call zmq_send continuously on
Router to send messages to specific Dealer, if Dealer is in mute
state, because the Router socket send is blocked and cannot dequeue a
message, it's going to reach high water mark in Router. After it
reached high water mark in Router, rest of the message I tried to
enqueue using zmq_send in Router is dropped. But do the previous data
enqueued already in ZMQ queue of Router would be able to reach Dealer
for sure if Dealer move out from mute state?
Thanks,
Yuya
On Tue, Oct 18, 2016 at 10:48 AM, Joshua Foster wrote:
Lets see if I can explain it. Each socket has its own queue/buffer. This is
what is used to track how many messages. PUB/SUB uses a drop approach when
its queue gets full. If a subscriber is slow (has a full queue), the
publisher continues publishing and does not block. The slow subscriber would
then miss all messages until it was able to clear some of its queue.
I don't know much about PGM, but I believe that it just provides more
reliability at the packet level so the behavior of PUB/SUB would be the
same.
PUSH/PULL block when their queues get full. This puts back-pressure on the
producers. Router and Dealers are sockets with a pair of Push/Pull sockets
so they behave the same. So if a Dealer is blocked, the router socket would
block on send until the dealer can accept.
The correct answer to avoiding queue drops is that you don't. The correct
approach is to create you solution that allows messages to drop. There are
MANY reasons for messages to drop and it is really just a tradeoff as to
what can drop and what performance is required. One simple way to handle
dropped messages is to use idempotent messages. The basic idea is that you
create messages that contain the most recent state of whatever you need.
That way if all previous messages get dropped, you still have all the data
needed. In contrast, transactional messages are not able to be dropped.
Idempotent Example:
{1}
{5}
{10}
Transactional Example:
{+1}
{+4}
{+5}
Joshua
On 10/18/2016 1:21 PM, 白木勇矢 wrote:
Hello everyone,
I'm trying to use ZeroMQ Pub/Sub and Router/Dealer for my school
project. I have a question about reliability of these sockets.
According to the API Page(http://api.zeromq.org/), when queue reached
a high water mark and move to mute state, Dealer is going to block,
Router is going to drop, Sub I'm not sure, and Pub is going to drop
the message. But I cannot clearly see the behavior of each socket.
When Dealer send message to Router via TCP and Router is in mute
state, Dealer would be able to know this and resend message again
after some interval? If Router sends the message and Dealer is in mute
state and blocked, how Router is going to act?
Also when Pub sends the message to Sub via TCP and if Sub is in mute
state, how Pub is going to behave? If Pub wants to know whether the
message is correctly enqueued in Sub, PGM is going to solve this?
If someone would be able to answer these questions and tell me how to
avoid queue drops, I appreciate it so much.
Thanks,
Yuya
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Question about reliability in ZeroMQ
Thank you for your response Joshua. So I cannot avoid queue drop when
sending packet from Pub to Sub or from Dealer to Router unless I
create some logic to solve this issue.
According to what you explained, when I call zmq_send continuously on
Router to send messages to specific Dealer, if Dealer is in mute
state, because the Router socket send is blocked and cannot dequeue a
message, it's going to reach high water mark in Router. After it
reached high water mark in Router, rest of the message I tried to
enqueue using zmq_send in Router is dropped. But do the previous data
enqueued already in ZMQ queue of Router would be able to reach Dealer
for sure if Dealer move out from mute state?
Thanks,
Yuya
On Tue, Oct 18, 2016 at 10:48 AM, Joshua Foster wrote:
> Lets see if I can explain it. Each socket has its own queue/buffer. This is
> what is used to track how many messages. PUB/SUB uses a drop approach when
> its queue gets full. If a subscriber is slow (has a full queue), the
> publisher continues publishing and does not block. The slow subscriber would
> then miss all messages until it was able to clear some of its queue.
>
> I don't know much about PGM, but I believe that it just provides more
> reliability at the packet level so the behavior of PUB/SUB would be the
> same.
>
> PUSH/PULL block when their queues get full. This puts back-pressure on the
> producers. Router and Dealers are sockets with a pair of Push/Pull sockets
> so they behave the same. So if a Dealer is blocked, the router socket would
> block on send until the dealer can accept.
>
> The correct answer to avoiding queue drops is that you don't. The correct
> approach is to create you solution that allows messages to drop. There are
> MANY reasons for messages to drop and it is really just a tradeoff as to
> what can drop and what performance is required. One simple way to handle
> dropped messages is to use idempotent messages. The basic idea is that you
> create messages that contain the most recent state of whatever you need.
> That way if all previous messages get dropped, you still have all the data
> needed. In contrast, transactional messages are not able to be dropped.
>
> Idempotent Example:
> {1}
> {5}
> {10}
>
> Transactional Example:
> {+1}
> {+4}
> {+5}
>
> Joshua
>
>
> On 10/18/2016 1:21 PM, 白木勇矢 wrote:
>>
>> Hello everyone,
>>
>> I'm trying to use ZeroMQ Pub/Sub and Router/Dealer for my school
>> project. I have a question about reliability of these sockets.
>> According to the API Page(http://api.zeromq.org/), when queue reached
>> a high water mark and move to mute state, Dealer is going to block,
>> Router is going to drop, Sub I'm not sure, and Pub is going to drop
>> the message. But I cannot clearly see the behavior of each socket.
>>
>> When Dealer send message to Router via TCP and Router is in mute
>> state, Dealer would be able to know this and resend message again
>> after some interval? If Router sends the message and Dealer is in mute
>> state and blocked, how Router is going to act?
>>
>> Also when Pub sends the message to Sub via TCP and if Sub is in mute
>> state, how Pub is going to behave? If Pub wants to know whether the
>> message is correctly enqueued in Sub, PGM is going to solve this?
>>
>> If someone would be able to answer these questions and tell me how to
>> avoid queue drops, I appreciate it so much.
>>
>> Thanks,
>> Yuya
>> ___
>> zeromq-dev mailing list
>> zeromq-dev@lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Question about reliability in ZeroMQ
Lets see if I can explain it. Each socket has its own queue/buffer. This
is what is used to track how many messages. PUB/SUB uses a drop approach
when its queue gets full. If a subscriber is slow (has a full queue),
the publisher continues publishing and does not block. The slow
subscriber would then miss all messages until it was able to clear some
of its queue.
I don't know much about PGM, but I believe that it just provides more
reliability at the packet level so the behavior of PUB/SUB would be the
same.
PUSH/PULL block when their queues get full. This puts back-pressure on
the producers. Router and Dealers are sockets with a pair of Push/Pull
sockets so they behave the same. So if a Dealer is blocked, the router
socket would block on send until the dealer can accept.
The correct answer to avoiding queue drops is that you don't. The
correct approach is to create you solution that allows messages to drop.
There are MANY reasons for messages to drop and it is really just a
tradeoff as to what can drop and what performance is required. One
simple way to handle dropped messages is to use idempotent messages. The
basic idea is that you create messages that contain the most recent
state of whatever you need. That way if all previous messages get
dropped, you still have all the data needed. In contrast, transactional
messages are not able to be dropped.
Idempotent Example:
{1}
{5}
{10}
Transactional Example:
{+1}
{+4}
{+5}
Joshua
On 10/18/2016 1:21 PM, 白木勇矢 wrote:
Hello everyone,
I'm trying to use ZeroMQ Pub/Sub and Router/Dealer for my school
project. I have a question about reliability of these sockets.
According to the API Page(http://api.zeromq.org/), when queue reached
a high water mark and move to mute state, Dealer is going to block,
Router is going to drop, Sub I'm not sure, and Pub is going to drop
the message. But I cannot clearly see the behavior of each socket.
When Dealer send message to Router via TCP and Router is in mute
state, Dealer would be able to know this and resend message again
after some interval? If Router sends the message and Dealer is in mute
state and blocked, how Router is going to act?
Also when Pub sends the message to Sub via TCP and if Sub is in mute
state, how Pub is going to behave? If Pub wants to know whether the
message is correctly enqueued in Sub, PGM is going to solve this?
If someone would be able to answer these questions and tell me how to
avoid queue drops, I appreciate it so much.
Thanks,
Yuya
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] Question about reliability in ZeroMQ
Hello everyone, I'm trying to use ZeroMQ Pub/Sub and Router/Dealer for my school project. I have a question about reliability of these sockets. According to the API Page(http://api.zeromq.org/), when queue reached a high water mark and move to mute state, Dealer is going to block, Router is going to drop, Sub I'm not sure, and Pub is going to drop the message. But I cannot clearly see the behavior of each socket. When Dealer send message to Router via TCP and Router is in mute state, Dealer would be able to know this and resend message again after some interval? If Router sends the message and Dealer is in mute state and blocked, how Router is going to act? Also when Pub sends the message to Sub via TCP and if Sub is in mute state, how Pub is going to behave? If Pub wants to know whether the message is correctly enqueued in Sub, PGM is going to solve this? If someone would be able to answer these questions and tell me how to avoid queue drops, I appreciate it so much. Thanks, Yuya ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
