Re: [zeromq-dev] How many messages are in queue now?
It was never possible, due to the fact that pipes are being written and read asynchronously. You cannot measure the free space except by stopping everything. The workaround is to use credit based flow control. I've a section in the Guide that explains how this works. You can in effect know what % of the pipe from sender to receiver (including all ZeroMQ and network buffers) is filled. -Pieter On Wed, Apr 15, 2015 at 4:30 PM, Ilja Golshtein ilej...@narod.ru wrote: Hello. I used to think that it is possible to retrieve number or queued messages per socket in recent versions of 0mq, while I failed to find correspond getsockopt in 4.0.5 Use case: I need to alter behavior of my application (drop certain type of messages) if a queue is, say, 90% of HWM. Please, advise if this facility is implemented or planned, or suggest a workaround which is more elegant than having atomic variable (it is inproc) maintained by reader and writer threads which is what I plan to do. Thanks. -- Best regards Ilja Golshtein ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] How many messages are in queue now?
Hello Pieter, thank you for your answer. I am in process of reading http://zguide.zeromq.org/hx:chapter7 , while atomic counter seems more natural choice for inproc so far. Could you please explain why it is possible to detect that we are at 100% of HWM and it is not possible to detect that we are e.g. at 90% ? 17.04.2015, 11:54, Pieter Hintjens p...@imatix.com: It was never possible, due to the fact that pipes are being written and read asynchronously. You cannot measure the free space except by stopping everything. The workaround is to use credit based flow control. I've a section in the Guide that explains how this works. You can in effect know what % of the pipe from sender to receiver (including all ZeroMQ and network buffers) is filled. -Pieter On Wed, Apr 15, 2015 at 4:30 PM, Ilja Golshtein ilej...@narod.ru wrote: Hello. I used to think that it is possible to retrieve number or queued messages per socket in recent versions of 0mq, while I failed to find correspond getsockopt in 4.0.5 Use case: I need to alter behavior of my application (drop certain type of messages) if a queue is, say, 90% of HWM. Please, advise if this facility is implemented or planned, or suggest a workaround which is more elegant than having atomic variable (it is inproc) maintained by reader and writer threads which is what I plan to do. Thanks. -- Best regards Ilja Golshtein ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev -- Best regards Ilja Golshtein ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] How many messages are in queue now?
At the point a message is placed on the queue it is either at HWM or has at least space for one message, this can be evaluated atomically, claiming the 'slot' under the HWM for the message to be placed on the queue. Any other threads attempting to send would synchronize with this operation and the HWM will be respected, no overshoot. Any observation of an atomic count, as it is being changed, will be the state of the count, at some point in the past. So you could have a HWM of 1000, read a count of 850, make some decision, yah, cool to send, 90%, and have a failing subsequent send, because in the time it took you to do all of that, another thread may have enqueued 30 messages, and yet another thread may have tried to enqueue 50 messages. Any such observations of queue depth with multiple producers and consumers are at best, estimates of queue state at some point in the past, and it is very difficult IME to build reliable logic around that information. On Friday, April 17, 2015, Ilja Golshtein ilej...@narod.ru wrote: Hello Pieter, thank you for your answer. I am in process of reading http://zguide.zeromq.org/hx:chapter7 , while atomic counter seems more natural choice for inproc so far. Could you please explain why it is possible to detect that we are at 100% of HWM and it is not possible to detect that we are e.g. at 90% ? 17.04.2015, 11:54, Pieter Hintjens p...@imatix.com javascript:;: It was never possible, due to the fact that pipes are being written and read asynchronously. You cannot measure the free space except by stopping everything. The workaround is to use credit based flow control. I've a section in the Guide that explains how this works. You can in effect know what % of the pipe from sender to receiver (including all ZeroMQ and network buffers) is filled. -Pieter On Wed, Apr 15, 2015 at 4:30 PM, Ilja Golshtein ilej...@narod.ru javascript:; wrote: Hello. I used to think that it is possible to retrieve number or queued messages per socket in recent versions of 0mq, while I failed to find correspond getsockopt in 4.0.5 Use case: I need to alter behavior of my application (drop certain type of messages) if a queue is, say, 90% of HWM. Please, advise if this facility is implemented or planned, or suggest a workaround which is more elegant than having atomic variable (it is inproc) maintained by reader and writer threads which is what I plan to do. Thanks. -- Best regards Ilja Golshtein ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org javascript:; http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org javascript:; http://lists.zeromq.org/mailman/listinfo/zeromq-dev -- Best regards Ilja Golshtein ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org javascript:; http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] How many messages are in queue now?
Thomas, Agree that it makes perfect sense for multiple producers and consumers, while in my case it is ZMQ_PAIR. Thanks. 17.04.2015, 16:49, "Thomas Rodgers" rodg...@twrodgers.com:At the point a message is placed on the queue it is either at HWM or has at least space for one message, this can be evaluated atomically, claiming the 'slot' under the HWM for the message to be placed on the queue. Any other threads attempting to send would synchronize with this operation and the HWM will be respected, no overshoot. Any observation of an atomic count, as it is being changed, will be the state of the count, at some point in the past. So you could have a HWM of 1000, read a count of 850, make some decision, yah, cool to send, 90%, and have a failing subsequent send, because in the time it took you to do all of that, another thread may have enqueued 30 messages, and yet another thread may have tried to enqueue 50 messages. Any such observations of queue depth with multiple producers and consumers are at best, estimates of queue state at some point in the past, and it is very difficult IME to build reliable logic around that information.On Friday, April 17, 2015, Ilja Golshtein ilej...@narod.ru wrote:Hello Pieter, thank you for your answer. I am in process of reading http://zguide.zeromq.org/hx:chapter7 , while atomic counter seems more natural choice for inproc so far. Could you please explain why it is possible to detect that we are at 100% of HWM and it is not possible to detect that we are e.g. at 90% ? 17.04.2015, 11:54, "Pieter Hintjens" p...@imatix.com: It was never possible, due to the fact that pipes are being written and read asynchronously. You cannot measure the free space except by stopping everything. The workaround is to use credit based flow control. I've a section in the Guide that explains how this works. You can in effect know what % of the pipe from sender to receiver (including all ZeroMQ and network buffers) is filled. -Pieter On Wed, Apr 15, 2015 at 4:30 PM, Ilja Golshtein ilej...@narod.ru wrote: Hello. I used to think that it is possible to retrieve number or queued messages per socket in recent versions of 0mq, while I failed to find correspond getsockopt in 4.0.5 Use case: I need to alter behavior of my application (drop certain type of messages) if a queue is, say, 90% of HWM. Please, advise if this facility is implemented or planned, or suggest a workaround which is more elegant than having atomic variable (it is inproc) maintained by reader and writer threads which is what I plan to do. Thanks. -- Best regards Ilja Golshtein ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev -- Best regards Ilja Golshtein ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev,___zeromq-dev mailing listzeromq-dev@lists.zeromq.orghttp://lists.zeromq.org/mailman/listinfo/zeromq-dev -- Best regardsIlja Golshtein ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] How many messages are in queue now?
Yes, this is true. If a sysadmin detects that a disk storage is 90% full, she notifies users and do other things to obtain more room, while it is quite possible that the disk is only 89% full at the moment.So, ability to calculate disk usage is "a likely source of confusion and gnashing of teeth".Right. 17.04.2015, 17:06, "Thomas Rodgers" rodg...@twrodgers.com:I suppose then it is a question of general utility. Even with ZMQ_PAIR, it is still at best and estimate, likely to have changed (perhaps considerably) from the time you query it to the the time you did something based on that information. Even under the best of circumstances it seems of such limited utility (and a likely source of confusion and gnashing of teeth) that I don't see the value in adding the feature.On Fri, Apr 17, 2015 at 9:00 AM, Ilja Golshtein ilej...@narod.ru wrote:Thomas, Agree that it makes perfect sense for multiple producers and consumers, while in my case it is ZMQ_PAIR. Thanks. 17.04.2015, 16:49, "Thomas Rodgers" rodg...@twrodgers.com:At the point a message is placed on the queue it is either at HWM or has at least space for one message, this can be evaluated atomically, claiming the 'slot' under the HWM for the message to be placed on the queue. Any other threads attempting to send would synchronize with this operation and the HWM will be respected, no overshoot. Any observation of an atomic count, as it is being changed, will be the state of the count, at some point in the past. So you could have a HWM of 1000, read a count of 850, make some decision, yah, cool to send, 90%, and have a failing subsequent send, because in the time it took you to do all of that, another thread may have enqueued 30 messages, and yet another thread may have tried to enqueue 50 messages. Any such observations of queue depth with multiple producers and consumers are at best, estimates of queue state at some point in the past, and it is very difficult IME to build reliable logic around that information.On Friday, April 17, 2015, Ilja Golshtein ilej...@narod.ru wrote:Hello Pieter, thank you for your answer. I am in process of reading http://zguide.zeromq.org/hx:chapter7 , while atomic counter seems more natural choice for inproc so far. Could you please explain why it is possible to detect that we are at 100% of HWM and it is not possible to detect that we are e.g. at 90% ? 17.04.2015, 11:54, "Pieter Hintjens" p...@imatix.com: It was never possible, due to the fact that pipes are being written and read asynchronously. You cannot measure the free space except by stopping everything. The workaround is to use credit based flow control. I've a section in the Guide that explains how this works. You can in effect know what % of the pipe from sender to receiver (including all ZeroMQ and network buffers) is filled. -Pieter On Wed, Apr 15, 2015 at 4:30 PM, Ilja Golshtein ilej...@narod.ru wrote: Hello. I used to think that it is possible to retrieve number or queued messages per socket in recent versions of 0mq, while I failed to find correspond getsockopt in 4.0.5 Use case: I need to alter behavior of my application (drop certain type of messages) if a queue is, say, 90% of HWM. Please, advise if this facility is implemented or planned, or suggest a workaround which is more elegant than having atomic variable (it is inproc) maintained by reader and writer threads which is what I plan to do. Thanks. -- Best regards Ilja Golshtein ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev -- Best regards Ilja Golshtein ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev,___zeromq-dev mailing listzeromq-dev@lists.zeromq.orghttp://lists.zeromq.org/mailman/listinfo/zeromq-dev -- Best regardsIlja Golshtein ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ,___zeromq-dev mailing listzeromq-dev@lists.zeromq.orghttp://lists.zeromq.org/mailman/listinfo/zeromq-dev -- Best regardsIlja Golshtein ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] How many messages are in queue now?
Patches welcome. On Friday, April 17, 2015, Ilja Golshtein ilej...@narod.ru wrote: Yes, this is true. If a sysadmin detects that a disk storage is 90% full, she notifies users and do other things to obtain more room, while it is quite possible that the disk is only 89% full at the moment. So, ability to calculate disk usage is a likely source of confusion and gnashing of teeth. Right. 17.04.2015, 17:06, Thomas Rodgers rodg...@twrodgers.com javascript:_e(%7B%7D,'cvml','rodg...@twrodgers.com');: I suppose then it is a question of general utility. Even with ZMQ_PAIR, it is still at best and estimate, likely to have changed (perhaps considerably) from the time you query it to the the time you did something based on that information. Even under the best of circumstances it seems of such limited utility (and a likely source of confusion and gnashing of teeth) that I don't see the value in adding the feature. On Fri, Apr 17, 2015 at 9:00 AM, Ilja Golshtein ilej...@narod.ru javascript:_e(%7B%7D,'cvml','ilej...@narod.ru'); wrote: Thomas, Agree that it makes perfect sense for multiple producers and consumers, while in my case it is ZMQ_PAIR. Thanks. 17.04.2015, 16:49, Thomas Rodgers rodg...@twrodgers.com javascript:_e(%7B%7D,'cvml','rodg...@twrodgers.com');: At the point a message is placed on the queue it is either at HWM or has at least space for one message, this can be evaluated atomically, claiming the 'slot' under the HWM for the message to be placed on the queue. Any other threads attempting to send would synchronize with this operation and the HWM will be respected, no overshoot. Any observation of an atomic count, as it is being changed, will be the state of the count, at some point in the past. So you could have a HWM of 1000, read a count of 850, make some decision, yah, cool to send, 90%, and have a failing subsequent send, because in the time it took you to do all of that, another thread may have enqueued 30 messages, and yet another thread may have tried to enqueue 50 messages. Any such observations of queue depth with multiple producers and consumers are at best, estimates of queue state at some point in the past, and it is very difficult IME to build reliable logic around that information. On Friday, April 17, 2015, Ilja Golshtein ilej...@narod.ru javascript:_e(%7B%7D,'cvml','ilej...@narod.ru'); wrote: Hello Pieter, thank you for your answer. I am in process of reading http://zguide.zeromq.org/hx:chapter7 , while atomic counter seems more natural choice for inproc so far. Could you please explain why it is possible to detect that we are at 100% of HWM and it is not possible to detect that we are e.g. at 90% ? 17.04.2015, 11:54, Pieter Hintjens p...@imatix.com: It was never possible, due to the fact that pipes are being written and read asynchronously. You cannot measure the free space except by stopping everything. The workaround is to use credit based flow control. I've a section in the Guide that explains how this works. You can in effect know what % of the pipe from sender to receiver (including all ZeroMQ and network buffers) is filled. -Pieter On Wed, Apr 15, 2015 at 4:30 PM, Ilja Golshtein ilej...@narod.ru wrote: Hello. I used to think that it is possible to retrieve number or queued messages per socket in recent versions of 0mq, while I failed to find correspond getsockopt in 4.0.5 Use case: I need to alter behavior of my application (drop certain type of messages) if a queue is, say, 90% of HWM. Please, advise if this facility is implemented or planned, or suggest a workaround which is more elegant than having atomic variable (it is inproc) maintained by reader and writer threads which is what I plan to do. Thanks. -- Best regards Ilja Golshtein ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev -- Best regards Ilja Golshtein ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev , ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org javascript:_e(%7B%7D,'cvml','zeromq-dev@lists.zeromq.org'); http://lists.zeromq.org/mailman/listinfo/zeromq-dev -- Best regards Ilja Golshtein ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org javascript:_e(%7B%7D,'cvml','zeromq-dev@lists.zeromq.org'); http://lists.zeromq.org/mailman/listinfo/zeromq-dev , ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org
Re: [zeromq-dev] How many messages are in queue now?
... On Fri, Apr 17, 2015 at 5:12 PM, Thomas Rodgers rodg...@twrodgers.com wrote: Patches welcome. On Friday, April 17, 2015, Ilja Golshtein ilej...@narod.ru wrote: Yes, this is true. If a sysadmin detects that a disk storage is 90% full, she notifies users and do other things to obtain more room, while it is quite possible that the disk is only 89% full at the moment. So, ability to calculate disk usage is a likely source of confusion and gnashing of teeth. Right. 17.04.2015, 17:06, Thomas Rodgers rodg...@twrodgers.com: I suppose then it is a question of general utility. Even with ZMQ_PAIR, it is still at best and estimate, likely to have changed (perhaps considerably) from the time you query it to the the time you did something based on that information. Even under the best of circumstances it seems of such limited utility (and a likely source of confusion and gnashing of teeth) that I don't see the value in adding the feature. On Fri, Apr 17, 2015 at 9:00 AM, Ilja Golshtein ilej...@narod.ru wrote: Thomas, Agree that it makes perfect sense for multiple producers and consumers, while in my case it is ZMQ_PAIR. Thanks. 17.04.2015, 16:49, Thomas Rodgers rodg...@twrodgers.com: At the point a message is placed on the queue it is either at HWM or has at least space for one message, this can be evaluated atomically, claiming the 'slot' under the HWM for the message to be placed on the queue. Any other threads attempting to send would synchronize with this operation and the HWM will be respected, no overshoot. Any observation of an atomic count, as it is being changed, will be the state of the count, at some point in the past. So you could have a HWM of 1000, read a count of 850, make some decision, yah, cool to send, 90%, and have a failing subsequent send, because in the time it took you to do all of that, another thread may have enqueued 30 messages, and yet another thread may have tried to enqueue 50 messages. Any such observations of queue depth with multiple producers and consumers are at best, estimates of queue state at some point in the past, and it is very difficult IME to build reliable logic around that information. On Friday, April 17, 2015, Ilja Golshtein ilej...@narod.ru wrote: Hello Pieter, thank you for your answer. I am in process of reading http://zguide.zeromq.org/hx:chapter7 , while atomic counter seems more natural choice for inproc so far. Could you please explain why it is possible to detect that we are at 100% of HWM and it is not possible to detect that we are e.g. at 90% ? 17.04.2015, 11:54, Pieter Hintjens p...@imatix.com: It was never possible, due to the fact that pipes are being written and read asynchronously. You cannot measure the free space except by stopping everything. The workaround is to use credit based flow control. I've a section in the Guide that explains how this works. You can in effect know what % of the pipe from sender to receiver (including all ZeroMQ and network buffers) is filled. -Pieter On Wed, Apr 15, 2015 at 4:30 PM, Ilja Golshtein ilej...@narod.ru wrote: Hello. I used to think that it is possible to retrieve number or queued messages per socket in recent versions of 0mq, while I failed to find correspond getsockopt in 4.0.5 Use case: I need to alter behavior of my application (drop certain type of messages) if a queue is, say, 90% of HWM. Please, advise if this facility is implemented or planned, or suggest a workaround which is more elegant than having atomic variable (it is inproc) maintained by reader and writer threads which is what I plan to do. Thanks. -- Best regards Ilja Golshtein ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev -- Best regards Ilja Golshtein ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev , ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev -- Best regards Ilja Golshtein ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev , ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev -- Best regards Ilja Golshtein ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] How many messages are in queue now?
On Apr 17, 2015, at 10:00 AM, Ilja Golshtein ilej...@narod.ru wrote: Agree that it makes perfect sense for multiple producers and consumers, while in my case it is ZMQ_PAIR. Simple solution for your use case: Add an incrementing serial number to each of your outgoing messages. The sending thread keeps track of the last serial number it sent. The receiving thread keeps track of the last serial number it received. Periodically, a third thread: Collects both numbers for the current point in time (via shared variables or sockets, it doesn't matter). Subtracts last_sent - last_received. The difference is the approximate number of messages in the queue at that moment. Compares that answer against the HWM and takes whatever action is appropriate. Does that help? Have Fun, Steve Eley smime.p7s Description: S/MIME cryptographic signature ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] How many messages are in queue now?
Hello. I used to think that it is possible to retrieve number or queued messages per socket in recent versions of 0mq, while I failed to find correspond getsockopt in 4.0.5 Use case: I need to alter behavior of my application (drop certain type of messages) if a queue is, say, 90% of HWM. Please, advise if this facility is implemented or planned, or suggest a workaround which is more elegant than having atomic variable (it is inproc) maintained by reader and writer threads which is what I plan to do. Thanks. -- Best regards Ilja Golshtein ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
