That's ok, this is only for a simulation test. I'm not focused on congestion issues right now. I'm getting a measure of how much the implementation decisions have improved scalability in both space and speed across the entire software system.
On Tue, Jul 28, 2015 at 8:12 PM, Justin Karneges <[email protected]> wrote: > Sure thing. And yeah, if you're always polling forever for writability > before writing, then you effectively have a backpressure strategy. > > Note that if you begin polling for read and write at the same time, then > you open the door to potentially processing new data while you've still got > pending outbound data to write, and then you will face those three choices > I mentioned. > > On Tue, Jul 28, 2015, at 05:07 PM, Kenneth Adam Miller wrote: > > Thank you, by the way. > > On Tue, Jul 28, 2015 at 8:07 PM, Kenneth Adam Miller < > [email protected]> wrote: > > There's no need to take any of those options because in my case, because > the producer polls every time before sending. > > I think I realize with your first sentence what I missed when I wasn't > vocalizing my issue. I think I now know that it's not because the other > clients aren't concurrent that there is an issue, it's the lack of > isolating the location in code for polling into doing both read and write > polling in one call. If it were one call, then the send lockup could never > happen, but right now, it occurs in sequence, where if you return from > polling for read, you read, do work, and then poll before sending that back > out. In any case, I think I am highly confident I have the correct > implementation path now. > > On Tue, Jul 28, 2015 at 7:43 PM, Justin Karneges <[email protected]> > wrote: > > > You can poll for read and write at the same time, so there shouldn't be > any issue with both peers sending simultaneously and then failing to read. > > Note that if you can't write, then you need to think about how to manage > your outbound messages. The three basic choices you have are: > 1) Backpressure (while waiting to write, stop doing anything that causes > new messages to be produced) > 2) Drop messages > 3) Queue every message, indefinitely (caution to the wind!) > > The third option is almost never what you want. It sucks having to choose > between the first and second options but that's reality. Once you've > decided on your strategy then you'll know what code to write. > > On Tue, Jul 28, 2015, at 04:28 PM, Kenneth Adam Miller wrote: > > I can't use ZeroMQ for this, because I know if I could this wouldn't be a > problem. But since there are so many systems engineers on the channel and > fundamentally it's what ZMQ aims to solve, I think it's the appropriate > place. > > For a scalability benchmark meant to emulate our real world scenario, I > have a cycle created between a producer and a consumer, where the producer > spams a fifo object that the consumer is listening on. The consumer does > some validation work on the data that it was hit with and sends it back out > over a different fifo object that the producer is concurrently listening to > (rather it's a chain of four processes linked in this way). > > <- process_echo <- > Producer consumer > -> process_echo -> > > This works fine from the perspective of the producer, because listening > and writing requires no coordination between threads. But the producer and > echoing processes were written singled threaded, and from the perspective > that it would poll only for input and not for when it writes. > > Well, as my testing went along I discovered that I had to make both the > producer and consumer poll on output so that no data could be lost when the > sender goes to send and the fifo isn't being read from fast enough. > (possibly that's wrong) > > But using poll on the sender has caused hangups on the system as a whole. > Poll returns for readers, I think, when someone has already written or is > writing. And for writers, when someone is blocking (whether polling or in > read) to read. > > But the problem that this solves brings up a different problem: if the > single threaded programs poll to write, then the cycle could deadlock where > every side is polling to write, before even one message makes it fully > round circle. > > My thought is, I could break it by making the consumer concurrent in > reading and writing, that way, lockup on send cannot happen, because there > will always be a reader to allow execution to continue in the echo tool and > consumer. But that's hard because the code base is large. > > I said all of that just to ask-does anybody have any alternative > suggestions regarding how to approach this exact problem? > > All fifo objects are in non-blocking mode. > *_______________________________________________* > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > *_______________________________________________* > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > >
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
