On Thu, 2010-04-01 at 11:23 +1300, Tim Beale wrote: > Hi Steve, > > End-to-end flow control is something I'd really love to see. It sounds like > your proposal won't fix all the problems we're seeing with flow control > though. > > A problem we've seen is kind of permanent congestion - the receiver gets a > burst of several hundred CPG messages queued up and never really recovers. The > sender continues sending enough CPG messages that the receiver never clears > out > its queue, but doesn't run out of memory either. The receiver's queue could > hover in this state indefinitely. On our setup, a healthcheck mechanism > detects > the receiver has locked up (some operations are blocking due to flow control > congestion) and eventually restarts the process. > (As an interim workaround for this on our setup, I fudged the token backlog > calculation to gradually force the sender to backoff, so the sender's totem > message queue fills up and it starts getting TRY_AGAIN errors). > > I was wondering whether end-to-end flow control at the CPG group level is a > possible/feasible option that'd solve both this case and the oom one? E.g. in > the CPG library code it sends an internal message to notify the rest of the > CPG > group whenever the flow control status for an application changes? >
Tim, Also more directly addressing this last point, I have tried that in the past and it didn't operate as I would expect, but alot of things have changed since I tried (openais 0.80.2ish... which used a different non-shared memory ipc system). Likely worth some more investigation. In the case of memory overusage on one node, lib user gets try again on all nodes if that hasn't happened, but cpg flow control is active, lib user gets try again on all nodes Double flow control features (one based on protecting against oom, one to protect the individual cpg group). Regards -steve > Regards, > Tim > _______________________________________________ > Openais mailing list > [email protected] > https://lists.linux-foundation.org/mailman/listinfo/openais _______________________________________________ Openais mailing list [email protected] https://lists.linux-foundation.org/mailman/listinfo/openais
