Hi Michal, regarding limits, there already is a configurable hard limit on the mailbox size. What I want to add is a soft limit to print a warning, plus analogous limits for message age. For my use case, I'm primarily interested in the soft limits, to get a log message as soon as things start to go wrong.
Regarding a) (for the rest of the audience, fty-rest is our "gateway" between HTTP and malamute): It certainly is possible, but the respective HTTP request can time out anyway and delivering the mailbox message at this point is a waste of cycles... Regarding b), that's what I considered in my original email, but it's not as simple as it looks: The broker would need to keep track of past ephemeral clients, so as not to queue mail for them, so either its memory consumption would gradually grow beyond limits, or it would have to forget clients after some time, which means that there would have to be an arbitrary limit... Which means we are back to square one ;). That said, I can go for either solution, just let me know which one is preferable. But keep in mind that it's a choice between two evils: Either a configurable pattern match, which works fine, but is admittedly a hack, or a per-client ephemeral flag, which is a cleaner interface, but the implementation would have to deal with a growing set of past client connections. Thanks, Michal On 2017-12-12 11:42, Michal Vyskocil wrote: > Hi Michal, > > I am not happy with limits, they are hard to get right and will probably > break legal cases. My proposal is to > > a) Fix fty-rest to use malamute client pool > https://github.com/42ity/fty-rest/blob/master/src/shared/tntmlm.cc > exclusively. This way you will have fixed number of mlm clients, which > will regularly read their mailboxes. Theoretically it is possible that > their mailbox will grow indefinitely, but that'd be very pathological > case - like only one thread issuing long replies and timeouts all the time. > > b) Extend malamute protocol, so clients can mark themselves as > "ephemeral". Adding broker based configuration just brings additional > hidden protocol. Client always knows if he wants to have persistent > mailbox or not. Don't forget this does not work with mlm client pool :-) > See following example (and please come with better names :-) > > |// Set client's mailbox as temporary, so messages won't be queued if > client is not connected. > // Return 0 if successful, -1 otherwise (interrupted, not supported) > int mlm_client_set_temporary_mailbox (mlm_client_t *); > > //... > > mlm_client_t *client = mlm_client_new (); > mlm_client_set_temporary_malbox (client); mlm_client_connect (...); | > > > The limits inside broker looks really complicated and impossible to > setup in real use cases. Plus if you really know that if client is gone > you're not interested in messages, "ephemeral" clients is what you want to. > > Bye > Michal > > > On Mon, Dec 11, 2017 at 12:23 PM, Osiris Pedroso <[email protected] > <mailto:[email protected]>> wrote: > > Sounds like a good improvement on Malamute's current implementation. > Go ahead a submit your PR wit these changes. > If someone deems them improvable, they will do so by doing a new PR. > > PS: About the limits, I suggest a much smaller limit for the warning: > size-limit-warn = 65536 > > > > On Fri, Dec 8, 2017 at 8:07 AM Marek, Michal <[email protected] > <mailto:[email protected]>> wrote: > > Hi, > > I've been debugging issues in our project, where malamute's > memory usage > increases due to undelivered mailbox messages. The > mailbox/size-limit > config option helps in some cases, but it does not cover everything. > E.g., we have client connections that are ephemeral -- they > connect with > a randomly generated address, send a request to some other client's > mailbox and either receive a response or disconnect after a > timeout. The > result is that if the system is under higher load, malamute ends > up with > bunch of queues holding one message each. Another issue is that it's > difficult to debug such cases, because the mailbox engine is rather > silent. Would you guys take patches that > > A) add some "soft" limit for the queue size to log a warning > when the > mailbox grows too much? I'm thinking of something like > mlm_server > mailbox > size-limit = 524288 > size-limit-warn = 262144 > > B) add an aging mechanism for messages? The idea is that a message > undelivered for a long period of time can a sign of a > problem, either > on the sender or the receiver. In fact, in our use case, all the > clients are supposed to be connected, so actually any queued > message > is an anomaly. > > C) implement some special handling of ephemeral clients, so that > messages directed to them are either delivered directly or > dropped. > Now the problem is that if the clients are to advertise their > ephemeral nature, malamute would have to keep track of past > clients, > so the memory usage would just accumulate elsewhere :). How about > keying this off the client address somehow, e.g. via a > configurable > regex: > mlm_server > mailbox > no-queue-pattern = ^temp\..* > and the clients would set their address to "temp.XXXXXX". > > The above also applies to service queues to some extent, we just > do not > use this model in our project. What do you think? > > Thanks, > Michal -- Michal Marek Eaton European Innovation Center Open Source Developer Bořivojova 2380 42ity.org 252 63 Roztoky Tel: +420 211 151 532 www.eaton.cz ----------------------------- Eaton Elektrotechnika s.r.o. ~ Sídlo společnosti, jak je zapsáno v rejstříku: Komárovská 2406, Praha 9 - Horní Počernice, 193 00, Česká Republika ~ Jméno, místo, kde byla společnost zaregistrována: Praha ~ Identifikační číslo (IČO): 498 11 894 ----------------------------- _______________________________________________ zeromq-dev mailing list [email protected] https://lists.zeromq.org/mailman/listinfo/zeromq-dev
