Dear Qpid users, In our mission critical software for the Dutch government we use Qpid proton 0.34.0 in our C++-client software together with the Qpid dispatch router 1.16.0. We updated to these versions not so long ago, before we used proton 0.25.0 and dispatch 1.3.0. Our application runs on several VM's with a router on each VM. All clients connect to the local router only and the routers connect to eachother in a hub spoke pattern. In both the client configuration as the router configuration we have configured an idle timeout of 30 seconds.
Two weeks ago we were confronted with an incident in production where a lot of our client processes reported problems regarding the idle timeouts. These client processes were already running stable for more than 3 weeks. The problem appeared in two flavors: 1. Transport error "error: amqp:resource-limit-exceeded: local-idle-timeout expired" 2. epoll proactor failure in epoll_timer.c:263: "idle timeout sequencing error" On each VM at least 3 processes showed one of these problems in a time window of less than a minute. We haven't found any cause in the underlying hardware, hypervisor, network or operating system until now. Although we don't know the root cause of the problems, we can solve the first situation by using the proper reconnect settings. However the second situation is more odd because it results in an abort within proton itself. The comments in epoll_timer.c explain that this error occurs when a connection timer is moved backwards a second time. We don't understand how this can happen suddenly. Does anyone have experienced similar problems using recent proton versions (the epoll_timer.c module is introduced in version 0.33.0). And even more important is there a solution or workaround? Looking forward to any reaction. Thanks in advance! With kind regards, Jaap Wiggelinkhuizen Software architect & Systeem integrator [cid:image001.jpg@01D77971.55B01F30] E jaap.wiggelinkhui...@intraffic.nl<mailto:jaap.wiggelinkhui...@intraffic.nl> W intraffic.nl<https://www.intraffic.nl/> [Afbeelding met tekening, bord Automatisch gegenereerde beschrijving]<https://www.linkedin.com/company/intraffic>[cid:image004.png@01D77971.55B01F30]<https://twitter.com/InTrafficNL> [cid:image005.png@01D77971.55B01F30] <https://www.youtube.com/channel/UCPQeh0v2U2v2hBRlNFXNY9A> [cid:image006.png@01D77971.55B01F30] <https://www.facebook.com/InTrafficNL/> Visiting address: Iepenhoeve 11, 3438 MR Nieuwegein [cid:image007.jpg@01D77971.55B01F30]<https://ictgroup.eu/>
