zdravko123 commented on issue #2441: URL: https://github.com/apache/couchdb/issues/2441#issuecomment-671647975
Thanks for that, i had already suggested we do restarts of the couch service rather than the reboots during our daily chron jobs. We though it could be daily chron jobs causing this and some adverse effect in the connections. The reason why its so big of an issue for us is because we run some pretty massive clusters, 4 x 32 core instances with 64GB ram each, when the ram dumps it dumps hard, ie 50gb, during this time the node becomes unavailable, I suspect due to the fact it needs to then reload into ram the data to effectively serve requests. This takes time, it ends up taking longer to serve these requests and holds on to the connections, this then has a cascading effect that it exhausts the connection limit maximum threshold resulting in 502 Bad gateway errors as other requests are still coming in. Now for most of the users of couchDB this isn't a problem as they are not running a massive cluster, but sadly for us we have half the country running of it, where a small glitch like this is noticeable to say the least. We have had to manually reboot the system in the past and has resulted on someone being on-call 24/7 ... me... While scheduled reboots would fix it a patch would be greatly appreciated, esp since its just a few lines of code. We are investigating migrating to couch 3.x but thats still in the planing stages and not sure it will happen this side of the year, and will prob result in 2 clusters, 1 remaining on 2.3 due to other limits related to connection, ect. since we have over 6,000 databases atm and twice as many replications. if this can be done a paid hot fix please contact me also. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
