nickva commented on PR #4729: URL: https://github.com/apache/couchdb/pull/4729#issuecomment-1689333326
> updated again. I've expanded fabric_streams existing 'cleanup' process to be a watchdog (in the classic sense). > > before starting a fabric_stream you must call `fabric_stream:enable_watchdog().` If you do, the cleanup process starts a timer. If the watchdog is not kicked at least once (with `fabric_stream:kick_watchdog().`) before the timer fires, the watchdog kills the coordinator process (for us, the mochiweb/httpd process) and its workers. > > by default this is not active, so all calls to fabric:all_docs etc are unaffected. > > in mango_httpd I enable the watchdog and only when adding a row to the response do I kick the watchdog. If we go a long time between rows, the watchdog fires. > > similar enable/kick can be done on other endpoints. That's a decent solution. It would detect inactivity on the application side (workers are slow or no index defined or if anything else gets stuck and we don't emit rows). The timeout might be a bit low as there might be existing (even if perhaps pathological) queries which could emit rows every 61 seconds, or if we raise to 5 minutes, there might still be a some cases which we might still break. It's sort of a balancing act as if we make it too large, it might reduce the effect of this watchdog (requests might pile up). I still think it might be better to be able to detect if the client socket has closed so I kept pressing with the tcp_info options check to see how many OS-es we it can cover and to make it a bit less ugly. So far I couldn't figure how to get windows to do it but got all the others I think. Here we can be more bold with a polling interval even at 5 or 10 seconds if needed as we can confidently clean if the client disconnects. Another direction to take it is to emit newline (heartbeats) between json rows. I couldn't quite see where to do that without disturbing the views and all_docs callback mechanism too much. Maybe have a last timestamp on the cursor in https://github.com/apache/couchdb/blob/e60e27554708f46a9e6528fa6049d025c1aba859/src/mango/src/mango_cursor_view.erl#L487-L491 and we can issue a handle_doc(...) callback with a `timeout` and then in the mango_http we'd emit a `\n`? That would maintain the connection alive and quit it as soon as a send failure happens? I am not too familiar with mango so that may be completely off, too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
