pgj opened a new pull request, #4735: URL: https://github.com/apache/couchdb/pull/4735
## Overview In case of map-reduce views, the arrival of the `complete` message is not guaranteed for the view callback (at the shard) when a `stop` is issued during the aggregation (at the coordinator). Due to that, internally collected shard-level statistics may not be fed back to the coordinator which can cause data loss hence inaccuracy in the overall execution statistics. Address this issue by switching to a "rolling" model where row-level statistics are immediately streamed back to the coordinator. Support mixed-version cluster upgrades by activating this model only if requested through the map-reduce arguments and the given shard supports that. This is only a proposal, a way explore the approach, comments and feedback are welcome. Remarks: - It clearly uses more bandwidth as it will send more `execution_stats` messages. Is this acceptable? - It does not require changes in fabric itself, the whole logic is kept on the Mango side and it is compatible with the previous solution. - Are there other ways to stop the aggregation as soon as the limit is reached? - Are there other kind of messages coming from the shards that could be reliably used to signal the end of statistics collection (alternatives to capturing `complete`)? ## Testing recommendations Running the respective Mango unit and integration test suites might suffice (which is done by the CI): ```shell make eunit apps=mango make mango-test MANGO_TEST_OPTS="15-execution-stats-test" ``` But there is a detailed description in related the ticket (see below) on how to trigger the problem. Feel free to kick the tires. ## Related Issues or Pull Requests Fixes #4560 ## Checklist - [x] Code is written and works correctly - [x] Changes are covered by tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@couchdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
