rnewson opened a new pull request, #4749:
URL: https://github.com/apache/couchdb/pull/4749
## Overview
This is a performance only pull request, but a complicated one.
I benchmarked nouveau against clouseau for the same database and found it
was slower (2-4x slower depending on how large the documents were, the smaller
the docs the bigger the difference).
This is fundamentally because clouseau's overhead between erlang and the jvm
is much lower, as they are exchanging messages over an established, dedicated
connection.
To improve nouveau performance I have adopted a similar approach, while
ensuring index consistency safety.
1) nouveau on the jvm side requires each index update request (whether a
document update, delete or purge, or a request to advance update_seq or
purge_seq independently from an index change) to pass the current update_seq or
purge_seq of the index. This is critical in detecting missed updates in the
events of failure.
2) nouveau_index_updater spawns its own ibrowse worker (and, thus, its own
dedicated http connection) when it starts to update an index. This ensures that
all requests issued from the erlang side and received in the same order on the
jvm side. Using the ibrowse pool would have allowed for a reordering.
3) finally, the performance improvement itself. nouveau_index_updater sets
the `[{stream_to, self()}]` option, so does not wait for the response,
successful or otherwise, before making the next one. This is why item 1 is
critical. If there is a failure for one update, all subsequent updates will
also fail, as the update/purge sequence on the jvm side will not have advanced.
At the start of each loop of the callback fun we do a selective receive to
collect any and all async responses we might have received. If any are errors,
we terminate the update process with that error, otherwise the messages are
discarded. We also check after then callback fun is finished but this time we
block to receive all the outstanding responses, to ensure the index is in the
state we require before issuing the request.
I've separated out these changes across multiple commits for readability.
## Testing recommendations
covered by the test suite
## Related Issues or Pull Requests
N/A
## Checklist
- [x] Code is written and works correctly
- [x] Changes are covered by tests
- [ ] Any new configurable parameters are documented in
`rel/overlay/etc/default.ini`
- [ ] Documentation changes were made in the `src/docs` folder
- [ ] Documentation changes were backported (separated PR) to affected
branches
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
