witash opened a new pull request, #6138:
URL: https://github.com/apache/couchdb/pull/6138
<!-- Thank you for your contribution!
Please file this form by replacing the Markdown comments
with your text. If a section needs no action - remove it.
Also remember, that CouchDB uses the Review-Then-Commit (RTC) model
of code collaboration. Positive feedback is represented +1 from
committers
and negative is a -1. The -1 also means veto, and needs to be addressed
to proceed. Once there are no objections, the PR can be merged by a
CouchDB committer.
See: http://couchdb.apache.org/bylaws.html#decisions for more info.
Artificial Intelligence and Large Language Models Contributions Policy
It is expressly forbidden to contribute material generated by
AI, LLMs, and similar technologies, to the CouchDB project.
This includes, but is not limited to, source code, documentation,
commit messages, or any other areas of the project.
-->
## Overview
When updating nouveau indexes, `nouveau_index_updater:update/1` runs two
phases: first the purge phase (`purge_index`), which walks the purge infos
since the index's `purge_seq`, then the changes phase, which walks the changes
feed since the index's `update_seq`.
If the document for a purge info still exists in the database, the current
behavior is to re-index the document's current winning revision during the
purge phase. This is sent to nouveau as a normal index update, using the
document's current update seq, so it moves the index's `update_seq` to that
seq, and the changes phase then starts from the moved `update_seq`.
Purge infos are processed in purge order, but the current seqs of the
documents that still exist don't follow that order, and can even be lower than
the index's `update_seq`. So these updates can go backwards, and nouveau
rejects them with "Index updates applied in the wrong order". When the update
does go forward, `update_seq` can skip past changes that haven't been indexed
yet, and the changes phase never indexes them, with no error.
This branch stops re-indexing documents that still exist during the purge
phase, so the purge phase never changes `update_seq`. If the document for a
purge info still exists, the purge phase compares the document's current update
seq with the index's `update_seq` from the start of the update:
- If the document's seq is greater, the document was changed after the index
last saw it (a purge that removes revisions gives the document a new seq). It's
removed from the index with a purge delete, which only moves `purge_seq`, and
the changes phase then indexes its current winning revision.
- If the document's seq is less than or equal, the purge didn't change the
document and the index already has its current winning revision, so nothing is
sent.
The exclude list, which let the changes phase skip documents already
re-indexed in the purge phase, is removed, since nothing is re-indexed there
any more. And the purge phase no longer needs a JS process, since it doesn't
run the index function.
This keeps the index consistent with the rules stated in the documentation
(`src/docs/src/api/database/misc.rst`, "Updating Indexes"):
- If the winner of the revision tree is not specified in the purge request,
there is no change to the index record of this document.
- If the winner of the revision tree is specified in the purge request, and
there is still a revision left after purging, the index record of the document
will be built according to the new winner of the revision tree.
- If all revisions of the document are specified in the purge request, the
index record of the document will be deleted. The document will no longer be
found in searches.
But there is a small change in behavior; between the purge phase and the
changes phase, documents that were purged and then updated are not in the
index. This is basically consistent with applying purges and then updates in
order; first apply the purge (the old document is no longer indexed), then
apply the update (the new document is indexed). But it's different from the
current behavior, where the new document is indexed as soon as the purge is
applied.
I didn't see any way around this, other than a larger change to nouveau so
that it can apply updates without changing `update_seq`.
## Testing recommendations
`make nouveau-test`. The new elixir test `purge with out of order edits` (in
`test/elixir/test/nouveau_test.exs`) creates a single-shard database, indexes 4
documents, purges two of them (A, then B, in separate requests), creates a new
document C, re-creates B and then A, and checks that the search returns all 5
documents and that the index's `update_seq` and `purge_seq` match the database.
Without this change, the final search fails with "Index updates applied in the
wrong order (current seq: 10, match seq: 10, attempted seq: 9)".
The existing `purge with conflicts` test covers documents where the purge
removes the winning revision and a revision is left, which now goes through the
purge delete and the changes phase.
## Related Issues or Pull Requests
Fixes #6118. #6096 fixed the `{invalid_start_purge_seq, 0, N}` error from
the same issue; this fixes the "Index updates applied in the wrong order" error
that still happens after it, when documents that were purged still exist.
## Checklist
- [ ] This is my own work, I did not use AI, LLM's or similar technology
- [ ] Code is written and works correctly
- [ ] Changes are covered by tests
- [ ] Any new configurable parameters are documented in
`rel/overlay/etc/default.ini`
- [ ] Documentation changes were made in the `src/docs` folder
- [ ] Documentation changes were backported (separated PR) to affected
branches
## AI Disclosure
I used Claude code to review changes that I authored, and to help with
Erlang syntax, but all of the code in this PR was written by me directly, not
Claude.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
