https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43584
Bug ID: 43584
Summary: Elasticsearch mapping changes that require a full
reindex silently leave the live index broken until an
admin manually reindexes
Initiative type: ---
Sponsorship ---
status:
Product: Koha
Version: Main
Hardware: All
OS: All
Status: NEW
Severity: normal
Priority: P5 - low
Component: Searching
Assignee: [email protected]
Reporter: [email protected]
QA Contact: [email protected]
Target Milestone: ---
When a library admin changes a search field's mapping in Administration >
Search engine configuration (Elasticsearch)
(admin/searchengine/elasticsearch/mappings.pl) in a way that isn't purely
additive (e.g. changing a field's type), Koha does not reindex or roll back -
it leaves the live Elasticsearch index in a broken, mismatched state
indefinitely, with no visible warning to end users and no automatic recovery.
The only fix is a library admin/sysadmin noticing the problem and manually
running a full misc/search_tools/rebuild_elasticsearch.pl -d (or -r), which for
a large catalogue can mean a long window of degraded/incorrect search results.
Root cause (traced in Koha/SearchEngine/Elasticsearch/Indexer.pm and
admin/searchengine/elasticsearch/mappings.pl):
1. Saving the mappings form immediately calls Indexer->update_mappings(), which
issues a live Elasticsearch PUT _mapping against the existing index (Indexer.pm
update_mappings, around line 262).
2. Elasticsearch's put_mapping API can only add genuinely new fields - it
cannot change an already-mapped field's type/analyzer. When the change isn't
purely additive, put_mapping throws. Koha catches this, sets the index status
to "recreate required" (INDEX_STATUS_RECREATE_REQUIRED), and shows an error
banner - but only on the mappings.pl admin screen itself.
3. That index status is never checked anywhere else in the codebase - not in
search.pl, not in the OPAC, not in the live cataloguing indexing pipeline
(Koha::SearchEngine::Elasticsearch::Indexer::update_index, called on every
biblio/item save). So cataloguing and searching both continue to hit the
now-inconsistent index with no gating or warning.
4. Nothing schedules or forces the required full reindex. It's entirely a
manual, unscheduled, blocking operation the admin must remember to run.
I reproduced this on main (2026-09-21, es8) as follows:
Test plan / reproduction steps:
1. Set up KTD with Elasticsearch (e.g. ktd --search-engine es8 up -d), set the
SearchEngine system preference to "Elasticsearch", and run a full reindex
(misc/search_tools/rebuild_elasticsearch.pl -r -v) so the index and DB mapping
config agree.
2. Confirm a normal search works, e.g. search for a known title in the OPAC or
staff interface.
3. Go to Administration > Search engine configuration (Elasticsearch) >
Bibliographic records.
4. Edit an existing, already-mapped field that has real indexed data - for
example change "title" from type "String" to type "Number" - and save.
5. Observe: the page reports an error and Koha logs (or, reproduced directly in
Perl):
Unable to update mappings for index "koha_<instance>_biblios". Reason was:
"mapper [title] cannot be changed from type [text] to [integer]". Index needs
to be recreated and reindexed
6. Note that this error is only visible on the mappings.pl page itself. Go to
the OPAC or staff client and search - searches continue to run against the
still-mismatched index with no warning to the person searching, and
results/behaviour for the affected field are now undefined/inconsistent with
the field's newly configured type and options (facets, sort, filters depending
on the field silently misbehave).
7. The only way to recover is for someone to notice the admin banner and run,
e.g.: misc/search_tools/rebuild_elasticsearch.pl -d -b -v (drop & recreate +
reindex biblios), which for a large catalogue can take a long time, during
which search functionality for the affected field(s) stays broken.
Note this isn't limited to "hard failure" cases like the type-change example
above. Even a purely additive mapping change (e.g. adding a new facet on an
existing field) succeeds immediately at the Elasticsearch level, but every
record indexed before the change has no value in the new field - so
facets/filters/sort on it are silently incomplete for the whole existing
catalogue until a full reindex populates it. There is no indication to a
searching user (or even the admin) that results are incomplete.
Suggested directions (for discussion, not yet implemented):
A. Blue/green reindex via index aliases: point the Elasticsearch index name
Koha reads/writes as an alias (rather than the literal <instance>_biblios index
it uses today - see Koha::SearchEngine::Elasticsearch::Indexer
create_index/update_mappings/index_name) at a concrete versioned index
(<instance>_biblios_v3). When a mapping change requires a rebuild, build a new
concrete index with the new mappings in the background, reindex into it, then
atomically flip the alias once it's caught up (capturing any records changed
during the rebuild window in a final catch-up pass), and drop the old index.
This is the standard Elasticsearch zero-downtime reindex pattern and would mean
searches keep working against the old, still-consistent index throughout, with
no admin-visible downtime at all.
B. Scheduled off-hours full reindex with fallback: rather than applying an
incompatible mapping change immediately, queue it, keep serving searches
against the current (old-mapping) index/config, and run the full reindex as a
background/cron job during a configured off-hours window, then apply the new
mapping config atomically once the reindex completes.
B is strictly weaker than A (still involves a window where either old or new
config must be chosen, and doesn't help instances that want the change live
sooner), so A is the preferred direction, but is a bigger architectural change
(introducing alias-based indices, a background full-reindex job, and a "reindex
in progress" status) than B. Either would be a large enough change to warrant
an RFC / community discussion before a patch is written - filing this now
primarily to document the problem, root cause, and reproduction steps, and to
open discussion on which direction (or another one entirely) the community
prefers.
At minimum, independent of A/B, two smaller, cheap wins are worth considering
regardless of the larger direction chosen:
- Surface the "recreate required" / "reindex required" index status somewhere
visible beyond the mappings.pl page (e.g. a staff client system alert), so
admins aren't only warned if they happen to revisit that specific screen.
- Have mappings.pl warn before saving when a proposed type change is likely to
be non-additive (best-effort - comparing old vs new type), rather than only
after the fact.
--
You are receiving this mail because:
You are watching all bug changes.
You are the assignee for the bug.
_______________________________________________
Koha-bugs mailing list -- [email protected]
To unsubscribe send an email to [email protected]
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
