[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Julian Maurice changed: What|Removed |Added Blocks||36009 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36009 [Bug 36009] Document koha-worker --queue elastic_index -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Nick Clemens changed: What|Removed |Added Blocks||35086 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35086 [Bug 35086] Koha::SearchEngine::Elasticsearch::Indexer->update_index needs to commit in batches -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Lucas Gass changed: What|Removed |Added Status|Pushed to stable|RESOLVED CC||lu...@bywatersolutions.com Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Bug 32594 depends on bug 32992, which changed state. Bug 32992 Summary: Move background worker script to misc/workers https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32992 What|Removed |Added Status|Pushed to stable|RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #46 from Martin Renvoize --- Many hands makes light work, thankyou everyone! Pushed to 22.11.x for the next release -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Martin Renvoize changed: What|Removed |Added Version(s)|23.05.00|23.05.00,22.11.06 released in|| Status|Pushed to master|Pushed to stable -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Martin Renvoize changed: What|Removed |Added Blocks||33486 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33486 [Bug 33486] Remove Koha::BackgroundJob::UpdateElasticIndex->process routine -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Tomás Cohen Arazi changed: What|Removed |Added Keywords|additional_work_needed | --- Comment #45 from Tomás Cohen Arazi --- Follow-up in master. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Martin Renvoize changed: What|Removed |Added Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Martin Renvoize changed: What|Removed |Added Status|ASSIGNED|Passed QA -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Martin Renvoize changed: What|Removed |Added Status|Pushed to stable|ASSIGNED -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Jonathan Druart changed: What|Removed |Added Attachment #148350|0 |1 is obsolete|| Attachment #148571|0 |1 is obsolete|| --- Comment #44 from Jonathan Druart --- Created attachment 148588 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=148588=edit Bug 32594: Mark jobs as started and finished Signed-off-by: Nick Clemens Restored authorship so that it shows the patch has been signed off. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #43 from Nick Clemens --- (In reply to Jonathan Druart from comment #41) > (In reply to Nick Clemens from comment #39) > > Created attachment 148350 [details] [review] [review] > > Bug 32594: POC > > I don't think we should loop over the jobs, we are adding additional (almost > useless) processing. Here we need to do things fast, so let's update them > all at once. To prevent other scripts/jobs to use this, we should not > provide a method (that does not feel correct). I was trying to set the number processed to the numebr in the job, I forgot we made all a size of 1 I squashed and mark yours SO > > Now we need to decide if we allow indexing jobs from another worker. The > detail page of the job is still wrong " / records have successfully been > reindexed. Some errors occurred.". Bug 33319 filed to improve As for processing: > IMO we should not provide 2 ways to process the jobs, only the new worker > should do it. So let's remove the process method and simplify the job detail > view. > Sounds like we should move that to a separate bug... I think that's a third job - we could have some plugins or some outside system wanting to handle indexation for some reason? But I am okay to remove too -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Nick Clemens changed: What|Removed |Added Attachment #148513|0 |1 is obsolete|| --- Comment #42 from Nick Clemens --- Created attachment 148571 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=148571=edit Bug 32594: Mark jobs as started and finished Signed-off-by: Nick Clemens -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Nick Clemens changed: What|Removed |Added See Also||https://bugs.koha-community ||.org/bugzilla3/show_bug.cgi ||?id=33316 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #41 from Jonathan Druart --- (In reply to Nick Clemens from comment #39) > Created attachment 148350 [details] [review] > Bug 32594: POC I don't think we should loop over the jobs, we are adding additional (almost useless) processing. Here we need to do things fast, so let's update them all at once. To prevent other scripts/jobs to use this, we should not provide a method (that does not feel correct). Now we need to decide if we allow indexing jobs from another worker. The detail page of the job is still wrong " / records have successfully been reindexed. Some errors occurred.". This is because it expects Koha::BackgroundJob::UpdateElasticIndex->process to have processed the job, and provided the correct info. IMO we should not provide 2 ways to process the jobs, only the new worker should do it. So let's remove the process method and simplify the job detail view. Sounds like we should move that to a separate bug... -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #40 from Jonathan Druart --- Created attachment 148513 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=148513=edit Bug 32594: Mark jobs as started and finished -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #39 from Nick Clemens --- Created attachment 148350 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=148350=edit Bug 32594: POC -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #38 from Matt Blenkinsop --- Backport to 22.11.04 reversed while additional work is needed -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Matt Blenkinsop changed: What|Removed |Added CC||matt.blenkinsop@ptfs-europe ||.com Version(s)|23.05.00,22.11.04 |23.05.00 released in|| -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #37 from Jonathan Druart --- Should we remove Koha::BackgroundJob::UpdateElasticIndex->progress? -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #36 from Jonathan Druart --- Nick, I am seeing " / records have successfully been reindexed. Some errors occurred. " when showing a job detail. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Tomás Cohen Arazi changed: What|Removed |Added Keywords||rel_22_11_candidate --- Comment #35 from Tomás Cohen Arazi --- POD fix pushed to master. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #34 from Tomás Cohen Arazi --- Created attachment 147562 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=147562=edit Bug 32594: (QA follow-up) Fix POD Signed-off-by: Tomas Cohen Arazi -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Tomás Cohen Arazi changed: What|Removed |Added Blocks||33108 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33108 [Bug 33108] We need a way to launch the ES indexer automatically -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #33 from Matt Blenkinsop --- Nice work everyone! Pushed to stable for 22.11.x -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Matt Blenkinsop changed: What|Removed |Added Version(s)|23.05.00|23.05.00,22.11.04 released in|| Status|Pushed to master|Pushed to stable -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #32 from Tomás Cohen Arazi --- (In reply to Jonathan Druart from comment #31) > I think we forget an important part here, there is no daemon around the new > script, and so it's not running (and records are not indexed!) Maybe bake it into koha-indexer? -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Jonathan Druart changed: What|Removed |Added Keywords||additional_work_needed --- Comment #31 from Jonathan Druart --- I think we forget an important part here, there is no daemon around the new script, and so it's not running (and records are not indexed!) -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #30 from Tomás Cohen Arazi --- Created attachment 147460 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=147460=edit Bug 32594: (QA follow-up) Adjust tests Signed-off-by: Tomas Cohen Arazi -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #29 from Tomás Cohen Arazi --- Pushed to master for 23.05. Nice work everyone, thanks! -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Tomás Cohen Arazi changed: What|Removed |Added Version(s)||23.05.00 released in|| Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Jonathan Druart changed: What|Removed |Added Status|Signed Off |Passed QA --- Comment #28 from Jonathan Druart --- Not sure where my patches disappear, but that's not important. Ready to go! -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Jonathan Druart changed: What|Removed |Added Attachment #146402|0 |1 is obsolete|| Attachment #146831|0 |1 is obsolete|| --- Comment #27 from Jonathan Druart --- Created attachment 147141 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=147141=edit Bug 32594: Add a dedicated Elasticsearch biblio indexing background queue Currently we generate large numebrs if single record reindex for circulation and other actions. It can take a long time to process these as we need to load the ES settings for each. This patch updates the Elasticsearch background jobs to throw records into a new queue that can be processed by it's own worker and adds a dedicated worker that batches the jobs every 1 second. To test: 1 - Apply patches, set SearchEngine system preference to 'Elasticsearch' 2 - perl misc/search_tools/es_indexer_daemon.pl 3 - Leave the running in terminal and perform actions in staff interface: - Checking out a bib - Returning a bib - Editing a single bib - Editing a single item - Batch editing bibs - Batch editing items 4 - Confirm for each action that records are updated in search/search results 5 - Stop the script 6 - set SearchEngine system preference to 'Zebra' 7 - perl misc/search_tools/es_indexer_daemon.pl 8 - Script dies as Elasticsearch not enabled Signed-off-by: David Nind Signed-off-by: Emily Lamancusa Bug 32594: (follow-up) Adjust logging per bug 32612 JD amended patch: tidy! There were tabs here... Signed-off-by: Jonathan Druart -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Nick Clemens changed: What|Removed |Added Depends on||32992 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32992 [Bug 32992] Move background worker script to misc/workers -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #26 from Nick Clemens --- Created attachment 146831 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=146831=edit Bug 32594: (follow-up) Adjust logging per bug 32612 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Bug 32594 depends on bug 32393, which changed state. Bug 32393 Summary: background job worker explodes if JSON is incorrect https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32393 What|Removed |Added Status|Pushed to oldstable |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Fridolin Somers changed: What|Removed |Added CC||fridolin.som...@biblibre.co ||m -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Jonathan Druart changed: What|Removed |Added See Also||https://bugs.koha-community ||.org/bugzilla3/show_bug.cgi ||?id=32920 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #25 from Emily Lamancusa --- Bug 32920 created for the error noted in comment #20 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #24 from Emily Lamancusa --- Testing notes (using KTD): The path to the script for steps 2 and 7 is: perl misc/workers/es_indexer_daemon.pl -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Emily Lamancusa changed: What|Removed |Added Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Emily Lamancusa changed: What|Removed |Added Attachment #145711|0 |1 is obsolete|| --- Comment #23 from Emily Lamancusa --- Created attachment 146402 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=146402=edit Bug 32594: Add a dedicated Elasticsearch biblio indexing background queue Currently we generate large numebrs if single record reindex for circulation and other actions. It can take a long time to process these as we need to load the ES settings for each. This patch updates the Elasticsearch background jobs to throw records into a new queue that can be processed by it's own worker and adds a dedicated worker that batches the jobs every 1 second. To test: 1 - Apply patches, set SearchEngine system preference to 'Elasticsearch' 2 - perl misc/search_tools/es_indexer_daemon.pl 3 - Leave the running in terminal and perform actions in staff interface: - Checking out a bib - Returning a bib - Editing a single bib - Editing a single item - Batch editing bibs - Batch editing items 4 - Confirm for each action that records are updated in search/search results 5 - Stop the script 6 - set SearchEngine system preference to 'Zebra' 7 - perl misc/search_tools/es_indexer_daemon.pl 8 - Script dies as Elasticsearch not enabled Signed-off-by: David Nind Signed-off-by: Emily Lamancusa -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #22 from Emily Lamancusa --- (In reply to Jonathan Druart from comment #21) > (In reply to Emily Lamancusa from comment #20) > > I encountered an error while testing this patch with batch item updates, in > > which one bad bib blocked the ES indexing update for every item in the > > batch. > > Hello Emily, I confirm the problem, but that's not directly linked with this > patch. I am pretty sure this issue exists without this patch. > > When you create a new batch, 2 new jobs are created, the first one is > modifying the items in database, the second one is updating elastic's index. > If you look at the job list you will see that the second one has in error > (still not clear, the status is "finished", but there is an error in the > detail: " / records have successfully been reindexed. Some errors occurred.") > > That's not ideal but we should deal with that on a separate bug report in my > opinion. Good to know - I didn't think to check whether the error was present without the patch (though it seems obvious in retrospect). Everything worked as intended on edits that didn't touch the broken bib, so I'll sign off here and open a new bug. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Jonathan Druart changed: What|Removed |Added Status|Failed QA |Needs Signoff --- Comment #21 from Jonathan Druart --- (In reply to Emily Lamancusa from comment #20) > I encountered an error while testing this patch with batch item updates, in > which one bad bib blocked the ES indexing update for every item in the batch. Hello Emily, I confirm the problem, but that's not directly linked with this patch. I am pretty sure this issue exists without this patch. When you create a new batch, 2 new jobs are created, the first one is modifying the items in database, the second one is updating elastic's index. If you look at the job list you will see that the second one has in error (still not clear, the status is "finished", but there is an error in the detail: " / records have successfully been reindexed. Some errors occurred.") That's not ideal but we should deal with that on a separate bug report in my opinion. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Emily Lamancusa changed: What|Removed |Added Status|Patch doesn't apply |Failed QA --- Comment #20 from Emily Lamancusa --- I encountered an error while testing this patch with batch item updates, in which one bad bib blocked the ES indexing update for every item in the batch. To reproduce the error: 1. Apply patches and run ktd --es7 up 2. Set SearchEngine system preference to 'Elasticsearch' 3. perl misc/search_tools/es_indexer_daemon.pl (leave running in the terminal while performing subsequent steps) 4. Perform an item search for all items where Collection is Reference 5. Export the results to a barcode file (898 items) 6. Upload this file to the Batch Item Modification Tool 7. Make an edit to the Full Call Number* and click Save 8. Notice that es_indexer_daemon prints an error to the console 9. Perform a catalog search 10. Limit the results with the Collection - Reference facet 11. Note that none of the results display the updated Call Number value *Or, as far as I can tell, any edit that will produce a change in the record for biblionumber 369 Error message printed to the console: ^ at /kohadevbox/koha/Koha/Biblio/Metadata.pm line 114. DEBUG - Update of elastic index failed with: Invalid data, cannot decode metadata object (biblio_metadata.id=368, biblionumber=369, format=marcxml, schema=MARC21, decoding_error=':8: parser error : PCDATA invalid Char value 31 00aD15937 ^ :9: parser error : PCDATA invalid Char value 31 00satmrnu0 ^ :9: parser error : PCDATA invalid Char value 31 00satmrnu0 ^ :9: parser error : PCDATA invalid Char value 31 00satmrnu0 ^ :9: parser error : PCDATA invalid Char value 31 00satmrnu0 ^ :10: parser error : PCDATA invalid Char value 31 00ar19881981bdkldan ^ :10: parser error : PCDATA invalid Char value 31 00ar19881981bdkldan ^ :10: parser error : PCDATA invalid Char value 31 00ar19881981bdkldan ^') -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #19 from David Cook --- (In reply to Jonathan Druart from comment #18) > Shouldn't we move misc/background_jobs_worker.pl to misc/workers/ as well? > On another bug? +1 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #18 from Jonathan Druart --- Shouldn't we move misc/background_jobs_worker.pl to misc/workers/ as well? On another bug? -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Jonathan Druart changed: What|Removed |Added QA Contact||jonathan.druart+koha@gmail. ||com -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Jonathan Druart changed: What|Removed |Added Status|Signed Off |Patch doesn't apply --- Comment #17 from Jonathan Druart --- error: sha1 information is lacking or useless (misc/background_jobs_worker.pl). -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Emily Lamancusa changed: What|Removed |Added CC||emily.lamancusa@montgomeryc ||ountymd.gov -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 David Nind changed: What|Removed |Added CC||da...@davidnind.com --- Comment #16 from David Nind --- Testing notes (using KTD): 1. The path to the script for steps 2 and 7 is: perl misc/workers/es_indexer_daemon.pl -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 David Nind changed: What|Removed |Added Attachment #145704|0 |1 is obsolete|| --- Comment #15 from David Nind --- Created attachment 145711 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145711=edit Bug 32594: Add a dedicated Elasticsearch biblio indexing background queue Currently we generate large numebrs if single record reindex for circulation and other actions. It can take a long time to process these as we need to load the ES settings for each. This patch updates the Elasticsearch background jobs to throw records into a new queue that can be processed by it's own worker and adds a dedicated worker that batches the jobs every 1 second. To test: 1 - Apply patches, set SearchEngine system preference to 'Elasticsearch' 2 - perl misc/search_tools/es_indexer_daemon.pl 3 - Leave the running in terminal and perform actions in staff interface: - Checking out a bib - Returning a bib - Editing a single bib - Editing a single item - Batch editing bibs - Batch editing items 4 - Confirm for each action that records are updated in search/search results 5 - Stop the script 6 - set SearchEngine system preference to 'Zebra' 7 - perl misc/search_tools/es_indexer_daemon.pl 8 - Script dies as Elasticsearch not enabled Signed-off-by: David Nind -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 David Nind changed: What|Removed |Added Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Nick Clemens changed: What|Removed |Added Attachment #145145|0 |1 is obsolete|| Attachment #145197|0 |1 is obsolete|| --- Comment #14 from Nick Clemens --- Created attachment 145704 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145704=edit Bug 32594: Add a dedicated Elasticsearch biblio indexing background queue Currently we generate large numebrs if single record reindex for circulation and other actions. It can take a long time to process these as we need to load the ES settings for each. This patch updates the Elasticsearch background jobs to throw records into a new queue that can be processed by it's own worker and adds a dedicated worker that batches the jobs every 1 second. To test: 1 - Apply patches, set SearchEngine system preference to 'Elasticsearch' 2 - perl misc/search_tools/es_indexer_daemon.pl 3 - Leave the running in terminal and perform actions in staff interface: - Checking out a bib - Returning a bib - Editing a single bib - Editing a single item - Batch editing bibs - Batch editing items 4 - Confirm for each action that records are updated in search/search results 5 - Stop the script 6 - set SearchEngine system preference to 'Zebra' 7 - perl misc/search_tools/es_indexer_daemon.pl 8 - Script dies as Elasticsearch not enabled -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Nick Clemens changed: What|Removed |Added Status|In Discussion |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #13 from David Cook --- I don't use Elasticsearch with Koha yet, so I'm probably not the best positioned to comment. (In reply to Jonathan Druart from comment #9) > Also I think it is confusing to have > misc/search_tools/es_indexer_daemon.pl > and > misc/background_jobs_worker.pl I don't think there's an issue having a separate worker program, but "search_tools" probably isn't the right place for it. Perhaps we should have something like "misc/workers/background_jobs_worker.pl" and "misc/workers/es_indexer_daemon.pl" -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594
--- Comment #12 from David Cook ---
Comment on attachment 145197
--> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145197
Bug 32594: Improve the ES daemon indexer
Review of attachment 145197:
-->
(https://bugs.koha-community.org/bugzilla3/page.cgi?id=splinter.html=32594=145197)
-
::: misc/search_tools/es_indexer_daemon.pl
@@ +126,4 @@
>
> +my @records;
> +for my $job ( @jobs ) {
> +my $args = $job->json->decode($job->data);
I suppose this line should have a try/catch since we're doing that for the
other job handling code.
--
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Jonathan Druart changed: What|Removed |Added CC||m.de.r...@rijksmuseum.nl, ||martin.renvoize@ptfs-europe ||.com, tomasco...@gmail.com --- Comment #11 from Jonathan Druart --- No-one else here? -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Jonathan Druart changed: What|Removed |Added Severity|enhancement |critical --- Comment #10 from Jonathan Druart --- We need this ASAP for 22.11. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #9 from Jonathan Druart --- (In reply to Jonathan Druart from comment #7) > Adding my bit to the discussion. We are not ready however: > > - I think we should deal with authorities here > - We need to make the constant configurable (options of the script enough?) > - Koha::Logger is logging to the opac log (this is true for bug 32393 as > well). Also I think it is confusing to have misc/search_tools/es_indexer_daemon.pl and misc/background_jobs_worker.pl -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Jonathan Druart changed: What|Removed |Added Attachment #145196|0 |1 is obsolete|| --- Comment #8 from Jonathan Druart --- Created attachment 145197 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145197=edit Bug 32594: Improve the ES daemon indexer -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Jonathan Druart changed: What|Removed |Added Status|ASSIGNED|In Discussion Assignee|jonathan.druart+koha@gmail. |n...@bywatersolutions.com |com | -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #7 from Jonathan Druart --- Adding my bit to the discussion. We are not ready however: - I think we should deal with authorities here - We need to make the constant configurable (options of the script enough?) - Koha::Logger is logging to the opac log (this is true for bug 32393 as well). -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Jonathan Druart changed: What|Removed |Added Attachment #145195|0 |1 is obsolete|| --- Comment #6 from Jonathan Druart --- Created attachment 145196 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145196=edit Bug 32594: Improve the ES daemon indexer -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #5 from Jonathan Druart --- Created attachment 145195 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145195=edit Bug 32594: Improve the ES daemon indexer -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Jonathan Druart changed: What|Removed |Added Status|Failed QA |ASSIGNED Assignee|n...@bywatersolutions.com |jonathan.druart+koha@gmail. ||com Depends on||32393, 32481 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32393 [Bug 32393] background job worker explodes if JSON is incorrect https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594
--- Comment #4 from Jonathan Druart ---
On the other hand... we are having the Koha module loaded already, and it
cannot be more efficient than that. I wanted to avoid a different script for a
specific job type, but actually I am inclined to agree with the approach.
My first idea was to have a Koha::BackgroundJobs::UpdateElasticIndex (notice
the *s*) to process several jobs. We would send the job ids from an array
constructed from the worker, with something in the config like {
background_jobs: { update_elastic_index: { batch: 10 } } } to process per batch
of 10. But it makes things more complicated and we don't want that for now.
--
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #3 from Jonathan Druart --- I don't think this is the correct way. First I think we should have an entry in koha-conf to configure the workers, depending on their type. Here, for update_elastic_index jobs, we actually want to batch consume and process them all every X seconds (or we could decide every X jobs). I am trying something. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 David Cook changed: What|Removed |Added Status|Needs Signoff |Failed QA -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #2 from David Cook --- (In reply to Nick Clemens from comment #0) > When pushing the ES indexing jobs to the background queue, we have found it > is hard for the workers to keep up in production. > > During high circulation times there are many single record reindexes > produced. > > It would be nice to have a dedicated ES worker that can batch these to > reduce the processing time and reduce the hits to the ES server I like the idea of batching together the single record indexes. I think I'd prefer if this worker created a new message that contained multiple job_ids, but background_jobs_worker.pl only supports a single job_id. Maybe we should enhance background_jobs_worker.pl to be able to take multiple job IDs... I'm not super familiar with alarm, so I'm curious at what point in code it is able to interrupt. It seems like it could introduce a race condition... (If you received the frame, and then the alarm handler fired, the frame would be acknowledged before it's processed. But the job should be handled in the next alarm. I suppose you might double acknowledge the same frame by accident. I don't know if that creates a warning or an error...) -- Marking "Failed QA" as there's some unused copy/paste code that needs to be removed (e.g. "process_job"). You'll also need to take into account Bug 32481. I suspect that you might also run into memory problems over time since you're not doing any forking. I reckon you could fork in the commit_records() function to do the actual indexing, since that's the point that will be most subject to spikes in memory usage. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 David Cook changed: What|Removed |Added See Also||https://bugs.koha-community ||.org/bugzilla3/show_bug.cgi ||?id=32481 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Nick Clemens changed: What|Removed |Added Assignee|koha-b...@lists.koha-commun |n...@bywatersolutions.com |ity.org | Blocks||32572 CC||dc...@prosentient.com.au, ||jonathan.druart+koha@gmail. ||com Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32572 [Bug 32572] We should have a background queue for each job type -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #1 from Nick Clemens --- Created attachment 145145 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145145=edit Bug 32594: Add a dedicated Elasticsearch biblio indexing background queue Currently we generate large numebrs if single record reindex for circulation and other actions. It can take a long time to process these as we need to load the ES settings for each. This patch updates the Elasticsearch background jobs to throw records into a new queue that can be processed by it's own worker and adds a dedicated worker that batches the jobs every 1 second. To test: 1 - Apply patches, set SearchEngine system preference to 'Elasticsearch' 2 - perl misc/search_tools/es_indexer_daemon.pl 3 - Leave the running in terminal and perform actions in staff interface: - Checking out a bib - Returning a bib - Editing a single bib - Editing a single item - Batch editing bibs - Batch editing items 4 - Confirm for each action that records are updated in search/search results 5 - Stop the script 6 - set SearchEngine system preference to 'Zebra' 7 - perl misc/search_tools/es_indexer_daemon.pl 8 - Script dies as Elasticsearch not enabled -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Nick Clemens changed: What|Removed |Added Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
