[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 Jonathan Druart changed: What|Removed |Added Blocks||35089 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35089 [Bug 35089] Background worker: Should prefetch-count be activemq.prefetchSize ? -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 wainuiwitikap...@catalyst.net.nz changed: What|Removed |Added CC||wainuiwitikapark@catalyst.n ||et.nz --- Comment #40 from wainuiwitikap...@catalyst.net.nz --- Not backported to 21.05.x -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 Arthur Suzuki changed: What|Removed |Added Status|Pushed to oldstable |Pushed to oldoldstable Version(s)|23.05.00,22.11.02,22.05.09 |23.05.00,22.11.02,22.05.09, released in||21.11.16 CC||arthur.suz...@biblibre.com --- Comment #39 from Arthur Suzuki --- applied to 21.11.x for 21.11.16 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 Lucas Gass changed: What|Removed |Added Status|Pushed to stable|Pushed to oldstable Version(s)|23.05.00,22.11.02 |23.05.00,22.11.02,22.05.09 released in|| CC||lu...@bywatersolutions.com --- Comment #38 from Lucas Gass --- Backported to 22.05.x for upcoming 22.05.09 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 --- Comment #37 from Jacob O'Mara --- Nice work, thanks everyone! Pushed to 22.11.x for the next release. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 Jacob O'Mara changed: What|Removed |Added Status|Pushed to master|Pushed to stable Version(s)|23.05.00|23.05.00,22.11.02 released in|| -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 --- Comment #36 from Tomás Cohen Arazi --- Pushed to master for 23.05. Nice work everyone, thanks! -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 Tomás Cohen Arazi changed: What|Removed |Added Version(s)||23.05.00 released in|| Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 Nick Clemens changed: What|Removed |Added Attachment #145173|0 |1 is obsolete|| --- Comment #35 from Nick Clemens --- Created attachment 145208 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145208&action=edit Bug 32481: Limit prefetch size for background jobs worker This patch adds a prefetch size of 1 to the background jobs worker, so that it fetches 1 message at a time. Without this change, the RabbitMQ connection timeout when too many messages for slow tasks are fetched at the same time. To test: 0. Apply patch 1. Run background worker 2. Rapidly enqueue multiple jobs that in total will take longer than 30 minutes to process Bug 32481: Use correct prefetch syntax for RabbitMQ According to https://www.rabbitmq.com/stomp.html the header to use for managing the prefetch is "prefetch-count". You can verify the number of delivered and unacknowledged messages on a channel on a connection by running "rabbitmqctl list_channels" on the RabbitMQ host. This will tell you how many messages have been delivered and are awaiting acknowledgement Signed-off-by: Jonathan Druart Signed-off-by: Nick Clemens -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 Nick Clemens changed: What|Removed |Added Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 Jonathan Druart changed: What|Removed |Added Blocks||32594 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 [Bug 32594] Add a dedicated ES indexing background worker -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 Jonathan Druart changed: What|Removed |Added Attachment #144809|0 |1 is obsolete|| Attachment #145082|0 |1 is obsolete|| Attachment #145134|0 |1 is obsolete|| --- Comment #34 from Jonathan Druart --- Created attachment 145173 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145173&action=edit Bug 32481: Limit prefetch size for background jobs worker This patch adds a prefetch size of 1 to the background jobs worker, so that it fetches 1 message at a time. Without this change, the RabbitMQ connection timeout when too many messages for slow tasks are fetched at the same time. To test: 0. Apply patch 1. Run background worker 2. Rapidly enqueue multiple jobs that in total will take longer than 30 minutes to process Bug 32481: Use correct prefetch syntax for RabbitMQ According to https://www.rabbitmq.com/stomp.html the header to use for managing the prefetch is "prefetch-count". You can verify the number of delivered and unacknowledged messages on a channel on a connection by running "rabbitmqctl list_channels" on the RabbitMQ host. This will tell you how many messages have been delivered and are awaiting acknowledgement Signed-off-by: Jonathan Druart -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 Jonathan Druart changed: What|Removed |Added Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 David Cook changed: What|Removed |Added See Also||https://bugs.koha-community ||.org/bugzilla3/show_bug.cgi ||?id=32594 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 --- Comment #33 from David Cook --- That 3rd patch "Bug 32481: Use correct prefetch syntax for RabbitMQ" uses the correct syntax, so the worker only fetches 1 message at a time ( see https://www.rabbitmq.com/stomp.html#pear.p ) Jonathan's test_bg.pl script completes perfectly with it. No time outs. You can verify it using "rabbitmqctl list_channels" as well. Once testers are happy, I'd suggest obsoleting Jonathan's patch and then squashing my 2 patches together. -- Of course, as I note above, the 30 minute timeout will still be an issue for any 1 job that takes longer than 30 minutes. But now I'm confident that Nick's case of 6 jobs of 6 minutes each will be resolved. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 --- Comment #32 from David Cook --- (In reply to Jonathan Druart from comment #25) > I am giving up, need help. I am terribly sorry for wasting your time on this one, Jonathan. I solved this problem on a different project using https://metacpan.org/pod/Net::AMQP::RabbitMQ but couldn't directly translate it across, and just assumed the first thing that said "prefetch" on https://metacpan.org/pod/Net::Stomp#subscribe would be correct... Thanks for providing your testing patch, as that helped me to figure out the prefetch config wasn't working. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 --- Comment #31 from David Cook --- Created attachment 145134 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145134&action=edit Bug 32481: Use correct prefetch syntax for RabbitMQ According to https://www.rabbitmq.com/stomp.html the header to use for managing the prefetch is "prefetch-count". You can verify the number of delivered and unacknowledged messages on a channel on a connection by running "rabbitmqctl list_channels" on the RabbitMQ host. This will tell you how many messages have been delivered and are awaiting acknowledgement -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 --- Comment #30 from David Cook --- I'm thinking that the prefetch config I provided isn't working with Net::Stomp... I enqueue 100 messages and then I look at the channel. root@kohadevbox:kohadevbox$ rabbitmqctl list_channels Listing channels ... pid userconsumer_count messages_unacknowledged guest 1 0 guest 1 99 According to https://www.rabbitmq.com/rabbitmqctl.8.html, messages_unacknowledged means that many messages have been delivered but haven't been acknowledged. The Net::Stomp library was set up to work with ActiveMQ, and I made the mistake of assuming that some of the config option naming was labelled for ActiveMQ but could be used for other providers. That's my bad. I'm working on a better fix now... -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 --- Comment #29 from David Cook --- Ok I see the timeout happening 10 seconds after it probably got the first message on the connection/channel. Although according to the logs it processed 51 tasks. That seems odd... I'm going to dive deeper... 622 2023-01-09 00:14:38.291 [info] <0.2408.0> accepting STOMP connection <0.2408.0> (127.0.0.1:58134 -> 127.0.0.1:61613) 623 2023-01-09 00:15:39.033 [warning] <0.2420.0> Consumer Q_/queue/koha_kohadev-default on channel 1 has timed out waiting on consumer acknowledgement. Timeout used: 1 ms 624 2023-01-09 00:15:39.170 [error] <0.2420.0> Channel error on connection <0.2411.0> (127.0.0.1:58134 -> 127.0.0.1:61613, vhost: '/', user: 'guest'), channel 1: 625 operation none caused a channel exception precondition_failed: consumer ack timed out on channel 1 626 2023-01-09 00:15:39.182 [error] <0.2408.0> STOMP error frame sent: 627 Message: precondition_failed 628 Detail: "PRECONDITION_FAILED - consumer ack timed out on channel 1\n" 629 Server private detail: none 630 2023-01-09 00:15:39.307 [info] <0.2408.0> closing STOMP connection <0.2408.0> (127.0.0.1:58134 -> 127.0.0.1:61613) -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 --- Comment #28 from David Cook --- Looks like the test plan is missing a step since without restarting the koha background job worker you'll just get the following: Exception 'Koha::Exception' thrown 'test is not a valid job_type' -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 --- Comment #27 from David Cook --- (In reply to Jonathan Druart from comment #25) > I am giving up, need help. > > I wrote this script to help adjusting the worker code and see how things > went. > > To test: > Edit /etc/rabbitmq/rabbitmq.conf > > consumer_timeout = 1 > > # Delete existing jobs - not needed but better to track down what's going on > MariaDB [koha_kohadev]> delete from background_jobs; > > # Watch the logs > % sudo tail -f /var/log/koha/kohadev/* /var/log/rabbitmq/*.log > > # Run the script > % perl test_bg.pl > > Wait 100 seconds I'm not sure I understand. That timeout is 10 seconds but the problem happens after 100 seconds? I'll give the test a go, since those missing frames are disturbing... -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 --- Comment #26 from David Cook --- (In reply to Jonathan Druart from comment #17) > And what about the 30 minutes timeout? That's always a danger. If 1 message takes 30+ minutes to process, then we'll have problems when using a "worker queue" (as described at https://www.rabbitmq.com/tutorials/tutorial-two-python.html) style of background job processing. Options that come to mind are increasing the timeout value (which is tough to do programmatically with RabbitMQ via the koha-common package) or breaking down long running tasks into smaller easier to process chunks. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 --- Comment #25 from Jonathan Druart --- I am giving up, need help. I wrote this script to help adjusting the worker code and see how things went. To test: Edit /etc/rabbitmq/rabbitmq.conf consumer_timeout = 1 # Delete existing jobs - not needed but better to track down what's going on MariaDB [koha_kohadev]> delete from background_jobs; # Watch the logs % sudo tail -f /var/log/koha/kohadev/* /var/log/rabbitmq/*.log # Run the script % perl test_bg.pl Wait 100 seconds -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 --- Comment #24 from Jonathan Druart --- Created attachment 145082 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145082&action=edit Test background jobs -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 --- Comment #23 from Jonathan Druart --- (In reply to Jonathan Druart from comment #22) > The best solution I found is the 2 patches from bug 32393. But still not idea, I still get gaps. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 --- Comment #22 from Jonathan Druart --- The best solution I found is the 2 patches from bug 32393. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 Kyle M Hall changed: What|Removed |Added CC||k...@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481
Jonathan Druart changed:
What|Removed |Added
Severity|normal |critical
--- Comment #21 from Jonathan Druart ---
There is something scary happening here.
I've added 3 debug statements, one displays the frame, 2 others are before the
process and ack calls.
{"record_ids":[295],"record_server":"biblioserver","job_id":1824} at
/kohadevbox/koha/misc/background_jobs_worker.pl line 97.
Processing 1824 at /kohadevbox/koha/misc/background_jobs_worker.pl line 105.
Acking 1824 at /kohadevbox/koha/misc/background_jobs_worker.pl line 107.
{"record_ids":[296],"job_id":1825,"record_server"CONNECTED
server at /kohadevbox/koha/misc/background_jobs_worker.pl line 97.
':' expected, at character offset 49 (before "CONNECTED\nserver") at
/kohadevbox/koha/misc/background_jobs_worker.pl line 98.
{"record_server":"biblioserver","job_id":1879,"record_ids":[350]} at
/kohadevbox/koha/misc/background_jobs_worker.pl line 97.
Processing 1879 at /kohadevbox/koha/misc/background_jobs_worker.pl line 105.
Acking 1879 at /kohadevbox/koha/misc/background_jobs_worker.pl line 107.
=> Frame for job 1825 is incorrect (?!), and we are losing jobs 1826-1878.
--
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 --- Comment #20 from Jonathan Druart --- > operation basic.ack caused a channel exception precondition_failed: unknown > delivery tag 245 I think it's the fork, I removed it and it seems to be ok now. Could someone confirm? > Detail: "PRECONDITION_FAILED - consumer ack timed out on channel 1\n" I am still seeing this after I removed the fork, which means this patch does not fix the timeout problem. I am also seeing this: ':' expected, at character offset 49 (before "CONNECTED\nserver") at /kohadevbox/koha/misc/background_jobs_worker.pl line 97. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 Jonathan Druart changed: What|Removed |Added See Also||https://bugs.koha-community ||.org/bugzilla3/show_bug.cgi ||?id=32393 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 --- Comment #19 from Nick Clemens --- (In reply to Jonathan Druart from comment #18) > I think I recreate the problem this way: > > % more /etc/rabbitmq/rabbitmq.conf > consumer_timeout = 15000 > > Add a sleep 10 in Koha::BackgroundJob::BatchUpdateItem->process > > Then enqueue several batch item mod jobs. > > However: > 1. I don't see jobs stuck in "new" (without the patch) > > 2. I am seeing the following in the rabbitmq logs, with and without the patch > > 2023-01-04 11:01:03.041 [warning] <0.2774.0> Consumer > Q_/queue/koha_kohadev-long_tasks on channel 1 has timed out waiting on > consumer acknowledgement. Timeout used: 15000 ms > > 2023-01-04 11:01:03.041 [error] <0.2774.0> Channel error on connection > <0.2764.0> (127.0.0.1:39578 -> 127.0.0.1:61613, vhost: '/', user: 'guest'), > channel 1: > operation none caused a channel exception precondition_failed: consumer ack > timed out on channel 1 > > 2023-01-04 11:01:03.042 [error] <0.2758.0> STOMP error frame sent: > > Message: precondition_failed > Detail: "PRECONDITION_FAILED - consumer ack timed out on channel 1\n" > Server private detail: none > 2023-01-04 11:01:03.042 [info] <0.2758.0> closing STOMP connection > <0.2758.0> (127.0.0.1:39578 -> 127.0.0.1:61613) I played with this a bunch, and things seem to sometimes recover and sometimes not Setting rabbitmq.conf with: consumer_timeout = 1 Adding "sleep 1;" in the Koha/BackgroundJob/UpdateElasticIndex.pm sudo koha-mysql kohadev DELETE FROM biblio WHERE biblionumber=269; DELETE FROM biblio WHERE biblionumber=72; Set SearchEngine syspref to 'Elastic' perl misc/maintenance/touch_all_biblios.pl With or without patch, I get errors like: 2023-01-04 16:49:40.689 [warning] <0.7712.0> Consumer Q_/queue/koha_kohadev-default on channel 1 has timed out waiting on consumer acknowledgement. Timeout used: 1 ms 2023-01-04 16:49:40.692 [error] <0.7712.0> Channel error on connection <0.7703.0> (127.0.0.1:60178 -> 127.0.0.1:61613, vhost: '/', user: 'guest'), channel 1: operation none caused a channel exception precondition_failed: consumer ack timed out on channel 1 2023-01-04 16:49:40.692 [error] <0.7700.0> STOMP error frame sent: Message: precondition_failed Detail: "PRECONDITION_FAILED - consumer ack timed out on channel 1\n" Server private detail: none 2023-01-04 16:49:40.693 [info] <0.7700.0> closing STOMP connection <0.7700.0> (127.0.0.1:60178 -> 127.0.0.1:61613) 2023-01-04 16:49:41.224 [info] <0.14720.0> accepting STOMP connection <0.14720.0> (127.0.0.1:60376 -> 127.0.0.1:61613) 2023-01-04 16:49:41.235 [error] <0.14732.0> Channel error on connection <0.14723.0> (127.0.0.1:60376 -> 127.0.0.1:61613, vhost: '/', user: 'guest'), channel 1: operation basic.ack caused a channel exception precondition_failed: unknown delivery tag 245 2023-01-04 16:49:41.239 [error] <0.14720.0> STOMP error frame sent: Message: precondition_failed Detail: "PRECONDITION_FAILED - unknown delivery tag 245\n" Server private detail: none 2023-01-04 16:49:41.239 [info] <0.14720.0> closing STOMP connection <0.14720.0> (127.0.0.1:60376 -> 127.0.0.1:61613) 2023-01-04 16:49:42.477 [info] <0.14736.0> accepting STOMP connection <0.14736.0> (127.0.0.1:60388 -> 127.0.0.1:61613) And varied number of jobs that remain in new -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 --- Comment #18 from Jonathan Druart --- I think I recreate the problem this way: % more /etc/rabbitmq/rabbitmq.conf consumer_timeout = 15000 Add a sleep 10 in Koha::BackgroundJob::BatchUpdateItem->process Then enqueue several batch item mod jobs. However: 1. I don't see jobs stuck in "new" (without the patch) 2. I am seeing the following in the rabbitmq logs, with and without the patch 2023-01-04 11:01:03.041 [warning] <0.2774.0> Consumer Q_/queue/koha_kohadev-long_tasks on channel 1 has timed out waiting on consumer acknowledgement. Timeout used: 15000 ms 2023-01-04 11:01:03.041 [error] <0.2774.0> Channel error on connection <0.2764.0> (127.0.0.1:39578 -> 127.0.0.1:61613, vhost: '/', user: 'guest'), channel 1: operation none caused a channel exception precondition_failed: consumer ack timed out on channel 1 2023-01-04 11:01:03.042 [error] <0.2758.0> STOMP error frame sent: Message: precondition_failed Detail: "PRECONDITION_FAILED - consumer ack timed out on channel 1\n" Server private detail: none 2023-01-04 11:01:03.042 [info] <0.2758.0> closing STOMP connection <0.2758.0> (127.0.0.1:39578 -> 127.0.0.1:61613) -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 --- Comment #17 from Jonathan Druart --- And what about the 30 minutes timeout? -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 --- Comment #16 from Nick Clemens --- (In reply to David Cook from comment #15) > The test plan is a bit lacklustre because it's probably going to be hard for > people to test. > > I suppose we can just test that it doesn't cause any problems with the > current background jobs, and then Nick can confirm whether or not it fixes > the problem for him. I'll try to see if I can reproduce on a test environment, thanks David! -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 --- Comment #12 from Nick Clemens --- (In reply to David Cook from comment #4) > (In reply to Nick Clemens from comment #0) > > We saw this happen several times - a library would queue up 6 batch > > modifications that would take roughly 6 minutes each. > > Is that 1 job for 6 items, or is that 6 different jobs? Why were the jobs > taking 6 minutes? 6 batch jobs of several hundred items, submitted one after the other > > > The jobs would process normally, until the 30 minute configured timeout > > would hit - then the remaining job would fail, and rabbit would disconnect > > and not reconnect, and jobs would pile up > > When you say "rabbit would disconnect and not reconnect", are you talking > about the RabbitMQ server disconnecting, and then not allowing the > koha-worker to reconnect? Rabbit crashed for all intents and purposes - koha worker could not connect, only a restart would beign to process jobs again > > > It seems that koha receives all of the jobs immediately, but only > > acknowledges when complete - so the final job was waiting over 30 minutes > > for acknowledgement and this caused issues > > What do you mean by "koha"? Do you mean "koha-worker"? I think we need more > information here. Yes, the koha-worker - it seems to get the frame, but not send an ack -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 Magnus Enger changed: What|Removed |Added CC||mag...@libriotech.no -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 --- Comment #7 from Julian Maurice --- > as soon as it receives it... or at least as soon as it has validated that the message correspond to an existing pending job, or something like that. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 Julian Maurice changed: What|Removed |Added CC||julian.maur...@biblibre.com --- Comment #6 from Julian Maurice --- (In reply to David Cook from comment #3) > According to the RabbitMQ Work Queues tutorial, we're doing the right thing > by ACKing after the message has been processed. According to that tutorial, this allows RabbitMQ to redeliver the message to another worker. But do we really want that ? A background job should not be restarted if it failed, because it might fail forever, or because by running it more than once it might do unwanted modifications to the database. I believe that the background job worker should acknowledge the message as soon as it receives it. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 --- Comment #5 from Katrin Fischer --- A timeout seems arbitrary when you ultimately want all jobs to be processed and nothing be lost. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 --- Comment #4 from David Cook --- (In reply to Nick Clemens from comment #0) > We saw this happen several times - a library would queue up 6 batch > modifications that would take roughly 6 minutes each. Is that 1 job for 6 items, or is that 6 different jobs? Why were the jobs taking 6 minutes? > The jobs would process normally, until the 30 minute configured timeout > would hit - then the remaining job would fail, and rabbit would disconnect > and not reconnect, and jobs would pile up When you say "rabbit would disconnect and not reconnect", are you talking about the RabbitMQ server disconnecting, and then not allowing the koha-worker to reconnect? > It seems that koha receives all of the jobs immediately, but only > acknowledges when complete - so the final job was waiting over 30 minutes > for acknowledgement and this caused issues What do you mean by "koha"? Do you mean "koha-worker"? I think we need more information here. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 David Cook changed: What|Removed |Added CC||dc...@prosentient.com.au --- Comment #3 from David Cook --- (In reply to Katrin Fischer from comment #2) > I am slightly confused by this one, wasn't the idea of RabbitMQ to avoid > just this kind of thing? Having a server dealing with the jobs so nothing > would be lost and it all done in due time? It seems strange that we run into > limitations so soon. Generally speaking, Koha developers are not experienced with using advanced tools and techniques. Even the most hardcore of Koha developers are still just beginners with things like distributed computing. According to the RabbitMQ Work Queues tutorial, we're doing the right thing by ACKing after the message has been processed. I think options would be increasing the timeout, or changing how we're enqueuing tasks. In this case, are the "6 batch modifications" actually 1 batch modification job for 6 items? Why were they taking 6 minutes? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 --- Comment #2 from Katrin Fischer --- I am slightly confused by this one, wasn't the idea of RabbitMQ to avoid just this kind of thing? Having a server dealing with the jobs so nothing would be lost and it all done in due time? It seems strange that we run into limitations so soon. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481
Jonathan Druart changed:
What|Removed |Added
CC||jonathan.druart+koha@gmail.
||com
--- Comment #1 from Jonathan Druart ---
https://www.rabbitmq.com/consumers.html#acknowledgement-timeout
"""
The timeout can be deactivated using advanced.config. This is not recommended:
%% advanced.config
[
{rabbit, [
{consumer_timeout, undefined}
]}
].
Instead of disabling the timeout entirely, consider using a high value (for
example, a few hours).
"""
--
You are receiving this mail because:
You are the assignee for the bug.
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
