[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 --- Comment #47 from Martin Renvoize --- Nice work everyone! Pushed to master for 20.05 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 Joy Nelson changed: What|Removed |Added CC||j...@bywatersolutions.com --- Comment #46 from Joy Nelson --- does not apply to 19.11.x branch please rebase if needed. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 Nick Clemens changed: What|Removed |Added See Also||https://bugs.koha-community ||.org/bugzilla3/show_bug.cgi ||?id=25265 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 Martin Renvoize changed: What|Removed |Added CC||martin.renvoize@ptfs-europe ||.com Keywords|rel_20_05_candidate | -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 --- Comment #45 from Martin Renvoize --- Nice work everyone! Pushed to master for 20.05 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 Martin Renvoize changed: What|Removed |Added Status|Passed QA |Pushed to master Version(s)||20.05.00 released in|| -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 --- Comment #43 from Jonathan Druart --- Created attachment 102580 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=102580=edit Bug 24027: Add POD about the new parameter in Koha::Item->store Signed-off-by: Jonathan Druart -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 --- Comment #42 from Jonathan Druart --- Created attachment 102579 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=102579=edit Bug 24027: (QA follow-up) Fix POD warning Signed-off-by: Tomas Cohen Arazi Signed-off-by: Jonathan Druart -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027
Jonathan Druart changed:
What|Removed |Added
Attachment #102567|0 |1
is obsolete||
Attachment #102568|0 |1
is obsolete||
Attachment #102572|0 |1
is obsolete||
--- Comment #41 from Jonathan Druart
---
Created attachment 102578
-->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=102578=edit
Bug 24027: Call ModZebra once after all items added/deleted in a batch
Issue description:
- call to ModZebra was unconditional inside 'store' method for Koha::Item,
so it was after each item added, or deleted.
- ModZebra called with param biblionumber, so it is the same parameter
across calls for each items with same biblionumber, especially when we
adding/removing in a batch.
- with ElasticSearch enabled this makes even more significant load
and it is also progressively grows when more items already in DB
Solution:
- to add extra parameter 'skip_modzebra_update' and propagate it down to
'store' method call to prevent call of ModZebra,
- but to call ModZebra once after the whole batch loop in the upper layer
Test plan / how to replicate:
- make sure that you have in the admin settings "SearchEngine" set to
"Elasticsearch" and your ES is configured and working
( /cgi-bin/koha/admin/preferences.pl?op=search=SearchEngine )
- select one of biblioitems without items
( /cgi-bin/koha/cataloguing/additem.pl?biblionumber=XXX )
- press button "add multiple copies of this item",
- enter 200 items, start measuring time and submit the page/form...
On my test machine when adding 200 items 3 times in a row (so 600 in
total, but to show that time grows with every next batch gradually):
WHEN ElasticSearch DISABLED (only Zebra queue):
- 9s, 12s, 13s
WHEN ElasticSearch ENABLED:
- 1.3m, 3.2m, 4.8m
WITH PATCH WHEN ElasticSearch ENABLED:
- 10s, 13s, 15s
Same slowness (because also same call to ModZebra) happens when you try
to delete all items ("op=delallitems"). And same fix.
Signed-off-by: Tomas Cohen Arazi
Signed-off-by: Jonathan Druart
Amended commit message: the fix does not include the batch item
deletion.
--
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027
--- Comment #44 from Jonathan Druart
---
(In reply to Jonathan Druart from comment #40)
> From the commit message:
> """
> Same slowness (because also same call to ModZebra) happens when you try to
> delete items in a batch or delete all items ("op=delallitems"). And same fix.
> """
>
> What do you mean by "delete items in a batch"?
Confirmed by Andrew (pm), I amended the commit message.
--
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 Jonathan Druart changed: What|Removed |Added Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027
--- Comment #40 from Jonathan Druart
---
From the commit message:
"""
Same slowness (because also same call to ModZebra) happens when you try to
delete items in a batch or delete all items ("op=delallitems"). And same fix.
"""
What do you mean by "delete items in a batch"?
--
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 --- Comment #39 from Jonathan Druart --- Created attachment 102572 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=102572=edit Bug 24027: Add POD about the new parameter in Koha::Item->store -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027
--- Comment #37 from Tomás Cohen Arazi ---
Created attachment 102567
-->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=102567=edit
Bug 24027: Call ModZebra once after all items added/deleted in a batch
Issue description:
- call to ModZebra was unconditional inside 'store' method for Koha::Item,
so it was after each item added, or deleted.
- ModZebra called with param biblionumber, so it is the same parameter
across calls for each items with same biblionumber, especially when we
adding/removing in a batch.
- with ElasticSearch enabled this makes even more significant load
and it is also progressively grows when more items already in DB
Solution:
- to add extra parameter 'skip_modzebra_update' and propagate it down to
'store' method call to prevent call of ModZebra,
- but to call ModZebra once after the whole batch loop in the upper layer
Test plan / how to replicate:
- make sure that you have in the admin settings "SearchEngine" set to
"Elasticsearch" and your ES is configured and working
( /cgi-bin/koha/admin/preferences.pl?op=search=SearchEngine )
- select one of biblioitems without items
( /cgi-bin/koha/cataloguing/additem.pl?biblionumber=XXX )
- press button "add multiple copies of this item",
- enter 200 items, start measuring time and submit the page/form...
On my test machine when adding 200 items 3 times in a row (so 600 in
total, but to show that time grows with every next batch gradually):
WHEN ElasticSearch DISABLED (only Zebra queue):
- 9s, 12s, 13s
WHEN ElasticSearch ENABLED:
- 1.3m, 3.2m, 4.8m
WITH PATCH WHEN ElasticSearch ENABLED:
- 10s, 13s, 15s
Same slowness (because also same call to ModZebra) happens when you try
to delete items in a batch or delete all items ("op=delallitems"). And same
fix.
Signed-off-by: Tomas Cohen Arazi
--
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 --- Comment #38 from Tomás Cohen Arazi --- Created attachment 102568 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=102568=edit Bug 24027: (QA follow-up) Fix POD warning Signed-off-by: Tomas Cohen Arazi -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 Tomás Cohen Arazi changed: What|Removed |Added Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 Tomás Cohen Arazi changed: What|Removed |Added Attachment #102474|0 |1 is obsolete|| -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 --- Comment #36 from Tomás Cohen Arazi --- Talked to the author of most of Koha::Item->store (Jonathan) and we agreed this should move forward with a FIXME or similar. I'm testing this ATM for sign-off -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027
--- Comment #35 from Tomás Cohen Arazi ---
I don't know you, Andrew, and I think your patches are doing a good job (well
spotted the bottleneck, the loop design issue, etc. Well done!). But I think it
is time for all of us to think more in the long term.
ModZebra (in the Elasticsearch case) is fetching the MARC record so it gets
indexed by ES. The main problem here is that this is not an async call, but a
blocking one. It doesn't hurt Zebra because in that case we have task queue and
a script that takes care of the queue.
I think it is the way to go: have a task queue and leverage on that.
That said, I never liked that we call ModZebra from Koha::Item->store, and
while I understand the reasons for that and agree to just move forward and fix
later, I wouldn't add even more things related to ModZebra to the method
signature like this. We would just be polluting things even more.
To put it clear, my POV is that this post-success-actions (i.e. actions that
take place if storing the new/updated Item is successful) should be something
that happens in the controller code. If Koha::Item->store didn't call ModZebra
by itself, I'd do something like:
try {
# This is just fiction code
while ( my $item = $items_to_store->next ) {
$item->store;
}
}
catch {
handle_exception($_);
}
finally {
if (@_) {
notify_the_error();
}
else {
# No error
if (ES) {
trigger_es_reindex( $item->biblionumber );
}
elsif (Zebra) {
ModZebra(...);
}
else {
Koha::Exception::WTF->throw();
}
}
};
So, I understand the optimization, I fear from my QA POV that we are
introducing even more code that we will need to remove, in the place we
wouldn't want to. And this is always the excuse we use for not doing things
right!
If I was to fix this, I would look into moving the code for ES from plain
triggering the indexing, into using zebraqueue (probably renaming some things
while on it).
I hope this discussion doesn't discourage you, Andrew! I don't want to block
this if the QA team thinks we could live with this.
--
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 Tomás Cohen Arazi changed: What|Removed |Added CC||tomasco...@gmail.com --- Comment #34 from Tomás Cohen Arazi --- Lets implement pagination as all the framework is completed to accomplish that! No more Max items. For references on using the API to render (paginated) datatables, see bug 20212. The important bits are embedding related objects for rendering and filtering, and rendering properly :-D So you need to add embeds and build the right renderers for each column. Regarding my work on the performance front, I identified this situations: - We retrieve (cached or not) the frameworks inside loops without a real need, and really deep in the code. Take a biblio, as the outer situation: it only has one framework assigned, we can pass it down to the last portion of the code we call instead of doing it inside the loop (and many times as we do). - The same applies for code related to OpacHiddenItems and friends. Bug 23247 is a good example of how I think it should be done. It might still need fetching the framework outside now that I look at it with fresh eyes :-D -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 --- Comment #32 from Andrew Nugged --- Agreed. There will be another ticket from me for this new "Max pref" and post-POST page. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 --- Comment #33 from Andrew Nugged --- ... so I support conclusion that we can proceed with this ticket as finished / to be signed+tested. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 Jonathan Druart changed: What|Removed |Added Keywords||rel_20_05_target -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 Jonathan Druart changed: What|Removed |Added QA Contact|testo...@bugs.koha-communit |jonathan.dru...@bugs.koha-c |y.org |ommunity.org Severity|normal |major -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 --- Comment #31 from Jonathan Druart --- I do not think we should have an additional click. I guess cataloguers want to see the result of their last changes after the POST. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 --- Comment #30 from Andrew Nugged --- (In reply to Jonathan Druart from comment #28) Jonathan, which your nickname (for PM) on IRC / Koha if you're there? (I'm 'nugged') -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 --- Comment #29 from Andrew Nugged --- (In reply to Jonathan Druart from comment #28) ... so we can proceed with this single ModZebra update patch? and queue the Ajax-one for later (and to combine with Tomas'es updates?) And in my prev. message I proposed two options (it was unclear as one single solution): 1. ajax 2. OR simply post-POST page to make it without listing all added items on the same page So about p2 above: > - straight simple ... just NOT to generate list but show on post-POST page > "xxx items added" + link or forward to list page i.e. no "prefs", I will just add after items added: "xxx items added. Follow this link to return to the list: ..." message on post-POST page without list generation. Do you think this p2 (quick-patch) needed here in this ticket? Should I create? -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 --- Comment #28 from Jonathan Druart --- (In reply to Andrew Nugged from comment #26) > but CORRECT solution (two options) I propose/see: > > - complex but beautiful: create ajax pagination as it made on patrons page > (for example) > > - straight simple dumb but robust without any extra pref-parameters: > unconditionally, without any "prefs", just NOT to generate items list on > post-POST page, but add the link (or forward with delay) to full list+form > page in non-POST mode (i.e. usual post-POST page "xxx items added" + link or > forward) Yes, that's definitely the way to go, but not trivial. Make sure you are aware of the latest moves from Tomas in this area (see bug 20212) before starting! -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 --- Comment #27 from Andrew Nugged --- s/feet/feel/ :P -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 --- Comment #26 from Andrew Nugged --- (In reply to Jonathan Druart from comment #23) But second patch not based nor related to the first one, that's why I propose to separate it and do it right: - it was an UGLY tradeoff solution with not so good UX improvement: it just not shown all number of items on the post-POST page if it is more than that parameter, just saying "XXX items added" only. Adding extra branching in the template and pre-template call, but not adding so much improvement because of the next page: - it anyway had plain normal mode page which loaded all 1000 (or how much there) items on a single page in non-POST request then, so anyway page gen time is long and usefulness of displaying all-items-on-one page for an operator is very questionable, - that's why I propose to separate this into another ticket and do it, - but (and) yes, I can re-create it with the separate new pref-parameter as it was on the beginning if you want, but I feet myself "bad senior coder" with this "plumber" solution :) but CORRECT solution (two options) I propose/see: - complex but beautiful: create ajax pagination as it made on patrons page (for example) - straight simple dumb but robust without any extra pref-parameters: unconditionally, without any "prefs", just NOT to generate items list on post-POST page, but add the link (or forward with delay) to full list+form page in non-POST mode (i.e. usual post-POST page "xxx items added" + link or forward) -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 --- Comment #25 from Andrew Nugged --- (In reply to Jonathan Druart from comment #23) About FIRST patch which I just posted – it was "full separate one" and it is the same one, but more clear and proper: I re-created and re-tested/re-described, because: - I now found the real cause (Elasticsearch enabled) why it happens to be tested well, - I also found that "deleteallitems" had the same issue so change also added, - the parameter name (skip_modzebra_update) and propagation down below to "store" made better now - overall rebase (with rework) was needed because old patch used "AddItem" sub which now removed in favor of Koha::Item->...->store. And this first patch had main and most SERIOUS impact on execution speed. (but second patch: ... ) -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 --- Comment #24 from Andrew Nugged --- Those were anyway two patches, which not related to each other. One for "ModZebra" calls and another for GUI changes. Yet both improving the same site page(s) section. so: -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 --- Comment #23 from Jonathan Druart --- Hi Andrew, can you explain why did you change your mind? The patches were almost ready, I was only asking for a new pref. The changes to add a new pref is minimal, I can help if needed. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027
Andrew Nugged changed:
What|Removed |Added
Attachment #102464|0 |1
is obsolete||
--- Comment #22 from Andrew Nugged ---
Created attachment 102474
-->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=102474=edit
Bug 24027: Call ModZebra once after all items added/deleted in a batch
Issue description:
- call to ModZebra was unconditional inside 'store' method for Koha::Item,
so it was after each item added, or deleted.
- ModZebra called with param biblionumber, so it is the same parameter
across calls for each items with same biblionumber, especially when we
adding/removing in a batch.
- with ElasticSearch enabled this makes even more significant load
and it is also progressively grows when more items already in DB
Solution:
- to add extra parameter 'skip_modzebra_update' and propagate it down to
'store' method call to prevent call of ModZebra,
- but to call ModZebra once after the whole batch loop in the upper layer
Test plan / how to replicate:
- make sure that you have in the admin settings "SearchEngine" set to
"Elasticsearch" and your ES is configured and working
( /cgi-bin/koha/admin/preferences.pl?op=search=SearchEngine )
- select one of biblioitems without items
( /cgi-bin/koha/cataloguing/additem.pl?biblionumber=XXX )
- press button "add multiple copies of this item",
- enter 200 items, start measuring time and submit the page/form...
On my test machine when adding 200 items 3 times in a row (so 600 in
total, but to show that time grows with every next batch gradually):
WHEN ElasticSearch DISABLED (only Zebra queue):
- 9s, 12s, 13s
WHEN ElasticSearch ENABLED:
- 1.3m, 3.2m, 4.8m
WITH PATCH WHEN ElasticSearch ENABLED:
- 10s, 13s, 15s
Same slowness (because also same call to ModZebra) happens when you try
to delete items in a batch or delete all items ("op=delallitems"). And same
fix.
--
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 --- Comment #21 from Andrew Nugged --- (+ did rebase to latest master) -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027
Andrew Nugged changed:
What|Removed |Added
Attachment #98252|0 |1
is obsolete||
Attachment #98253|0 |1
is obsolete||
--- Comment #20 from Andrew Nugged ---
Created attachment 102464
-->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=102464=edit
Bug 24027: Call ModZebra once after all items added/deleted in a batch
Issue description:
- call to ModZebra was unconditional inside 'store' method for Koha::Item,
so it was after each item added, or deleted.
- ModZebra called with param biblionumber, so it is the same parameter
across calls for each items with same biblionumber, especially when we
adding/removing in a batch.
- with ElasticSearch enabled this makes even more significant load
and it is also progressively grows when more items already in DB
Solution:
- to add extra parameter 'skip_modzebra_update' and propagate it down to
'store' method call to prevent call of ModZebra,
- but to call ModZebra once after the whole batch loop in the upper layer
Test plan / how to replicate:
- make sure that you have in the admin settings "SearchEngine" set to
"Elasticsearch" and your ES is configured and working
( /cgi-bin/koha/admin/preferences.pl?op=search=SearchEngine )
- select one of biblioitems without items
( /cgi-bin/koha/cataloguing/additem.pl?biblionumber=XXX )
- press button "add multiple copies of this item",
- enter 200 items, start measuring time and submit the page/form...
On my test machine when adding 200 items 3 times in a row (so 600 in
total, but to show that time grows with every next batch gradually):
WHEN ElasticSearch DISABLED (only Zebra queue):
- 9s, 12s, 13s
WHEN ElasticSearch ENABLED:
- 1.3m, 3.2m, 4.8m
WITH PATCH WHEN ElasticSearch ENABLED:
- 10s, 13s, 15s
Same slowness (because also same call to ModZebra) happens when you try
to delete items in a batch or delete all items ("op=delallitems"). And same
fix.
--
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 Andrew Nugged changed: What|Removed |Added Status|In Discussion |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 Andrew Nugged changed: What|Removed |Added Status|Failed QA |In Discussion --- Comment #19 from Andrew Nugged --- (In reply to Jonathan Druart from comment #18) > We should not reuse OpacMaxItemsToDisplay but create a new pref here. I will separate this ticket into two – First one, current, will be without GUI update, but to fix this "ModZebra" call with ElasticSearch enabled which takes really crazy amount of time to perform. Second one – I will create another ticket – with pref or with API/ajax pagination or some other solution (we will decide I will think + ask in IRC/etc), it's really separated update so ok to have major speed affecting this one ModZebra issue, publishing updated patch + explanation: ... -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 Jonathan Druart changed: What|Removed |Added Status|Signed Off |Failed QA --- Comment #18 from Jonathan Druart --- We should not reuse OpacMaxItemsToDisplay but create a new pref here. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027
--- Comment #17 from Michal Denar ---
Created attachment 98253
-->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=98253=edit
Bug 24027: (follow-up) Don't combine multiple items add POST page with list
generation
When hundreds or thousands of items added, it takes time to add to DB.
Then generation of items list in the SAME request also takes time.
This "doubles" page generation time.
This patch proposes to show only the number of added and total items on
POST-generated page instead of list generation, but not always:
only if some limit are reached ("OpacMaxItemsToDisplay" // 50 is used),
and propose navigation link for the user to see the list after.
It's much more sensible on big databases and libraries hosting/clouds
with limited resources even up to page generation timeout.
This stacks up with previous patch for the combined speed up effect.
Other improvements with this patch:
because added counters for both:
- added now items,
- overall items for this biblioid in DB,
it is now shows:
- number of added items after adding in "dialog message" style box,
- number of total displayed items on the beginning of the list table.
Signed-off-by: Michal Denar
--
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 Michal Denar changed: What|Removed |Added Attachment #98250|0 |1 is obsolete|| Attachment #98251|0 |1 is obsolete|| --- Comment #16 from Michal Denar --- Created attachment 98252 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=98252=edit Bug 24027: Call ModZebra only once after all items have been added Why happened: - call to ModZebra was done after EACH item added, but it was called only with main biblionumber only, so call was the same on each of requests - and also time spent in that ModZebra sub increased with every next hundred items in DB for that element: so adding every next 100 was slower and slower, Solved: - now it's called only once (by adding some extra parameter to "AddItem*" sub set in postponed mode) - and now adding of elements not so heavily depends from how much items was in DB before. Test plan / How to replicate the issue: (test depends from how already populated DB and Zebra/Elasticsearch DBs so might be not so noticeably fast on empty DBs) - go to one of bibliotitems cataloguing/additem.pl?biblionumber=XXX pages - press button "add multiple copies of this item", put 1000 on slower machines, 5000 on faster - start measuring time + submit page/form It takes some amount of time even up to timeout. After applying patch, run the same procedure for same number of items. Note: it goes fast in both variants on empty database so time not so noticeable (also depents how ModZebra-related stuff is configured). Also it slows more linearly when number of items grows. But, again, it heavily depends how database is populated and indexes are rebuild but it's obvious that it will me much more efficient to call "ModZebra" once after 1000 addings, then call it 1000 times on each item created in the loop. Signed-off-by: Michal Denar -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027
Michal Denar changed:
What|Removed |Added
Attachment #95663|0 |1
is obsolete||
--- Comment #15 from Michal Denar ---
Created attachment 98251
-->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=98251=edit
Bug 24027: (follow-up) Don't combine multiple items add POST page with list
generation
When hundreds or thousands of items added, it takes time to add to DB.
Then generation of items list in the SAME request also takes time.
This "doubles" page generation time.
This patch proposes to show only the number of added and total items on
POST-generated page instead of list generation, but not always:
only if some limit are reached ("OpacMaxItemsToDisplay" // 50 is used),
and propose navigation link for the user to see the list after.
It's much more sensible on big databases and libraries hosting/clouds
with limited resources even up to page generation timeout.
This stacks up with previous patch for the combined speed up effect.
Other improvements with this patch:
because added counters for both:
- added now items,
- overall items for this biblioid in DB,
it is now shows:
- number of added items after adding in "dialog message" style box,
- number of total displayed items on the beginning of the list table.
Signed-off-by: Michal Denar
--
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 Michal Denar changed: What|Removed |Added Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 Michal Denar changed: What|Removed |Added Attachment #95473|0 |1 is obsolete|| --- Comment #14 from Michal Denar --- Created attachment 98250 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=98250=edit Bug 24027: Call ModZebra only once after all items have been added Why happened: - call to ModZebra was done after EACH item added, but it was called only with main biblionumber only, so call was the same on each of requests - and also time spent in that ModZebra sub increased with every next hundred items in DB for that element: so adding every next 100 was slower and slower, Solved: - now it's called only once (by adding some extra parameter to "AddItem*" sub set in postponed mode) - and now adding of elements not so heavily depends from how much items was in DB before. Test plan / How to replicate the issue: (test depends from how already populated DB and Zebra/Elasticsearch DBs so might be not so noticeably fast on empty DBs) - go to one of bibliotitems cataloguing/additem.pl?biblionumber=XXX pages - press button "add multiple copies of this item", put 1000 on slower machines, 5000 on faster - start measuring time + submit page/form It takes some amount of time even up to timeout. After applying patch, run the same procedure for same number of items. Note: it goes fast in both variants on empty database so time not so noticeable (also depents how ModZebra-related stuff is configured). Also it slows more linearly when number of items grows. But, again, it heavily depends how database is populated and indexes are rebuild but it's obvious that it will me much more efficient to call "ModZebra" once after 1000 addings, then call it 1000 times on each item created in the loop. Signed-off-by: Michal Denar -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 Michal Denar changed: What|Removed |Added CC||blac...@gmail.com --- Comment #13 from Michal Denar --- Hi, any test plan? -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 Andrew Nugged changed: What|Removed |Added Status|ASSIGNED|Needs Signoff -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027
Andrew Nugged changed:
What|Removed |Added
Attachment #95417|0 |1
is obsolete||
--- Comment #12 from Andrew Nugged ---
Created attachment 95663
-->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=95663=edit
Bug 24027: (follow-up) Don't combine multiple items add POST page with list
generation
When hundreds or thousands of items added, it takes time to add to DB.
Then generation of items list in the SAME request also takes time.
This "doubles" page generation time.
This patch proposes to show only the number of added and total items on
POST-generated page instead of list generation, but not always:
only if some limit are reached ("OpacMaxItemsToDisplay" // 50 is used),
and propose navigation link for the user to see the list after.
It's much more sensible on big databases and libraries hosting/clouds
with limited resources even up to page generation timeout.
This stacks up with previous patch for the combined speed up effect.
Other improvements with this patch:
because added counters for both:
- added now items,
- overall items for this biblioid in DB,
it is now shows:
- number of added items after adding in "dialog message" style box,
- number of total displayed items on the beginning of the list table.
--
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 Andrew Nugged changed: What|Removed |Added Attachment #95446|0 |1 is obsolete|| --- Comment #11 from Andrew Nugged --- Created attachment 95473 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=95473=edit Bug 24027: Call ModZebra only once after all items have been added Why happened: - call to ModZebra was done after EACH item added, but it was called only with main biblionumber only, so call was the same on each of requests - and also time spent in that ModZebra sub increased with every next hundred items in DB for that element: so adding every next 100 was slower and slower, Solved: - now it's called only once (by adding some extra parameter to "AddItem*" sub set in postponed mode) - and now adding of elements not so heavily depends from how much items was in DB before. Test plan / How to replicate the issue: (test depends from how already populated DB and Zebra/Elasticsearch DBs so might be not so noticeably fast on empty DBs) - go to one of bibliotitems cataloguing/additem.pl?biblionumber=XXX pages - press button "add multiple copies of this item", put 1000 on slower machines, 5000 on faster - start measuring time + submit page/form It takes some amount of time even up to timeout. After applying patch, run the same procedure for same number of items. Note: it goes fast in both variants on empty database so time not so noticeable (also depents how ModZebra-related stuff is configured). Also it slows more linearly when number of items grows. But, again, it heavily depends how database is populated and indexes are rebuild but it's obvious that it will me much more efficient to call "ModZebra" once after 1000 addings, then call it 1000 times on each item created in the loop. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 Andrew Nugged changed: What|Removed |Added Attachment #95413|0 |1 is obsolete|| Attachment #95414|0 |1 is obsolete|| --- Comment #10 from Andrew Nugged --- Created attachment 95446 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=95446=edit Bug 24027: ModZebra should be called once after all items added, not on each item add Why happened: - call to ModZebra was done after EACH item added, but it was called only with main biblionumber only, so call was the same on each of requests - and also time spent in that ModZebra sub increased with every next hundred items in DB for that element: so adding every next 100 was slower and slower, Solved: - now it's called only once (by adding some extra parameter to "AddItem*" sub set in postponed mode) - and now adding of elements not so heavily depends from how much items was in DB before. Test plan / How to replicate the issue: (test depends from how already populated DB and Zebra/Elasticsearch DBs so might be not so noticeably fast on empty DBs) - go to one of bibliotitems cataloguing/additem.pl?biblionumber=XXX pages - press button "add multiple copies of this item", put 1000 on slower machines, 5000 on faster - start measuring time + submit page/form It takes some amount of time even up to timeout. After applying patch, run the same procedure for same number of items. Note: it goes fast in both variants on empty database so time not so noticeable (also depents how ModZebra-related stuff is configured). Also it slows more linearly when number of items grows. But, again, it heavily depends how database is populated and indexes are rebuild but it's obvious that it will me much more efficient to call "ModZebra" once after 1000 addings, then call it 1000 times on each item created in the loop. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 Jonathan Druart changed: What|Removed |Added Blocks||15262 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15262 [Bug 15262] Run Koha Run. Koha search should be fast again. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 Jonathan Druart changed: What|Removed |Added Severity|minor |normal -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 --- Comment #8 from Jonathan Druart --- (for the OPAC we have OpacMaxItemsToDisplay, to limit the number of items to display on the biblio detail view) -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 --- Comment #6 from Jonathan Druart --- I understand the need of the UI patch, but I am worry it will not fit all needs, as it adds an extra step. I am pretty sure catalogers want to see if the last action did what they wanted :) We have something similar for the batch item edit/del tools, driven by a sysprefs. We could hardcoded it for now (50, 100?) and see if it passes QA :) -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 --- Comment #1 from Andrew Nugged --- Created attachment 95413 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=95413=edit Bug 24027: dbh handler not passed through all chain preparatory fix for next patch to have all parameters aligned: $dbh handler passed through as parameter in AddItem but forgotten in AddItemFromMarc. Added. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 --- Comment #7 from Jonathan Druart --- (In reply to Jonathan Druart from comment #6) > I understand the need of the UI patch, but I am worry it will not fit all > needs, as it adds an extra step. I am pretty sure catalogers want to see if > the last action did what they wanted :) > > We have something similar for the batch item edit/del tools, driven by a > sysprefs. We could hardcoded it for now (50, 100?) and see if it passes QA :) I meant: do not display the item list if X items are added in one go. But it will not fix the problem if there are already thousands of items. So maybe a pref to limit the display of the number of items. I feel like we should ask the list to get feedbacks from catalogers. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 --- Comment #2 from Andrew Nugged --- Created attachment 95414 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=95414=edit Bug 24027: (follow-up) ModZebra should be called once after all items added, not on each item add Why happened: - call to ModZebra was done after EACH item added, but it was called only with main biblionumber only, so call was the same on each of requests - and also time spent in that ModZebra sub increased with every next hundred items in DB for that element: so adding every next 100 was slower and slower, Solved: - now it's called only once (by adding some extra parameter to "AddItem*" sub set in postponed mode) - and now adding of elements not so heavily depends from how much items was in DB before. Test plan / How to replicate the issue: (test depends from how already populated DB and Zebra/Elasticsearch DBs so might be not so noticeably fast on empty DBs) - go to one of bibliotitems cataloguing/additem.pl?biblionumber=XXX pages - press button "add multiple copies of this item", put 1000 on slower machines, 5000 on faster - start measuring time + submit page/form It takes some serious amount of time even up to timeout. After applying patch, same procedure for same number of items. Also it much less slows down when number of items rises, more linearly. But, again, it heavily depends how database is populated and indexes are rebuild but it's obvious that it will me much more efficient call "ModZebra" once after 1000 addings, then 1000 times on each item copied. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 Jonathan Druart changed: What|Removed |Added Blocks|15262 | See Also||https://bugs.koha-community ||.org/bugzilla3/show_bug.cgi ||?id=15262 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15262 [Bug 15262] Run Koha Run. Koha search should be fast again. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027
Jonathan Druart changed:
What|Removed |Added
CC||jonathan.dru...@bugs.koha-c
||ommunity.org
--- Comment #3 from Jonathan Druart
---
Hi Andrew,
Thanks for the patch, that's a great idea.
Is it ready for signoff?
Only one thing, we will certainly reject the first patch, $dbh must be
retrieved from C4::Context->dbh only.
Also we prefer to have a hashref as parameters to make the flag explicit:
{ postpone_indexes_update => 1 }
Cheers,
Jonathan
--
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 --- Comment #4 from Andrew Nugged --- Created attachment 95417 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=95417=edit Bug 24027: (follow-up) Don't combine long items-add POST page with list generation When hundreds or thousands of items added, it takes time. Then generation of items list also takes time, and more items more items list generation needed, so sometimes user gets even browser timeout. It's much more sensible on big databases and libraries hosting/clouds with limited resources. So - showing only counter of added items on POST-generated page instead of list generation shortens page generation time and then user can navigate further. This stacks up with previous patches in this ticket for the combined effect. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027
--- Comment #5 from Andrew Nugged ---
Thanks Jonathan,
> Only one thing, we will certainly reject the first patch, $dbh must be
> retrieved from C4::Context->dbh only.
so I will rework in opposite direction, removing mine + old dbh in params: that
> Also we prefer to have a hashref as parameters to make the flag explicit:
> { postpone_indexes_update => 1 }
yes, this is good coding style for not to seding unknown ", 1" or ", 0" in
params :),
thought about that but was afraid to announce it because it is more bigger
change then (calling style from sequential params to hash with options). But
both thumbs up for this!
I will recreate now patches/chain,
please then give feedback on last GUI change one:
Bug 24027: (follow-up) Don't combine long items-add POST page with list
generation
... so I will rework them in a batch :), and then I will ask for Sign-Off.
--
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 --- Comment #9 from Andrew Nugged --- > OpacMaxItemsToDisplay thought to make it configurable but hezitated :). Will be more brave then. Okay will use your vision and present rework for all the chain this week. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 Andrew Nugged changed: What|Removed |Added Assignee|koha-b...@lists.koha-commun |nug...@gmail.com |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 24027] Adding multiple items is slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24027 Andrew Nugged changed: What|Removed |Added Status|NEW |ASSIGNED -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
