https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21872
david holoshka <[email protected]> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |[email protected] --- Comment #35 from david holoshka <[email protected]> --- We were force to rewrite rebuild_elastic_search.pl as it just died after a couple days never finishing to index our 2.4 million bibliographic records. Our version forks a copy of the process to each machine core using biblio_metadata based limits precalculated by the parent process (this has been upgraded since I sent you a copy of the code, David to make sure each core gets the same number of records to index). My old algorithm didn't distribute the load well as the metadata ids gaps were create by biblio updates with time. With 8 cores the indexing completes in 50 minutes with elastic search running on the same virtual machine. We speed up the process a great deal by accessing the metadata table directly instead of through the iterator. The only draw back is memory usage due to needing to put the 952 item data (coincidentally also 2.4 million items) in hashes. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug. _______________________________________________ Koha-bugs mailing list [email protected] http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
