https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39537
Bug ID: 39537
Summary: bulkmarcimport.pl fails to import large files
Change sponsored?: ---
Product: Koha
Version: unspecified
Hardware: All
OS: All
Status: NEW
Severity: normal
Priority: P5 - low
Component: Command-line Utilities
Assignee: [email protected]
Reporter: [email protected]
QA Contact: [email protected]
CC: [email protected], [email protected],
[email protected], [email protected]
After applying bug 29440 (entered in 24.05), bulkmarcimport.pl fails to import
large files.
The reason is that, after the changes made there, there are now two loops
iterating the records to be imported, first of which reads the file and places
the read records in an Perl array in memory. The second loop takes the records
from that array array and actually inserts them into Koha database.
Now, imagine what happens with limited RAM on the system and a large file
(100K+, 1M+, 5M+ records, depending on system configuration) -- the script
consumes more and more memory and at the end destroys the system stability,
possibly being killed because of oom.
I must say, bulkmarcimport.pl has a lot of useful features, but has become
completely useless for larger datasets and one have now to write own simple
scripts for each use case to cover actual needs and adapting elements from the
original script. This is tiring.
I am wondering what was the reason of such a design (splitting the main loop
and collecting the records in a growing array in memory). Should it not be
corrected? I would be happy to know David's (the original author's) opinion.
--
You are receiving this mail because:
You are watching all bug changes.
You are the assignee for the bug.
_______________________________________________
Koha-bugs mailing list
[email protected]
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
