Hi Alex and all, While I'm not sure how exactly to solve this, this sounds like a possible Hibernate caching issue in DSpace 6. If a large number of objects are loaded into the Hibernate "Session" (which roughly corresponds to a database connection), future queries will be less efficient.
The way to avoid this issue is to ensure objects are either *committed* or *uncached* after they are no longer being used. Much more info on how this all works in DSpace 6 is documented here (See especially the "Development tips" section): https://wiki.duraspace.org/display/DSPACE/DSpace+Database+Access#DSpaceDatabaseAccess-TheHibernateSession(Cache)andtheContextObject So, my best guess here is that there *might* be a bug in the AIP packager import process. It sounds like it isn't properly managing this cache, and should be uncaching (i.e. calling Context.uncacheEntity()) objects after it has imported them successfully. That way, the cache is kept small, and the import speed should be more consistent. But, I'll admit, I'm not 100% sure *where* the issue in the code may be, and/or if others have seen similar import behavior that could help us narrow down the problem. So, I'm hoping others here are listening in, and can report back any findings/suspicions they may have. That way we can get a bug report created in our ticketing system and get it resolve as soon as possible. In the meantime, decreasing the amount of objects imported at once (as mentioned by Edmund) likely could be a way to "work around" the problem, as that likely won't encounter the same caching issues. (Finally, as a sidenote, you do not need to export and reimport all your content when upgrading from DSpace 5.4 to 6.2. So, if you are just trying to upgrade, you could do so by simply upgrading your content in place, by following the upgrade procedure at https://wiki.duraspace.org/display/DSDOC6x/Upgrading+DSpace) - Tim On Tue, Dec 26, 2017 at 5:39 AM Alex D <[email protected]> wrote: > Hi! > We have an issue with import process as we're tryings to migrate from 5.4 > to 6.2 Dspace (fresh install) > When I migrated from Dspace 4 to 5 I have imported about 9000 items for > couple of hours and it was fair. > But now I have 17000 items and it will take FOREVER to import all of them. > It already took more than 12 hours and after looking carefullty to logs I > can see that there's no point to wait - the time distance between > "[email protected]" checkpoints starts from 11s (17000*11s = over > 50 hours!) and after 10 hours of import time distance between items > increased 3x times - and by now it imported only 3000 items. It will take > forever so I just stoped it. > I believe there's a big problem but I don't know how to diagnose the > bottleneck. > > There's a screen of log in the middle of process - it takes on avrg 18s to > import just 1 item (( > https://prnt.sc/hse940 > > Maybe it will help someone to diagnose what is going on on background. I > have no fancy customizations or 3rd party on my freshly installed Dspace > 6.2 - just minimal starting settings. > Virtual Machine info (it's on SSD): > prntscr.com/hsedud > > Command I use to import: > sudo JAVA_OPTS="-Xms512m -Xmx8192m -XX:+UseGCOverheadLimit -server" / > dspace/bin/dspace packager -r -a -f -t AIP -o skipIfParentMissing=true -e > [email protected] -i lib/0 /home/alex/import-dspace/sitewide-aip.zip > > Please help to identify probrem and how to resolve it. Thanks in advance! > > Ужгородський національний університет > > http://www.uzhnu.edu.ua > > -- > You received this message because you are subscribed to the Google Groups > "DSpace Technical Support" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/dspace-tech. > For more options, visit https://groups.google.com/d/optout. > -- Tim Donohue Technical Lead for DSpace & DSpaceDirect DuraSpace.org | DSpace.org | DSpaceDirect.org -- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/dspace-tech. For more options, visit https://groups.google.com/d/optout.
