Although you're right that this should be fixed, the extent to which we can fix this will be limited.
Each batch import is basically a DB-level transaction. As it gets bigger, there will be some slowdown which is understandable (up to some extent). By isolating the whole import, we can roll it back as a whole if it fails. We could split it down into manageable parts (e.g. 1000 rows) which would run in separate transactions, but we would lose the benefits of the isolation. That's why it's probably better to split your import into smaller parts yourself. You can use the command-line "split" command on your csv file (don't forget to copy the header, too). Of course, we should investigate whether the slowdown comes fully from the transaction size or may have other reason, too. Feel free to file a Jira issue. Regards, ~~helix84 Compulsory reading: DSpace Mailing List Etiquette https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette On Wed, Aug 2, 2017 at 12:27 PM, João Silva <[email protected]> wrote: > Hi, > > I am trying to batch ingest around ~19k records from a CSV into a DSpace > 6.x, clean instance. It stayed online the whole night and is currently at > record ~7300. > > It seems like the system just gets slower and slower. I then tried > increasing the heap size in bin/dspace to -Xms256m -Xmx1024m, no luck. > > Any tips on how to improve this? > > Thanks, > João Rocha. > > -- > You received this message because you are subscribed to the Google Groups > "DSpace Technical Support" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/dspace-tech. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/dspace-tech. For more options, visit https://groups.google.com/d/optout.
