I get it and understand your reasons for doing it like that. I am going to write a small script to split the csv and import, elbeit losing the isolation.
I will post the script here after I am done. Thanks! quarta-feira, 2 de Agosto de 2017 às 11:43:56 UTC+1, helix84 escreveu: > > Although you're right that this should be fixed, the extent to which > we can fix this will be limited. > > Each batch import is basically a DB-level transaction. As it gets > bigger, there will be some slowdown which is understandable (up to > some extent). By isolating the whole import, we can roll it back as a > whole if it fails. > > We could split it down into manageable parts (e.g. 1000 rows) which > would run in separate transactions, but we would lose the benefits of > the isolation. > > That's why it's probably better to split your import into smaller > parts yourself. You can use the command-line "split" command on your > csv file (don't forget to copy the header, too). > > Of course, we should investigate whether the slowdown comes fully from > the transaction size or may have other reason, too. Feel free to file > a Jira issue. > > > Regards, > ~~helix84 > > Compulsory reading: DSpace Mailing List Etiquette > https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette > > > On Wed, Aug 2, 2017 at 12:27 PM, João Silva <[email protected] > <javascript:>> wrote: > > Hi, > > > > I am trying to batch ingest around ~19k records from a CSV into a DSpace > > 6.x, clean instance. It stayed online the whole night and is currently > at > > record ~7300. > > > > It seems like the system just gets slower and slower. I then tried > > increasing the heap size in bin/dspace to -Xms256m -Xmx1024m, no luck. > > > > Any tips on how to improve this? > > > > Thanks, > > João Rocha. > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "DSpace Technical Support" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to [email protected] <javascript:>. > > To post to this group, send email to [email protected] > <javascript:>. > > Visit this group at https://groups.google.com/group/dspace-tech. > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/dspace-tech. For more options, visit https://groups.google.com/d/optout.
