Dear Michael, Your assetstore doesn't seem unpractically large to me. We have ~90,000 items in our DSpace repository and our assetstore looks like this:
# du -sh /home/dspace/assetstore/ 76G /home/dspace/assetstore/ # find /home/dspace/assetstore/ -type f | wc -l 143062 We do a simple `s3cmd sync` to Amazon S3 every night and have never, ever had an issue in ten years. I regularly rsync our production assetstore to dev and it's super fast... The overwhelming majority of your assetstore won't change every day so backups should only copy new files and those that have changed. If your colleagues are doing file-level backups it should be trivial to only sync what has changed...? BTW even rsync can be parallelized¹ with a little bit of shell scripting. Regards, ¹ https://mjanja.ch/2014/07/parallelizing-rsync/ On Thu, Aug 13, 2020 at 10:46 AM Michael White <[email protected]> wrote: > Hi, > > I just want to thank folk for the replies and discussion around this issue > - I will pass all this on to my network colleague and hopefully it will > give him some food for thought and a way forwards. > > Cheers, > > Mike > > On Tuesday, August 11, 2020 at 2:34:52 PM UTC+1, Michael White wrote: >> >> Hi, >> >> >> >> Our DSpace instance has been steadily growing over the years and now has >> over 18,000 records, all with one or more full text files attached (taking >> up around 106GB of disk space). We are on DSpace 6.2 and currently only >> have one assetstore configured. >> >> >> >> Whilst we have no issues with dspace operation/performance, my System and >> Network colleagues are reporting problems when their scheduled backup jobs >> of the DSpace server are trying to run as they are timing out, which they >> suspect is due to the large number of files on the server. >> >> >> >> To resolve this, they have asked me about the possibility of splitting >> the current dspace filesystem in to a number of smaller filesystems (which >> they can then back up in parallel, reducing the overall time to back up the >> dspace server). >> >> >> >> To that end they have asked about the possibility of splitting the >> assetstore in to a number of filesystems and creating softlinks to these >> from the assetstore - e.g. adding 9 new filesystems for clusters of 10 >> assetstore subdirectories - i.e.: >> >> >> >> .../dspace/assetstore/10 -> .../assetstores10-19/10 >> >> .../dspace/assetstore/11 -> .../assetstores10-19/11 >> >> .../dspace/assetstore/12 -> .../assetstores10-19/12 >> >> .... ... >> >> .../dspace/assetstore/20 -> ../assetstores20-29/20 >> >> >> >> Has anyone ever done anything like this? Any reasons why that wouldn't >> work? >> >> >> >> I can, of course, add another assetstore alongside the existing one, but >> my understanding is that this would only be used going forwards, and so >> wouldn't address the large number of files in the current assetstore - >> unless there is a way to then distribute the existing assetstore across a >> number of smaller assetstores? >> >> >> >> Has anyone else in the community had issues like this? And, if so, any >> (other) suggestions on how best to address the problem? >> >> >> >> Cheers, >> >> >> >> Mike >> >> >> >> >> *Michael White Senior Developer* >> >> >> *Business Applications and Integrations Information Services* >> >> >> 4B19, Cottrell >> >> University of Stirling >> Stirling >> FK9 4LA >> >> Tel: +44 (0)1786 466877 >> Email: [email protected] >> Web: stir.ac.uk/informationservices >> <http://www.stir.ac.uk/informationservices> >> >> <https://www.facebook.com/stirlinglibrary/> >> <https://twitter.com/isstirling> <https://www.instagram.com/isstirling/> >> <https://www.youtube.com/user/infoservicesatstir> >> >> [image: Banner] <https://www.stir.ac.uk/> >> >> >> >> >> ------------------------------ >> The University achieved an overall 5 stars in the QS World University >> Rankings 2020 >> UK Sports University of the Year 2020 (Times Higher Good University Guide) >> The University of Stirling is a charity registered in Scotland, number SC >> 011159. >> > -- > All messages to this mailing list should adhere to the DuraSpace Code of > Conduct: https://duraspace.org/about/policies/code-of-conduct/ > --- > You received this message because you are subscribed to the Google Groups > "DSpace Technical Support" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/dspace-tech/99a08863-e58e-4954-9cfe-79b5fabe5f3bo%40googlegroups.com > <https://groups.google.com/d/msgid/dspace-tech/99a08863-e58e-4954-9cfe-79b5fabe5f3bo%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Alan Orth [email protected] https://picturingjordan.com https://englishbulgaria.net https://mjanja.ch -- All messages to this mailing list should adhere to the DuraSpace Code of Conduct: https://duraspace.org/about/policies/code-of-conduct/ --- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/CAKKdN4XbXk%2BLL6q9mvit4KPPNcZUdMRhpoQb5%3D3e3jd126_-hw%40mail.gmail.com.
