On Tue, Aug 11, 2020 at 01:34:48PM +0000, Michael White wrote: > Our DSpace instance has been steadily growing over the years and now has over > 18,000 records, all with one or more full text files attached (taking up > around 106GB of disk space). We are on DSpace 6.2 and currently only have one > assetstore configured. > > Whilst we have no issues with dspace operation/performance, my System and > Network colleagues are reporting problems when their scheduled backup jobs of > the DSpace server are trying to run as they are timing out, which they > suspect is due to the large number of files on the server. > > To resolve this, they have asked me about the possibility of splitting the > current dspace filesystem in to a number of smaller filesystems (which they > can then back up in parallel, reducing the overall time to back up the dspace > server). > > To that end they have asked about the possibility of splitting the assetstore > in to a number of filesystems and creating softlinks to these from the > assetstore - e.g. adding 9 new filesystems for clusters of 10 assetstore > subdirectories - i.e.: > > .../dspace/assetstore/10 -> .../assetstores10-19/10 > .../dspace/assetstore/11 -> .../assetstores10-19/11 > .../dspace/assetstore/12 -> .../assetstores10-19/12 > .... ... > .../dspace/assetstore/20 -> ../assetstores20-29/20 > > Has anyone ever done anything like this? Any reasons why that wouldn't work?
I see no obvious reason why it wouldn't work. "They" must think that the backup storage device is the bottleneck, if they believe that parallel backups will improve throughput. I would do some exploring to see if this is true, before doing a lot of work to redistribute all those files. Is there another lurking bottleneck, nearly as narrow, currently covered by the one they are working on? Each of those ten volumes will continue to grow, but at 1/10 the current rate, so this puts the problem off for some time but doesn't solve it in the long run. That will give you some time to ponder whether there is a smarter way to back up your repository. > I can, of course, add another assetstore alongside the existing one, but my > understanding is that this would only be used going forwards, and so wouldn't > address the large number of files in the current assetstore - unless there is > a way to then distribute the existing assetstore across a number of smaller > assetstores? Your understanding is correct: all new submissions go into the "current assetstore". There is a "BitStore migration tool" which moves *all* content from one assetstore to another: https://wiki.lyrasis.org/display/DSDOC6x/Storage+Layer#StorageLayer-MigrateBitStores That isn't quite what you want, but it might serve as a model for a tool to split up an assetstore. If you do go this route, you might look at changing the way you do backup. The materials in the "old" assetstores are probably subject to very infrequent deletions but are otherwise static storage -- all new and replacement content goes into the "current" assetstore. You might devise a very different backup schedule for the old assetstores vs. current. -- Mark H. Wood Lead Technology Analyst University Library Indiana University - Purdue University Indianapolis 755 W. Michigan Street Indianapolis, IN 46202 317-274-0749 www.ulib.iupui.edu -- All messages to this mailing list should adhere to the DuraSpace Code of Conduct: https://duraspace.org/about/policies/code-of-conduct/ --- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/20200811150732.GA30122%40IUPUI.Edu.
signature.asc
Description: PGP signature
