On Wed, Aug 12, 2020 at 05:59:02AM +0000, Tony Brian Albers wrote: > On Tue, 2020-08-11 at 11:07 -0400, Mark H. Wood wrote: > > On Tue, Aug 11, 2020 at 01:34:48PM +0000, Michael White wrote: > > > Our DSpace instance has been steadily growing over the years and > > > now has over 18,000 records, all with one or more full text files > > > attached (taking up around 106GB of disk space). We are on DSpace > > > 6.2 and currently only have one assetstore configured. > > > > > > Whilst we have no issues with dspace operation/performance, my > > > System and Network colleagues are reporting problems when their > > > scheduled backup jobs of the DSpace server are trying to run as > > > they are timing out, which they suspect is due to the large number > > > of files on the server. > > > > > > To resolve this, they have asked me about the possibility of > > > splitting the current dspace filesystem in to a number of smaller > > > filesystems (which they can then back up in parallel, reducing the > > > overall time to back up the dspace server). > > > > > > To that end they have asked about the possibility of splitting the > > > assetstore in to a number of filesystems and creating softlinks to > > > these from the assetstore - e.g. adding 9 new filesystems for > > > clusters of 10 assetstore subdirectories - i.e.: > > > > > > .../dspace/assetstore/10 -> .../assetstores10-19/10 > > > .../dspace/assetstore/11 -> .../assetstores10-19/11 > > > .../dspace/assetstore/12 -> .../assetstores10-19/12 > > > .... ... > > > .../dspace/assetstore/20 -> ../assetstores20-29/20 > > > > > > Has anyone ever done anything like this? Any reasons why that > > > wouldn't work? > > Not really. However one should consider using mountpoints in the > assetstore itself instead. That would make it easier to administrate.
I thought about that, but you'd need 100 of them, not 10. The first-level assetstore directories are 00 through 99. They are named by the first two decimal digits of a checksum of the Bitstream's content. > > > > I see no obvious reason why it wouldn't work. > > > > "They" must think that the backup storage device is the bottleneck, > > if > > they believe that parallel backups will improve throughput. > > Ehm, not following you here, Mark. > What I think they want to achieve is a higher client parallelism. This > means that instead of one save stream running on one file system they > can have several savestreams running, one on each filesystem. If they > had backup storage performance issues that would not make sense. > So, going for higher parallelism by having several smaller file systems > makes sense. If the bottleneck is network, higher parallelism doesn't help. If the bottleneck is CPU or memory, higher parallelism doesn't help. If the bottleneck is disk, higher parallelism only helps if those ten volumes are on separate physical disks. (Where do you buy 10GB disks these days?) Same for the storage controller(s). That left me to consider the backup system itself, where the slowest thing in the entire process, by far, is the tape drive(s). The problem *could* be the local disks. I'd want to prove that first. -- Mark H. Wood Lead Technology Analyst University Library Indiana University - Purdue University Indianapolis 755 W. Michigan Street Indianapolis, IN 46202 317-274-0749 www.ulib.iupui.edu -- All messages to this mailing list should adhere to the DuraSpace Code of Conduct: https://duraspace.org/about/policies/code-of-conduct/ --- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/20200812131451.GA20729%40IUPUI.Edu.
signature.asc
Description: PGP signature
