Hi Peter. I'm interested in running the application server on a separated machine from the assets (more space, fast updates). I think using S3 (or something like Minio.io) could be a good start. Do you have more information about S3 and DSpace 5? I think DSpace 6 it's not ready for production yet. Is there another way besides S3 protocol?
Thank you very much. El lunes, 11 de enero de 2016, 16:59:04 (UTC-5), Peter Dietz escribió: > > Hi Adam, > > A followup to DSpace + S3. Last week, the code to add S3 storage layer was > merged into DSpace's master code base, (for the upcoming DSpace 6 release). > > S3 Pull Request on GitHub > https://github.com/DSpace/DSpace/pull/1159 > > DSpace 6.x Storage Layer documentation > https://wiki.duraspace.org/display/DSDOC6x/Storage+Layer > > Since your last email, here are some changes that have happened to the S3 > code. > > For running checksums, it is able to have the S3 service compute a > checksum of the object, (remotely on S3), as opposed to having to GET each > file, parse through the bits, and compute the hash. So, a 100GB repository, > previously would have had to GET 100GB of assets, just to do the checksum, > with S3 computing the hash, its just request/response with metadata. > (Normal filesystem based assetstores will have to read all content to do > the checksum.) > > There is a bitstore migrate command that I've added, to allow your DSpace > site to move all of your assets from one bitstore (assetstore) to another. > i.e. local disk assetstore to S3 bucket s3://dspace-nist-prod. The transfer > operation is fairly robust, I've disconnect network midtransfer, closed lid > on MBP, sent exit code to operation, and it all seemed to work just fine. > Since it is "new" code for an upcoming version of DSpace, I would encourage > early adopters, and perhaps large adopters to test this out, and if you > encounter anything, to share your feedback so that things can be fixed > early in the cycle. > > A known issue at this moment is that DSpace's S3 code only supports a > maximum single-file size of 5GB. If this is important to sites, then feel > free to submit a contribution to increase that. S3's maximum theoretical > file size limit for a single file is 5 Terabytes, you just need to change > the code we use to interface with it, to transfer content in chunks. > > Performance is great, we have several sites in production using an earlier > version of the S3 code, and haven't run into anything concerning. > > We can foresee future optimizations. Such as if a user is requesting an > object that, instead of S3 sending the binary to your DSpace application > server, then through your webserver, then to the end user, that you could > instead send the user a one-time signed URL to directly to access the file > from S3. But then we might run into issues with crawlers indexing our > content, and if the url for content keeps changing, then they might have > trouble indexing content?? I suppose if you throw HTTP 305 Use Proxy, then > actually, I have no idea, but I suppose this could be solved, to eek out a > bit more performance, and reduce load on your servers. > > Also, adding S3 as a storage layer implementation, means that we cleaned > up / refactored the BitStore interface, so if you had some other desired > cloud storage service, then other implementations could fit in easily. > > > ________________ > Peter Dietz > Longsight > www.longsight.com > [email protected] <javascript:> > p: 740-599-5005 x809 > > On Thu, Sep 3, 2015 at 4:42 PM, Peter Dietz <[email protected] > <javascript:>> wrote: > >> Hi Adam, >> >> S3 is going to be a great asset for DSpace to ease the march to the >> cloud. Unlimited painless storage. We've launched new production sites >> (starting fresh with zero content) using the S3 storage, and haven't run >> into any issues. >> >> The only gotchas at this time are: >> - There's no migration script to move your existing assetstore to S3. So, >> we have launched new production sites with S3 storage, but haven't yet >> migrated our existing sites. We (Longsight) are developing a migration >> script, but if anyone wants to contribute here, then feel free to jump in. >> >> - If you are hosting DSpace outside of EC2, then you'll notice network >> latency as data has to go in and out of AWS. ( I noticed this when I did a >> batch import on my localhost laptop, and writing to local disk is very >> fast, but having to send it over the wire, through a vpn, etc was slower.) >> >> It would be nice to do a proper benchmark comparison between DSpace in >> different architectures. 1M writes / 1M reads with S3 vs EBS (traditional >> file based asset store), and see if there is any noticeable difference. >> >> I haven't yet run into an issue, but I've been wondering if media-filter >> and indexing make too many calls to the assetstore. >> >> We developed this first on the 5x branch, and then ported it to 6x >> branch. >> >> Another random thought is that S3 does allow you to write "metadata" when >> you write a file. So, we could store additional information about that >> asset (filename, filetype, parent item, ...), when we send it to S3. >> >> Yet another random thought is that since S3 has the offline / cheaper >> Glacier storage. That perhaps in the future DSpace could mark certain types >> of content that is either infrequently accessed, or stored for archival >> purpose rather than end-user access. I'm thinking of a site that maybe >> uploads large video master copied, and perhaps serves a YouTube version >> instead of that mpeg. Or large datasets. But that would require some type >> of workflow to deal with the storage lifecycle (8 hours to retrieve, or >> policy that moves files to cold storage). >> >> But overall S3 is a very stable product. The storage interface in DSpace >> is pretty clean / abstracted / minimal. And we're using the AWS SDK to use >> S3, as opposed to some third party wrapper. >> >> I hope all of this helps, and feel free to reach out if you need more >> information or assistance. >> >> ________________ >> Peter Dietz >> Longsight >> www.longsight.com >> [email protected] <javascript:> >> p: 740-599-5005 x809 >> >> On Thu, Sep 3, 2015 at 4:09 PM, Adam Morey <[email protected] >> <javascript:>> wrote: >> >>> I'm trying to learn about incorporating AWS S3 with DSpace and would >>> like to contact anyone using this in their current configuration. I've >>> read the longsight blog, http://duraspace.org/articles/2583 on >>> contributing an S3 plugin to DSpace 6. I've seen a few link to folks using >>> duraspace with jungledisk, anyone tried it with Dspace? >>> >>> We're contemplating migrating our repository (materialsdata.nist.gov) >>> to Amazon AWS, but the costs only scale if we store the data in S3. >>> >>> Any help would be appreciated. >>> >>> Is this a "not ready for production" config? >>> What are the limitations of using S3 with Dspace? File upload size >>> restrictions? Performance? Indexing? >>> >>> Adam >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "DSpace Community" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected] <javascript:>. >>> To post to this group, send email to [email protected] >>> <javascript:>. >>> Visit this group at http://groups.google.com/group/dspace-community. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> > -- You received this message because you are subscribed to the Google Groups "DSpace Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/dspace-community. For more options, visit https://groups.google.com/d/optout.
