The immediate mechanism that comes to mind for your case is Bitstream
Registration
https://wiki.duraspace.org/display/DSDOC18/Registering+(not+Importing)+Bitstreams+via+Simple+Archive+Format
But if SWORD is not aware of this mechanism some customization may be in
order to support it.
Mark
On Mon, Nov 28, 2011 at 10:19 AM, Leggett, Pete <p.f.legg...@exeter.ac.uk>wrote:
> Stuart,****
>
> ** **
>
> Can you provide any links to examples of using ‘deposit by reference’ ?***
> *
>
> ** **
>
> I am looking at feasibility of depositing very large items (tar.gz or
> zip’d data files), say up to 16TB, into Dspace 1.6.x with the obvious
> problems of doing this using a web interface.****
>
> Wondering if EasyDeposit can be adapted to do ‘deposit by reference’ with
> either a utility of some kind on the dspace server looking for large items
> to injest or a client pushing the data onto a directory on the dspace
> server from where it can be injested. Ideally want to minimise any copies
> of the data.****
>
> ** **
>
> Really want to avoid copying the item once it’s on the Dspace server.
> Could item be uploaded directly into asset store maybe ?****
>
> The other problem is how anyone could download the item once it’s in
> Dspace ?****
>
> ** **
>
> Anyone else doing this sort of very large item ( i.e. TB’s ) injest ?****
>
> ** **
>
> Thank you,****
>
>
> Pete****
>
> ** **
>
> ** **
>
> *From:* David FLANDERS [mailto:d.fland...@jisc.ac.uk]
> *Sent:* 21 November 2011 18:10
> *To:* Ben O'Steen; Stuart Lewis
>
> *Cc:* <, sword-app-tech@lists.sourceforge.net>,
> *Subject:* Re: [sword-app-tech] How to send large fiels****
>
> ** **
>
> To second that, some amazing things being done down here in Australia as
> they take *large* data off of scientific instruments. /dff****
>
> ** **
>
> *From:* Ben O'Steen [mailto:bost...@gmail.com]
> *Sent:* 14 November 2011 23:54
> *To:* Stuart Lewis
> *Cc:* <, sword-app-tech@lists.sourceforge.net>,
> *Subject:* Re: [sword-app-tech] How to send large fiels****
>
> ** **
>
> +1 for deposit by reference. It is almost like giving a metadata receipt
> for a deposit not happening via the http route.****
>
> I would also highly recommend looking at High-Performance SSH or HPN-SSH.
> On comparable hardware, I have been shown that it outpaces even grid-ftp
> for file transfer speeds, but is a backward compatible patch for the
> openssh library.****
>
> This means that if the server and client are both patched, the transfer is
> multithreaded and otherwise highly optimized. It means that Unix tools
> which use SSH benefit as well - rsync, ssh -X, and so on.****
>
> Ben****
>
> On Nov 15, 2011 10:37 AM, "Stuart Lewis" <s.le...@auckland.ac.nz> wrote:**
> **
>
> Hi Jesús,
>
> The method that seems to work in this setting is to use 'deposit by
> reference'. That is, you deposit a description of the item, including
> details of where the item can be found. It is then up to ingesting system
> to pull the data - perhaps via an offline queue process, using some other
> method (ftp, scp, nfs, etc).
>
> SWORD v2 might be useful here too, because the SWORD statement could be
> requested to find out the status of the file upload (for example, using a
> status such as pending, in-process, complete, failed, etc). This would
> allow the sender/depositor to be able to find out the status of the item.
>
> Let us know how you get on - it is interesting to see the protocol being
> pushed to its limits with use cases such as yours.
>
> Thanks,
>
>
> Stuart Lewis
> Digital Development Manager
> Te Tumu Herenga The University of Auckland Library
> Auckland Mail Centre, Private Bag 92019, Auckland 1142, New Zealand
> Ph: +64 (0)9 373 7599 x81928
>
>
>
> On 15/11/2011, at 11:27 AM, Jesús García Crespo wrote:
>
> > Hi everyone,
> >
> > We are using SWORD to deposit DIPs in ICA-AtoM from Archivematica, but
> we have encountered some problems with large files (8GB). HTTP requests can
> be very resource intensive and unmanageable when the contents are very big.
> Does anyone have any recommendations, for depositing large files via the
> SWORD protocol? Like maybe sending related files using SFTP and then
> indicating the local file route? (That is something we are considering.)
> >
> > Thank you in advance,
> >
> > --
> > Jesús García Crespo
> >
> ------------------------------------------------------------------------------
> > RSA(R) Conference 2012
> > Save $700 by Nov 18
> > Register now
> >
> http://p.sf.net/sfu/rsa-sfdev2dev1_______________________________________________
> > sword-app-tech mailing list
> > sword-app-tech@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/sword-app-tech
>
>
>
>
>
> ------------------------------------------------------------------------------
> RSA(R) Conference 2012
> Save $700 by Nov 18
> Register now
> http://p.sf.net/sfu/rsa-sfdev2dev1
> _______________________________________________
> sword-app-tech mailing list
> sword-app-tech@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/sword-app-tech****
>
>
> ------------------------------------------------------------------------------
> All the data continuously generated in your IT infrastructure
> contains a definitive record of customers, application performance,
> security threats, fraudulent activity, and more. Splunk takes this
> data and makes sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-novd2d
> _______________________________________________
> sword-app-tech mailing list
> sword-app-tech@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/sword-app-tech
>
>
--
[image: @mire Inc.]
*Mark Diggory*
*2888 Loker Avenue East, Suite 305, Carlsbad, CA. 92010*
*Esperantolaan 4, Heverlee 3001, Belgium*
http://www.atmire.com
------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
sword-app-tech mailing list
sword-app-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sword-app-tech
