Re: [Dspace-tech] Large files and DSpace
Most large files are audio/video files. Downloading the whole file may not be a good solution here. Maybe adapting the large files to suit the users' needs, e.g., if the user's bandwidth or usage environment is limited, then adapting the audio/video files into lower quality streams, would be a more sustainable approach? Besides, DSpace does not necessarily just give whatever it holds to the users. The dissemination can be adaptive based on various factors. Building such an adaptive dissemination framework shall be very interesting. Zhiwu Xie -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Cory Snavely Sent: Tuesday, April 17, 2007 6:55 AM To: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Large files and DSpace Interesting thought, but using bittorrent would require the setup of several peer sites in order to do its thing. Probably a good idea from a preservation standpoint but I would suspect not practical for many, and there are of course easier ways to support large transfer demands. I've thought that if a need emerged for us to handle this type of media, the DAS we use for the assetstore would be just fine, as of course would a SAN or NAS arrangement, but submissions would probably need to come on removable media and be loaded by staff, and distribution would probably need to be via streaming, which I know has been discussed on this list. To me those are the indicated approaches for this issue. c On Mon, 2007-04-16 at 22:30 -0300, Afonso Comba de Araujo Neto wrote: The problem is very intriguing and I felt like giving my 2 cents. I don't even think the problem is where you'll put it or how you'll integrate such files to DSpace. The main problem is how a regular user would download such a gigantic file. My first try would be to use another technology which is focused on handling such downloads. The best technology I can think of for this kind of thing is bit torrent. If I had to do that, I would include on DSpace just a .torrent file and instruct the users how to download using the bit torrent protocol (links to free clients, etc.). Not only it would be way better than a simple http download, but it could alleviate the strain on your server, which certainly would build up with such lengthy downloads. Regards, Afonso Araujo Neto Citando Gary Browne [EMAIL PROTECTED]: We have an assetstore residing on a SAN which solves the capacity issues, but as Cory says it is more the logistics of getting items into and out of the assetstore which is the problem. Regards Gary Gary Browne Development Programmer Library IT Services University of Sydney Australia ph: 61-2-9351 5946 -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Cory Snavely Sent: Tuesday, 17 April 2007 10:55 AM To: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Large files and DSpace I'd be interested to know how using SRB addresses the problem, which I understand to be the logistics of handling such a large file in both the user interface and the back end. Does it? Cory Snavely University of Michigan Library IT Core Services - Original Message - From: Ekaterina Pechekhonova [EMAIL PROTECTED] To: Gary Browne [EMAIL PROTECTED] Cc: dspace-tech@lists.sourceforge.net Sent: Monday, April 16, 2007 8:12 PM Subject: Re: [Dspace-tech] Large files and DSpace Hi Gary, you can configure Dspace to use SRB instead of regular assetstore. Some basic information can be found in the docs which come with Dspace.Also you can check this link: http://wiki.dspace.org/index.php//DspaceSrbIntegration Kate Ekaterina Pechekhonova Digital Library Programmer/Analyst New York University Libraries email: [EMAIL PROTECTED] phone: 212-992-9993 - Original Message - From: Gary Browne [EMAIL PROTECTED] Date: Monday, April 16, 2007 7:41 pm Subject: [Dspace-tech] Large files and DSpace To: dspace-tech@lists.sourceforge.net Hello All I think I posted a question like this last year but I've just become a dad for the first time and have a bit of brain meltdown. I tried searching for answers on the annoying sourceforge list archive (should I start a separate thread about this...?) but didn't find much. My question is a general one in that I'm wondering how people are handling large files in DSpace (getting them onto the server, submissions and publication/access)? Is the SymLink stuff the only option at this point? For example, we have (and will be getting lots more of) a 12GB video file to be used in one of our collections. I'd like to nut out what the possible options are before I try anything. Thanks and kind regards Gary Gary Browne Development Programmer Library IT Services University of Sydney Australia ph
Re: [Dspace-tech] Large files and DSpace
Thanks to all those who replied to this question - looks like people have varying opinions and implementations. I might start with SRB/registration and see how that goes. Regards Gary Gary Browne Development Programmer Library IT Services University of Sydney Australia ph: 61-2-9351 5946 - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Large files and DSpace
Hi We had done some preliminary testing using registration of large files 2GB into DSpace and it worked. Our approach was to utilize multiple asset stores within DSpace, and large files were handled by SRB asset store. The ingest was done through SRB and registration was done into DSpace. We basically worked off David Little's DSpace SRB integration patch, additionally my colleague Kate Pechekhonova did some re-factoring to make it work alongside non-SRB managed asset store. In our PR (preservation repository) which is managed by DSpace, we don't plan on allowing direct downloads of large files, since the same I/O bandwidth issues that prevent DSpace from directly ingesting large files, are also true for exporting of large files. Currently we plan on streaming proxies of the corresponding lower res AV files, through appropriate streaming server, and for qualified users the large files would be made available for download through some appropriate client application outside DSpace. Summary: Instead of ingesting individual large files, think of ingesting the large file with associated smaller proxy file into your collection which can be downloadable or stream-able depending upon your collection policy. Unni Unni Pillai Programmer/Analyst Tel: (212)-992-9741 [EMAIL PROTECTED] AIM: [EMAIL PROTECTED] Jabber:[EMAIL PROTECTED] Digital Library Program NDIIPP-Preserving Digital Public Television On Apr 17, 2007, at 8:55 AM, Mark H. Wood wrote: My position on huge AV files has been that they are too large for the average user to fiddle with if he has to download the whole thing before seeing/hearing anything. We ought to have the bitstreams in DSpace for archival purposes, but we also ought to mount them on a streaming service and point to them, so that casual users can just open them with a streaming client. It's fairly easy to build a trivial SMIL document pointing to a stream and tuck that into an item. This doesn't address the ingestion performance issue, of course. Does the registration mechanism not suffice for this? -- Mark H. Wood, Lead System Programmer [EMAIL PROTECTED] Typically when a software vendor says that a product is intuitive he means the exact opposite. -- --- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Large files and DSpace
Hi We had done some preliminary testing using registration of large files 2GB into DSpace and it worked. Our approach was to utilize multiple asset stores within DSpace, and large files were handled by SRB asset store. The ingest was done through SRB and registration was done into DSpace. We basically worked off David Little's DSpace SRB integration patch, additionally my colleague Kate Pechekhonova did some re-factoring to make it work alongside non-SRB managed asset store. In our PR (preservation repository) which is managed by DSpace, we don't plan on allowing direct downloads of large files, since the same I/O bandwidth issues that prevent DSpace from directly ingesting large files, are also true for exporting of large files. Currently we plan on streaming proxies of the corresponding lower res AV files, through appropriate streaming server, and for qualified users the large files would be made available for download through some appropriate client application outside DSpace. Summary: Instead of ingesting individual large files, think of ingesting the large file with associated smaller proxy file into your collection which can be downloadable or stream-able depending upon your collection policy. Unni Pillai Programmer/Analyst Tel: (212)-992-9741 [EMAIL PROTECTED] AIM: [EMAIL PROTECTED] Jabber:[EMAIL PROTECTED] Digital Library Program NDIIPP-Preserving Digital Public Television On Apr 17, 2007, at 8:55 AM, Mark H. Wood wrote: My position on huge AV files has been that they are too large for the average user to fiddle with if he has to download the whole thing before seeing/hearing anything. We ought to have the bitstreams in DSpace for archival purposes, but we also ought to mount them on a streaming service and point to them, so that casual users can just open them with a streaming client. It's fairly easy to build a trivial SMIL document pointing to a stream and tuck that into an item. This doesn't address the ingestion performance issue, of course. Does the registration mechanism not suffice for this? -- Mark H. Wood, Lead System Programmer [EMAIL PROTECTED] Typically when a software vendor says that a product is intuitive he means the exact opposite. -- --- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Large files and DSpace
My position on huge AV files has been that they are too large for the average user to fiddle with if he has to download the whole thing before seeing/hearing anything. We ought to have the bitstreams in DSpace for archival purposes, but we also ought to mount them on a streaming service and point to them, so that casual users can just open them with a streaming client. It's fairly easy to build a trivial SMIL document pointing to a stream and tuck that into an item. This doesn't address the ingestion performance issue, of course. Does the registration mechanism not suffice for this? -- Mark H. Wood, Lead System Programmer [EMAIL PROTECTED] Typically when a software vendor says that a product is intuitive he means the exact opposite. pgpnsImbrUysh.pgp Description: PGP signature - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
[Dspace-tech] Large files and DSpace
Hello All I think I posted a question like this last year but I've just become a dad for the first time and have a bit of brain meltdown. I tried searching for answers on the annoying sourceforge list archive (should I start a separate thread about this...?) but didn't find much. My question is a general one in that I'm wondering how people are handling large files in DSpace (getting them onto the server, submissions and publication/access)? Is the SymLink stuff the only option at this point? For example, we have (and will be getting lots more of) a 12GB video file to be used in one of our collections. I'd like to nut out what the possible options are before I try anything. Thanks and kind regards Gary Gary Browne Development Programmer Library IT Services University of Sydney Australia ph: 61-2-9351 5946 - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Large files and DSpace
Hi Gary, you can configure Dspace to use SRB instead of regular assetstore. Some basic information can be found in the docs which come with Dspace.Also you can check this link: http://wiki.dspace.org/index.php//DspaceSrbIntegration Kate Ekaterina Pechekhonova Digital Library Programmer/Analyst New York University Libraries email: [EMAIL PROTECTED] phone: 212-992-9993 - Original Message - From: Gary Browne [EMAIL PROTECTED] Date: Monday, April 16, 2007 7:41 pm Subject: [Dspace-tech] Large files and DSpace To: dspace-tech@lists.sourceforge.net Hello All I think I posted a question like this last year but I've just become a dad for the first time and have a bit of brain meltdown. I tried searching for answers on the annoying sourceforge list archive (should I start a separate thread about this...?) but didn't find much. My question is a general one in that I'm wondering how people are handling large files in DSpace (getting them onto the server, submissions and publication/access)? Is the SymLink stuff the only option at this point? For example, we have (and will be getting lots more of) a 12GB video file to be used in one of our collections. I'd like to nut out what the possible options are before I try anything. Thanks and kind regards Gary Gary Browne Development Programmer Library IT Services University of Sydney Australia ph: 61-2-9351 5946 - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Large files and DSpace
I'd be interested to know how using SRB addresses the problem, which I understand to be the logistics of handling such a large file in both the user interface and the back end. Does it? Cory Snavely University of Michigan Library IT Core Services - Original Message - From: Ekaterina Pechekhonova [EMAIL PROTECTED] To: Gary Browne [EMAIL PROTECTED] Cc: dspace-tech@lists.sourceforge.net Sent: Monday, April 16, 2007 8:12 PM Subject: Re: [Dspace-tech] Large files and DSpace Hi Gary, you can configure Dspace to use SRB instead of regular assetstore. Some basic information can be found in the docs which come with Dspace.Also you can check this link: http://wiki.dspace.org/index.php//DspaceSrbIntegration Kate Ekaterina Pechekhonova Digital Library Programmer/Analyst New York University Libraries email: [EMAIL PROTECTED] phone: 212-992-9993 - Original Message - From: Gary Browne [EMAIL PROTECTED] Date: Monday, April 16, 2007 7:41 pm Subject: [Dspace-tech] Large files and DSpace To: dspace-tech@lists.sourceforge.net Hello All I think I posted a question like this last year but I've just become a dad for the first time and have a bit of brain meltdown. I tried searching for answers on the annoying sourceforge list archive (should I start a separate thread about this...?) but didn't find much. My question is a general one in that I'm wondering how people are handling large files in DSpace (getting them onto the server, submissions and publication/access)? Is the SymLink stuff the only option at this point? For example, we have (and will be getting lots more of) a 12GB video file to be used in one of our collections. I'd like to nut out what the possible options are before I try anything. Thanks and kind regards Gary Gary Browne Development Programmer Library IT Services University of Sydney Australia ph: 61-2-9351 5946 - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Large files and DSpace
We have an assetstore residing on a SAN which solves the capacity issues, but as Cory says it is more the logistics of getting items into and out of the assetstore which is the problem. Regards Gary Gary Browne Development Programmer Library IT Services University of Sydney Australia ph: 61-2-9351 5946 -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Cory Snavely Sent: Tuesday, 17 April 2007 10:55 AM To: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Large files and DSpace I'd be interested to know how using SRB addresses the problem, which I understand to be the logistics of handling such a large file in both the user interface and the back end. Does it? Cory Snavely University of Michigan Library IT Core Services - Original Message - From: Ekaterina Pechekhonova [EMAIL PROTECTED] To: Gary Browne [EMAIL PROTECTED] Cc: dspace-tech@lists.sourceforge.net Sent: Monday, April 16, 2007 8:12 PM Subject: Re: [Dspace-tech] Large files and DSpace Hi Gary, you can configure Dspace to use SRB instead of regular assetstore. Some basic information can be found in the docs which come with Dspace.Also you can check this link: http://wiki.dspace.org/index.php//DspaceSrbIntegration Kate Ekaterina Pechekhonova Digital Library Programmer/Analyst New York University Libraries email: [EMAIL PROTECTED] phone: 212-992-9993 - Original Message - From: Gary Browne [EMAIL PROTECTED] Date: Monday, April 16, 2007 7:41 pm Subject: [Dspace-tech] Large files and DSpace To: dspace-tech@lists.sourceforge.net Hello All I think I posted a question like this last year but I've just become a dad for the first time and have a bit of brain meltdown. I tried searching for answers on the annoying sourceforge list archive (should I start a separate thread about this...?) but didn't find much. My question is a general one in that I'm wondering how people are handling large files in DSpace (getting them onto the server, submissions and publication/access)? Is the SymLink stuff the only option at this point? For example, we have (and will be getting lots more of) a 12GB video file to be used in one of our collections. I'd like to nut out what the possible options are before I try anything. Thanks and kind regards Gary Gary Browne Development Programmer Library IT Services University of Sydney Australia ph: 61-2-9351 5946 - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
