Hi Adam, Thanks for the response. From your info it sounds like a shouldn't have too much of a problem. Some of our video ingests will need to happen through some kind of web interface, so I was thinking ahead to user experience, and whether or not I would have to build-in any progress screens while a checksum was created. Looks like I won't have to worry.
It'd still be a good idea to build some kind of progress interface - if only because the API request for the ingest might time out before the ingest is completed. I've been meaning to do some more testing in this area but haven't yet done so, so once you get to that stage, anything you can share would be useful. Our really large video files will most likely be ingested via scripts or a some kind of network drop box, so timing isn't so much of an issue there. That sounds reasonable. We've experimented with encoding our video straight into a WebDAV share, which Fedora can access. That way, we can build FoXMLs with '<foxml:contentLocation>' tags pointing to the HTTP URLs of the encoded items and Fedora will download them, rather than relying on POSTing large files through the REST or API-M interfaces. An alternative is the file:// URI schema, which could be faster if the files are already locally addressable. The other bug which you may run into on 3.3 is FCREPO-696 [1] - also related to checksums. This is fixed in 3.4. Also, the Flex client has an upload bug relating to managed content datastreams [2], though this doesn't affect the API. Regards, Graeme [1] https://jira.duraspace.org/browse/FCREPO-696 [2] https://jira.duraspace.org/browse/FCREPO-605 On 25 Oct 2010, at 17:48, Adam Wead wrote: Hi Graeme, Thanks for the response. From your info it sounds like a shouldn't have too much of a problem. Some of our video ingests will need to happen through some kind of web interface, so I was thinking ahead to user experience, and whether or not I would have to build-in any progress screens while a checksum was created. Looks like I won't have to worry. Our really large video files will most likely be ingested via scripts or a some kind of network drop box, so timing isn't so much of an issue there. I'm also currently developing with version 3.3 and will soon move to 3.4 to avoid the bugs you mentioned. thanks again! ...adam On Mon, Oct 25, 2010 at 4:59 AM, West, Graeme <graeme.w...@gcu.ac.uk<mailto:graeme.w...@gcu.ac.uk>> wrote: Hi Adam, I can't answer the question regarding threading in any technical detail, but I gather that in general Fedora is multi-threaded, so multiple ingest requests will be handled separately. I think it's the case though that when ingesting multiple objects simultaneously, the shell utilities provided with Fedora (e.g. passing a 'd' flag to fedora-ingest.sh ) work sequentially - so you may want to make API requests directly rather than using these if you want to do things in parallel. We're using Fedora for storage of video files - mostly compressed MPEG-2, so in the 800MB - 4GB file size range. We use SHA-512 checksums without any issues. I generally monitor ingests through a combination of Java VisualVM and iotop to get an impression of what's IO-centric and what's CPU-centric: on ingest of 2 - 4GB files, checksumming is pretty quick - 1 to 5 seconds at the end of an ingest cycle. That's on a Xen VM running on a dual-quad core Xeon physical machine. Just to note, Fedora 3.4 fixes a bug in the REST interface in relation to files over 2GB [1]. Also, versions prior to 3.2 (I think) will generally attempt to read entire datastreams into RAM on ingest, which isn't a great idea with large files. There is also an outstanding JIRA issue relating to streaming files directly into the back-end filestore, which if implemented would speed up large ingests considerably [2]. We have tested 3.4 ingesting up to 45GB files with no particular performance issues, and these ingests included checksumming. Regards, Graeme [1] https://jira.duraspace.org/browse/FCREPO-704 [2] https://jira.duraspace.org/browse/FCREPO-495 On 22 Oct 2010, at 19:02, Adam Wead wrote: Hi all, How does the checksum generation process for very large files, like video files of 50 GB, affect Fedora's performance? The scenario I'm thinking about is if you're ingesting a batch of large files into Fedora, will it fork a new process to generate the checksum in the background while proceeding on to the next file, or will it only proceed to the next file after the checksum for the first file has been generated. Anyone care to chime in on checksums in general for video files? I've used them before on audio files without too much trouble, but the size of video files might present problems. thanks, ...adam Email has been scanned for viruses by Altman Technologies' email management service<http://www.altman.co.uk/emailsystems> ------------------------------------------------------------------------------ Nokia and AT&T present the 2010 Calling All Innovators-North America contest Create new apps & games for the Nokia N8 for consumers in U.S. and Canada $10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store http://p.sf.net/sfu/nokia-dev2dev Email has been scanned for viruses by Altman Technologies' email management service - www.altman.co.uk/emailsystems<http://www.altman.co.uk/emailsystems> _______________________________________________ Fedora-commons-users mailing list Fedora-commons-users@lists.sourceforge.net<mailto:Fedora-commons-users@lists.sourceforge.net> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users Email has been scanned for viruses by Altman Technologies' email management service - www.altman.co.uk/emailsystems<http://www.altman.co.uk/emailsystems> Glasgow Caledonian University is a registered Scottish charity, number SC021474 Winner: Times Higher Education's Widening Participation Initiative of the Year 2009 and Herald Society's Education Initiative of the Year 2009 http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,en.html ------------------------------------------------------------------------------ Nokia and AT&T present the 2010 Calling All Innovators-North America contest Create new apps & games for the Nokia N8 for consumers in U.S. and Canada $10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store http://p.sf.net/sfu/nokia-dev2dev _______________________________________________ Fedora-commons-users mailing list Fedora-commons-users@lists.sourceforge.net<mailto:Fedora-commons-users@lists.sourceforge.net> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users -- ~/amsterdam Email has been scanned for viruses by Altman Technologies' email management service<http://www.altman.co.uk/emailsystems> ------------------------------------------------------------------------------ Nokia and AT&T present the 2010 Calling All Innovators-North America contest Create new apps & games for the Nokia N8 for consumers in U.S. and Canada $10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store http://p.sf.net/sfu/nokia-dev2dev Email has been scanned for viruses by Altman Technologies' email management service - www.altman.co.uk/emailsystems _______________________________________________ Fedora-commons-users mailing list Fedora-commons-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/fedora-commons-users Email has been scanned for viruses by Altman Technologies' email management service - www.altman.co.uk/emailsystems Glasgow Caledonian University is a registered Scottish charity, number SC021474 Winner: Times Higher Education's Widening Participation Initiative of the Year 2009 and Herald Society's Education Initiative of the Year 2009 http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,en.html ------------------------------------------------------------------------------ Nokia and AT&T present the 2010 Calling All Innovators-North America contest Create new apps & games for the Nokia N8 for consumers in U.S. and Canada $10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store http://p.sf.net/sfu/nokia-dev2dev _______________________________________________ Fedora-commons-users mailing list Fedora-commons-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
