Hi Graeme,
Thanks for the response. From your info it sounds like a shouldn't have too
much of a problem. Some of our video ingests will need to happen through
some kind of web interface, so I was thinking ahead to user experience, and
whether or not I would have to build-in any progress screens while a
checksum was created. Looks like I won't have to worry.
Our really large video files will most likely be ingested via scripts or a
some kind of network drop box, so timing isn't so much of an issue there.
I'm also currently developing with version 3.3 and will soon move to 3.4 to
avoid the bugs you mentioned.
thanks again!
...adam
On Mon, Oct 25, 2010 at 4:59 AM, West, Graeme <graeme.w...@gcu.ac.uk> wrote:
> Hi Adam,
> I can't answer the question regarding threading in any technical detail,
> but I gather that in general Fedora is multi-threaded, so multiple ingest
> requests will be handled separately. I think it's the case though that when
> ingesting multiple objects simultaneously, the shell utilities provided with
> Fedora (e.g. passing a 'd' flag to fedora-ingest.sh ) work sequentially - so
> you may want to make API requests directly rather than using these if you
> want to do things in parallel.
>
> We're using Fedora for storage of video files - mostly compressed MPEG-2,
> so in the 800MB - 4GB file size range. We use SHA-512 checksums without any
> issues. I generally monitor ingests through a combination of Java VisualVM
> and iotop to get an impression of what's IO-centric and what's CPU-centric:
> on ingest of 2 - 4GB files, checksumming is pretty quick - 1 to 5 seconds at
> the end of an ingest cycle. That's on a Xen VM running on a dual-quad core
> Xeon physical machine.
>
> Just to note, Fedora 3.4 fixes a bug in the REST interface in relation to
> files over 2GB [1]. Also, versions prior to 3.2 (I think) will generally
> attempt to read entire datastreams into RAM on ingest, which isn't a great
> idea with large files. There is also an outstanding JIRA issue relating to
> streaming files directly into the back-end filestore, which if implemented
> would speed up large ingests considerably [2].
>
> We have tested 3.4 ingesting up to 45GB files with no particular
> performance issues, and these ingests included checksumming.
>
> Regards,
>
> Graeme
>
>
> [1] https://jira.duraspace.org/browse/FCREPO-704
> [2] https://jira.duraspace.org/browse/FCREPO-495
>
> On 22 Oct 2010, at 19:02, Adam Wead wrote:
>
> Hi all,
>
> How does the checksum generation process for very large files, like video
> files of 50 GB, affect Fedora's performance? The scenario I'm thinking
> about is if you're ingesting a batch of large files into Fedora, will it
> fork a new process to generate the checksum in the background while
> proceeding on to the next file, or will it only proceed to the next file
> after the checksum for the first file has been generated.
>
> Anyone care to chime in on checksums in general for video files? I've used
> them before on audio files without too much trouble, but the size of video
> files might present problems.
>
> thanks,
>
> ...adam
>
> Email has been scanned for viruses by Altman Technologies' email management
> service<http://www.altman.co.uk/emailsystems>
>
>
> ------------------------------------------------------------------------------
> Nokia and AT&T present the 2010 Calling All Innovators-North America
> contest
> Create new apps & games for the Nokia N8 for consumers in U.S. and Canada
> $10 million total in prizes - $4M cash, 500 devices, nearly $6M in
> marketing
> Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
> http://p.sf.net/sfu/nokia-dev2dev
> Email has been scanned for viruses by Altman Technologies' email management
> service - www.altman.co.uk/emailsystems
> _______________________________________________
> Fedora-commons-users mailing list
> Fedora-commons-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
>
> Email has been scanned for viruses by Altman Technologies' email management
> service - www.altman.co.uk/emailsystems
>
>
> Glasgow Caledonian University is a registered Scottish charity, number
> SC021474
>
> Winner: Times Higher Education's Widening Participation Initiative of the
> Year 2009 and Herald Society's Education Initiative of the Year 2009
>
> http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,en.html
>
>
> ------------------------------------------------------------------------------
> Nokia and AT&T present the 2010 Calling All Innovators-North America
> contest
> Create new apps & games for the Nokia N8 for consumers in U.S. and Canada
> $10 million total in prizes - $4M cash, 500 devices, nearly $6M in
> marketing
> Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
> http://p.sf.net/sfu/nokia-dev2dev
> _______________________________________________
> Fedora-commons-users mailing list
> Fedora-commons-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
>
--
~/amsterdam
------------------------------------------------------------------------------
Nokia and AT&T present the 2010 Calling All Innovators-North America contest
Create new apps & games for the Nokia N8 for consumers in U.S. and Canada
$10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
http://p.sf.net/sfu/nokia-dev2dev
_______________________________________________
Fedora-commons-users mailing list
Fedora-commons-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
