I wanted to let you guys know that once I upgraded to FC 3.6.2, all these issues were resolved. I think there was a bug in 3.4.x where ingested files did not get garbage collected properly.
After upgrading to FC3.6.2, I was able to ingest 4GB files easily into Islandora 6.x. I have not tried files > 4GB though. Also, Islandora 6.x uses SOAP APIs and Islandora 7.x uses REST APIs for accessing Fedora server. I think for large files, REST API is better and might support ingestion of even larger files. Thanks. Diwakar Timilsina Systems Analyst, UITS University of Arkansas, Fayetteville Email: diwa...@uark.edu Phone: 479.575.6160 -----Original Message----- From: Nick Ruest [mailto:rue...@gmail.com] Sent: Monday, March 25, 2013 7:01 PM To: fedora-commons-users@lists.sourceforge.net Subject: Re: [fcrepo-user] Performance issues when ingesting large files I don't see anything too crazy with `lsof -a -p`. I could use another set of eyes on this[1] in case I completely missed something :-) -nruest [1] https://gist.github.com/ruebot/4ba99bb186d2699a50fa On 13-03-25 04:51 PM, Adam Wead wrote: > Can you dig around your process list and figure out which file is > involved with those two processes that are taking up 99% CPU? I might > guess that Fedora is computing some checksums on two large (?) files, > which would explain the two procs with high cpu usage, and that's > bottlenecking everything else up, hence all the other threads that are > taking up little to no cpu. > ____________________________________________ > Adam Wead > Systems and Digital Collections Librarian Rock and Roll Hall of Fame > and Museum > 216.515.1960 (t) > 215.515.1964 (f) > > > On Mon, Mar 25, 2013 at 2:38 PM, Nick Ruest <rue...@gmail.com > <mailto:rue...@gmail.com>> wrote: > > I'm definitely experience very similar issues. > > Today I replaced a few datastreams on some video objects, and it has > basically taken fcrepo down. I can't access it via Tomcat (web). Both > /describe and /admin time out for me. But, htop is showing everything is > running with *lots* of resource usage. See these[1] screenshots of htop. > > I've come across this before, and usually I just wait it out. Should > that be the course of action here? Or, is something up? > > Any input would be great. > > cheers! > > -nruest > > [1] http://imgur.com/a/q2VgF > > > On 13-02-05 09:08 AM, Diwakar Timilsina wrote: > > Hi, > > > > I am trying to use the Islandora Video Solution Pack to > > upload video files that are 1.6 GB each. I am having an issue where > > once I ingest a video file, Islandora becomes very slow and stays > like > > that. Prior to ingesting any video file, Islandora loads pretty fast. > > The Fedora Commons JVM heap space is almost all used up and the > garbage > > collection occurs very frequently, almost every 3-4 seconds. The heap > > usage never goes down after that. It's causing Islandora/Fedora > Commons > > to perform poorly. When I try to ingest additional videos, the > ingestion > > fails with the following message: > > > > Error Trying to call SOAP function "ingest". Exception: Error > Fetching > > http headers in > > > > /usr/local/islandora/drupal/modules/islandora/api/fedora_item.inc(1226)\n#0 > > [internal function]: SoapClient->__doRequest('<?xml version="...', > > 'http://doratst....', 'http://www.fedo...', 1, 0)..... > > > > I have tried the following JVM Options without any luck. The > server has > > 8 GB of RAM and 4 CPU. We are using Fedora Commons 3.5. > > > > -Xms5g -Xmx5g -XX:MaxPermSize=256m -XX:NewRatio=1 > -XX:+DisableExplicitGC > > -XX:SurvivorRatio=10 -XX:TargetSurvivorRatio=90 > > -XX:MaxTenuringThreshold=30 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC > > > > -Xms5g -Xmx5g -XX:MaxPermSize=256m -XX:+DisableExplicitGC > > > > -Xms5g -Xmx5g -XX:MaxPermSize=256m > > > > I appreciate any feedback. > > > > Thanks. > > > > *Diwakar Timilsina* > > > > Systems Analyst, UITS > > > > University of Arkansas, Fayetteville > > > > Email: diwa...@uark.edu <mailto:diwa...@uark.edu> > <mailto:diwa...@uark.edu <mailto:diwa...@uark.edu>> > > > > Phone: 479.575.6160 <tel:479.575.6160> > > > > > > > > > > ------------------------------------------------------------------------------ > > Free Next-Gen Firewall Hardware Offer > > Buy your Sophos next-gen firewall before the end March 2013 > > and get the hardware for free! Learn more. > > http://p.sf.net/sfu/sophos-d2d-feb > > > > > > > > _______________________________________________ > > Fedora-commons-users mailing list > > Fedora-commons-users@lists.sourceforge.net > <mailto:Fedora-commons-users@lists.sourceforge.net> > > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users > > > > > ------------------------------------------------------------------------------ > Everyone hates slow websites. So do we. > Make your web apps faster with AppDynamics > Download AppDynamics Lite for free today: > http://p.sf.net/sfu/appdyn_d2d_mar > _______________________________________________ > Fedora-commons-users mailing list > Fedora-commons-users@lists.sourceforge.net > <mailto:Fedora-commons-users@lists.sourceforge.net> > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users > > > > > ---------------------------------------------------------------------- > -------- Everyone hates slow websites. So do we. > Make your web apps faster with AppDynamics Download AppDynamics Lite > for free today: > http://p.sf.net/sfu/appdyn_d2d_mar > > > > _______________________________________________ > Fedora-commons-users mailing list > Fedora-commons-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users > -- -nruest ------------------------------------------------------------------------------ Own the Future-Intel® Level Up Game Demo Contest 2013 Rise to greatness in Intel's independent game demo contest. Compete for recognition, cash, and the chance to get your game on Steam. $5K grand prize plus 10 genre and skill prizes. Submit your demo by 6/6/13. http://p.sf.net/sfu/intel_levelupd2d _______________________________________________ Fedora-commons-users mailing list Fedora-commons-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/fedora-commons-users ------------------------------------------------------------------------------ Own the Future-Intel® Level Up Game Demo Contest 2013 Rise to greatness in Intel's independent game demo contest. Compete for recognition, cash, and the chance to get your game on Steam. $5K grand prize plus 10 genre and skill prizes. Submit your demo by 6/6/13. http://p.sf.net/sfu/intel_levelupd2d _______________________________________________ Fedora-commons-users mailing list Fedora-commons-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
