I've also heard talk of people extracting the resource index from Fedora core, and running a standalone instance of Mulgara in its own JVM for the resource index. I recall some recent traffic on the lists on that topic. Not sure if that's feasible with current versions of Fedora, though.
-- Scott On 07/14/2011 09:08 AM, Benjamin Armintor wrote: > Ludovic- > Another option, if you're worried about the initial load, is to turn > off the resource index during the ingest. Then, when the ingest is > complete, enable the resource index and run the rebuilder. > > - Ben > > On 7/14/11, Ludovic Deravet<ludovicdera...@gmail.com> wrote: >> Hello Scott, >> >> Yes. We do use it with the following value: >> >> -XX:MaxPermSize=256m >> >> We came to the conclusion that since we create thousands of digital objects >> during consecutive hours (with of course creation of content datastreams and >> setting relationships between digital objects) this lead to the creation >> within the JVM of many objects by Mulgara (this is our understanding). >> >> And since there's no pause during the initial load phase, the JVM comes to >> an end where there's no memory left and has to execute too many small GC >> operation during a short period of time. This is why we get this >> error: *java.lang.OutOfMemoryError: >> GC overhead limit exceeded* >> >> >> After some research and tuning, the only option we see right now is to make >> some pauses during the initial load process to let the JVM takes times to >> execute normal GC operations. >> >> Cheers, >> >> *Ludovic* >> >> >> On Wed, Jul 13, 2011 at 11:07 PM, Scott Prater<pra...@wisc.edu> wrote: >> >>> Ludovic, >>> >>> The -Xmx option determines the size of the heap. Did you also set the >>> -XX:MaxPermSize option? The Sun JVM also uses this parameter to >>> allocate memory space, in addition to the heap. >>> >>> >>> http://www.sumoc.com/blog/index.cfm?mode=entry&entry=CDCDBF8B-5004-2066-B7460CDEAB79328F >>> >>> http://www.javaperformancetuning.com/news/qotm036.shtml >>> >>> -- Scott >>> >>> On 07/08/2011 10:08 AM, Ludovic Deravet wrote: >>>> Dear all, >>>> >>>> We are experiencing a serious issue with Fedora when ingesting thousands >>> of >>>> packages into the repository. Indeed, the development of our project is >>>> almost finished and our customer is trying to perform massive load of >>>> objects into Fedora before going into production. >>>> >>>> However an instability problem that manifest itself through *java heap >>> out >>>> of memory *errors appears when the customer wants to ingest about 10.000 >>> of >>>> METS envelopes. >>>> >>>> Internally, our application is transforming all METS envelopes into >>>> FOXML >>>> files that are then sent to the Fedora's REST's API. And documents are >>>> defined as content managed datastreams. >>>> >>>> To give some figures about these envelopes: >>>> >>>> - A METS envelope defines in average 80 digital objects >>>> - 10% of the envelopes reference documents where their total size is >>>> lower than 1MB >>>> - 65% of the envelopes reference documents where their total size is >>>> between 1MB and 5MB >>>> - 20% of the envelopes reference documents where their total size is >>>> between 5MB and 10MB >>>> - 5% of the envelopes reference documents where their total size is >>>> bigger than 10MB >>>> >>>> Fedora is installed on a production-like hardware: >>>> >>>> - Sun Oracle Enterprise M5000 - 2 clusters of 4 CPUs 2.53GHZ SPARC64 >>>> USVII (so 32 cores in total) >>>> - JVM is configured with 2GB of memory (-Xmx2048m) >>>> - JVM is in server mode on Solaris 10 (64bits) >>>> - Fedora-commons 3.4.2 with default configuration options activated >>>> >>>> In a production-like environment, our application is in fact launching >>>> 10 >>>> threads where each thread is processing one METS envelope at a time. We >>>> realize that this is kind of stressful for Fedora but the customer is >>>> expecting us to ingest all their METS envelopes in a reasonable short >>> period >>>> of time. >>>> >>>> We have performed many different tests in our development environment >>>> (of >>>> course quite less powerful than the production-like environment) and >>> cannot >>>> reproduce the instability problem we are experiencing in production. >>>> >>>> We have analyzed in our development environment how Fedora is managing >>> the >>>> memory and everything seems normal. The Garbage is working fine even if >>> we >>>> set the heap size to 255MB and try to ingest several MB of data. >>>> >>>> I really doubt that this problem lies within Fedora itself. Instead, I >>> think >>>> that the problem is more a configuration issue somewhere but right now >>>> we >>>> don't know which one exactly. Is there someone that could share their >>>> experience with such issue and help us try to find a solution to this >>>> problem? >>>> >>>> Thank you in advance for any help. >>>> >>>> Kind regards, >>>> >>>> >>>> >>>> >>> ------------------------------------------------------------------------------ >>>> All of the data generated in your IT infrastructure is seriously >>> valuable. >>>> Why? It contains a definitive record of application performance, >>>> security >>>> threats, fraudulent activity, and more. Splunk takes this data and makes >>>> sense of it. IT sense. And common sense. >>>> http://p.sf.net/sfu/splunk-d2d-c2 >>>> >>>> >>>> >>>> _______________________________________________ >>>> Fedora-commons-users mailing list >>>> Fedora-commons-users@lists.sourceforge.net >>>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users >>> >>> >>> -- >>> Scott Prater >>> Library, Instructional, and Research Applications (LIRA) >>> Division of Information Technology (DoIT) >>> University of Wisconsin - Madison >>> pra...@wisc.edu >>> >>> >>> ------------------------------------------------------------------------------ >>> AppSumo Presents a FREE Video for the SourceForge Community by Eric >>> Ries, the creator of the Lean Startup Methodology on "Lean Startup >>> Secrets Revealed." This video shows you how to validate your ideas, >>> optimize your ideas and identify your business strategy. >>> http://p.sf.net/sfu/appsumosfdev2dev >>> _______________________________________________ >>> Fedora-commons-users mailing list >>> Fedora-commons-users@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users >>> >> >> >> >> -- >> *Ludovic* >> > > ------------------------------------------------------------------------------ > AppSumo Presents a FREE Video for the SourceForge Community by Eric > Ries, the creator of the Lean Startup Methodology on "Lean Startup > Secrets Revealed." This video shows you how to validate your ideas, > optimize your ideas and identify your business strategy. > http://p.sf.net/sfu/appsumosfdev2dev > _______________________________________________ > Fedora-commons-users mailing list > Fedora-commons-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users -- Scott Prater Library, Instructional, and Research Applications (LIRA) Division of Information Technology (DoIT) University of Wisconsin - Madison pra...@wisc.edu ------------------------------------------------------------------------------ AppSumo Presents a FREE Video for the SourceForge Community by Eric Ries, the creator of the Lean Startup Methodology on "Lean Startup Secrets Revealed." This video shows you how to validate your ideas, optimize your ideas and identify your business strategy. http://p.sf.net/sfu/appsumosfdev2dev _______________________________________________ Fedora-commons-users mailing list Fedora-commons-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
