Hello Scott,
Yes. We do use it with the following value:
-XX:MaxPermSize=256m
We came to the conclusion that since we create thousands of digital objects
during consecutive hours (with of course creation of content datastreams and
setting relationships between digital objects) this lead to the creation
within the JVM of many objects by Mulgara (this is our understanding).
And since there's no pause during the initial load phase, the JVM comes to
an end where there's no memory left and has to execute too many small GC
operation during a short period of time. This is why we get this
error: *java.lang.OutOfMemoryError:
GC overhead limit exceeded*
After some research and tuning, the only option we see right now is to make
some pauses during the initial load process to let the JVM takes times to
execute normal GC operations.
Cheers,
*Ludovic*
On Wed, Jul 13, 2011 at 11:07 PM, Scott Prater <pra...@wisc.edu> wrote:
> Ludovic,
>
> The -Xmx option determines the size of the heap. Did you also set the
> -XX:MaxPermSize option? The Sun JVM also uses this parameter to
> allocate memory space, in addition to the heap.
>
>
> http://www.sumoc.com/blog/index.cfm?mode=entry&entry=CDCDBF8B-5004-2066-B7460CDEAB79328F
>
> http://www.javaperformancetuning.com/news/qotm036.shtml
>
> -- Scott
>
> On 07/08/2011 10:08 AM, Ludovic Deravet wrote:
> > Dear all,
> >
> > We are experiencing a serious issue with Fedora when ingesting thousands
> of
> > packages into the repository. Indeed, the development of our project is
> > almost finished and our customer is trying to perform massive load of
> > objects into Fedora before going into production.
> >
> > However an instability problem that manifest itself through *java heap
> out
> > of memory *errors appears when the customer wants to ingest about 10.000
> of
> > METS envelopes.
> >
> > Internally, our application is transforming all METS envelopes into FOXML
> > files that are then sent to the Fedora's REST's API. And documents are
> > defined as content managed datastreams.
> >
> > To give some figures about these envelopes:
> >
> > - A METS envelope defines in average 80 digital objects
> > - 10% of the envelopes reference documents where their total size is
> > lower than 1MB
> > - 65% of the envelopes reference documents where their total size is
> > between 1MB and 5MB
> > - 20% of the envelopes reference documents where their total size is
> > between 5MB and 10MB
> > - 5% of the envelopes reference documents where their total size is
> > bigger than 10MB
> >
> > Fedora is installed on a production-like hardware:
> >
> > - Sun Oracle Enterprise M5000 - 2 clusters of 4 CPUs 2.53GHZ SPARC64
> > USVII (so 32 cores in total)
> > - JVM is configured with 2GB of memory (-Xmx2048m)
> > - JVM is in server mode on Solaris 10 (64bits)
> > - Fedora-commons 3.4.2 with default configuration options activated
> >
> > In a production-like environment, our application is in fact launching 10
> > threads where each thread is processing one METS envelope at a time. We
> > realize that this is kind of stressful for Fedora but the customer is
> > expecting us to ingest all their METS envelopes in a reasonable short
> period
> > of time.
> >
> > We have performed many different tests in our development environment (of
> > course quite less powerful than the production-like environment) and
> cannot
> > reproduce the instability problem we are experiencing in production.
> >
> > We have analyzed in our development environment how Fedora is managing
> the
> > memory and everything seems normal. The Garbage is working fine even if
> we
> > set the heap size to 255MB and try to ingest several MB of data.
> >
> > I really doubt that this problem lies within Fedora itself. Instead, I
> think
> > that the problem is more a configuration issue somewhere but right now we
> > don't know which one exactly. Is there someone that could share their
> > experience with such issue and help us try to find a solution to this
> > problem?
> >
> > Thank you in advance for any help.
> >
> > Kind regards,
> >
> >
> >
> >
> ------------------------------------------------------------------------------
> > All of the data generated in your IT infrastructure is seriously
> valuable.
> > Why? It contains a definitive record of application performance, security
> > threats, fraudulent activity, and more. Splunk takes this data and makes
> > sense of it. IT sense. And common sense.
> > http://p.sf.net/sfu/splunk-d2d-c2
> >
> >
> >
> > _______________________________________________
> > Fedora-commons-users mailing list
> > Fedora-commons-users@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
>
>
> --
> Scott Prater
> Library, Instructional, and Research Applications (LIRA)
> Division of Information Technology (DoIT)
> University of Wisconsin - Madison
> pra...@wisc.edu
>
>
> ------------------------------------------------------------------------------
> AppSumo Presents a FREE Video for the SourceForge Community by Eric
> Ries, the creator of the Lean Startup Methodology on "Lean Startup
> Secrets Revealed." This video shows you how to validate your ideas,
> optimize your ideas and identify your business strategy.
> http://p.sf.net/sfu/appsumosfdev2dev
> _______________________________________________
> Fedora-commons-users mailing list
> Fedora-commons-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
>
--
*Ludovic*
------------------------------------------------------------------------------
AppSumo Presents a FREE Video for the SourceForge Community by Eric
Ries, the creator of the Lean Startup Methodology on "Lean Startup
Secrets Revealed." This video shows you how to validate your ideas,
optimize your ideas and identify your business strategy.
http://p.sf.net/sfu/appsumosfdev2dev
_______________________________________________
Fedora-commons-users mailing list
Fedora-commons-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
