Dear all,
We are experiencing a serious issue with Fedora when ingesting thousands of
packages into the repository. Indeed, the development of our project is
almost finished and our customer is trying to perform massive load of
objects into Fedora before going into production.
However an instability problem that manifest itself through *java heap out
of memory *errors appears when the customer wants to ingest about 10.000 of
METS envelopes.
Internally, our application is transforming all METS envelopes into FOXML
files that are then sent to the Fedora's REST's API. And documents are
defined as content managed datastreams.
To give some figures about these envelopes:
- A METS envelope defines in average 80 digital objects
- 10% of the envelopes reference documents where their total size is
lower than 1MB
- 65% of the envelopes reference documents where their total size is
between 1MB and 5MB
- 20% of the envelopes reference documents where their total size is
between 5MB and 10MB
- 5% of the envelopes reference documents where their total size is
bigger than 10MB
Fedora is installed on a production-like hardware:
- Sun Oracle Enterprise M5000 - 2 clusters of 4 CPUs 2.53GHZ SPARC64
USVII (so 32 cores in total)
- JVM is configured with 2GB of memory (-Xmx2048m)
- JVM is in server mode on Solaris 10 (64bits)
- Fedora-commons 3.4.2 with default configuration options activated
In a production-like environment, our application is in fact launching 10
threads where each thread is processing one METS envelope at a time. We
realize that this is kind of stressful for Fedora but the customer is
expecting us to ingest all their METS envelopes in a reasonable short period
of time.
We have performed many different tests in our development environment (of
course quite less powerful than the production-like environment) and cannot
reproduce the instability problem we are experiencing in production.
We have analyzed in our development environment how Fedora is managing the
memory and everything seems normal. The Garbage is working fine even if we
set the heap size to 255MB and try to ingest several MB of data.
I really doubt that this problem lies within Fedora itself. Instead, I think
that the problem is more a configuration issue somewhere but right now we
don't know which one exactly. Is there someone that could share their
experience with such issue and help us try to find a solution to this
problem?
Thank you in advance for any help.
Kind regards,
--
*Ludovic*
------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
Fedora-commons-users mailing list
Fedora-commons-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
