Hi Nicolas,
My name is Jason Varghese and I'm a senior developer at the New York Public
Library. I think you are doing work similar to what I am presently
doing based on reading some of your posts.
We have a relatively large scale Fedora implementation here. We've had all
the hardware in place for some time and are in the process of migrating
from a large homegrown repository to a Fedora based platform. We have a
single Fedora ingest machine and 3 Fedora readers. The ingest machine
alone is 4 x 6 core processors w/ 128GB RAM. I'm in the process
of generating about 1 million+ digital objects and attaching to each DO all
the metadata (as managed content datastreams) and all the digital assets
(as external content datastreams). The digital assets currently are about
183 TB of content (this is replicated at two sites). I have a
multithreaded java client I wrote to accomplish the task for the Fedora
ingest/DO generation and I use the Mediashelf REST API client for
connectivity to Fedora. I was able to successfully ingest 10's of thousands
of digital objects, but really need ensure this process performs optimally
and scales for millions of objects. What bottlenecks were you able to
identify when running your multithreaded ingest process? Look forward to
learning/sharing experiences from this process with you and the
community and possibly collaborating. Thanks
Jason Varghese
NYPL
On Wed, Sep 12, 2012 at 3:56 AM, Nicolas Hervé <nhe...@ina.fr> wrote:
> Hi,
>
> here is the code that reproduces the stress tests I've made.
>
> https://github.com/nrv/FedoraCommonsPerf
>
> You have to modify the connection parameters, the number of threads and
> choose wheter you want a single FedoraClient instance for all the threads
> or an instance per thread. Both configurations leads to the same
> conclusions.
>
> The test has two steps. First it purges all the fake documents, then it
> recreates them. Run it twice and take a look at the server CPU, network and
> I/O wait.
>
> I'm running Fedora Commons 3.5 on a 12 cores server.
>
> I've already increased the memory for the tomcat process (-Xms1g -Xmx4g).
>
> Thanks
>
> Nicolas
> ________________________________________
> De : Serhiy Polyakov [sp0...@gmail.com]
> Envoyé : mercredi 29 août 2012 20:16
> À : Support and info exchange list for Fedora users.
> Objet : Re: [fcrepo-user] Ingest performances
>
> May be the following can give some clues... I had export performance
> increased after increasing memory parameters in
> client/bin/env-client.sh
> Original was:
> exec_cmd="exec \"$java\" -Xms64m -Xmx96m \
>
> Serhiy Polyakov
> University of North Texas
>
>
> On Wed, Aug 22, 2012 at 4:56 AM, Nicolas Hervé <nhe...@ina.fr> wrote:
> > Hi,
> >
> > I'm running a 3.5 instance and use the Java fedora client library to
> access
> > it. I'm doing massive parallel operations to manipulate huge amount of
> > objects. I've writen a multi-threaded application that is able to ingest
> and
> > purge.
> >
> > When purging, I have all the CPU on my server that are running, that's
> fine.
> > But when ingesting, only 20% of them are used and I don't undestand where
> > the bottleneck is. I've modified the Tomcat connector to allow massive
> > parallel connections. I've done the same for MySQL. I've desactivated DC
> > fields indexing and Resource indexing. By the way, as it's working fine
> when
> > purging, I suspect a problem in the ingestion process rather than a
> "maximum
> > number of connections" configuration issue. Finally, I don't have I/O
> wait,
> > so the HDD shouldn't be the problem either.
> >
> > Any clue on what is happening ?
> >
> > Thanks
> >
> > Nicolas
> >
> >
> ------------------------------------------------------------------------------
> > Live Security Virtual Conference
> > Exclusive live event will cover all the ways today's security and
> > threat landscape has changed and how IT managers can respond. Discussions
> > will include endpoint security, mobile security and the latest in malware
> > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> > _______________________________________________
> > Fedora-commons-users mailing list
> > Fedora-commons-users@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
> >
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Fedora-commons-users mailing list
> Fedora-commons-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
>
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Fedora-commons-users mailing list
> Fedora-commons-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
>
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://ad.doubleclick.net/clk;258768047;13503038;j?
http://info.appdynamics.com/FreeJavaPerformanceDownload.html
_______________________________________________
Fedora-commons-users mailing list
Fedora-commons-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
