I certainly agree with establishing a library of reusable components (a soapbox of mine too).
One note however, on the YourKit testing. Testing for speed of a single operation may or may not help overall throughput. We must be clear on our goals including usability. In the aggregate performance tests time was dominated by streaming the content and XML processing was a trivial part of the overall load. It seemed that the limits on thread handling set the overall system limits. Of course for bottleneck analysis you are constantly pushing the bottleneck somewhere else. So you can make a single process run faster and possibly make the system run slower for multiple users. You can fix ingest and bottleneck access, or vice versa. You can make the overall, multi-user operations of the system faster possibly making individual operations slower improving throughput. Small improvements in speed for a single operation may make the system harder to use. Its nice if you make all facets better but there is usually a trade-off. I am not making any conclusions about the findings so far but we need a wider scope to judge the trade=offs (if any). Dan Davis On 1/26/2012 8:36 AM, Stephen Bayliss wrote: > Adam - that would be great, thanks! > > Steve > >> -----Original Message----- >> From: aj...@virginia.edu [mailto:aj...@virginia.edu] >> Sent: 25 January 2012 15:47 >> To: fedora-commons-developers@lists.sourceforge.net >> Subject: Re: [fcrepo-dev] Ingest performance testing >> >> >> The committers are going to groan and roll their eyes when >> they hear this (it's a favorite theme of mine on committers' >> calls), but it might be a good step towards the kind of >> SOA-oriented modularization that would bear fruit in other >> efforts (e.g. in the short term: refactoring testing, in the >> long term: migration to OSGi). >> >> Steve-- I'd be happy to help on this. (Got to put my money >> where my mouth is!) >> >> --- >> A. Soroka >> Online Library Environment >> the University of Virginia Library >> >> >> >> >> On Jan 25, 2012, at 10:39 AM, Stephen Bayliss wrote: >> >>> +1 for me on Adam's suggestion (I do recall you suggesting this >>> +previously), >>> a re-usable component for this so that other parts of XML >>> transforming/processing/validation could be migrated over >> would be an >>> excellent implementation option. Maybe I could provide at least a >>> starting-point as part of this. >>> >>> Steve >>> >>>> -----Original Message----- >>>> From: aj...@virginia.edu [mailto:aj...@virginia.edu] >>>> Sent: 25 January 2012 15:32 >>>> To: fedora-commons-developers@lists.sourceforge.net >>>> Subject: Re: [fcrepo-dev] Ingest performance testing >>>> >>>> >>>> I've occasionally made the following suggestion before, >>>> although I recognize that it implies a great deal of work: It >>>> might be a good technique to establish a registry/pool of >>>> prepared transformations and validations in Fedora, in order >>>> to avoid as much as possible the expensive construction of >>>> such machinery in the course of an actual operation against >>>> the repository. Essentially this is Steve's suggestion below >>>> ("explore caching of the prepared stylesheets") writ very large. >>>> >>>> --- >>>> A. Soroka >>>> Online Library Environment >>>> the University of Virginia Library >>>> >>>> >>>> >>>> >>>> On Jan 25, 2012, at 10:29 AM, Stephen Bayliss wrote: >>>> >>>>> Following the call timing test that Chris did a little >>>> while ago [1, >>>>> 2] I have now had chance to have a quick play-around with >>>> YourKit myself. >>>>> Initial results from some not-very-extensive tests confirm the >>>>> findings that Chris made - I was seeing that the "prepare" >>>> of the stylesheet for the Schematron validation taking around >>>> 25% of DefaultManagement.ingest total time (so this is a >>>> little lower than Chris' results, but still significant). >>>>> So should we just go ahead and raise an issue for this? >>>>> >>>>> I was planning on taking a look at FCREPO-1026 and as part >>>> of this can >>>>> explore caching of the prepared stylesheets as it is >>>> related (and including a configuration option to disable >>>> Schematron validation entirely). I was also planning taking >>>> a look at FCREPO-1029, FCREPO-1030 and FCREPO-1031 (and >>>> possibly FCREPO-1027) as part of this piece of work as a lot >>>> of them are in the same code area. >>>>> Any thoughts welcomed! >>>>> >>>>> Steve >>>>> >>>>> [1] >>>>> >>>> https://wiki.duraspace.org/display/FCREPO/2011-11-01+-+Fedora+ >>> Committer+Meeting >>>> [2] >> https://wiki.duraspace.org/download/attachments/30216221/YourKitCallTi >>> mingEx >>> ample.png >>>> >> --------------------------------------------------------------------- >>>> - >>>> -------- >>>> Keep Your Developer Skills Current with LearnDevNow! >>>> The most comprehensive online learning library for >> Microsoft developers >>>> is just $99.99! Visual Studio, SharePoint, SQL - plus >> HTML5, CSS3, MVC3, >>>> Metro Style Apps, more. Free future releases when you >> subscribe now! >> http://p.sf.net/sfu/learndevnow-d2d___________________________________ >>> ______ >>> ______ >>>> Fedora-commons-developers mailing list >>>> Fedora-commons-developers@lists.sourceforge.net >>>> >> https://lists.sourceforge.net/lists/listinfo/fedora-commons-developer >>>> s >>> >>> >> ---------------------------------------------------------------------- >>> ------ >>> -- >>> Keep Your Developer Skills Current with LearnDevNow! >>> The most comprehensive online learning library for >> Microsoft developers is >>> just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, >> CSS3, MVC3, Metro >>> Style Apps, more. Free future releases when you subscribe now! >>> http://p.sf.net/sfu/learndevnow-d2d >>> _______________________________________________ >>> Fedora-commons-developers mailing list >>> Fedora-commons-developers@lists.sourceforge.net >>> >> https://lists.sourceforge.net/lists/listinfo/fedora-commons-developers >>> >>> >> ---------------------------------------------------------------------- >>> -------- >>> Keep Your Developer Skills Current with LearnDevNow! >>> The most comprehensive online learning library for >> Microsoft developers >>> is just $99.99! Visual Studio, SharePoint, SQL - plus >> HTML5, CSS3, MVC3, >>> Metro Style Apps, more. Free future releases when you subscribe now! >>> http://p.sf.net/sfu/learndevnow-d2d >>> _______________________________________________ >>> Fedora-commons-developers mailing list >>> Fedora-commons-developers@lists.sourceforge.net >>> >> https://lists.sourceforge.net/lists/listinfo/fedora-commons-developers >> >> >> -------------------------------------------------------------- >> ---------------- >> Keep Your Developer Skills Current with LearnDevNow! >> The most comprehensive online learning library for Microsoft >> developers is just $99.99! Visual Studio, SharePoint, SQL - >> plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future >> releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d >> _______________________________________________ >> Fedora-commons-developers mailing list >> Fedora-commons-developers@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/fedora-commons-developers >> > > ------------------------------------------------------------------------------ > Keep Your Developer Skills Current with LearnDevNow! > The most comprehensive online learning library for Microsoft developers > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, > Metro Style Apps, more. Free future releases when you subscribe now! > http://p.sf.net/sfu/learndevnow-d2d > _______________________________________________ > Fedora-commons-developers mailing list > Fedora-commons-developers@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/fedora-commons-developers ------------------------------------------------------------------------------ Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d _______________________________________________ Fedora-commons-developers mailing list Fedora-commons-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/fedora-commons-developers
