I think I know what this is. I had some additional changes exactly in this area that I simply forgot to apply this morning. I should have it fixed sometime today.
Cheers, David On 14 May 2015 at 14:03, David Bosschaert <david.bosscha...@gmail.com> wrote: > Hi Pierre, > > I'll take a look today. > > Cheers, > > David > > On 14 May 2015 at 14:00, Pierre De Rop <pierre.de...@gmail.com> wrote: >> I just committed the benchmark tool in >> http://svn.apache.org/viewvc/felix/sandbox/pderop/loadtest/, if you can >> take a look. >> >> To run the scenario: >> >> - install jdk8: >> >> [nxuser@nx0012 pderop]$ java -version >> java version "1.8.0_40" >> Java(TM) SE Runtime Environment (build 1.8.0_40-b26) >> Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode) >> >> - checkout the loadtest from >> http://svn.apache.org/viewvc/felix/sandbox/pderop/loadtest/ >> >> - go the the "loadtest" directory and start the test, just like this: >> >> $ java -server -jar bin/felix.jar >> Welcome to Apache Felix Gogo >> >> g! Starting benchmarks (each tested bundle will add/remove 630 components >> during bundle activation). >> >> [Starting benchmarks with no processing done in components start >> methods] >> >> Benchmarking bundle: >> org.apache.felix.dependencymanager.benchmark.dependencymanager >> .................................................. >> -> results in nanos: [139,129,744 | 143,957,687 | 152,157,581 | 319,631,722 >> | 919,838,078] >> >> Benchmarking bundle: >> org.apache.felix.dependencymanager.benchmark.dependencymanager.parallel . >> >> >> Here, the first >> "org.apache.felix.dependencymanager.benchmark.dependencymanager" test >> (single-threaded) passes OK. But the next one hangs >> (org.apache.felix.dependencymanager.benchmark.dependencymanager.parallel). >> it uses a fork join pool with size=4. >> >> and when typing "log warn", we see: >> >> "log warn" >> >> 2015.05.14 13:56:10 ERROR - Bundle: >> org.apache.felix.dependencymanager.benchmark.dependencymanager.parallel - >> [ForkJoinPool-1-worker-3] Error processing tasks - >> java.util.ConcurrentModificationException >> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429) >> at java.util.HashMap$KeyIterator.next(HashMap.java:1453) >> at java.util.AbstractCollection.addAll(AbstractCollection.java:343) >> at >> org.apache.felix.framework.capabilityset.CapabilitySet.match(CapabilitySet.java:245) >> at >> org.apache.felix.framework.capabilityset.CapabilitySet.match(CapabilitySet.java:212) >> at >> org.apache.felix.framework.capabilityset.CapabilitySet.match(CapabilitySet.java:189) >> at >> org.apache.felix.framework.ServiceRegistry.getServiceReferences(ServiceRegistry.java:269) >> at >> org.apache.felix.framework.Felix.getServiceReferences(Felix.java:3577) >> at >> org.apache.felix.framework.Felix.getAllowedServiceReferences(Felix.java:3655) >> at >> org.apache.felix.framework.BundleContextImpl.getServiceReferences(BundleContextImpl.java:434) >> at >> org.apache.felix.dm.tracker.ServiceTracker.getInitialReferences(ServiceTracker.java:422) >> at >> org.apache.felix.dm.tracker.ServiceTracker.open(ServiceTracker.java:375) >> at >> org.apache.felix.dm.tracker.ServiceTracker.open(ServiceTracker.java:319) >> at >> org.apache.felix.dm.tracker.ServiceTracker.open(ServiceTracker.java:295) >> at >> org.apache.felix.dm.impl.ServiceDependencyImpl.start(ServiceDependencyImpl.java:226) >> at >> org.apache.felix.dm.impl.ComponentImpl.startDependencies(ComponentImpl.java:657) >> at >> org.apache.felix.dm.impl.ComponentImpl.performTransition(ComponentImpl.java:535) >> at >> org.apache.felix.dm.impl.ComponentImpl.handleChange(ComponentImpl.java:492) >> at >> org.apache.felix.dm.impl.ComponentImpl.access$5(ComponentImpl.java:482) >> at >> org.apache.felix.dm.impl.ComponentImpl$3.run(ComponentImpl.java:227) >> at >> org.apache.felix.dm.impl.DispatchExecutor.runTask(DispatchExecutor.java:182) >> at >> org.apache.felix.dm.impl.DispatchExecutor.run(DispatchExecutor.java:165) >> at >> java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402) >> at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) >> at >> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) >> at >> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1689) >> at >> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) >> >> >> (I will investigate also in my code to check if the problem does not come >> from me ?) >> >> cheers; >> /Pierre >> >> >> On Thu, May 14, 2015 at 1:47 PM, Pierre De Rop <pierre.de...@gmail.com> >> wrote: >> >>> Hi David, >>> >>> I don't know if it's me (a bug in my benchmark tool) or if if there is a >>> regression somewhere in the framework, by my parallel test does not pass >>> anymore. >>> >>> The test first starts with a single-threaded scenario, which passes OK >>> (org.apache.felix.dependencymanager.benchmark.dependencymanager), then when >>> the parallel test starts >>> (org.apache.felix.dependencymanager.benchmark.dependencymanager.parallel) >>> it suddenly hangs, and when I type "log warn" under the gogo shell, I see >>> the following exception: >>> >>> (I'm using java8): >>> >>> $ java -server -Xmx4g -Xms4g -jar bin/felix.jar >>> ____________________________ >>> Welcome to Apache Felix Gogo >>> >>> Benchmarking bundle: >>> org.apache.felix.dependencymanager.benchmark.dependencymanager.parallel . >>> >>> (here, the dependencymanager.parallel test hangs and when I type "log >>> warn", I see this:) >>> >>> g! log warn >>> 2015.05.14 13:31:03 ERROR - Bundle: >>> org.apache.felix.dependencymanager.benchmark.dependencymanager.parallel - >>> [ForkJoinPool-1-worker-3] Error processing tasks - >>> java.util.ConcurrentModificationException >>> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429) >>> at java.util.HashMap$KeyIterator.next(HashMap.java:1453) >>> at java.util.AbstractCollection.addAll(AbstractCollection.java:343) >>> at >>> org.apache.felix.framework.capabilityset.CapabilitySet.match(CapabilitySet.java:245) >>> at >>> org.apache.felix.framework.capabilityset.CapabilitySet.match(CapabilitySet.java:212) >>> at >>> org.apache.felix.framework.capabilityset.CapabilitySet.match(CapabilitySet.java:189) >>> at >>> org.apache.felix.framework.ServiceRegistry.getServiceReferences(ServiceRegistry.java:269) >>> at >>> org.apache.felix.framework.Felix.getServiceReferences(Felix.java:3577) >>> at >>> org.apache.felix.framework.Felix.getAllowedServiceReferences(Felix.java:3655) >>> at >>> org.apache.felix.framework.BundleContextImpl.getServiceReferences(BundleContextImpl.java:434) >>> at >>> org.apache.felix.dm.tracker.ServiceTracker.getInitialReferences(ServiceTracker.java:422) >>> at >>> org.apache.felix.dm.tracker.ServiceTracker.open(ServiceTracker.java:375) >>> at >>> org.apache.felix.dm.tracker.ServiceTracker.open(ServiceTracker.java:319) >>> at >>> org.apache.felix.dm.tracker.ServiceTracker.open(ServiceTracker.java:295) >>> at >>> org.apache.felix.dm.impl.ServiceDependencyImpl.start(ServiceDependencyImpl.java:226) >>> at >>> org.apache.felix.dm.impl.ComponentImpl.startDependencies(ComponentImpl.java:657) >>> at >>> org.apache.felix.dm.impl.ComponentImpl.performTransition(ComponentImpl.java:535) >>> at >>> org.apache.felix.dm.impl.ComponentImpl.handleChange(ComponentImpl.java:492) >>> at >>> org.apache.felix.dm.impl.ComponentImpl.access$5(ComponentImpl.java:482) >>> at >>> org.apache.felix.dm.impl.ComponentImpl$3.run(ComponentImpl.java:227) >>> at >>> org.apache.felix.dm.impl.DispatchExecutor.runTask(DispatchExecutor.java:182) >>> at >>> org.apache.felix.dm.impl.DispatchExecutor.run(DispatchExecutor.java:165) >>> at >>> java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402) >>> at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) >>> at >>> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) >>> at >>> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1689) >>> at >>> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) >>> >>> (If I configure my threadpool to 1, I have no problems, but with >>> threadpool=4, then I have the problem) >>> >>> I will investigate, but Ideally, may be it would be helpful if you could >>> also run the test by yourself; so I will commit soon something to reproduce >>> the problem in my sandbox. >>> >>> cheers; >>> /Pierre >>> >>> On Thu, May 14, 2015 at 11:11 AM, David Bosschaert < >>> david.bosscha...@gmail.com> wrote: >>> >>>> I've committed this now in >>>> http://svn.apache.org/viewvc?view=revision&revision=1679327 >>>> >>>> Curious to see what others are measuring. My tests were focused on >>>> multiple bundles/threads obtaining the same service, as that's were I >>>> saw a bit of contention. >>>> >>>> Cheers, >>>> >>>> David >>>> >>>> On 13 May 2015 at 15:10, Pierre De Rop <pierre.de...@gmail.com> wrote: >>>> > Hi David, >>>> > >>>> > I'm looking forward to test your improvements using the >>>> dependencymanager >>>> > benchmark tool ([1]). >>>> > >>>> > >>>> > [1] >>>> > >>>> http://svn.apache.org/viewvc/felix/trunk/dependencymanager/org.apache.felix.dependencymanager.benchmark/ >>>> > >>>> > /Pierre >>>> > >>>> > On Wed, May 13, 2015 at 3:02 PM, David Bosschaert < >>>> > david.bosscha...@gmail.com> wrote: >>>> > >>>> >> I have implemented the performance improvements that I was thinking of >>>> >> using Java 5 concurrency tools, they can be viewed at [1]. >>>> >> >>>> >> I wrote a little performance test suite [2] that tests multithreaded >>>> >> service registry performance (10 threads) from single / multiple >>>> >> bundles with either singleton services and Prototype Service Factory >>>> >> services and the results are quite impressive. I'm getting performance >>>> >> improvements compared to the current trunk from 8 times better than >>>> >> the original (800%) to more than 30 times better (3000%). >>>> >> >>>> >> Carsten has already reviewed the code (thanks Carsten!) and I'm >>>> >> planning to commit it to Felix tomorrow if nobody objects. >>>> >> >>>> >> Cheers, >>>> >> >>>> >> David >>>> >> >>>> >> [1] >>>> >> >>>> https://github.com/bosschaert/felix/commit/e6a1b06c6e66d9c98e6d81b91ef7003c8e725450 >>>> >> [2] >>>> >> >>>> https://github.com/bosschaert/coderthoughts/tree/master/service-registry-perftest/srperf >>>> >> >>>> >> On 23 March 2015 at 15:39, Richard S. Hall <he...@ungoverned.org> >>>> wrote: >>>> >> > On 3/23/15 10:17 , David Bosschaert wrote: >>>> >> >> >>>> >> >> On 23 March 2015 at 13:39, Richard S. Hall <he...@ungoverned.org> >>>> >> wrote: >>>> >> >>> >>>> >> >>> On 3/23/15 03:55 , Guillaume Nodet wrote: >>>> >> >>>> >>>> >> >>>> There's a call to interrupt() in Felix#acquireBundleLock(), not >>>> sure >>>> >> if >>>> >> >>>> it >>>> >> >>>> can be the culprit though. >>>> >> >>>> Interrupts could also be caused by a bundle being shutdown while >>>> one >>>> >> of >>>> >> >>>> its >>>> >> >>>> thread is waiting for a service, which should is a valid use case >>>> >> imho. >>>> >> >>>> Anyway, I think sanely reacting to a thread being interrupted >>>> would be >>>> >> >>>> good. >>>> >> >>> >>>> >> >>> >>>> >> >>> Yes, threads can be interrupted if they are holding a bundle lock >>>> and >>>> >> the >>>> >> >>> global lock holder needs the bundle lock. >>>> >> >>> >>>> >> >>> I admit that I do not recall why we ignore the interrupt here, but >>>> >> didn't >>>> >> >>> we >>>> >> >>> implement service lookup so that a bundle lock wasn't necessary? I >>>> >> >>> thought >>>> >> >>> we just checked for the validity of the bundle context before >>>> returning >>>> >> >>> or >>>> >> >>> something. Perhaps we felt there was no reason to be interrupted in >>>> >> that >>>> >> >>> case. I really don't know. >>>> >> >> >>>> >> >> I think that the Service Registry could be rewritten to be >>>> completely >>>> >> >> free of synchronized blocks using the Java 5 concurrency libraries, >>>> >> > >>>> >> > >>>> >> > Well, that just moves the sync blocks to the library, but yeah sure. >>>> >> > >>>> >> >> which I think would really be a better approach. There is too much >>>> >> >> locking going on in the current SR implementation IMHO. >>>> >> > >>>> >> > >>>> >> > I don't really think there is too much, but it is complicated. >>>> >> > Unfortunately, it is complicated to make sure that locks aren't held >>>> >> while >>>> >> > do service lookups and this is complicated because you can run into >>>> >> cycles, >>>> >> > etc. >>>> >> > >>>> >> > But feel free to try to simplify it. >>>> >> > >>>> >> >> >>>> >> >> This brings the question: can we move to Java 5 (or Java 6) for the >>>> >> >> Framework codebase? AFAIK we're currently still JDK 1.4 compatible >>>> but >>>> >> >> I would be surprised if there is anyone who still needs a JDK that >>>> >> >> went end-of-life 7 years ago. >>>> >> > >>>> >> > >>>> >> > At this point, it doesn't really matter to me. >>>> >> > >>>> >> > -> richard >>>> >> > >>>> >> >> >>>> >> >> Best regards, >>>> >> >> >>>> >> >> David >>>> >> > >>>> >> > >>>> >> >>>> >>> >>>
