This is getting off-topic.... but... I think the issue is that there's a single connection or a single thread accessing MongoDB. I'd suggest trying a connection pool to MongoDB or to otherwise see why the MongoDB stuff is not getting parallelized.
Also, Scala Actors do a lot of fancy stuff in terms of scheduling... you might want to try the same benchmarks with Lift Actors and see if there's a difference. On Thu, Nov 12, 2009 at 4:48 PM, Lincoln <[email protected]> wrote: > Hi, thank you very much for your help so far I really appreciate it. > Apologies in advance if this thread has become inappropriate since it's not > directly related to Lift. Just let me know if that's the case and I will > attempt to solve this problem another way. But, I'm hoping you can help me > some more... > > Warning: Long email ahead... > > Here's the summary of my load testing results: > > Each request in my load test consists of a single save() to mongodb, > followed by a findOne() for the record just written and then an arbitrary > find() for 1000 docs. > > I've been experimenting with 3 scenarios: > > 1. A single actor processing all requests (this effectively makes > things single-threaded) > 2. A pool of 50 actors handling requests that are dispatched to via a > round robin scheduler > 3. Create a new actor on demand for every request. > > The load I've been using for all tests is 2000 requests at a concurrency > level of 75. All the results I'm reporting are averages over 3 executions. > > As a benchmark, I ran all these scenarios with the worker stubbed out to > simply sleep for 100ms and then return. Here are the results when I ran the > benchmark: > > - Single shared worker: 7534.439 ms > - Shared Pool of 50 workers: 156.190 ms > - On-demand actor for each request: 127.066 ms > > These results are about what I expected. In the single worker case the > queue quickly backed up and stayed at 74 requests, thus the results were > about the concurrency level (75) multiplied by the work length (100 ms). > The shared pool was obviously a huge improvement, but the actor on demand > was even better because the concurrency level (75) was greater than the > number of actors in the shared pool (50) and thus there was still some > queueing in that case. > > I expected when I ran the tests with the mongodb work plugged in that I > would see results that were roughly proportional to the benchmark. However, > here's what I got: > > - Single shared worker: 1968.113 ms > - Shared Pool of 50 workers: 1955.218 ms > - On-demand actor for each request: 1951.352 ms > > For the vast majority of these requests, most of the time for the request > is spent waiting for the actor to receive the work job. This is the case > regardless of whether I'm using a single actor, a pool, or creating actors > on demand. In most cases, the mongo work takes 0 - 100 ms but with a > meaningful minority in the 100 - 300ms range. I think this means the JVM > must be busy with something else while that waiting is happening. > > I guess it makes sense that the single-threaded case goes faster since most > of the time the mongodb operations take far less than 100 ms. However, I > don't understand why I'm not seeing proportional improvement in the pooled > and on-demand scenarios. > > Here is the method I'm using to create my actors, whether it be 1, the pool > of 50, or a new one on demand every time: > > def worker(n: Int) = actor { > loop { > react { > case msg @ LoadReq(timer, stats, cont) => > timer.split("react") > stats("mailboxSize") = mailboxSize.toLong > val coll = staticColl > val dbo = ( > _msg -> "blah blah blah" << > _event -> "blah blah blah" << > _user -> "blah blah blah" << > _name -> "Lincoln" > ) > coll.save(dbo) > timer.split("save") > coll.findOne(_name -> "Lincoln") > timer.split("findOne") > val res = coll.find().limit(1000).toArray > timer.split("findAll") > > // timer.split("sleep") > cont.setAttribute(Base.CONTINUATION_RESP, LoadResp(timer, stats)) > cont.resume > case other => trace("worker discarded: "+other) > } > } > } > > I took a kill -3 while my load test was running and I saw a lot of these: > > "qtp1491907201-104" prio=5 tid=0x0000000102278800 nid=0x153199000 runnable > [0x0000000153198000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x0000000108bca370> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963) > at > org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:319) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:450) > at java.lang.Thread.run(Thread.java:637) > > I also used jstat throughout: > > Timestamp S0 S1 E O P YGC YGCT FGC > FGCT GCT > 161.8 100.00 0.00 0.00 68.15 60.18 102 9.215 2 > 0.022 9.237 > 166.8 100.00 0.00 10.04 28.97 60.32 134 12.083 4 > 0.049 12.132 > 171.8 100.00 0.00 60.79 68.23 60.06 172 14.904 4 > 0.049 14.952 > 176.8 0.00 100.00 100.00 4.04 59.87 191 17.437 7 > 1.362 18.799 > 181.8 100.00 0.00 2.12 4.56 59.88 196 18.363 8 > 1.712 20.075 > 186.8 100.00 0.00 4.62 4.56 59.88 196 18.363 8 > 1.712 20.075 > 191.8 100.00 0.00 4.65 4.56 59.88 196 18.363 8 > 1.712 20.075 > > Am I reading this correctly that it's basically doing a ridiculous amount > of garbage collection? If so, I would imagine that's what all my threads > might be waiting on. > > Do you have any suggestions for what I should try next? > > Thanks, > Lincoln > > > > On Sun, Nov 8, 2009 at 11:32 PM, Lincoln <[email protected]> wrote: > >> Thanks for the feedback David... I have also created a pool of actors >> and dispatch requests round-robin style to it. Oddly enough, I see >> absolutely no difference in the performance characteristics - in both >> cases 95% of the time is spent between the suspend and the actor >> receiving the request, but the mailbox never grows past a size of 1 or >> 2. I will try and send along a simplified version of my code to >> illustrate what I'm doing. >> >> Thanks, >> Lincoln >> >> On Sun, Nov 8, 2009 at 11:02 PM, David Pollak >> <[email protected]> wrote: >> > If you have all your requests going to 1 actor, then you've reduced your >> > application to a single threaded app. >> > >> > If you're using Scala Actors, there are all kinds of wacky things that >> > happen with the Fork-Join library that the Actors use for scheduling and >> a >> > whole lot of knobs to turn to get the scheduling right depending on your >> > number of CPUs, etc. >> > >> > But, at the end of the day, if you're just pushing work from one thread >> (the >> > thread that the HTTP request comes in on) to another thread (the thread >> > that's waiting for the RDBMS request), you're not saving any time or >> > threads, in fact you're only increasing the amount of thrash between >> > scheduling the Actor, suspending the HTTP request thread, etc. >> > >> > But, without seeing your code, it's hard to give you a better analysis. >> > >> > On Sun, Nov 8, 2009 at 7:08 PM, Lincoln <[email protected]> wrote: >> >> >> >> Hi, >> >> >> >> Currently I have a jetty webapp that uses continuations and actors to >> >> scale to many connections. While this isn't a lift question per se I >> >> was hoping the folks on this list could help since it's my >> >> understanding that Lift does similar stuff under certain setups. >> >> >> >> Basically, the functionality that I'm testing receives a request, >> >> suspends it via jetty continuations, and fires it off to an actor for >> >> processing. Once the work is done (some benchmark database requests, >> >> usually on the order of 10 to 100ms) the continuation is resumed and >> >> results are returned. >> >> >> >> As I scale up to 1000+ concurrents in apache benchmark, I notice that >> >> the vast majority of time from receipt of the request to response, is >> >> between when the continuation is suspended and when my actor receives >> >> the message. This strikes me as bizarre since I'm monitoring the >> >> mailbox and it never grows to a size larger than 1 or 2. It gets to >> >> the point where requests are taking 10 seconds to come back and 9.5 of >> >> those seconds are my actor waiting to receive the work request. >> >> >> >> Throughout the load test the database does not become stressed. >> >> >> >> Any thoughts on this? I guess I'm hoping there is something basic >> >> that I'm doing wrong. Just in case, I've tried creating an actor pool >> >> to receive the work requests but that apparently has no effect on the >> >> results. >> >> >> >> My only speculation at this point is that perhaps the reason messages >> >> are taking so long to get to my actor is because jetty itself is >> >> overloaded trying to queue requests but that is just a guess. >> >> >> >> Thanks, >> >> Lincoln >> >> >> >> >> > >> > >> > >> > -- >> > Lift, the simply functional web framework http://liftweb.net >> > Beginning Scala http://www.apress.com/book/view/1430219890 >> > Follow me: http://twitter.com/dpp >> > Surf the harmonics >> > >> > > >> > >> > > > > > -- Lift, the simply functional web framework http://liftweb.net Beginning Scala http://www.apress.com/book/view/1430219890 Follow me: http://twitter.com/dpp Surf the harmonics --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Lift" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/liftweb?hl=en -~----------~----~----~----~------~----~------~--~---
