David,

> My immediate problem 
> is:http://groups.google.com/group/liftweb/browse_thread/thread/b3783e24b...
>
> This has been a persistent problem with Scala Actors and I identified it
> last year in November or December.

This is indeed supposed to be fixed in Scala 2.7.7 (to be released
later today). It replaces the old (as opposed to the new one for JDK7)
FJ framework with java.util.concurrent.ThreadPoolExecutor.

> Philipp did the 2.7.4 release which did not address the issue.  The 2.7.5
> release was supposed to address the issue, but the use of Lift Actors masked
> the issue until the above issue was raised.  I left my 2.7.5 related
> discussions with Philipp with the impression that the java.util.concurrent
> library was being used for thread pooling rather than the FJ library.  On
> this, I backed out the Lift Actor changes from powering Lift's CometActors
> (Lift Actors power the long polling part of Lift).

So, I guess there was a mis-understanding. 2.7.5 tried to fix several
issues related to memory leakage; however, it did not replace the use
of the FJ library. This was, however, done in trunk a couple of weeks
before 2.7.5 was released. At the time of JavaOne (beginning of June),
the plan was still to release (a beta of) Scala 2.8 in July.

> >  . The ForkJoinPool in 2.8 is completely different from FJTask in
> >   2.7.5; it's the version that's going into JDK7. It has been
> >   battle-tested and should not suffer from any memory leaks.
>
> > The reason why Scala actors use the FJ framework is performance, in
> > particular on multi-core hardware. So we do not think it's a good idea
> > to go back to java.util.concurrent, except maybe for applications with
> > very specialized demands.
>
> Do you have specific benchmarks that justify the tradeoff of baking in an
> external library (that could be as buggy as the one that's currently baked
> into Scala) versus java.util.concurrent?

We did some experiments that show that, indeed, a work-stealing thread
pool is better than a java.util.concurrent.Executor with a central
entry queue when running on an 8-core machine. You find a plot in this
tech report:
http://lamp.epfl.ch/~phaller/doc/haller07actorsunify.pdf

I am talking about figure 7 on page 16. It shows the performance of a
heat diffusion simulation implemented using actors. In the "global"
version, actors use a thread pool with a global task queue (like
java.util.concurrent.Executor). The "local" version uses FJ for task
execution (each pool thread has its own local queue). One can see that
when using more than 4 hardware threads, the performance of "global"
decreases because the global task queue is contended. This is just one
concrete experiment that I did. There is of course a lot more work out
there that shows experimental results on the performance of work-
stealing (for instance, Cilk, data-parallel Haskell, etc).

So, I firmly believe that the ForkJoinPool of JDK 7 is the way to go.
Doug Lea told me in person that the new version has virtually no line
of code in common with the old FJTask library; it has been completely
redesigned and rewritten. However, by default it compiles and runs
only on JDK 1.6. It is possible to backport it to JDK 1.5; however, if
your 1.5 JVM is not up-to-date, you can encounter unexpected
InterruptedExceptions being thrown. That's the main reason why I am
not in favor of using it on JDK 1.5.

> > The
> > result is that people have not noticed the changes. For example, most
> > of the issues that Erik raises in his blog post no longer apply to
> > Scala 2.8. Initially we wanted 2.8 to be out by now, but it's taken
> > much longer than we have foreseen, because some of the problems were
> > harder than initially thought. We are sorry to have left the 2.7
> > branch relatively unattended for so long. It's difficult for us,
> > though, to provide the resources to support two diverging branches in
> > parallel. More community support with backports etc could help.
>
> While this is a nice thought, given the choice between debating with EPFL as
> to whether my changes are good enough or doing a Lift Actor implementation,
> I will opt for continuing to develop and maintain the Lift Actor library.
>  This gives Lift users the assurance that bugs will be fixed in a timely
> manner, that we can add features based on community need, and that we can
> work with other library authors to insure common interfaces.

I'd be happier if more people would debate with us. :-)

> But there is a difference between a research vehicle and something that can
> be used in production systems.  What Philipp has done is delivered a
> prototype and a vision of concurrency to the Scala community.  What Jonas
> and I have done is built production-quality systems based on that vision,
> our experience, and other systems we have used or observed.

I second Martin's view. You and Jonas are doing really great work.
But, I also think that it is important to have a core set of
concurrency abstractions in the standard library that can be used in
production systems. There are several people in the Scala community
that could help achieve that.

> I would much prefer to free Philipp to do his pi calculus work (or whatever
> else inspires him) than maintain a library for me.  I want to see the next
> great thing he comes up with rather than making him my bug-fixing slave.

I'd certainly like to spend more time trying to do that. (I'm really
excited about my current project; but that's for another mail.)

Thanks,
Philipp

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups 
"Lift" group.
To post to this group, send email to liftweb@googlegroups.com
To unsubscribe from this group, send email to 
liftweb+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/liftweb?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply via email to