Hello,
I once developed an interest in STM and surfed a bit through the
Internet to find more information about it. I discovered that JBoss
Infinispan is - according to the JBoss Infinispan people themselves -
very STM-like. It has MVCC (already JBoss Cache had MVCC). Infinispan
seems to be an option as a central store, through which actors can do
lock-free communication. Infinispan also supports asynchronous event
notification like JavaSpaces. Large parts of JBoss Cache had been
rewritten, because they also had to learn their lessons. Somewhere on
TheServerside.com I once found a remark by some developer saying that
they are seriously considering Infinispan since the license costs for
Coherence are killing them. There was also some comment where somebody
questioned the accurateness MVCC was implemented in Akka. I should
provide a link to that reference I know. But if you search the Internet
for "MVCC Akka Actors Infinispan Coherence, etc." you should find some
interesting comments.
So my suggestion would be to go with something like River or Infinispan
and add some actor framework to it or develop your own little
queue-based thread pool-aware mini actor solution to it. I like very
much the idea of actors and STM used in conjunction. With STM a commit
is guaranteed to be deadlock free. Eventually, you can get an optimistic
locking exception. There is some overhead here, because you then have to
refetch your data from your central store and then restart your unit of
work. For someone who knows what a pain deadlocks can be in a true
production environment the cost is little compared to the gain.
I think actors on their own are not that useful. Shared-nothing is
really hard to achieve sometimes unless things can be well isolated. In
that case you don't need actors as you need nothing more than some
map-reduce framework. But actors combined with STM makes really sense to
me. Through the STM space actors are able to coordinate themselves in a
lock-free way. Let's say executing a job in the actor queue creates an
exception. Then how do you react propperly? First thing would be to ask
all other actors to stop execution. How else can you unwind things
propperly if everyone else continues to do it's job. Once your actors
understands what went wrong and wants to set itself up again the
surrounding environment has already been modified in the meanwhile by
the other actors. Then where do you restart? It then gets really
difficult to continue correctly after some exception or some logical error.
What you never hear from any of the actor proponents is that
asynchronous communication is not easy. Messages don't need to arrive in
the order they were sent. The receiving actor needs to make sure some
meaningful order is established. Asynchronous programming is not new and
hence people have already discovered hierarchical state machines can be
used well for this job. But I never saw any comment on any homepage from
some actor framework project group dropping a note about this. And how
to you proceed in case of an exception? This is also never mentioned? I
think in case of an exception the STM is really useful. Not changing
some flag after an exception tells the actors that something went wrong
somewhere else and that they have to return to some safe point, re-start
their unit of work or continue on their own or whatever.
Actually, JavaSpaces is well-suited for actor-based distributed
programming in my opinion. Only the actor libraries are a bit lacking.
Most actor frameworks are hobby projects or thesis works where some work
remains to be done till ready for production use. Pity is that River or
Blitz are not clustered and GigaSpaces is only free if there is only one
server in the cluster (same meanwhile for Terracotta). I fear for
JavaSpaces to do well in the future a clustered free implementation is
needed. Otherwise people simply go with something like Infinispan or
EHCache. Some backing company like JBoss in addition would be nice as well.
I thought about combining Apache Hadoop with JavaSpaces lately (only the
Hadoop distributed files system without the map-reduce stuff). Hadoop is
clustered (the name server is a single point of failure, I know ... but
it is nonetheless clustered and scales well). I know the idea is a bit
crazy since Hadoop is based on a read-many/write-once approach which
might be fine for data analsysis, but is nonetheless very contrary to
the idea in JavaSpaces. But I don't want to give up that quickly on the
idea. Maybe write-once is a more practical interpretation of shared
nothing? Yah, now it gets really crazy... Time to finish off and enjoy
the rest of my Sunday :-).
Cheers, Oliver
On 22.10.2011 09:27, Dan Creswell wrote:
Hey Patrick,
On 21 October 2011 21:44, Patrick Logan<[email protected]> wrote:
Hi all,
I was involved in a jini/javaspaces project five years ago, with good
success. Recently I've been trying Scala including the Akka "actors"
library.
Akka has promise but has fewer capabilities than Jini and Javaspaces,
and is certainly less mature.
I am wondering if anyone on the River list has been using Akka with or
without Scala?
Yes and I know some other people doing similar. Quite a lot of
interest in the banks....
Might folks see River as a more mature, capable system worthy of a
revival now that Scala and Akka are attracting a lot of interest in a
related distributed programming model?
I'm afraid I think it's unlikely to help bridge the gap for several reasons:
(1) The programming models are substantially different -
single-threaded, message passing with queues and no clear
differentiation of remote ( I'll say a bit more about that in a
minute.) from local vs services and any threading model you like.
(2) I'll no doubt get into trouble for saying this: The Akka folks are
making all the same mistakes made by others previously. In particular
they've attempted uniform I/O models for different grades of
consistency, they consider various bits and pieces as optimisations
rather than critical requirements for stability/maintainability and
they're attempting to retrofit high availability and replication to a
non-distributed memory model (something that Terracotta have attempted
for example).
(3) I'll likely get into trouble for saying this too: It's not just
the Akka folks, the community of users has made the same old mistake
of "do distributed looking like local". Akka is nothing new (it's just
single-threaded executors and queues for the most part) but many think
it is. The same assumptions are being made with the same "discoveries"
to follow.
Fundamentally, people are buying in to the same old "there is remote
that looks like local" snake oil. River has never been in that camp,
it could move there but would be a completely different beast. Whilst
it remains a different beast and people still want the snake oil,
adoption will be limited (I think for various other reasons that will
always be the case, another story there).
Back to programming models as promised.....
Doug Lea has some interesting stuff to say on transactional memory,
actors and such here: http://days2011.scala-lang.org/node/138/274 -
I'd summarise as "horses for courses and no silver bullet". Watch it
though, it's a great talk.
And remoteness:
The number of services in a large system is likely very much smaller
than the number of actors one would see in a similar system. That will
require a re-think of monitoring and debugging. No one seems to be
worried about that even though most existing messaging systems have
considerably fewer moving parts than one will find in an actor based
system of comparable size (there are very few large-scale message
based systems other than the likes of TCP, UDP and IP networks to take
learnings from).
Whilst most hold that introducing asynchronous operation is sufficient
to paper over the cracks of local vs remote, it's not true. One must
also account for lost messages which means introducing timeouts. Note
that messages aren't lost in the local model but will be across
networks requiring resends. That introduces highly variable latency
where on networks with no problems, things are fine as they are on the
local box but not for the failing network. The actor model has no
accounting for such latency, nor does it hint it's an issue. Resending
of messages implies build up of queues and resource exhaustion which
needs tackling via the likes of flow control or limited queue sizes
and dropped messages. Again, the actor model lacks in this area.
Similar resource exhaustion problems exist when actors don't process
queues fast enough (perhaps because they are updating persistent
storage which is substantially slower than network I/O and memory
operations), again limited discussion.
To summarise, there's a lot of noise about actors and plenty of
interest. Unfortunately, IMHO there's nothing more to it than adopting
a discipline of single-threaded executors with queues and dealing with
the well-known consequences. Sure, Akka has "remote queues" but they
don't work right/well and without a substantial mindset change that
will break the original promise of "easy remoteness" won't ever work.
Has anyone tried Scala with River (or related services such as Bllitz)?
I have, I know that Paul Snively was looking at it way back (Scala on
Blitz) and in fact there is Fly (which isn't Jini/JavaSpaces but
similar) which has Scala bindings and is being used by a few people.
Wouldn't say there's widespread usage tho'
-Patrick
--
www.objectscape.org
