David Blevins wrote:
Hey All,
We have a few people who've murmured the word clustering in the last
few weeks, so I thought it would be good to put people in touch so
murmurs can turn into discussion and code.
Jeremy Whitlock: In his words, "working on the first crude integration
of JCACHE into OpenEJB"
Jules Gosnell: Mr. WADI
Gianny Damour: Mr. CMP
Matt Hogstrom: Mr. Know-It-All :)
Matt works for IBM, so make sure and tease him.
I don't know that everyone is subscribed to the list, so make sure to
reply all.
-David
I've just subscribed to all the OpenEJB lists.
WADI (http://wadi.codehaus.org) started as a distributable (i.e.
clustered) HttpSession impl, but I am hoping that we will be able to
reuse the code under openEJB to cluster Stateful Session Beans.
I mailed an update to geronimo-dev recently - I'll cut-n-paste from it
for the benefit of those who haven't seen it:
---------------------------------------------------------------------
Guys,
I thought that it was high time that I brought you up to date with my
efforts in building a clustering layer for Geronimo.
The project, wadi.codehaus.org, started as an effort to build a
scalable clustered HttpSession implementation, but in doing this, I
have built components that should be useful in clustering the state
held in any tier of Geronimo e.g. OpenEJB SFSBs etc.
WADI (Web Application Distribution Infrastructure) has two main
elements - the vertical and the horizontal.
Vertically, WADI comprises a stack of pluggable stores. Each store has
a pluggable Evicter responsible for demoting aging Sessions
downwards. Requests arriving at the container are fed into the top of
the stack and progress downwards, until their corresponding Session is
found and promoted to the top, where the request is correctly rendered
in its presence.
Typically the top-level store is in Memory. Aging Sessions are demoted
downwards onto exclusively owned LocalDisc. The bottom-most store is a
database shared between all nodes in the Cluster. The first node
joining the Cluster promotes all Sessions from the database into
exclusively-owned store - e.g. LocalDisc. The last node to leave the
Cluster demotes all Sessions down back into the database.
Horizontally, all nodes in a WADI Cluster are connected (p2p) via a
Clustered Store component within this stack. This typically sits at
the boundary between exclusive and shared Stores. As requests fall
through the stack, looking for their corresponding Session they arrive
at the Clustered store, where, if the Session is present anywhere in
the Cluster, its location may be learnt. At this point, the Session
may be migrated in, underneath the incoming request, or, if its
current location is considered advantageous, the request may be
proxied or redirected to its remote location. As a node leaves the
Cluster, all its Sessions are evacuated to other nodes via this store,
so that they may continue to be actively maintained.
The space in which Session ids are allocated is divided into a fixed
number of Buckets. This number should be large enough such that
management of the Buckets may be divided between all nodes in the
Cluster roughly evenly. As nodes leave and join the Cluster, a single
node, the Coordinator, is responsible for re-Bucketing the Cluster -
i.e. reorganising who manages which Buckets and ensuring the safe
transfer of the minimum number of Buckets to implement the new
layout. The Coordinator is elected via a Pluggable policy. If the
Coordinator leaves or fails, a new one is elected. If a node leaves or
joins, buckets emigrate from it or immigrate into it, under the
control of the Coordinator, to/from the rest of the Cluster.
A Session may be efficiently mapped to a Bucket by simply %-ing its
ID's hashcode() by the number of Buckets in the Cluster.
A Bucket is a map of SessionID:Location, kept up to date with the
Location of every Session in the Cluster, of which the id falls into
its space. i.e. as Sessions are created, destroyed or moved around the
Cluster notifications are sent to the node managing the relevant
Bucket, informing it of the change.
In this way, if a node receives a request for a Session which it does
not own locally, it may pass a message to it, in a maximum of
typically two hops, by sending the message to the Bucket owner, who
then does a local lookup of the Sessions actual location and forwards
the message to it. If Session and Bucket can be colocated, this can
reduced to a single hop.
Thus, WADI provides a fixed and scalable substrate over the more fluid
arrangement that Cluster membership comprises, on top of which further
Clustered services may be built.
The above functionality exists in WADI CVS and I am currently working
on hardening it to the point that I would consider it production
strength. I will then consider the addition of some form of state
replication, so that, even with the catastrophic failure of a member
node, no state is lost from the Cluster.
I plan to begin integrating WADI with Geronimo as soon as a certified
1.0-based release starts to settle down. Certification is the most
immediate goal and clustering is not really part of the spec, so I
think it best to satisfy the first before starting on subsequent
goals.
Further down the road we need to consider the unification of session
id spaces used by e.g. the web and ejb tiers and introduction of an
ApplicationSession abstraction - an object which encapsulates all
e.g. web and ejb sessions associated with a particular client talking
to a particular application. This will allow WADI to maintain the
colocation of associated state, whilst moving and replicating it
around the Cluster.
If anyone would like to know more about WADI, please feel free to ask
me questions here on geronimo-dev or on wadi-dev.
Thanks for listening,
Jules
-----------------------------------------------------------
WADI is currently in pieces on my floor - i.e. far from enterprise
ready, but coming along nicely.
I guess we need to decide whether OpenEJB is just interested in the
'horizontal' element (i.e. Clustering and Replication (NYI)), or whether
you are alos interested in the vertical aspect - stackable stores into
which sessions can be paged out according to various pluggable
algorithms etc...
JCache is an API that WADI will support, although I have found that this
API is inadequate for managing distributed sessions, in that it only
permits the user to pull state into a container at which an invocation
is arriving. It is often more efficient, for various reasons, to
relocate the invocation to the container that actually holds the state,
rather than relocating the state to the container that has the
invocation (and then 'restick' the client to this new container
instance). Thus, WADI's API passes the 'invocation' down the vertical
stack of 'Contextualisers' until one 'contextualises' the request
locally or remotely (via an http proxy or request redirection). I
discussed request/invocation relocation with David at J1 and he seemed
to think that it would be no problem to include a redirection protocol
between client and server into OpenEJB, rather than figuring out how to
proxy an RMI, but you could do both (as WADI does for HTTP) - your call....
On the wider subject of EJB clustering - you guys will need some form of
autodiscovery mechanism - so that a client can discover a clusters EJB
tier without having to be routed to a specific node within that cluster.
The client will need to be kept up to date with cluster membership
(probably via extra data returned with each RMI). This will enable the
client to failover if the node that it is talking to disappears. If all
the nodes that it knows about disappear it can fall back on the
autodiscovery, in case nodes about which it knows nothing have joined
the cluster etc... - WADI uses activecluster/activemq for all its
interconnectivity and this has autodiscovery. Whether it would be
compact enough to put into a client-side EJB proxy, I am not sure....
I'm no JNDI guru, but we will need to decide what we are going to do
about clustering JNDI as well - so your clients can get refs/proxies etc...
Thats probably enough to be getting on with.
I don't know much about OpenEJB, so please bear with me if this doesn't
make much sense to you.
Let me know how much of WADI you would like to reuse and we can take the
conversation from there.
Jules
