Rajith Attapattu wrote:
Jules,
Sorry for the late reply and thanks for all the info!!! It's awesome.
I have more questions if you don't mind. Not very organized so bear with me :(
no problem.
Answers are greatly appreciated.
JNDI implementation
--------------------
I guess what you are talking about is our own version of (at the least)JNDI
Context implementation that is cluster aware and the lookup is based on the
various strategies outlined in your email. (This is where I guess Apache
directory can be leveraged to build our impl on top)
I think so.
ActiveSpace may be useful here.
The distributed caching problem space is large and varied. wadi-core is
designed very much with a particular subset of this space in mind. This
is the area concerned with numbers of distributed objects that are too
great to be held on a single node and that are frequently written and
read and require pessimistic locking policies to be used in their
manipulation. JNDI has slightly different requirements. Objects placed
in JNDI are generally few, rarely written (only on [un]deployment) and
frequently read. Optimistic locking policies may be sufficient.
Ultimately, wadi-core and ActiveSpace are complimentary technologies,
both providing solutions to different areas of this problem space. WADI
also contains lots of integration code to various other containers, so I
am imagining a situation where a layer of WADI code integrates Geronimo,
Apache Directory and ActiveSpace, all sitting on top of ActiveCluster
and ActiveMQ.
Plus you are taked about passing in membership information to the client via a
proprietary protocol or the client taking on itself to obtain membership info
via configuration or an auto discovery handle.
Correct. We may well settle on ActiveCluster as the API for membership.
I am not sure whether we can reuse autodiscovery code from ActiveMQ
(which uses it to connect peers when running on its peer:// protocol
stack), whether there is other suitable Geronimo or ASF-licensed code
available, or whether we will need to write our own WADI-autodiscovery
classes. The important thing is to impose as few dependencies on the
client as possible. The client side code should literally be a few
lines. Clients using clusters should not suddenly find themselves
sucking down e.g. the whole of activemq, just to do a once off
autodiscovery. Early versions of WADI had its own autodiscovery code. If
we need them, they could be resuscitated.
Instead can we have a proxy (which sits between the stub and skeleton) which
will sit on our JNDI server and handle any membership issues. Sort of a proxy
to a proxy :)
Client --> stub --> proxy (which handles membership issues) --> skeleton --> EJB
This will shield the client from having to have any knowledge whatsoever about
the cluster membership.
Sounds fine. We can solve this problem by aggregation and delegation
(your model, we delegate all cluster code to an additional proxy and
aggregate an instance to our client-side stub) or by inheritance and
specialisation (we subclass the stub and add cluster awareness - uses
one less object). If by the Aggregation/Delegation model, we can somehow
delay the snapshotting of cluster membership to the point when the stub
is passed to the client, rather than the point when the stub is
registered with the directory, then this would give significant advantage.
Assumptions
-------------
So the basic assumption is that it's a homogenous cluster. Does it mean that
only the EJB's are present in every cluster or is it EJB's plus + other
resources are available???
Assumptions are dangerous things :-) - lets just say that our initial
focus should probably be on homogeneous deployments, whilst bearing in
mind that heterogeneous deployments are bound to eventually impose
further requirements upon us. I am hoping that we can implement
heterogeneity in terms of intersecting sets of nodes. So, we specifiy a
set of nodes that provide 'service-a', a set that provide 'service-b'
etc. Initially, we will just talk in terms of one universal set that
provides all services, for the sake of simplicity.
For JNDI, we shouldn't just be thinking in terms of just EJBs, but of
any distributed component that wishes to register with JNDI.
Does it mean that they all have access to the same data?? Like they could be
pointing to the same database or a cluster of databases.
Hmmm... - I've lost you here - how about this ?
It is entirely up to the EJBs as to what they may map to and not
something that JNDI should concern itself with.
If two containers both need to publicise EJBs that map to the same
resource, then I would expect them to both register them with the same
JNDI name - in this way they are 'unified' and become different
incarnations of the same 'service'. If they map to different resources,
then they should be registered with different names, otherwise two
clients who believe that they are manipulating the same shared resource
may in fact have their own private copies...
In a homogeous deployment, you can make the assumption (that word again)
that if a service is available on one node it is available on all nodes.
In a heterogeneous deployment, we will have to calculate the set of
nodes on which each service is available and somehow include this info
in the stub.
This all needs much more thought, but I think that we are going in the
right direction - does it make sense to you ?
SLSB
----
I agree that in general we have a use case for non-sticky clustering. Again as
you pointed out the SLSB is usually a façade for other more complex operations
that may use system resources heavily.
So in such a case are we going to collocate the resources and use a per-Bean or
per-User (or per-Server?) sticky behavior here? If so that means we have to
make sure that particular SLSB is sticky in terms of clustering.
Well, I think we need to provide a variety of pluggable policy objects
at this point, so that the person deploying the EJB can decide which
policy best suits its intended use. The type of stickiness used at the
front end will be linked to the data representation at the back end, so
that they coordinate properly.
So it boils down to the fact that we are going to define a clustering strategy
per bean (not bean type for ex SLSB or SFSB). Is this correct???
I think that this is the only way to do it, We can provide sensible
defaults based on EJB type, but there will always be counter-examples
that we have to deal with and it is likely that whatever policies we
come up with initially will not be a closed set. Someone is bound to
come along with another form of e.g. 'randomness', or 'stickiness' that
they want to plug in for their bean.
SFSB
-----
As u mentioned most SFSB are going to be sticky, but if we do cluster the data
associated with the session then we can use a non sticky algo. But u mentioned
this is sub optimal? So is this really expensive?? Is it really worth looking
at this option?
If the policy is pluggable, then we can avoid having to make tricky
decisions like this and delegate it to the person doing the deployment.
In a stateful situation, stickiness is usually preferable as
serialisation and deserialisation are cpu intensive operations. By
sending an invocation to the wrong node, you may cause the relevant
session to be serialised on one node, sent over the network to the
invocation's location and be deserialised here underneath it, ready for
use. This is going to be much more expensive than sending the invocation
to the correct place in the first place, or even redirecting or
forwarding a badly placed invocation (as WADI does).
By making the policy pluggable you also allow people to test exceptional
cases - e.g. a non-sticky SFSB. This is something that you probably do
not want in production, but may want to stress test so that you can be
confident that if affinity is lost (e.g. a node is shutdown and a
session emigrates to a new location, unbeknownst to a number of
clients...) you know that your cluster is going to stand up to the
exceptional stress and not descend into deadlock and chaos.
Again I guess it depends on the amount of information that should be replicated
which bring us back to the strategy of choosing a clustering type per Bean not
by bean type.
exactly. We are on the same page :-)
Entity
--------
I guess if the underlying data sources are clustered then we can use a
non-sticky algo if not we have to use sticky. Is that correct for all
scenarios??
By delegating the decision to the deployer, we don't even have to ask
this question... but, in 99% of cases this is probably the case,
although, once again, I think that a non-sticky policy here would
generally be sub-optimal as Entity performance is all about caching, and
a non-sticky algorithm, whilst possibly working fine, will give you a
much reduced cache-hit rate.
More Questions/Answers and discussions will help us to flesh this out more.
Good - I hope my answers help - please keep the questions coming - this
is a very useful thread.
Jules
Regards,
Rajith Attapattu.
-----Original Message-----
From: Jules Gosnell [mailto:[EMAIL PROTECTED]
Sent: Tuesday, December 06, 2005 1:51 PM
To: [email protected]
Cc: [EMAIL PROTECTED]
Subject: Re: [Geronimo] Clustering
Rajith,
I'll have a crack at these and then if Jeff wants to add anything I'm
sure he will :-)
please see inline...
Rajith Attapattu wrote:
Jeff,
I am currently involved with JavaMail implementations so haven't had
much time to look at the clustering side. Will do soon after the mail
thingy is done.
However can please comment on the points stated below so when you are
done with the 1.0 release and me with the JavaMail thing, we can
continue with the clustering conversation.
Thanks,
Rajith
-----Original Message-----
From: Rajith Attapattu [mailto:[EMAIL PROTECTED]
Sent: Wednesday, November 30, 2005 7:29 PM
To: [email protected]; [EMAIL PROTECTED]
Subject: RE: [Geronimo] Clustering
Jeff,
We maybe able to leverage the Apache Directory for the underlying JNDI
aspect of it (I will look in to this, but might need help)
How about the policy management portion of clustering service??
For ex
Clustering strategy
======================
Whether to use Sticky vs Random vs other load balancing mechanisms or
are we allowing the user to choose a strategy from above.
OK - lets get down to brass tacks.
Ultimately we will need a number of different policies including those
that you have outlined above.
Different policies will suit different types of bean and different use
cases and it would be good to put together some sort of list of common
combinations.
Here is a start...
This is all straight off the top of my head - so be gentle :-)
I think we probably can assume homogeneous JNDI population - i.e. all
JNDI services carry exactly the same information - otherwise a client
would have to visit all of them to be sure that a service it required
did not exist. This means that deploying a new JNDI entry is expensive
(since it has to update all JNDI service replicas), but the deployment
of new entries is an exceptional event, whereas client lookup is a
common event - so we keep the common event cheap and the exceptional one
pays the price.
So, somehow (probably initially via some sort of autodiscovery
mechanism) the client hooks up with a jndi service replica and asks it
for a EJBHome stub.
This stub is returned from JNDI to the Client.
I guess the JNDI service could be cluster-aware and package recent
information about cluster membership with the stub back to the client
(proprietary protocol required?), or it could be dumb, in which case the
newly demarshalled stub may have to obtain this information for itself -
by having had some information about the cluster's membership and
auto-discovery handle (e.g. multicast address) serialised with it
(possibly a long time before it is finally used - so the membership data
may be stale and need refreshing via autodiscovery).
I guess we will at least want to have the opportunity of colocating all
serverside resources associated with a single user - since this will
greatly improve our response to said user. This also creates a handy way
to begin partitioning the problem that we might end up with if we just
created every ejb on a random box then hooked them all up together. So,
there is a good usecase here for requiring some sort of 'stickiness'
immediately.
So, it looks like we need two types of Home stub - a sticky one, which
might look at a unique userid held within the client (can we do this?)
and use this to map this user to a particular server and a non-sticky
type, which might use a variety of pluggable algorithms (or subtypes)
including random (but also e.g. trying to find the least loaded server,
the nearest server etc...). (there is also another type of stickiness
where we use the non-sticky algorithm to choose an initial server, then
go back to the same one thenceforth - but this does not give us such
finegrained control over the colocation of resources belonging to the
same user - perhaps we call this per-server stickiness).
So - when I talk about sticky here - I am talking about per-user
stickiness (all invocations from the same user go to the same target)
not just per-bean stickiness (all invocations for the same bean go to
the same server).
Lets say the client uses the Home stub to create an EJB instance for
which it receives a client-side stub. Lets examine the different
possibilities.
SLSB -
Logically it does not matter (in a homogeneous deployment) where
invocations for a SLSB (I'm open to correction on all of this - my EJB
is very rusty) might land. A SLSB should be lightweight and be able to
be transparently brought into existance underneath the incoming
invocation, service it and then be trashed (of course containers pool
etc. to reduce overheads associated with this sort of thing). So, we
have a good usecase for non-sticky behaviour in the SLSB stub here.
In reality, it is likely that this SLSB will access further resources
associated with the client that may be more expensive to pop in and out
of existance (e.g. Entities), so there is a good case for a per-Bean or
per-User (or per-Server?) sticky behaviour here.
SFSB -
The reverse of the SLSB.
Logically it does matter where invocations fall. If the state held in
the SFSB is tied to a single JVM then we would expect a
per-User/Bean/Server sticky algorithm in this sort of stub.
In reality, hopefully, the associated SFSB will be WADI-enabled and able
to travel to wherever it is needed. So, a non-sticky algorithm would
also work, but would probably be rather sub-optimal :-)
Entity -
More like an SLSB, I guess. Logically an Entity should be able to spring
into existance anywhere it is needed. In reality there is cost
associated with this. Cache hits are good, Cache misses are bad - So,
again, all algorithms should work, but a sticky variant is probably the
most common deployment option.
On the server-side, these different policies should tie up with some
ideas that I have discussed on the list about the way that a users
remote resources may be colocated in some sort of SuperSession (maybe a
UserSession or an ApplicationSession (maybe a per-Application stickiness
is also required) (colocates resources from various applications and
tiers that are all associated with the same application-and-user or just
user). So choosing a particular ServSide impl would constrain the
possible client-side load-balancing algorithms available to the
application i.e. they walk hand in hand. Unless we want to be really
outrageous (Geronimo-4.0?), these choices should be made at deployment
time, by humans with knowledge of the layout of the application
components within the cluster and their expected behaviour.....
I think that is it for the moment !
Apologies for the rather rambling nature of my thoughts on this - I
haven't really thought about this stuff rigorously yet, but this should
be enough to share where I am with the list.
Jules
P.S.
We need to figure out how the various forms of stickiness are maintained
when a node dies or voluntarily leaves a cluster. There is lots to talk
about.
We can represent each clustering strategy as a GBean which the user can
pick from (under the Clustering services GBean, I assume you have the
whole clustering feature represented as a GBean ).
So if somebody is not happy about the clustering strategy then simply
write there own and add that as a GBean.
Of course we will have to come up with a neat API for exposing the
aspects that should be open for improvement.
This will also help us to come up with better clustering strategies
later on in the future without a major impact on the code base.
What are your thoughts on this??? Everybody please help with ideas :)
Regards,
Rajith Attapattu.
-----Original Message-----
From: Jeff Genender [mailto:[EMAIL PROTECTED]
Sent: Wednesday, November 30, 2005 1:13 PM
To: [email protected]
Subject: Re: [Geronimo] Clustering
Rajith Attapattu wrote:
Jeff,
Apologize for late reply, down with flu.
Is high availability JNDI (or JNDI clustering) a concept brought up by
JBoss??
I don't know the answer to this question.
Frankly I am no expert on this, so any pointers will be very helpful.
I
see that WADI is yet to implement this. So do u have any documentation
on this?
This is an area we are all starting to look at. One area I would
recommend looking at is seeing if we can leverage the Apache Directory
to handle the HA component of JNDI. If so, this may be a much simpler
job.
Thanks for helping out...this is great to have more folks chipping in on
this.
I assume we will follow the jboss concept closely, but hopefully to
improve on it.
Any help is greatly appreciated.
Rajith.
-----Original Message-----
From: Jeff Genender [mailto:[EMAIL PROTECTED]
Sent: Monday, November 28, 2005 1:00 PM
To: [email protected]
Subject: Re: [Geronimo] Clustering
Rajith Attapattu wrote:
As per Jeff's request I am currently ramping up on WADI. I guess jeff
will shortly announce the integration or any other intermediate tasks
that needs to be done before WADI can be integrated.
I guess we will have some discussion on what areas we will work on
when the plan is announced??
Jeff can you pls comment on this?
I think we just need a couple of Gbeans to get it initially integrated
in the web tier...I will tackle that. It currently works under Tomcat
and Jetty in their standalone configurations. Gianni is currently
working on the OpenEJB session integration with WADI...and we look
forward to getting that.
We are interested in the HA JNDI...so lets definitely get some
discussion going on that.
Jeff
With Kind Regards,
Rajith Attapattu.
________________________________________
From: Panda, Akshaya Kumar (Cognizant)
[mailto:[EMAIL PROTECTED]
Sent: Sunday, November 27, 2005 11:42 PM
To: [email protected]
Subject: RE: [Geronimo] Road Map and TODO
Jeff/Rajith,
Yes i will be intersted to work on this. But before jumping in on, i
would like to know who all are involved in what areas so that it
can
help me to avoid any duplicate effort.
Rajith: can you pl outline what you have planned to do?
thanks
akshay
-----Original Message-----
From: Jeff Genender [mailto:[EMAIL PROTECTED]
Sent: Sat 11/26/2005 9:17 PM
To: [email protected]
Subject: Re: [Geronimo] Road Map and TODO
Panda, Akshaya Kumar (Cognizant) wrote:
I think WADI is going to provide web tier clustering, any initiative
for HA-JNDI?
thanks
Akshay
No, WADI (and probably should be renamed remove the web-only
connotations) will be providing all HA facets to Geronimo. If you
are
interested in this area, this is where we could use a helping hand,
so
feel free to jump in on this.
Jeff
--
"Open Source is a self-assembling organism. You dangle a piece of
string into a super-saturated solution and a whole operating-system
crystallises out around it."
/**********************************
* Jules Gosnell
* Partner
* Core Developers Network (Europe)
*
* www.coredevelopers.net
*
* Open Source Training & Support.
**********************************/
