Hello,
Here is a private thread that began between James Cook and myself about the design of
the currently implemented HA features for SLSB. As we both think this thread may be of
interest, I forward it to the JBoss-Dev ML. You'll find in attachment previous e-mails.
Don't hesitate to share your ideas!
Cheers,
Sacha
-----Message d'origine-----
De : James Cook [mailto:[EMAIL PROTECTED]]
Envoy� : samedi, 21 avril 2001 17:01
� : Sacha Labourey
Objet : RE: jBoss HA
> There's one thing I do not understand:
> 1) either people use "our" NS implementation
> 2) or people use "their own" NS implementation (sexy LDAP
> implementation)
To achieve the LB features, jBoss will have to bind to our NS. Our NS can
have the option of being persisted to any number of backends, preferably via
the JNDI interface. This would include LDAP.
This backend persistence is also what buys you the failover without
resorting to the network synchronization tactics. The same way that no one
synchronizes EJBs across VMs...they rely on the database. It always come
down to a single point of failure/synchronization.
> As for the replicated naming service, I agree that it is not
> mandatory but I do not think we can propose an HA implementation
> with such a miss. Furthermore, it would improve efficiency: all
> JBoss server would resolve through local lookups (and thus
> "eliminate" network latency). And as the NS is the EJB entry
> point, if you loose it, all your HA features go to trash unless
> you restart it (or repair the computer, or...) Because client
> nodes would only know the name of ONE naming server, it means
> that moving this name server is not really trivial (it could yes,
> but it means other complication I think)
I agree that if a HA NS is created the stubs that are retrieved must be
smart enough to know how to contact the next NS if the first one dies. I
would like to hear some details about how this is accomplished.
> > I'm not against a HA name service...I just think a load-balancing name
> > service would be a higher priority.
>
> Yes, but the HA name service will provide LB. First because each
> client can connect to a different NS and second because when the
> client downloads the remote+home proxy, the load-balancing
> features is already in.
>
> Am I missing something?
I don't think you are missing anything :-) I was referring to LB among the
jBoss servers and not the NS, that's all. The NS simply doles out a proxy to
the client. After that, the client will have no need to contact this NS
unless the proxy loses connection with the jBoss server.
> Yes, but you could imagine the client to connect to this name
> service and asks for a list of other available NS. We could also
> imagine to have a list of hosts instead of a single host in the
> property. This would change from the "standard" but could be
> easily done. In any cases, there is some kind of specific action
> to take to bind a JNDI provider to a client (such as setting the
> good class provider for example.)
Actually I don't think this approach will work. The client must be able to
discover the available NS on demand to achieve the level of HA that you
want. A list of NS at the time the proxy is generated is not sufficient. I
don't know how this can be done without some UDP broadcast, but we talked
earlier about its limitations.
> To be frank, I do not see (appart from location consideration)
> the real difference/gain between having a smart NS allowing
> multiple objects per name and a simple NS where you bind smart
> stubs. In both cases, you will need to have smart stubs => your
> container, when creating the stubs, will need to obtain the
> knowledge about other nodes and fill this smart stub with
> appropriate data.
I'll summarize how I would like to see HA be implemented. First of all, a
service is added to the container to replace the current Local NS (LNS).
This service has knowledge where the Global NS (GNS) is located. When the
LNS is started/stopped it notifies the GNS. As objects are bound/unbound
from the LNS, the GNS is notified and mirrors the actions.
When the jBoss server requires an object from the LNS, it always checks its
own list first. If the object is found it returns it, and if not, it
delegates to the GNS.
The GNS is backed by some permanent JNDI service. This can be any of the
SPIs supported by JNSI such as file store (the default), or database or
LDAP, etc. This way, if the GNS goes down and is restarted, all of the
information is current. At least after some sort of synchronization takes
place. The GNS will have to contact the LNS for a bulk update perhaps. Not
actually transmitting new proxies, but just checking to see if they are
alive.
When the GNS receives a request to bind an object, it stores it. More than
one object may be bound under the same name. (Note that the LNS does not
need this feature and the existing jBoss NS could be used with some small
mods.) When a request for an object is received, the GNS consults its LB
algorithm to decide which stub goes. This algorithm will default to a
round-robin, however more sophisticated techniques are possible. The LNS
could be equipped with their own pluggable module that reports system usage
or number of beans used, or other metrics. The GNS could expose these
methods and facilitate the call to the LNS on behalf of the LB plugin.
The client would have knowledge of the GNS. At this point, I only see how a
single GNS is possible, although there is nothing in the above design that
prevents a HA implementation. It would simply consist of additional GNS and
a smarter client proxy.
I just debate the benefit versus the complexity. There are many single
points of failure in any system. I believe the simplicity of a single GNS,
and one that can be auto-started will be an immediate win. I sincerely look
forward to seeing a HA implementation as well, but I would like to see it as
an enhancement to this base design. That is....if the base design makes
sense! ;-)
> Please understand that this was also my first idea but that now,
> I can't figure what would be the real advantage of this solution!
> Also understand that I am not trying to blindly argue just to
> save my design. I would be very happy to go in a better
> direction, and this current discussion is very helpful but what
> would help is a kind of list of pros/cons between both solutions
> because I am not sure that the smart NS way is smarter/easier/...
I think the summary should provide us with an opportunity to weigh these
pros and cons. More discussion!
> This is true for every imaginable solution: when everthing is
> dead, we need to restart them automagically.
I agree, but to emphasize a previous point, isn't Remote Object Activation
be a single point of failure? :-)
> > Actually, no commercial EJB implementation (AFAIK) supports serialized
> > access to entity beans across VMs. They all enforce serialization at the
> > database.
>
> But is this always possible? what about BMP? what about LDAP
> based persistance for example?
They do it. IMHO (and the opinion of these other vendors, weblogic, inprise,
oracle), there is no widespread technique for synchronizing across VMs that
outweighs the performance penalty of implementing it. Gemstone comes the
closest with their PCA, but they still don't synchronize in the EJB
container. They have just written a special database that locks across VMs.
> Please, provide me with "use-cases" where you would have a real
> benefit. The smart NS will need to deal with exactly the same
> issues that the HAConfigService solves today with JavaGroups but
> instead of having this logic in the NS directly, it is a distinct
> service launched with JBoss.
I understand that both implementations are similar conceptually, and you are
right to question the differences. I am not that familiar with the
HAConfigService so I can't speak to exact differences in the approaches. You
have basically created a cluster of LNS, whereas I am advocating a central
GNS. Sounds like the traditional master server versus peer-to-peer. I guess
I will lose from a marketing standpoint. P2P is too hot! :-)
I suppose that I advocate the GNS approach because of the simplicity it
affords the clients. I would also guess that it has a performance edge,
although I don't know enough about your implementation.
> BTW, what is your timezone? mine is CET (i.e. GMT+1 in winter and
> GMT+2 in summer). If I know this, I can decide when it is a good
> idea to answer quickly i.e. when you are probably on-line and not
> deeply sleepling!
I am GMT-5. I guess that puts me 7 hours behind you?
> Don't you think that this discussion could be moved (now or
> later) to the JBoss-dev ML?
I think that this is a valuable thread. Sure.
jim
> This was exactly my first idea if you look at the JBoss-Dev ML
> archive about
> clustering.
>
> First of all, I have to say that one of the next goal will probably be to
> develop a HA Naming Service (replicated, ...)
I agree that this would be nice, but probably HA on the name server will be
unnecessary for 90% of the installations. It could take too long for it to
be worthwhile. Maybe I'm wrong about this. There is always a single point of
failure in most systems and that is an acceptable risk. For example, most
large commercial applications will want their enterprise nameserver to be an
LDAP implementation, and often there is only one. Kind of like the database.
I'm not against a HA name service...I just think a load-balancing name
service would be a higher priority.
> Whatever solution is choosen, the client needs some code in its
> proxy and a
> set of reference. Otherwise, if the NS only gives him a single
> reference, if
> it fails, the client will have to explicitly resolve a new
> reference to get
> a new object. This need to be as much as possible transparent (i.e.
> transparent fail-over).
Agreed. The client should not have to do anything to get failover support.
In fact, Inprise uses a UDP broadcast to discover its name services. It is a
poor choice for most large networks however, because of the subnetting and
the fact that UDP broadcasts are often set up to not cross subnets. They
supply a routing table to get around these issues, but at that point it is
worthless. Most apps are designed to take a parameter indicating the host
name of the naming service.
> Now, imagine I get my home interface prepared by my smart naming
> service, I
> will most probably call the create method on it (for SLSB at least). This
> call will need to reach one of the running AS that in turn will
> return back
> a proxy object for the remote object. This proxy for the remote
> object ALSO
> needs to be HA i.e. to know multiple targets in order to transparently
> switch to another target. Consequently, the AS needs to know the list of
> available AS => communications between AS is necessary.
>
> In fact, it is not completely true. I could use the naming service as you
> suggest it but I do not think it would change much.
I think the name service would handle it all. It would be cool if the stubs
bound into the name service obtained a lease. Something where the name
service could remove them if their servers crashed or were shut down.
Alternatively, the name service could ping the stubs to see if they were
still alive. Kinda like a heartbeat.
> In fact, the next step is to "rewrite" the NS to be HA and to better
> integrate my HAConfigService with this in mind.
I understand that this is your focus. :-)
I think writing a COS-compliant name service that supports multiple binding
of stubs and a pluggable load-balancing algorithm would be the first
priority. You will need it anyway.
> I think the difference between the NS way and the HAConfigService way is
> that the NS way would have a list of proxies, each with a single target
> whereas the HAConfigService way has a single proxy, in which is a list of
> available targets (i.e. running distant containers). But it is also true
> that this could be re-engineered to have more logic in one part
> or another.
Not sure. I would argue that even in a HA naming service design, all servers
could fail and client is stuck. The real magic is in automated restarts! ;-)
> Maybe I have my head in the dust... Any critical feedback is welcome! This
> is the goal of this first step: to get feedback (implementation-feedback,
> design-feedback, ...) to go in the right direction ASAP!
>
> => thank you for your Feedback James! ;)
your welcome, and I too enjoy the discussion.
> For your global understanding:
> each Jboss instance has a JMX MBEAN called HAConfigService
> that, thanks to
> JavaGroups, maintains a list of other HAConfigServices
I really like the leverage you got from JavaGroups. Cool stuff.
> > The cluster would immediately support all types of beans.
>
> ?!? I guess you know we need to take in account many other problems in
> account for other beans such as synchronisation between nodes for entity
> beans, non-concurrent access for SFSB, ...
Actually, no commercial EJB implementation (AFAIK) supports serialized
access to entity beans across VMs. They all enforce serialization at the
database.
As far as Stateful Session Beans, this could be handled if the bean was
passivated to a shared storage mechanism at the end of each transaction.
Most commercial servers require this in a clustered environment.
> What I still don't know is wether the usage of a "dispatcher" or
> "server-proxy" i.e a software that takes place on the server and that
> dispatches calls would be interesting... not so sure...
I think it all comes down to a single name server that can support one or
more stubs bound to the same name. It sounds simple, but it would solve so
many of the HA problems in such a non-complicated manner.
jim
I'm glad you are addressing this important need. I just have a friendly question
on the design. :-)
It seems like the HA that you have implemented is 90% managed on the server-side
and involves a good deal of communication between server instances. Wouldn't a
name service that is shared among multiple containers be a simpler approach that
yields the same results? This name service would have the ability to store
multiple objects under the same name. When a client requested the object bound
to the name, the name service would hand them one of the bound references. This
would obviously be based on some pluggable algorithm.
Control of which servers are clustered would be a simple matter of registering
them with this name service.
The cluster would immediately support all types of beans.
I'm interested in your comments on this approach.
jim
Hello James,
> I'm glad you are addressing this important need. I just have a
> friendly question
> on the design. :-)
This is exactly the goal of this first release!
> It seems like the HA that you have implemented is 90% managed on
> the server-side
> and involves a good deal of communication between server
> instances. Wouldn't a
> name service that is shared among multiple containers be a
> simpler approach that
> yields the same results? This name service would have the ability to store
> multiple objects under the same name. When a client requested the
> object bound
> to the name, the name service would hand them one of the bound
> references. This
> would obviously be based on some pluggable algorithm.
This was exactly my first idea if you look at the JBoss-Dev ML archive about
clustering.
First of all, I have to say that one of the next goal will probably be to develop a HA
Naming Service (replicated, ...)
Whatever solution is choosen, the client needs some code in its proxy and a set of
reference. Otherwise, if the NS only gives him a single reference, if it fails, the
client will have to explicitly resolve a new reference to get a new object. This need
to be as much as possible transparent (i.e. transparent fail-over).
Now, imagine I get my home interface prepared by my smart naming service, I will most
probably call the create method on it (for SLSB at least). This call will need to
reach one of the running AS that in turn will return back a proxy object for the
remote object. This proxy for the remote object ALSO needs to be HA i.e. to know
multiple targets in order to transparently switch to another target. Consequently, the
AS needs to know the list of available AS => communications between AS is necessary.
In fact, it is not completely true. I could use the naming service as you suggest it
but I do not think it would change much.
In fact, the next step is to "rewrite" the NS to be HA and to better integrate my
HAConfigService with this in mind.
I think the difference between the NS way and the HAConfigService way is that the NS
way would have a list of proxies, each with a single target whereas the
HAConfigService way has a single proxy, in which is a list of available targets (i.e.
running distant containers). But it is also true that this could be re-engineered to
have more logic in one part or another.
Maybe I have my head in the dust... Any critical feedback is welcome! This is the goal
of this first step: to get feedback (implementation-feedback, design-feedback, ...) to
go in the right direction ASAP!
=> thank you for your Feedback James! ;)
For your global understanding:
each Jboss instance has a JMX MBEAN called HAConfigService that, thanks to
JavaGroups, maintains a list of other HAConfigServices (i.e. Jboss servers) and shares
information about HA deployed beans i.e. each bean, when deployed, has an id which is
built. In our case it is ~ cluster name + jar name + bean name. This list contains all
possible clusters, beans deployed in these clusters and for each bean, the list of
JBoss containers currently owning such a bean. Consequently, for example, you can ask
a HAConfigService for all available JRMPContainers for a bean called titi in cluster
toto in jar tata.ear. With this knowledge, any container can for example built a smart
proxy containing all possible targets. This service also detects view changes (when a
member dies) and dispatch appropriate information to appropriate objects. This service
is exported in the private JNDI tree for the container invoker for example and in the
public JNDI tree, thanks to a shorter interface, for proxies for example.
> Control of which servers are clustered would be a simple matter
> of registering
> them with this name service.
(You still need to detect when they die to unregister old references.)
In fact, registering a bean reference to the NS (+other info) or registering a bean to
the HAConfigService (+ container reference) is not so different! ;)
> The cluster would immediately support all types of beans.
?!? I guess you know we need to take in account many other problems in account for
other beans such as synchronisation between nodes for entity beans, non-concurrent
access for SFSB, ...
> I'm interested in your comments on this approach.
>
What I still don't know is wether the usage of a "dispatcher" or "server-proxy" i.e a
software that takes place on the server and that dispatches calls would be
interesting... not so sure...
Cheers,
Sacha
Hello James,
> > This was exactly my first idea if you look at the JBoss-Dev ML
> > archive about
> > clustering.
> >
> > First of all, I have to say that one of the next goal will
> probably be to
> > develop a HA Naming Service (replicated, ...)
>
> I agree that this would be nice, but probably HA on the name
> server will be
> unnecessary for 90% of the installations. It could take too long for it to
> be worthwhile. Maybe I'm wrong about this. There is always a
> single point of
> failure in most systems and that is an acceptable risk. For example, most
> large commercial applications will want their enterprise
> nameserver to be an
> LDAP implementation, and often there is only one. Kind of like
> the database.
There's one thing I do not understand:
1) either people use "our" NS implementation
2) or people use "their own" NS implementation (sexy LDAP implementation)
In the first case, we can propose whatever implementation fits our needs. In the
second case, we cannot manage everything as we don't receive lookup/bind calls.
Consequently, in the last case, we should verify which are the limitation for
providing HA through "multiple-bind-per-one-name"
As for the replicated naming service, I agree that it is not mandatory but I do not
think we can propose an HA implementation with such a miss. Furthermore, it would
improve efficiency: all JBoss server would resolve through local lookups (and thus
"eliminate" network latency). And as the NS is the EJB entry point, if you loose it,
all your HA features go to trash unless you restart it (or repair the computer, or...)
Because client nodes would only know the name of ONE naming server, it means that
moving this name server is not really trivial (it could yes, but it means other
complication I think)
> I'm not against a HA name service...I just think a load-balancing name
> service would be a higher priority.
Yes, but the HA name service will provide LB. First because each client can connect to
a different NS and second because when the client downloads the remote+home proxy, the
load-balancing features is already in.
Am I missing something?
> > Whatever solution is choosen, the client needs some code in its
> > proxy and a
> > set of reference. Otherwise, if the NS only gives him a single
> > reference, if
> > it fails, the client will have to explicitly resolve a new
> > reference to get
> > a new object. This need to be as much as possible transparent (i.e.
> > transparent fail-over).
>
> Agreed. The client should not have to do anything to get failover support.
> In fact, Inprise uses a UDP broadcast to discover its name
> services. It is a
> poor choice for most large networks however, because of the subnetting and
> the fact that UDP broadcasts are often set up to not cross subnets. They
> supply a routing table to get around these issues, but at that point it is
> worthless. Most apps are designed to take a parameter indicating the host
> name of the naming service.
Yes, but you could imagine the client to connect to this name service and asks for a
list of other available NS. We could also imagine to have a list of hosts instead of a
single host in the property. This would change from the "standard" but could be easily
done. In any cases, there is some kind of specific action to take to bind a JNDI
provider to a client (such as setting the good class provider for example.)
> > Now, imagine I get my home interface prepared by my smart naming
> > service, I
> > will most probably call the create method on it (for SLSB at
> least). This
> > call will need to reach one of the running AS that in turn will
> > return back
> > a proxy object for the remote object. This proxy for the remote
> > object ALSO
> > needs to be HA i.e. to know multiple targets in order to transparently
> > switch to another target. Consequently, the AS needs to know the list of
> > available AS => communications between AS is necessary.
> >
> > In fact, it is not completely true. I could use the naming
> service as you
> > suggest it but I do not think it would change much.
>
> I think the name service would handle it all. It would be cool if
> the stubs
> bound into the name service obtained a lease. Something where the name
> service could remove them if their servers crashed or were shut down.
> Alternatively, the name service could ping the stubs to see if they were
> still alive. Kinda like a heartbeat.
This functionality is already provided in JavaGroups.
To be frank, I do not see (appart from location consideration) the real
difference/gain between having a smart NS allowing multiple objects per name and a
simple NS where you bind smart stubs. In both cases, you will need to have smart stubs
=> your container, when creating the stubs, will need to obtain the knowledge about
other nodes and fill this smart stub with appropriate data.
> > In fact, the next step is to "rewrite" the NS to be HA and to better
> > integrate my HAConfigService with this in mind.
>
> I understand that this is your focus. :-)
> I think writing a COS-compliant name service that supports
> multiple binding
> of stubs and a pluggable load-balancing algorithm would be the first
> priority. You will need it anyway.
Please understand that this was also my first idea but that now, I can't figure what
would be the real advantage of this solution! Also understand that I am not trying to
blindly argue just to save my design. I would be very happy to go in a better
direction, and this current discussion is very helpful but what would help is a kind
of list of pros/cons between both solutions because I am not sure that the smart NS
way is smarter/easier/...
But again, I am strongly biaised and could be totally wrong. ;)
> > I think the difference between the NS way and the HAConfigService way is
> > that the NS way would have a list of proxies, each with a single target
> > whereas the HAConfigService way has a single proxy, in which is
> a list of
> > available targets (i.e. running distant containers). But it is also true
> > that this could be re-engineered to have more logic in one part
> > or another.
>
> Not sure. I would argue that even in a HA naming service design,
> all servers
> could fail and client is stuck. The real magic is in automated
> restarts! ;-)
This is true for every imaginable solution: when everthing is dead, we need to restart
them automagically.
> > Maybe I have my head in the dust... Any critical feedback is
> welcome! This
> > is the goal of this first step: to get feedback
> (implementation-feedback,
> > design-feedback, ...) to go in the right direction ASAP!
> >
> > => thank you for your Feedback James! ;)
>
> your welcome, and I too enjoy the discussion.
>
> > For your global understanding:
> > each Jboss instance has a JMX MBEAN called HAConfigService
> > that, thanks to
> > JavaGroups, maintains a list of other HAConfigServices
>
> I really like the leverage you got from JavaGroups. Cool stuff.
>
> > > The cluster would immediately support all types of beans.
> >
> > ?!? I guess you know we need to take in account many other problems in
> > account for other beans such as synchronisation between nodes for entity
> > beans, non-concurrent access for SFSB, ...
>
> Actually, no commercial EJB implementation (AFAIK) supports serialized
> access to entity beans across VMs. They all enforce serialization at the
> database.
But is this always possible? what about BMP? what about LDAP based persistance for
example?
> As far as Stateful Session Beans, this could be handled if the bean was
> passivated to a shared storage mechanism at the end of each transaction.
> Most commercial servers require this in a clustered environment.
As we would master the persistance mechanism this time (as opposed to the entity
beans), this would work, I do agree.
> > What I still don't know is wether the usage of a "dispatcher" or
> > "server-proxy" i.e a software that takes place on the server and that
> > dispatches calls would be interesting... not so sure...
>
> I think it all comes down to a single name server that can support one or
> more stubs bound to the same name. It sounds simple, but it would solve so
> many of the HA problems in such a non-complicated manner.
Please, provide me with "use-cases" where you would have a real benefit. The smart NS
will need to deal with exactly the same issues that the HAConfigService solves today
with JavaGroups but instead of having this logic in the NS directly, it is a distinct
service launched with JBoss.
BTW, what is your timezone? mine is CET (i.e. GMT+1 in winter and GMT+2 in summer). If
I know this, I can decide when it is a good idea to answer quickly i.e. when you are
probably on-line and not deeply sleepling!
Don't you think that this discussion could be moved (now or later) to the JBoss-dev ML?
Cheers,
Sacha
