Re: [jBoss-Dev] CL: Clustering, let's get started
Hi Stacy, I'm just about to post my code up here. So you can have a look at it, rip it apart, whatever. Looking forward to it :) -- Juha
RE: [jBoss-Dev] CL: Clustering, let's get started
On Mon, 12 Mar 2001, Sacha Labourey wrote: Hello Tom, I think that this last point is one which is being lost on some people. It means that there can be no central controller. Everything must work in a completely distributed way. I have tried to describe how this might work. For instance, there can not be a 'master' jndi repository, which is replicated, nor can there be a 'master' transaction manager, message queue... anything. This is why the private/public JNDI tree became necessary, so that some services could retain an identity at each node, while beans are federated across the nodes, and a lookup on them may return an instance at any node. OK, I see what you mean: difference between a bean and a service i.e. between a cluster-view and a node-view. Consequently, you suggest to have the public tree (cluster-view) being replicated at each node and all the private (i.e. node-view) trees, local to each node, being federated under the public tree. Right? (or we may as well have the public (cluster) tree federated under each private local tree) Yes, you are correct. 2nd, this "where-could-i-find-such-a-bean..." mechanism is probably not strict enough. What do you mean be this? How is it not 'strict' enough? I meant too lazy = not enough optimized. But I may have missed your point! Yes, it is possibly that. But the other option is to keep a replicated index of which PK can be found where (!) This is a clear memory-performance tradeoff. Having thought this through a little more, what I really meant was that objects in JNDI should be treated differently depending on whether they are services or EJBs. A service should have an identity at each node within the cluster. For a 10-node cluster, there will be 10 transaction managers, which are co-operating to provide distributed transactions. Each one still retains it's own identity. So there is a transaction manager in each 'private' JNDI, which is not visible to the client anyway. Then there are beans, which have to exist as one entity across all the nodes. Do you see what I am saying? There are two different sorts of clustering happening here. One is that of many individuals working independantly to provide a service, the other is one corporate body making up a service. The 'public/private' thing might be a bit confusing, it was just a way of implementing this. OK, I agree with need such local vs. global representation. The list of other live nodes would probably be best acheived by a regular ... ...). (in fact, the failure detection mechanism is a basic layer of the group communication problem) What do you think? I like this group communication thingy, except for that everyone I've heard explaining it seems to have advocated having a central controller, which gets messy in the case of the controller failing. I have tried to avoid this. ?!? the group communication scheme as generally implemented (like javagroups) is completely redundant. At a particular moment you have some kind of master that sequences the system. But, a completely distributed failure detection subsystem will detect its death (or the death of any other node). And if the "controler" dies, a distributed election mechanism automatically takes place. There no single POF. Have I answered your concern or am I (again ;) ) missing your point? No, that is cool. I just find the concept of having a controller, even a transitory one, a bit messy. I prefer the 'all are equals' approach, rather than the 'first among equals'. But that is for purely aesthetic reasons. Tom -- "If you mess with something for long enough it will break." - Schmidt's law of engineering
RE: [jBoss-Dev] CL: Clustering, let's get started
Alright... I am sorry but you guys completely missed the point. We are a week into this (good) discussion, everybody tries to impress everybody and we still are miles away from a single line of code. In a second we are going to +1/-1 discussions that never end... All I want is something we can bite on, a bit ambitious and a bit realistic... All I wanted was SSB failover as client server. Man this is going to be a long road... Does anyone step up to provide a simple "stateless bean" clustering, no b2b no nothing, just client server with failover of the stateless bean. JMX clustering will be good there:), deployer will be reworked right there.. naming will be investigated, and the failover logic will be coded. (btw the only point I will make is "smart stubs are good" MAKE IT SIMPLE) SO STOP SHOOTING FOR THE SKY, SHOOT AT THAT SHORT GOAL JUST STEP UP... THE KEYBOARD IS RIGHT THERE!! You wanna know the reason why? because I know the stuff you are going to write is going to crap. Believe me it is going to be absolute shite. IT ALWAYS IS. Or you think your code don't stink??? even rickard's was full of poopoos, mine? I won't even mention it, same with Sebastien and Juha, hiram, oleg, simone (;-) etc etc we ARE ALL THE SAME, the ONLY DIFFERENCE is that we stepped up THAT IS THE ONLY REAL THING, you must understand that your code will go to trash... so don't worry about getting it right YOU WON'T!!. I repeat "Clustering, let's get started", let's beat the mental barrier that is #1 in my mind ;-) Please get going, anything anything... please marc __ strip those feet of lead my friend raise the curtain, raise the roof, spirit's on tonight! --we love our audience 'bauhaus'-- __ |-Original Message- |From: [EMAIL PROTECTED] |[mailto:[EMAIL PROTECTED]]On Behalf Of Tom Cook |Sent: Monday, March 12, 2001 6:15 PM |To: JBoss-Dev |Subject: RE: [jBoss-Dev] CL: Clustering, let's get started | | |On Mon, 12 Mar 2001, Sacha Labourey wrote: | | Hello Tom, | | I think that this last point is one which is being lost on some | people. It means that there can be no central controller. Everything | must work in a completely distributed way. I have tried to |describe how | this might work. For instance, there can not be a 'master' jndi | repository, which is replicated, nor can there be a 'master' |transaction | manager, message queue... anything. This is why the |private/public JNDI | tree became necessary, so that some services could retain an |identity at | each node, while beans are federated across the nodes, and a lookup on | them may return an instance at any node. | | OK, I see what you mean: difference between a bean and a service i.e. | between a cluster-view and a node-view. | | Consequently, you suggest to have the public tree (cluster-view) being | replicated at each node and all the private (i.e. node-view) |trees, local to | each node, being federated under the public tree. Right? (or we |may as well | have the public (cluster) tree federated under each private local tree) | |Yes, you are correct. | | 2nd, this "where-could-i-find-such-a-bean..." mechanism is |probably not | strict enough. | | What do you mean be this? How is it not 'strict' enough? | | I meant too lazy = not enough optimized. But I may have missed |your point! | |Yes, it is possibly that. But the other option is to keep a replicated |index of which PK can be found where (!) This is a clear |memory-performance tradeoff. | | Having thought this through a little more, what I really meant was that | objects in JNDI should be treated differently depending on whether they | are services or EJBs. A service should have an identity at each node | within the cluster. For a 10-node cluster, there will be 10 |transaction | managers, which are co-operating to provide distributed | transactions. Each one still retains it's own identity. So there is a | transaction manager in each 'private' JNDI, which is not visible to the | client anyway. Then there are beans, which have to exist as one entity | across all the nodes. Do you see what I am saying? There are two | different sorts of clustering happening here. One is that of many | individuals working independantly to provide a service, the |other is one | corporate body making up a service. The 'public/private' |thing might be a | bit confusing, it was just a way of implementing this. | | OK, I agree with need such local vs. global representation. | |The list of other live nodes would probably be best acheived | by a regular | ... | ...). (in fact, the failure detection mechanism is a basic |layer of the | group communication problem) | | What do you think? | | I like this group communication thingy, except for that everyone I've | heard explaining it seems to have advocated having a central |con
Re: [jBoss-Dev] CL: Clustering, let's get started
I have been reading your discussions on Clustering the past few days and would like to throw my 2 cents in. To give a little background , I'm working for a company that is planning to deploy JBoss and Jetty for a production system in the coming months. We are porting from Weblogic to JBoss for monetary reasons. Not having clustering in JBoss/Jetty was a concern at first, but after further analysis of our requirements we determined it might be ok not to have clustering. I have a lot of random comments so, in advance, I apologize if the following points only apply to the concerns of my company or are ignorant or stupid in any way :-) - This is just an opinion, please no flames, but I would rather see the JBoss developer community focus on the quality of JBoss, the quality of the release cycle(an actual branching scheme and bug fixes branch), a clean 2.1 relase, and getting the docs up-to-date rather than new features like clustering. JBoss is already a very cool product, and I'm guessing a lot of us can work around the problems that clustering solves. - Distributed Transaction Processing. If the JBoss developers are planning not to tackle DTP along with other clustering features I hope great concern is taken not to break applications that use transactions within their EJBs. If a JBoss load-balancer is bouncing nested Bean method calls to many different nodes and you don't have DTP, many applications may break. A solution for this may be that nested Bean method calls are "sticky". That is, only the initiating client request be load-balanced. - IMHO, clustering EJBs is only half the problem. Servlet/jsp engines (Jetty/Tomcat/whatever) need to be clusterable as well for a complete solution and to be competitve against other Enterprise app servers like WebLogic. (That's the goal isn't, to beat WebLogic?). Don't most apps that use EJBs also use a servlet engine? - Also, IMHO, load-balancing is much less important than fail-over and DTP. As others have said, those of us who are writing Web-Apps are getting load-balancing from our routers already. - Our project is most concerned that all the ACID properties of Transactions are definable and reliable in a clustered or scaled environment. Hope these comments are useful. Best regards, Bill marc fleury wrote: Alright... I am sorry but you guys completely missed the point. We are a week into this (good) discussion, everybody tries to impress everybody and we still are miles away from a single line of code. In a second we are going to +1/-1 discussions that never end... All I want is something we can bite on, a bit ambitious and a bit realistic... All I wanted was SSB failover as client server. Man this is going to be a long road... Does anyone step up to provide a simple "stateless bean" clustering, no b2b no nothing, just client server with failover of the stateless bean. JMX clustering will be good there:), deployer will be reworked right there.. naming will be investigated, and the failover logic will be coded. (btw the only point I will make is "smart stubs are good" MAKE IT SIMPLE) SO STOP SHOOTING FOR THE SKY, SHOOT AT THAT SHORT GOAL JUST STEP UP... THE KEYBOARD IS RIGHT THERE!! You wanna know the reason why? because I know the stuff you are going to write is going to crap. Believe me it is going to be absolute shite. IT ALWAYS IS. Or you think your code don't stink??? even rickard's was full of poopoos, mine? I won't even mention it, same with Sebastien and Juha, hiram, oleg, simone (;-) etc etc we ARE ALL THE SAME, the ONLY DIFFERENCE is that we stepped up THAT IS THE ONLY REAL THING, you must understand that your code will go to trash... so don't worry about getting it right YOU WON'T!!. I repeat "Clustering, let's get started", let's beat the mental barrier that is #1 in my mind ;-) Please get going, anything anything... please marc __ strip those feet of lead my friend raise the curtain, raise the roof, spirit's on tonight! --we love our audience 'bauhaus'-- __ |-Original Message- |From: [EMAIL PROTECTED] |[mailto:[EMAIL PROTECTED]]On Behalf Of Tom Cook |Sent: Monday, March 12, 2001 6:15 PM |To: JBoss-Dev |Subject: RE: [jBoss-Dev] CL: Clustering, let's get started | | |On Mon, 12 Mar 2001, Sacha Labourey wrote: | | Hello Tom, | | I think that this last point is one which is being lost on some | people. It means that there can be no central controller. Everything | must work in a completely distributed way. I have tried to |describe how | this might work. For instance, there can not be a 'master' jndi | repository, which is replicated, nor can there be a 'master' |transaction | manager, message queue... anything. This is why the |private/public JNDI
RE: [jBoss-Dev] CL: Clustering, let's get started
On Fri, 9 Mar 2001, Sacha Labourey wrote: Hello Tom, * In normal operation both nodes are usable. I guess by "both" you mean "any" or "all" Yes, sorry, I got stuck thinking in two-node mode. I meant all. The meaning was really that this is an active, not passive, cluster. * In the case that one node fails, clients which are using objects on that node might see wierd things at this stage, but there are ways around that. * In normal operation, client requests on the _home_ interface are load balanced. The balancing policy is just that - a matter of policy which needn't concern us at this stage. * It should be possible to add and remove nodes from the cluster transparently. ok. I think that this last point is one which is being lost on some people. It means that there can be no central controller. Everything must work in a completely distributed way. I have tried to describe how this might work. For instance, there can not be a 'master' jndi repository, which is replicated, nor can there be a 'master' transaction manager, message queue... anything. This is why the private/public JNDI tree became necessary, so that some services could retain an identity at each node, while beans are federated across the nodes, and a lookup on them may return an instance at any node. 1. Semi-federated JNDI. --- I propose that some sort of delegation model be used for JNDI. There is a 'public' JNDI interface and a 'private' JNDI interface. The private interface is just an ordinary server as already exists, one on each node, which keeps a list of the remote objects registered in that node. This means that, rather than having one central instance of each service which co-ordinates the nodes, each node has it's own instance of each service and they all co-operate. This includes JNDI. The public interface is available on each node. It keeps a list of all other live nodes in the cluster, and delegates to them in a round-robin way. One request may be delegated to several servers before a match is found for the name. This inefficiency should only happen in exceptional circumstances (a node is going down for maintenance etc). The worst case is when no match for the name can be found, in which case every node will be checked. This is reasonable, since it indicates either the deployment is broken, or the user made an error. In 'normal' circumstances, this should not happen. In 'ideal' operating conditions (every node has every service available) there will be only one delegation for every request. IMO, this is not efficient. 1st, it would be nice to be able to determine that a particular application need to be deployed to only some particular nodes. Yes, but that would not be a huge hurdle once we had this system implemented. In the absolute simplest case, a node which does not have a bean deployed may simply report that it's pool for that bean is 100% full, so it will never get invocations on it. 2nd, this "where-could-i-find-such-a-bean..." mechanism is probably not strict enough. What do you mean be this? How is it not 'strict' enough? BTW, you say that you want a private and a public JNDI interface. Why? Is it to optimize local calls? No, like I said above, I want to be able to have federated and non-federated objects in JNDI. Having thought this through a little more, what I really meant was that objects in JNDI should be treated differently depending on whether they are services or EJBs. A service should have an identity at each node within the cluster. For a 10-node cluster, there will be 10 transaction managers, which are co-operating to provide distributed transactions. Each one still retains it's own identity. So there is a transaction manager in each 'private' JNDI, which is not visible to the client anyway. Then there are beans, which have to exist as one entity across all the nodes. Do you see what I am saying? There are two different sorts of clustering happening here. One is that of many individuals working independantly to provide a service, the other is one corporate body making up a service. The 'public/private' thing might be a bit confusing, it was just a way of implementing this. Have you read my previous proposition? I suggested something like you but we would have only one JNDI space replicated at different places (possibly and most probably on each node) with the possibility (extended JNDI semantic) to bind several (beans) reference for a same JNDI name. Consequently: - for whatever JNDI node a client decides to speak with, this node exactly knows which nodes have subscribed a reference to this name, without prompting its neighboors - local code optimisation is (can be) achieved - if an application is deployed only on a few nodes (not all nodes), the related JNDI names of this application will receive bind request only from these nodes
RE: [jBoss-Dev] CL: Clustering, let's get started
Hello Tom, I think that this last point is one which is being lost on some people. It means that there can be no central controller. Everything must work in a completely distributed way. I have tried to describe how this might work. For instance, there can not be a 'master' jndi repository, which is replicated, nor can there be a 'master' transaction manager, message queue... anything. This is why the private/public JNDI tree became necessary, so that some services could retain an identity at each node, while beans are federated across the nodes, and a lookup on them may return an instance at any node. OK, I see what you mean: difference between a bean and a service i.e. between a cluster-view and a node-view. Consequently, you suggest to have the public tree (cluster-view) being replicated at each node and all the private (i.e. node-view) trees, local to each node, being federated under the public tree. Right? (or we may as well have the public (cluster) tree federated under each private local tree) 2nd, this "where-could-i-find-such-a-bean..." mechanism is probably not strict enough. What do you mean be this? How is it not 'strict' enough? I meant too lazy = not enough optimized. But I may have missed your point! Having thought this through a little more, what I really meant was that objects in JNDI should be treated differently depending on whether they are services or EJBs. A service should have an identity at each node within the cluster. For a 10-node cluster, there will be 10 transaction managers, which are co-operating to provide distributed transactions. Each one still retains it's own identity. So there is a transaction manager in each 'private' JNDI, which is not visible to the client anyway. Then there are beans, which have to exist as one entity across all the nodes. Do you see what I am saying? There are two different sorts of clustering happening here. One is that of many individuals working independantly to provide a service, the other is one corporate body making up a service. The 'public/private' thing might be a bit confusing, it was just a way of implementing this. OK, I agree with need such local vs. global representation. The list of other live nodes would probably be best acheived by a regular ... ...). (in fact, the failure detection mechanism is a basic layer of the group communication problem) What do you think? I like this group communication thingy, except for that everyone I've heard explaining it seems to have advocated having a central controller, which gets messy in the case of the controller failing. I have tried to avoid this. ?!? the group communication scheme as generally implemented (like javagroups) is completely redundant. At a particular moment you have some kind of master that sequences the system. But, a completely distributed failure detection subsystem will detect its death (or the death of any other node). And if the "controler" dies, a distributed election mechanism automatically takes place. There no single POF. Have I answered your concern or am I (again ;) ) missing your point? Cheers, Sacha
RE: [jBoss-Dev] CL: Clustering, let's get started
Tom - Great stuff - your efforts to document your thoughts are good - hopefully we can build on and extend your thoughts in an ever more complete document. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Tom Cook Sent: Thursday, March 08, 2001 3:44 PM To: JBoss-Dev Subject: RE: [jBoss-Dev] CL: Clustering, let's get started OK, in all those hours I have free as a student ;-) I have been thinking about clustering, and this is what I've come up with. Four parts of the EJB server need to become distributed for clustering to work. By 'clustering' here I mean this: * In normal operation both nodes are usable. * In the case that one node fails, clients which are using objects on that node might see wierd things at this stage, but there are ways around that. * In normal operation, client requests on the _home_ interface are load balanced. The balancing policy is just that - a matter of policy which needn't concern us at this stage. * It should be possible to add and remove nodes from the cluster transparently. Here is how I propse to acheive this. 1. Semi-federated JNDI. --- I propose that some sort of delegation model be used for JNDI. There is a 'public' JNDI interface and a 'private' JNDI interface. The private interface is just an ordinary server as already exists, one on each node, which keeps a list of the remote objects registered in that node. This means that, rather than having one central instance of each service which co-ordinates the nodes, each node has it's own instance of each service and they all co-operate. This includes JNDI. The public interface is available on each node. It keeps a list of all other live nodes in the cluster, and delegates to them in a round-robin way. One request may be delegated to several servers before a match is found for the name. This inefficiency should only happen in exceptional circumstances (a node is going down for maintenance etc). The worst case is when no match for the name can be found, in which case every node will be checked. This is reasonable, since it indicates either the deployment is broken, or the user made an error. In 'normal' circumstances, this should not happen. In 'ideal' operating conditions (every node has every service available) there will be only one delegation for every request. The list of other live nodes would probably be best acheived by a regular broadcast ping. This sounds inefficient, but it is how commercial vendors (like Borland Visibroker) acheive similar ends. 2 Distributed J2EE deployer. When a package is deployed, it should be transferred to every node and deployed on them also. It may be safe to assume that if one node can deploy a package, then the others can too, or you might want to check them all and then undeploy if it fails. 3 Distributed home interfaces. -- The home interfaces need to be distributed to implement the sort of load balancing I have in mind. The idea is this. Pretty much any operation you can do on a home interface involves either activating an instance of a bean or finding an instance. So there are to activities: 3.1 Activating an instnace. --- Here the home interface finds the node which is least loaded (the definition of 'least loaded' depends on your load balancing policy - it might mean the next one in a list) and activates the instance on that node, returning the reference to the client. 3.2 Finding an instance. This might well require a sort of braodcast to see if anyone has an instance already activated as that bean. If so, return that instance. If not, use the activation described in 3.1. This sounds messy, but the aim is to simplify things by restricting an entity to only existing on one node at a time. This makes distributed transaction a hell of a lot easier, as far as I can see. 4 Distributed transaction manager. -- I don't think this is as bad as it sounds, if we can do a two phase commit on the beans. Then we need to find some way of propogating a unique transaction ID in bean to bean calls. The transaction management would look something like this: * Client makes a call on bean A. Transaction is started (given appropriate transaction attribute). Unique transaction ID is assigned (UUID might be necessary here). Say this is XID 111. * Bean A makes call on bean B. New transaction is started on bean B with the same transaction ID, ie. 111. This is independant of which node the transaction is started on. Bean A keeps a note that it has invoked bean B in this transaction. * Method on bean B completes, but bean B stays associated with transaction 111. * Method on bean A completes, and we decide whether to commit or rollback. - Commit Do a two phase commit on each node which bean A has accessed. This commit needs
RE: [jBoss-Dev] CL: Clustering, let's get started
So this is the scenario where information could be shared in the JNDI tree
for other purposes than just storing references to home interfaces.
===
| NODE 1 |NODE 2 | NODE3 |
===
JNDI |App1/BeanAHome |App1/BeanAHome |App1/BeanAHome |
SPACE|App1/BeanA |App1/BeanA |App1/BeanA |
|App2/BeanBHome |App2/BeanBHome |App2/BeanBHome |
|App2/BeanB |App2/BeanB |App2/BeanB |
|myapp.Snapshot |myapp.Snapshot | |
===
EJB | | | |
SERVER | App1 running | App1 running | App1 running |
| App2 deployed but | App2 running | |
| in standby backup | | |
===
CACHE| | | |
DATA | Data {A,B}| Data {B,C}| Data {D,E}|
===
I have made a monitor system built into my app. This monitor system consist
of an internal measurement/snapshot component that sits inside the EJB
application and an external report/analysis component.
The internal snapshot component gathers information about the EJB app
statistics and publishes them into an object into the JNDI tree.
In a non-clustered environment, the external report/analysis component will
go to each server and take the latest snap shot and compile the information
and produce a report. The snapshot is bound to the name myapp.Snapshot.
So far so good, the monitoring works great with my 3 different server nodes.
No client transparent fail-over has become a requirement from the EJB client
app (another app that accesses the EJB server, not the monitor report app).
The developers thought they could do this seamlessly, but as it turns out,
all the content in the JNDI tree gets replicated between the cluster nodes.
This means that object bound to the name "myapp.Snapshot" represents the
latest snapshot of each of the individual servers. But what happens if NODE
2 gets restarted, and a snapshot has not been performed (bound to the JNDI
tree)? Now there is a risk that a snapshot from NODE 1 or 3 gets replicated
into the JNDI tree of NODE 2 and the report component is accessing incorrect
data.
Also, lets say that the client is reading snapshot data as well and
collecting a history on client machine, so when NODE 2 fails, the request
gets automatically redirected to NODE 1 and the client data doesn't make
sense anymore :)
Now this is clearly a design that was not made with clustering (shared JNDI
contexts) in mind.
But it happens, and they probably already exist.
We could say, "hey dude, you should have thought of clustering at the time
you designed your report monitor" and the guy will reply
"well, there wasn't a clustering strategy in place for JBoss when I designed
my app, I had no idea that the JNDI context would replicate everything into
other server nodes."
Although this example is clearly not anything I would do (I know better than
to use JNDI for shared caching :) I'm sure similar scenarios actually do
exist.
Just trying to point out two things here
1. You will have name conflicts when replicating JNDI objects
2. If you have a strategy of resolving those name conflicts by applying some
FIFO overwrite rule, you may end up overwriting data that applications
depend upon.
3. When a client gets transparently redirected to another NODE, they may see
unexpected data in the JNDI tree from what they say in their previous call
One way to do clustering if you have an app like this is to use NODE 1 as
main server and NODE 2 as hot standby, ie no clients connect to NODE 2 until
NODE 1 crashes. this way the transparent failover is truly transparent
just some thoughts on JNDI replication issues
Filip
~
Namaste - I bow to the divine in you
~
Filip Hanik
Software Architect
[EMAIL PROTECTED]
www.filip.net
Re: [jBoss-Dev] CL: Clustering, let's get started
Sounds cool.. but I think that JMS CAN do most of that stuff. It's all in how it's implemented. In the JBossMQ CVS we already have a working Multicast JMS server. This server is different than the normal JBossMQ JMS server, I'm calling this one JBossMQ-Cluster. That's because I'm aiming to optimize it for operating in clustered environments. In other works JBossMQ-Cluster will not support persistence, queues, durable subscriptions, message priorities, or transactions to name a few missing features. It's going to provide FAST publish-subscribe services via multicast. What does it do now that java groups does? : * losses transmission of a message to all recipients (with retransmission of missing messages) * fragmentation of large messages into smaller ones and reassembly at the receiver's side ordering of messages, e.g. messages m1 and m2 sent by P will be received by all receivers in the same order, and not as m2, m1 (FIFO order) Using the above features and the standard JMS publish-subscribe services, I think we can build all the other features quickly. The only non standard JMS extension we would have to use is so that app could get notified when a peer failure is detected by the JMS Server. I've started working on a class that keeps a list which peers are active in the cluster. It should not be too hard to extend that so that it provides some similar "group" notification services such as the ones provided by javagroups. Regards, Hiram P.S. I have looked at the Javagroups site and they do have a WAY better multicast implementation. I think I'm going to have to steal a few Ideas from over there ;-) But the point of the e-mail was that: I think a framework around JMS can be implemented which will provide all the services that we will need. - Original Message - From: "Filip Hanik" [EMAIL PROTECTED] To: "JBoss-Dev" [EMAIL PROTECTED] Sent: Friday, March 09, 2001 11:52 AM Subject: RE: [jBoss-Dev] CL: Clustering, let's get started I wasn't aware that JMS could a) allow you to modify/customize the protocol stack b) provide notification of members that joins, leaves or crashes c) atomicity these are the big things, I would like to hear what info you have one JMS over multicast Filip --8--- JavaGroups is a group communication toolkit written entirely in Java. It is based on IP multicast, but extends it with reliability and group membership. Reliability includes * lossless transmission of a message to all recipients (with retransmission of missing messages) * fragmentation of large messages into smaller ones and reassembly at the receiver's side ordering of messages, e.g. messages m1 and m2 sent by P will be received by all receivers in the same order, and not as m2, m1 (FIFO order) * atomicity: a message will be received by all receivers, or none. Group Membership includes * Knowledge of who the members of a group are and * Notification when a new member joins, an existing member leaves, or an existing member has crashed ~ Namaste - I bow to the divine in you ~ Filip Hanik Software Architect [EMAIL PROTECTED] www.filip.net -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Hiram Chirino Sent: Friday, March 09, 2001 11:44 AM To: JBoss-Dev Subject: Re: [jBoss-Dev] CL: Clustering, let's get started I would like to know what javagroups give you that JMS over multicast does not? Regards, Hiram Yes, I read up on JavaGroups, it is a brilliant idea. if anyone of you haven't taken a look at it yet, goto www.javagroups.com the idea of building your own stack on top of their technology is very appealing to me. Furthermore, the idea of groups will blow away an implementation like Weblogic that uses IP multicast and heart beats to determine whether a server is up or down. Javagroups has that capability built in already. not trying to be competitive, just comparative. :) Tom and Sacha, great ideas and great work! ~ Namaste - I bow to the divine in you ~ Filip Hanik Software Architect [EMAIL PROTECTED] www.filip.net
Re: [jBoss-Dev] CL: Clustering, let's get started
Wow, I stand corrected. I did not understand that "view" concept. I see how that is very powerful and useful. This is definitely something that JMS will not be able to provide. I'm going to have to research it further. Thanks, Hiram - Original Message - From: "Sacha Labourey" [EMAIL PROTECTED] To: "JBoss-Dev" [EMAIL PROTECTED] Sent: Friday, March 09, 2001 1:57 PM Subject: RE: [jBoss-Dev] CL: Clustering, let's get started I wasn't aware that JMS could a) allow you to modify/customize the protocol stack b) provide notification of members that joins, leaves or crashes c) atomicity Exactly. Furthermore, it may also give you synchronous view group communication i.e. "- Agreement: correct processes deliver the same set of messages in any given view - Integrity: if a process P delivers message m, then it will not deliver m again - Validity: correct processes always deliver the messages that they send. If the system fails to deliver a message to any process q, then it notifies the suviving processes by delivering a new view with q excluded, immediately after the view in which any of them delivered the message. In a view-synchronous system, the delivery of a new view draws a conceptual line across the system and every message that is delivered at all is consistently devlivered one side or the other of that line. This enables the programmer to draw userful conclusions about the set of messages that other correct processes have delivered when it delivers a new view, based only on the local ordering of message delivery and view delivery events." (from Distributed systems, concepts and design, Coulouris-Dollimore-Kindbert, 3rd ed., Addison-Wesley) Some approximate definitions: - a group is a set of processes - a view is a set of process - delivering a message means receiving it - delivering a view means receiving the new set of processes that are corrects (i.e. alive, ...) Consequently, it is possible to exchange information in a group (new JNDI name binding for example...) and be sure that everyone correct will receive it (in the same order if necessary). As each view change (v0, v1, v2, ..., vn) and message (m1, m2, ..., mn) is numbered, it is possible for a process, when it wakes up, to ask the group to get all updates mi (i is the last message he got before collapsing). Furthermore, such a system needs to detect when a system has failed (to deliver a new view), which why such a subsystem is present in javagroups to detect these potential failures. The algorithm that determines which view is valid, ... is distributed and fault-tolerant in itself : if the "master" dies, an election automatically takes place to determine the new master. I understand it is a bit a quick introduction... ask if you have some questions! Cheers, Sacha
RE: [jBoss-Dev] CL: Clustering, let's get started
Hello, This naming service would eventually need to be some sort of federated/replicated thing, for HA. Each instance in the cluster might run its own NS. that would be my recommendation too, just like weblogic where each node runs its own NS and content gets replicated across the cluster using multicast. using a shared NS becomes to much of a headache. I aggree with you and sorry if my suggestion wasn't clear enough. We have many NS (one per server for example) each synchronized to (i.e. a replicate of) the other through group communication (hence the sourceforge.net/projects/javagroups link). This provide better HA, scalability, and performance (as resolution can be made locally). Cheers, Sacha
RE: [jBoss-Dev] CL: Clustering, let's get started
Yippee!! Or more precisely - clustering will make jBoss a 'real' commercial contender for obvious reasons. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of marc fleury Sent: Tuesday, March 06, 2001 4:27 PM To: jBoss Developer Subject: [jBoss-Dev] CL: Clustering, let's get started Ok, it's been promised, it's been discussed, it has already started, it is really wanted. Also it will put JBoss on the map for ever... The clustering. I would like to "officially" call the season for clustering "open". There have been some discussions already some public some private and I believe we should really take this take by step. So we won't shoot for the stars in the very beginning, just something simple. I will assume the interim leadership for the implementation since there is no clear lead yet and the team is yet to form. I invite all of you that have touched the subject to publicly post and explain your view. STEP ONE: -Start 2 servers do "round robin" on STATELESS SESSION. Simple right? That will be some work go! marc _ Marc Fleury, Ph.D [EMAIL PROTECTED] _
RE: [jBoss-Dev] CL: Clustering, let's get started
Hello, just some ideas to start a discussion... As said in a precedent post, administration tools will be very important in the clustering and they may also help, during the design phase, to see (as a kind of "use cases") what features the clustering should provide. I do not know much about WL, but have you checked Borland Appcenter tool? http://www.borland.com/appcenter/ It gives some nice ideas. Some general issues I have thought of: - should be clustering stuff be available to Java-RMI only clients or to all clients? If we want it to be available to all clients, we cannot rely on specific smart code (i.e. algo) in the dynamic stubs but instead need to have the logic on "the server side" (or approaching) and use some more standard principles such as IIOP multiple profiles IOR on the client side. We may also have a mix of solutions. - I suggest to have a (as someone mentioned earlier) a fault tolerant CORBA Naming Service able to store more than one reference for a particular binding. These extended binding features would be provided as extension of the basic IDL CORBA NS, thus allowing standard NS clients to get an appropriate IOR (selected by a pluggable algorithm) see next point for an implementation helper - it may be very interesting to keep an eye on JavaGroups (http://sourceforge.net/projects/javagroups), an open source project which may be of interest to provide the basic layer for group communication (sychrnonous view, group multicast, ...) It could be for example used to have the NS implementation (see point above) completely redundant and not using some kind of master-slave design where only one server can, at a time, receive write invocations The general idea would be to have load-balancing at the session level: the client connect to the NS and get a reference to its target (bean). As many containers may be bound to the same NS name, a pluggable algorithm in the NS would decide which particular container it wants the client to use (= load-balancing). Furthermore, in the case of IIOP, it may add some other profiles to other containers in the IOR to provide fail-over feature. In the case of RMI-Java client, the provided stub may contain the failover logic itself. If the client invocation fails because its server becomes corrupted, the client mechanism will automagically retry a new invocation. Thanks to the multi-profile IOR for IIOP and to the smart stub for Java-RMI. If a server is too busy to serve a request, it may send back a LOCATION_FORWARD IIOP message to the client (thus really helping the server to lower its load but creating a new consumming client-server network round-trip) or itself forward the request to another server in the cluster and act as a proxy. It is evident that all these ideas are just very general idea to "activate the pump" and concerns such as bean kind (entity, SFSB, SLSB), transactions consistency during failover, solving dependant object aliasing issue for EJB 2.0 entity beans, ... are not taken in consideration. Cheers, Sacha -Message d'origine- De : [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]De la part de marc fleury Envoy : mercredi, 7 mars 2001 01:27 : jBoss Developer Objet : [jBoss-Dev] CL: Clustering, let's get started Ok, it's been promised, it's been discussed, it has already started, it is really wanted. Also it will put JBoss on the map for ever... The clustering. I would like to "officially" call the season for clustering "open". There have been some discussions already some public some private and I believe we should really take this take by step. So we won't shoot for the stars in the very beginning, just something simple. I will assume the interim leadership for the implementation since there is no clear lead yet and the team is yet to form. I invite all of you that have touched the subject to publicly post and explain your view. STEP ONE: -Start 2 servers do "round robin" on STATELESS SESSION. Simple right? That will be some work go! marc _ Marc Fleury, Ph.D [EMAIL PROTECTED] _
RE: [jBoss-Dev] CL: Clustering, let's get started
Ok, time to be more precise... Assume a client talking directly to the bean, and load balance SSB. When THAT is done we will get B2B calls that need DTM. XP methodology. Don't stop. We know all that :) YES trivially if you deploy to n nodes, and you do nothing about "keep beans of an app in one node" and you do B2B, then you need a DTM. In clear I specifically want to avoid what mr cook is talking about. 10k feet "approach" talk. We need this and that oh but first bla bla.. XP! If DTM is your thing then please work on it since we will need it at the end. But in terms of approach I want to think of SSB direct clustering :) marc |-Original Message- |From: [EMAIL PROTECTED] |[mailto:[EMAIL PROTECTED]]On Behalf Of Jeremy Boynes |Sent: Wednesday, March 07, 2001 1:35 AM |To: [EMAIL PROTECTED] |Subject: RE: [jBoss-Dev] CL: Clustering, let's get started | | |Tom's question was really about approach rather than solutions and you did |ask for views... You might choose a "simple" scenario such as pure SSBs |first, but I tend to agree with Tom that you should consider the core |building blocks (e.g. DTM, cluster membership, resource location, state |replication, and many more) whilst doing so as in reality none of the |scenarios are actually that simple. | | |As for applicability of a DTM, one of the goals of a cluster is location |transparency (at least to the application). So consider a cluster with two |nodes, 1 2, and two SSBs A B; SSB A calls a method on SSB B; SSB A is |deployed to node 1, SSB B to node 2. | |So if a client invokes SSB A, it is directed by availability to node 1. SSB |A calls SSB B and is directed by availability to node 2. You now have more |than one node involved which requires a distributed transaction. | |Now make it more complex, deploy SSB A on both node 1 and node 2 so the |client could be load balanced to either node. If it goes to node 1 then you |have a distributed transaction as above whereas if it is directed to node 2 |then the transaction is local to node 2. | |Take it further and deploy both beans on both nodes. The client can get |load-balanced to either node; in theory so could the inter-bean |call but due |to the overhead of a remote call + DTM you would prefer it was directed to |the local machine. | |However, suppose SSB B becomes temporarily unavailable on one node for some |reason; availability would require that the inter-bean call is now directed |to the surviving node. The determination of which node will service the |request is dynamic, based on availability, and as a result so is the |determination of whether the transaction is local or distributed. | |If you plan to support scenarios beyond the most basic, you *will* need a |DTM so design it in early. | |Jeremy | |-Original Message- |From: [EMAIL PROTECTED] |[mailto:[EMAIL PROTECTED]]On Behalf Of marc fleury |Sent: Tuesday, March 06, 2001 7:18 PM |To: JBoss-Dev |Subject: RE: [jBoss-Dev] CL: Clustering, let's get started | | |what does a dtm have to do with Stateless session bean load balancing and |high availability | |? | |marc | |
RE: [jBoss-Dev] CL: Clustering, let's get started
-Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Jeremy Boynes Sent: 07 March 2001 06:35 To: [EMAIL PROTECTED] Subject: RE: [jBoss-Dev] CL: Clustering, let's get started Tom's question was really about approach rather than solutions and you did ask for views... You might choose a "simple" scenario such as pure SSBs first, but I tend to agree with Tom that you should consider the core building blocks (e.g. DTM, cluster membership, resource location, state replication, and many more) whilst doing so as in reality none of the scenarios are actually that simple. As for applicability of a DTM, one of the goals of a cluster is location transparency (at least to the application). So consider a cluster with two nodes, 1 2, and two SSBs A B; SSB A calls a method on SSB B; SSB A is deployed to node 1, SSB B to node 2. My definition of application server clustering does not allow a situation as you just described Jeremy. The beans (actually the application) would be deployed to the "cluster" and not to any particular node (or nodes) in a cluster. On an implemenatation level, I would still expect all nodes to host all applications once deployed. This removes the possibility of any application being unavailable even though a cluster is atill active. This could happen if either node 1 or node 2 failed in your example. Also I believe we need to think about the jBoss clustering effort in the wider context of clustering complete J2EE applications. Perhaps I am missing something here but shouldn't think about how a cluster-capable jBoss affects jBoss-Tomcat and/or jBoss-Jetty and other such (aggregated) J2EE stacks?. My Use Cases for a "JBoss Cluster User" includes : - Cluster Management: define/change the set of nodes (JBoss only or JBoss_+_Tomact/Jetty/etc) that belong together as a cluster - Application Deployment: deploy a JAR, WAR, EAR e.t.c on a node (any node?) in a jBoss-Tomcat (or jBoss-Jetty) cluster and have the application automagically deployed (and load-balanced) across all the nodes in the cluster. I presume that there would be a deployment tool for this purpose and that hot-deploy may or may not work (it would be great if it did though). Issues: - WHAT IS DEFINED AS A NODE IN THE CLUSTER? a) A physical computer running jBoss ? A cluster is then a named set of physical computers. This raises questions such as "must all instances (i.e. on different port nos) of jBoss on these computers be automatically clustered?". b) A specific instance (identified by port no) of JBoss running on a specified port on a physical computer?. A cluster may be all instances of jBoss running on the same port on a specified set of physical computers. If the instances are truly isolated, one can run a dev, test and staging cluster on the same set of physical computers for instance. I use multiple instances in this way with Oracle and SQL Server currently. - WHICH APPLICATION RESOURCES CAN BE CLUSTERED? * EJB Jars only ? * WAR, EAR ?? * other files ??? Cheers!, Micheal
RE: [jBoss-Dev] CL: Clustering, let's get started
Hello, My definition of application server clustering does not allow a situation as you just described Jeremy. The beans (actually the application) would be deployed to the "cluster" and not to any particular node (or nodes) in a cluster. On an implemenatation level, I would still expect all nodes to host all applications once deployed. This removes the possibility of any application being unavailable even though a cluster is atill active. This could happen if either node 1 or node 2 failed in your example. You are right, we need to set some terminology. IMO, we should be able to define a cluster has a set of Jboss instances (eg. A B C and D), we do not care if they are located on the same machine or not i.e. we may have a cluster of two instances running on the same machine. When we deploy an application, we can decide on which node of the cluster it should be running (A B only for example). In Borland tool, it is even possible to say, for example, that Application A1 should be working on at least 2 nodes, starting with nodes A and B but never on node D. Consequently, from the outside world, the cluster is seen as a single entity, but from the inside, we can decide what should run where. Also I believe we need to think about the jBoss clustering effort in the wider context of clustering complete J2EE applications. Perhaps I am missing something here but shouldn't think about how a cluster-capable jBoss affects jBoss-Tomcat and/or jBoss-Jetty and other such (aggregated) J2EE stacks?. Yes, it is a good idea, we could then share the administration tool for the presentation tier, but in fact, IMHO, this is not direclty related (but could gain in performance if again integrated). Cheers, Sacha
RE: [jBoss-Dev] CL: Clustering, let's get started
Lots of good feedback on the Clustering issue. Anybody thought about putting together a requirements document that captures these ideas and defines "The Path" for the effort? Does not have to be fancy - I would volunteer but don't have the chops (domain expertise). -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Micheal J Sent: Wednesday, March 07, 2001 6:41 AM To: JBoss-Dev Subject: RE: [jBoss-Dev] CL: Clustering, let's get started -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Jeremy Boynes Sent: 07 March 2001 06:35 To: [EMAIL PROTECTED] Subject: RE: [jBoss-Dev] CL: Clustering, let's get started Tom's question was really about approach rather than solutions and you did ask for views... You might choose a "simple" scenario such as pure SSBs first, but I tend to agree with Tom that you should consider the core building blocks (e.g. DTM, cluster membership, resource location, state replication, and many more) whilst doing so as in reality none of the scenarios are actually that simple. As for applicability of a DTM, one of the goals of a cluster is location transparency (at least to the application). So consider a cluster with two nodes, 1 2, and two SSBs A B; SSB A calls a method on SSB B; SSB A is deployed to node 1, SSB B to node 2. My definition of application server clustering does not allow a situation as you just described Jeremy. The beans (actually the application) would be deployed to the "cluster" and not to any particular node (or nodes) in a cluster. On an implemenatation level, I would still expect all nodes to host all applications once deployed. This removes the possibility of any application being unavailable even though a cluster is atill active. This could happen if either node 1 or node 2 failed in your example. Also I believe we need to think about the jBoss clustering effort in the wider context of clustering complete J2EE applications. Perhaps I am missing something here but shouldn't think about how a cluster-capable jBoss affects jBoss-Tomcat and/or jBoss-Jetty and other such (aggregated) J2EE stacks?. My Use Cases for a "JBoss Cluster User" includes : - Cluster Management: define/change the set of nodes (JBoss only or JBoss_+_Tomact/Jetty/etc) that belong together as a cluster - Application Deployment: deploy a JAR, WAR, EAR e.t.c on a node (any node?) in a jBoss-Tomcat (or jBoss-Jetty) cluster and have the application automagically deployed (and load-balanced) across all the nodes in the cluster. I presume that there would be a deployment tool for this purpose and that hot-deploy may or may not work (it would be great if it did though). Issues: - WHAT IS DEFINED AS A NODE IN THE CLUSTER? a) A physical computer running jBoss ? A cluster is then a named set of physical computers. This raises questions such as "must all instances (i.e. on different port nos) of jBoss on these computers be automatically clustered?". b) A specific instance (identified by port no) of JBoss running on a specified port on a physical computer?. A cluster may be all instances of jBoss running on the same port on a specified set of physical computers. If the instances are truly isolated, one can run a dev, test and staging cluster on the same set of physical computers for instance. I use multiple instances in this way with Oracle and SQL Server currently. - WHICH APPLICATION RESOURCES CAN BE CLUSTERED? * EJB Jars only ? * WAR, EAR ?? * other files ??? Cheers!, Micheal
RE: [jBoss-Dev] CL: Clustering, let's get started
ok, let me raise my voice :) the emails around clustering seem to focus a lot on loadbalancing as the primary reason to cluster servers. well, imho it is not. there is only one reason you would cluster servers, fail-over and high availability. if you want to do loadbalancing, you have much better performance using non-clustered servers and a stateful loadbalancer. as a matter of fact, you can buy a load balancing router for a couple of thousand bucks, so why try to replicate functionality that is already done. even with weblogic, if you are serious about loadbalancing because you have so much traffic, you do it the hardware way. again this is in my humble opinion :) so the whole round robin thing? let's focus on the main issue. let's focus on the important things 1. how do we share data in the JNDI tree 2. how do we solve JNDI naming conflicts (ie two server nodes try to bind different objects with the same name) 3. pinned vs. unpinned server objects (unpinned would be the goal, that the remote stub would automatically reconnect to a different server node if the main node fails) again, if you have full clustering on the server, the load balancing router can reconnect you to a different node, without any client logic, but this piece of hardware will not always be in place) 4. cluster management/communication 5. deployment management, does every node in a cluster have the same deployment configuration, or will this be automatically replicated Am I completely out in the blue? please let me know. Filip ~ Namaste - I bow to the divine in you ~ Filip Hanik Software Architect [EMAIL PROTECTED] www.filip.net -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Stefan Siprell Sent: Wednesday, March 07, 2001 8:26 AM To: 'JBoss-Dev' Subject: AW: [jBoss-Dev] CL: Clustering, let's get started to put a tie around all of my complaining, I would also volunteer to create a dcoument describing the main ideas and concepts: -Round Robin (kind of low, but the one that would create working versions the fastest) -Bean-level Request Dispatching -CORBA infrastructure -and maybe JINI and SPACES (my little idea) Might be easier to discuss (if required at all) the future path. -Ursprungliche Nachricht- Von: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]Im Auftrag von Peter F. Spicer Gesendet: Mittwoch, 7. Marz 2001 16:39 An: JBoss-Dev Betreff: RE: [jBoss-Dev] CL: Clustering, let's get started Lots of good feedback on the Clustering issue. Anybody thought about putting together a requirements document that captures these ideas and defines "The Path" for the effort? Does not have to be fancy - I would volunteer but don't have the chops (domain expertise). -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Micheal J Sent: Wednesday, March 07, 2001 6:41 AM To: JBoss-Dev Subject: RE: [jBoss-Dev] CL: Clustering, let's get started -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Jeremy Boynes Sent: 07 March 2001 06:35 To: [EMAIL PROTECTED] Subject: RE: [jBoss-Dev] CL: Clustering, let's get started Tom's question was really about approach rather than solutions and you did ask for views... You might choose a "simple" scenario such as pure SSBs first, but I tend to agree with Tom that you should consider the core building blocks (e.g. DTM, cluster membership, resource location, state replication, and many more) whilst doing so as in reality none of the scenarios are actually that simple. As for applicability of a DTM, one of the goals of a cluster is location transparency (at least to the application). So consider a cluster with two nodes, 1 2, and two SSBs A B; SSB A calls a method on SSB B; SSB A is deployed to node 1, SSB B to node 2. My definition of application server clustering does not allow a situation as you just described Jeremy. The beans (actually the application) would be deployed to the "cluster" and not to any particular node (or nodes) in a cluster. On an implemenatation level, I would still expect all nodes to host all applications once deployed. This removes the possibility of any application being unavailable even though a cluster is atill active. This could happen if either node 1 or node 2 failed in your example. Also I believe we need to think about the jBoss clustering effort in the wider context of clustering complete J2EE applications. Perhaps I am missing something here but shouldn't think about how a cluster-capable jBoss affects jBoss-Tomcat and/or jBoss-Jetty and other such (aggregated) J2EE stacks?. My Use Cases for a "JBoss Cluster User" includes : - Cluster Management: define/change the set of nodes (JBoss only or JBoss_+_Tomact/Jetty/etc) that belong together as a cluster - Application Deployment: deploy a JAR, WAR, EAR e.t.c o
RE: [jBoss-Dev] CL: Clustering, let's get started
Hello, ok, let me raise my voice :) the emails around clustering seem to focus a lot on loadbalancing as the primary reason to cluster servers. well, imho it is not. there is only one reason you would cluster servers, fail-over and high availability. "only one reason" is probably a bit strong... It depends of your business: you may need high availability or/and you may need very high throughput while keeping the cluster state consistent without worring much about lost clients when a server dies. if you want to do loadbalancing, you have much better performance using non-clustered servers and a stateful loadbalancer. as a matter of fact, you can buy a load balancing router for a couple of thousand bucks, so why try to replicate functionality that is already done. even with weblogic, if you are serious about loadbalancing because you have so much traffic, you do it the hardware way. again this is in my humble opinion :) Yes sure, if you are serious about fail-over, you buy a 1M$ TPM with a Sun Enterprise 1 ;), SAN replicated datastore, Oracle DB cluster, ... ;) I think we need to focus on both aspects without imposing that 99% of JBoss user only want fail-over. It is not because some other products also offer this feature that we should not provide it! (Furthermore, I hadn't the feeling that we were only speaking about LB.) But I agree, load-balancing without HA is not very functionaly interesting whereas HA without load-balancing is a bit "unfinished work". so the whole round robin thing? let's focus on the main issue. focus status=ON target=MAIN_ISSUE let's focus on the important things focus status=ON target=IMPORTANT_THINGS 1. how do we share data in the JNDI tree I was suggesting to use a standard CORBA NS and extend it to be able to share the same binding name more than once. = The container knows the extended IDL and is able to register it correctly. = When a client asks to resolve a name, the NS build a multi-profile IOR (here comes the HA) thanks to a particular algorith. The algo could be :" put the profiles from the least loaded server first" or whatever plugged algorithm is selected. We may use the JavaGroups framework to replicate the different instances of extended NS and thus insure that all NS can receive Read and Write invocations while staying consistent. For the egg and chicken problem... we need to be able to give to the client the complete list of NS which are available if one fails : either with an multi-profil IOR (beurk) or with any other bootstraping system (don't have enough info at the moment) 2. how do we solve JNDI naming conflicts (ie two server nodes try to bind different objects with the same name) conflicts would be at the heart of the system! ;) 3. pinned vs. unpinned server objects (unpinned would be the goal, that the remote stub would automatically reconnect to a different server node if the main node fails) again, if you have full clustering on the server, the load balancing router can reconnect you to a different node, without any client logic, but this piece of hardware will not always be in place) I think we need to develop an hardware-less solution as much as possible. For the stub, this would work for Java client but with standard CORBA client, it wouldn't = multi-profile IOR idea. 4. cluster management/communication base on JavaGroups? no real idea at the moment... 5. deployment management, does every node in a cluster have the same deployment configuration, or will this be automatically replicated I suggested to be able to choose, in the cluster, to which node an application is deployed (subset) /focus /focus Am I completely out in the blue? please let me know. No or we are two ;) Cheers, Sacha
RE: [jBoss-Dev] CL: Clustering, let's get started
Maybe you are addressing Marc's concern about finding a small enough bite? Clustering focused on fail-over first, then load-balancing? I believe so, thanks for you humble respons, having a rough day, hope I didn't bitch too much :) Filip ~ Namaste - I bow to the divine in you ~ Filip Hanik Software Architect [EMAIL PROTECTED] www.filip.net
Re: [jBoss-Dev] CL: Clustering, let's get started
Filip wrote: if you want to do loadbalancing, you have much better performance using non-clustered servers and a stateful loadbalancer. as a matter of fact, you can buy a load balancing router for a couple of thousand bucks, so why try to replicate functionality that is already done. even with weblogic, if you are serious about loadbalancing because you have so much traffic, you do it the hardware way. again this is in my humble opinion :) Sasha wrote: I think we need to develop an hardware-less solution as much as possible. My $.02 Contributed in the spirit of: "Are we sure we have a problem, and are we sure it hasn't already been "fixed"? A few weeks ago, Marc posted a message about the "Sys Admin" culture and how important it is to understand these guys if you're marketing to them. I just toured the hardware room of a fairly large Texas company developing large-scale internet applications. I know they're familiar with WL - they might even be using it. The place was stacked with Sun "pizza boxes", larger servers, and a rack full of (vendor name deleted - they're much too big and overpriced) guess what... load balancing routers. It's what these guys understand. If it "breaks", they replace the box. Tom
RE: [jBoss-Dev] CL: Clustering, let's get started
Cool. I haven't really the first idea where to start on it, but I do have time now, so I'll start looking into DTMs.. anyone else out there looking? Tom On Wed, 7 Mar 2001, marc fleury wrote: Ok, time to be more precise... Assume a client talking directly to the bean, and load balance SSB. When THAT is done we will get B2B calls that need DTM. XP methodology. Don't stop. We know all that :) YES trivially if you deploy to n nodes, and you do nothing about "keep beans of an app in one node" and you do B2B, then you need a DTM. In clear I specifically want to avoid what mr cook is talking about. 10k feet "approach" talk. We need this and that oh but first bla bla.. XP! If DTM is your thing then please work on it since we will need it at the end. But in terms of approach I want to think of SSB direct clustering :) marc |-Original Message- |From: [EMAIL PROTECTED] |[mailto:[EMAIL PROTECTED]]On Behalf Of Jeremy Boynes |Sent: Wednesday, March 07, 2001 1:35 AM |To: [EMAIL PROTECTED] |Subject: RE: [jBoss-Dev] CL: Clustering, let's get started | | |Tom's question was really about approach rather than solutions and you did |ask for views... You might choose a "simple" scenario such as pure SSBs |first, but I tend to agree with Tom that you should consider the core |building blocks (e.g. DTM, cluster membership, resource location, state |replication, and many more) whilst doing so as in reality none of the |scenarios are actually that simple. | | |As for applicability of a DTM, one of the goals of a cluster is location |transparency (at least to the application). So consider a cluster with two |nodes, 1 2, and two SSBs A B; SSB A calls a method on SSB B; SSB A is |deployed to node 1, SSB B to node 2. | |So if a client invokes SSB A, it is directed by availability to node 1. SSB |A calls SSB B and is directed by availability to node 2. You now have more |than one node involved which requires a distributed transaction. | |Now make it more complex, deploy SSB A on both node 1 and node 2 so the |client could be load balanced to either node. If it goes to node 1 then you |have a distributed transaction as above whereas if it is directed to node 2 |then the transaction is local to node 2. | |Take it further and deploy both beans on both nodes. The client can get |load-balanced to either node; in theory so could the inter-bean |call but due |to the overhead of a remote call + DTM you would prefer it was directed to |the local machine. | |However, suppose SSB B becomes temporarily unavailable on one node for some |reason; availability would require that the inter-bean call is now directed |to the surviving node. The determination of which node will service the |request is dynamic, based on availability, and as a result so is the |determination of whether the transaction is local or distributed. | |If you plan to support scenarios beyond the most basic, you *will* need a |DTM so design it in early. | |Jeremy | |-Original Message- |From: [EMAIL PROTECTED] |[mailto:[EMAIL PROTECTED]]On Behalf Of marc fleury |Sent: Tuesday, March 06, 2001 7:18 PM |To: JBoss-Dev |Subject: RE: [jBoss-Dev] CL: Clustering, let's get started | | |what does a dtm have to do with Stateless session bean load balancing and |high availability | |? | |marc | | -- "If you mess with something for long enough it will break." - Schmidt
Re: [jBoss-Dev] CL: Clustering, let's get started
Hmmm, game... Would it be better to do it piecemeal like this, developing a bit at a time, or to implement the underlying infrastructure first (like a distributed transaction manager)? Tom On Tue, 6 Mar 2001, marc fleury wrote: Ok, it's been promised, it's been discussed, it has already started, it is really wanted. Also it will put JBoss on the map for ever... The clustering. I would like to "officially" call the season for clustering "open". There have been some discussions already some public some private and I believe we should really take this take by step. So we won't shoot for the stars in the very beginning, just something simple. I will assume the interim leadership for the implementation since there is no clear lead yet and the team is yet to form. I invite all of you that have touched the subject to publicly post and explain your view. STEP ONE: -Start 2 servers do "round robin" on STATELESS SESSION. Simple right? That will be some work go! marc _ Marc Fleury, Ph.D [EMAIL PROTECTED] _ -- "If you mess with something for long enough it will break." - Schmidt
RE: [jBoss-Dev] CL: Clustering, let's get started
what does a dtm have to do with Stateless session bean load balancing and high availability ? marc |-Original Message- |From: [EMAIL PROTECTED] |[mailto:[EMAIL PROTECTED]]On Behalf Of Tom Cook |Sent: Tuesday, March 06, 2001 8:16 PM |To: JBoss-Dev |Subject: Re: [jBoss-Dev] CL: Clustering, let's get started | | |Hmmm, game... | |Would it be better to do it piecemeal like this, developing a bit at a |time, or to implement the underlying infrastructure first (like a |distributed transaction manager)? | |Tom | |On Tue, 6 Mar 2001, marc fleury wrote: | | Ok, it's been promised, it's been discussed, it has already |started, it is | really wanted. | | Also it will put JBoss on the map for ever... | | The clustering. | | I would like to "officially" call the season for clustering "open". | | There have been some discussions already some public some private and I | believe we should really take this take by step. So we won't |shoot for the | stars in the very beginning, just something simple. | | I will assume the interim leadership for the implementation |since there is | no clear lead yet and the team is yet to form. I invite all of you that | have touched the subject to publicly post and explain your view. | | STEP ONE: | -Start 2 servers do "round robin" on STATELESS SESSION. | | Simple right? That will be some work | | go! | | marc | | _ | Marc Fleury, Ph.D | [EMAIL PROTECTED] | _ | | | | |-- |"If you mess with something for long enough it will break." - Schmidt | |
RE: [jBoss-Dev] CL: Clustering, let's get started
Tom's question was really about approach rather than solutions and you did ask for views... You might choose a "simple" scenario such as pure SSBs first, but I tend to agree with Tom that you should consider the core building blocks (e.g. DTM, cluster membership, resource location, state replication, and many more) whilst doing so as in reality none of the scenarios are actually that simple. As for applicability of a DTM, one of the goals of a cluster is location transparency (at least to the application). So consider a cluster with two nodes, 1 2, and two SSBs A B; SSB A calls a method on SSB B; SSB A is deployed to node 1, SSB B to node 2. So if a client invokes SSB A, it is directed by availability to node 1. SSB A calls SSB B and is directed by availability to node 2. You now have more than one node involved which requires a distributed transaction. Now make it more complex, deploy SSB A on both node 1 and node 2 so the client could be load balanced to either node. If it goes to node 1 then you have a distributed transaction as above whereas if it is directed to node 2 then the transaction is local to node 2. Take it further and deploy both beans on both nodes. The client can get load-balanced to either node; in theory so could the inter-bean call but due to the overhead of a remote call + DTM you would prefer it was directed to the local machine. However, suppose SSB B becomes temporarily unavailable on one node for some reason; availability would require that the inter-bean call is now directed to the surviving node. The determination of which node will service the request is dynamic, based on availability, and as a result so is the determination of whether the transaction is local or distributed. If you plan to support scenarios beyond the most basic, you *will* need a DTM so design it in early. Jeremy -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of marc fleury Sent: Tuesday, March 06, 2001 7:18 PM To: JBoss-Dev Subject: RE: [jBoss-Dev] CL: Clustering, let's get started what does a dtm have to do with Stateless session bean load balancing and high availability ? marc
