Re: [JBoss-dev] Re: [jetty-discuss] HTTP loadbalancing
MMM all is not well in the state of java.nio! At least on linux, I'm getting lots of strange behaviour, which I suspect are bugs in the implementation or my understanding of it. Firstly it appears that the loadbalancer only works if debug is turned on. With debug turned off, it runs faster and I start seeing the following problems: + closing of a channel is not seen by the other end (client or server) until it tries to write back. + If such a lingering close happens, then following connections fail. the server reports a new connection, but the finishConnect() method returns false. It would be good if somebody could try this stuff on a win32 platform to see if it is well behaived there?? I have put a bit more failure handling in. The mickey mouse policy now keeps trying servers until they have all failed. Also server failure undoes InetAddress stickyness. cheers Greg Wilkins wrote: I have now added client InetAddress stickyness, which is probably going to be good enough for 90% of cases. But for those that want session stickyness, I have now allow delayed server allocation and pass the queued data into the Policy. This will allow a search for jsessionId up until the queue is full (at which time you had better allocated or you will just block). Only thing lacking is a monitor of the response channel so you can pick up session cookie sets. I'm now going to stop on the balancing policy and concentrate on the network stuff. Still no decent exception handling. There is also a funny in the nio (at least on Linux), where I close the Channel, the socket, the inputstream, the outputstream, etc. and the client still does not see the closed connection until it tries to write another packet! Note good for HTTP/1.0! This only happens sometimes - and never in a test harness!?!?!?!?! cheers -- Greg Wilkins[EMAIL PROTECTED] GB Phone: +44-(0)7092063462 Mort Bay Consulting Australia and UK.Mbl Phone: +61-(0)4 17786631 http://www.mortbay.com AU Phone: +61-(0)2 98107029 ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
Re: [JBoss-dev] Re: [jetty-discuss] HTTP loadbalancing
Bill Burke wrote:
I'll be your buddy
OK Bud!
I've just checked in a skeleton load balancer into
Jetty/src1.4/org/mortbay/loadbalancer
It has all the nio network plumbing for two way connections with
flow control - but not very good exception handling at the moment.
A trivial round robin policy is used:
public class Policy
{
private Server[] _server;
private int next;
public Policy(Server[] server)
{
_server=server;
}
public void allocate(Connection connection)
throws IOException
{
next=(next+1)%_server.length;
connection.allocate(_server[next]);
}
}
Note that this Policy class does not even have the API to do
anything real can't look for jsessionid and fiddly to get the
InetAddress of the client but it is at least a start.
The nio stuff is really rather cool - I think I may take some
timeout from load balancing to write SocketChannelListener for
Jetty, which should be a big winner for scalability!!!
So Bill or other Buddies, please have a look at the code and we
can talk Policies soon!
I use the org.mortbay.tools.Tenlet (telnet backwards) class
as a sample server - so I can type back at a telnet client.
to run with 1 listener on 8123 and two servers you say:
java org.mortbay.loadbalancer.Balancer :8123 - 127.0.0.1: 127.0.0.1:
remember this is a release-early-release-often piece of code... be gentle :-)
cheers
PS. If another buddy wants to get in on the scene - we have to make the
thing JMX manageable!
cheers
--
Greg Wilkins[EMAIL PROTECTED] GB Phone: +44-(0)7092063462
Mort Bay Consulting Australia and UK.Mbl Phone: +61-(0)4 17786631
http://www.mortbay.com AU Phone: +61-(0)2 98107029
___
Jboss-development mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/jboss-development
Re: [JBoss-dev] Re: [jetty-discuss] HTTP loadbalancing
I have now added client InetAddress stickyness, which is probably going to be good enough for 90% of cases. But for those that want session stickyness, I have now allow delayed server allocation and pass the queued data into the Policy. This will allow a search for jsessionId up until the queue is full (at which time you had better allocated or you will just block). Only thing lacking is a monitor of the response channel so you can pick up session cookie sets. I'm now going to stop on the balancing policy and concentrate on the network stuff. Still no decent exception handling. There is also a funny in the nio (at least on Linux), where I close the Channel, the socket, the inputstream, the outputstream, etc. and the client still does not see the closed connection until it tries to write another packet! Note good for HTTP/1.0! This only happens sometimes - and never in a test harness!?!?!?!?! cheers -- Greg Wilkins[EMAIL PROTECTED] GB Phone: +44-(0)7092063462 Mort Bay Consulting Australia and UK.Mbl Phone: +61-(0)4 17786631 http://www.mortbay.com AU Phone: +61-(0)2 98107029 ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
RE: [JBoss-dev] Re: [jetty-discuss] HTTP loadbalancing
Hello, I fully agree with Greg's message. - We need to do it in pure Java - we don't need to be 100% sticky (99.99% is enough) - Let's make a first version that don't have fancy things but, at least, that work! - java.nio is the way (plus giving a doc about tuning the TCP stack such as TIME_WAIT values and so on) Cheers, Sacha -Message d'origine- De : [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]De la part de Greg Wilkins Envoye : vendredi, 8 mars 2002 03:35 A : [EMAIL PROTECTED] Cc : [EMAIL PROTECTED] Objet : [JBoss-dev] Re: [jetty-discuss] HTTP loadbalancing While using DNS may well be a naive, I think it is more naive to think that user space code running on a general purpose computer is a good solution if you are concerned about load and/or failover. If you really want performance and reliability, then firmware is the solution for you and the 25K will do a little to help my Cisco shares :-) However, I do think we need a reasonable solution that works as a reasonable introduction to loadbalancing and failover. If my understanding of the clustering / session manager is correct, it does not really matter if sessions are not 100% sticky. The clustering stuff will cope, but it is just more efficient if it does not need to move everything to a new node. So this is the approach that I would like to take for this: + Write it in java, because we want to be 100% pure. If that is not fast enough - buy Cisco ( I will declare an interest here as Cisco are the prime sponsor of Jetty development !-) + Use java.nio as: * if you care about performance you should be using jdk1.4 anyway * if you want one machine to handle more connections than the sum of all your servers, then java.net.ServerSocket aint going to do the job! * I want to play with nio before I write the NioListener for Jetty! * Allow sticky sessions, sticky IP and sticky host options. * If sticky sessions are used, or all new connections look for jsessionid= in the first packet of the request and the response. If found, associate the ID with the current connecton allocation. * If sticky IP is set, use a hashmap of the source InetAddress to determine the connection association. * If sticky host option is set, use hashmap on InetAddress.getCanonicalHostName() to determine the connection association. * If multiple options are set, they are tried in the order they we defined until an association is found. * Stickyness applies to a listener, so you can have different settings for http and https ports. * If you have a new connection, an association to a server will be made via a pluggable balance policy. The default will be round robin or least connections. * If a connections are only re-associated as they break or fail to connect to a server. + A failing server will continue to be tried for new connections according to the balancing policy - so that it can come back into the fold. A fancy balancing policy may wish to exclude a failed server for a period of time, or until further notice. + At this stage, there will be no re-balancing of existing associations if the load becomes asymmetric due to random chance (or any other reason). How does that sound? It should at least be a start for load balancing and failover. I should have something working like this within a day or a month (Now there is a broad estimate for you). I'm happy to do this myself, but would also appreciate a keen buddy on this once the skeleton is in place. cheers Bill Burke wrote: I was looking at the way Apache+Tomcat works. They use a protocol between themselves. Apache decrypts the SSL message and extracts sticky session information as well as SSL information and passes this over a load-balanced socket over a proprietary protocol to the Tomcat instance. The message contains the HTTP request as well as all the SSL information. Tomcat unpacks the message, creates an HttpRequest, and sends the request on their local bus. I guess they call this a Tomcat Connector. --- Marc, I see no reason why you have to be separate from Apache. The issues here aren't trivial and they've already done all the work and they're open-source. Actually their license is much more liberal than LGPL. Plus for this kind of stuff you probably want something written in C for speed as Greg suggested. Jetty already leverages Jasper, why not mod_jk? --- As far as load-balancing based on client IP and AOL, it is even worse than you think. Supposedly, a client's IP can change in the middle of the session.(I said supposedly remember ;) You acutally have to stick on a mask of the IP. Sure, client IP isn't the best
RE: [JBoss-dev] Re: [jetty-discuss] HTTP loadbalancing
Cool! I hope nobody took offense to any of my comments! I was hoping to generate a discussion exactly like this! More comments follow... While using DNS may well be a naive, I think it is more naive to think that user space code running on a general purpose computer is a good solution if you are concerned about load and/or failover. If you really want performance and reliability, then firmware is the solution for you and the 25K will do a little to help my Cisco shares :-) FYI, I've had 2 different companies ask for a open-source solution for this that had tight budgets. I'd be really curious to compare a load-balancer running on a general purpose computer with a cisco switch. Anybody ever done this comparison? However, I do think we need a reasonable solution that works as a reasonable introduction to loadbalancing and failover. * If sticky sessions are used, or all new connections look for jsessionid= in the first packet of the request and the response. If found, associate the ID with the current connecton allocation. Is the jsessionid available through the URL in HTTPS requests? How does that sound? It should at least be a start for load balancing and failover. I should have something working like this within a day or a month (Now there is a broad estimate for you). I'm happy to do this myself, but would also appreciate a keen buddy on this once the skeleton is in place. I'll be your buddy ;). Hey, Let's talk about it at JBossOne? In the meantime, this might be a good source for requirements: http://www.cisco.com/univercd/cc/td/doc/pcat/11150.htm Regards, Bill ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
Re: [JBoss-dev] Re: [jetty-discuss] HTTP loadbalancing
This is maybe a bit offtopic, and a bit long, but this is one of those serindipitous occasions where a conversation strikes on something somebody else is already working on... Greg Wilkins wrote: + Use java.nio as: * if you care about performance you should be using jdk1.4 anyway * if you want one machine to handle more connections than the sum of all your servers, then java.net.ServerSocket aint going to do the job! * I want to play with nio before I write the NioListener for Jetty! Hmm. I've been writing a sort of general purpose TCP server using the NIO stuff that I'd actually rather like to try grafting onto the front end of Jetty, if you don't mind. Bear in mind that this is fairly early work yet, and there are probably lots of holes. Basically, the way it's structure is there is a Server (MBean) which contains listeners (one for each declared hostname:port). The Server has a single thread that uses an NIO selector to find out when it needs to accept on any of them. The resulting SocketChannels are handed off to a ReaderManager, which chooses a ReaderThread to handle that channel. Each reader thread selects on all of its channels, reads data and delegates to a Protocol implementation to build a Request and put appropriate data into it. Once the request gets to the state HEADERS_READ (the headers have all been read and there is enough there to be dispatched) the Reader thread hands gets a worker thread from a pool and hands the Request off to a ContextManager which uses information from the request to dispatch to the appropriate RequestHandler stack (which is where I _think_ Jetty would come in). When the request needs to be written to its output (because it's buffer is full, or the request has been completely handled) it (carrying its SocketChannel) gets handed off/hands itself to a WriterManager, which chooses a WriteThread to handle it in a similiar non-blocking fashion as the ReaderThread. Note that a POST request (or an SMTP command, or a JBossMQ message on a new protocol, or anything that can distinguish header/envelope from body/payload) can be dispatched and processing can begin while the request body is still being read. Likewise on the output: if the buffer is full, it will start to be written while processing continues. ReaderThreads hold onto SocketChannels until they're closed (to enable keep alive), WriterThreads do not. Direct Byte Buffers are used and pooled wherever possible, allowing the underlying implementation to optimize transfers. I think the biggest advantage here is the level of control it gives administrators over thread usage: you can configure read, write, and work pools separately. One of the things people notice in JBoss under linux is the number of threads used - they'd notice it under Windows or any commercial Unix as well if they looked, it's just that Linux puts its threads right out there for you to see and freak out on in ps and top et. al. Another thought I've had is to use this same framework as the basis for a new invoker for JBoss, so that remote client invocations don't put the server at the mercy of the whims of RMI's thread usage. thoughts, chortels, flames? thanks for your time, danch ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
Re: [JBoss-dev] Re: [jetty-discuss] HTTP loadbalancing
Bill Burke wrote: Cool! I hope nobody took offense to any of my comments! I was hoping to generate a discussion exactly like this! More comments follow... There have been several requests for something like this over the last year or so - this discussion was good to get me to actually start doing something. While using DNS may well be a naive, I think it is more naive to think that user space code running on a general purpose computer is a good solution if you are concerned about load and/or failover. If you really want performance and reliability, then firmware is the solution for you and the 25K will do a little to help my Cisco shares :-) However, I do think we need a reasonable solution that works as a reasonable introduction to loadbalancing and failover. * If sticky sessions are used, or all new connections look for jsessionid= in the first packet of the request and the response. If found, associate the ID with the current connecton allocation. Is the jsessionid available through the URL in HTTPS requests? In HTTPS everything including the URL is encrypted. The balancer will only see a stream of random bytes. I'll be your buddy ;). Hey, Let's talk about it at JBossOne? Well I hope to have it working before JBossOne. I'm close to getting the nio stuff working one way (Client to Server) and I should get the reverse path working shortly. So hopefully next week I'll be able to check in the network infrastructure which we can then work with to make a flexible/robust/simple loadbalancer! NB. I have previously bagged out nio on the Jetty-list- I think I will have to take a lot of that back (other than my complaint about anything with n in the name). It is rather cool and does have good links back to the old blocking stuff In fact a ChannelListener is now high on my priority list for Jetty. So another suggestion to the JBoss folks that you get the build environment able to optionally compile 1.4 based classes for these new bells and whistles! In the meantime, this might be a good source for requirements: http://www.cisco.com/univercd/cc/td/doc/pcat/11150.htm thanks -- Greg Wilkins[EMAIL PROTECTED] GB Phone: +44-(0)7092063462 Mort Bay Consulting Australia and UK.Mbl Phone: +61-(0)4 17786631 http://www.mortbay.com AU Phone: +61-(0)2 98107029 ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
[JBoss-dev] Re: [jetty-discuss] HTTP loadbalancing
While using DNS may well be a naive, I think it is more naive to think that user space code running on a general purpose computer is a good solution if you are concerned about load and/or failover. If you really want performance and reliability, then firmware is the solution for you and the 25K will do a little to help my Cisco shares :-) However, I do think we need a reasonable solution that works as a reasonable introduction to loadbalancing and failover. If my understanding of the clustering / session manager is correct, it does not really matter if sessions are not 100% sticky. The clustering stuff will cope, but it is just more efficient if it does not need to move everything to a new node. So this is the approach that I would like to take for this: + Write it in java, because we want to be 100% pure. If that is not fast enough - buy Cisco ( I will declare an interest here as Cisco are the prime sponsor of Jetty development !-) + Use java.nio as: * if you care about performance you should be using jdk1.4 anyway * if you want one machine to handle more connections than the sum of all your servers, then java.net.ServerSocket aint going to do the job! * I want to play with nio before I write the NioListener for Jetty! * Allow sticky sessions, sticky IP and sticky host options. * If sticky sessions are used, or all new connections look for jsessionid= in the first packet of the request and the response. If found, associate the ID with the current connecton allocation. * If sticky IP is set, use a hashmap of the source InetAddress to determine the connection association. * If sticky host option is set, use hashmap on InetAddress.getCanonicalHostName() to determine the connection association. * If multiple options are set, they are tried in the order they we defined until an association is found. * Stickyness applies to a listener, so you can have different settings for http and https ports. * If you have a new connection, an association to a server will be made via a pluggable balance policy. The default will be round robin or least connections. * If a connections are only re-associated as they break or fail to connect to a server. + A failing server will continue to be tried for new connections according to the balancing policy - so that it can come back into the fold. A fancy balancing policy may wish to exclude a failed server for a period of time, or until further notice. + At this stage, there will be no re-balancing of existing associations if the load becomes asymmetric due to random chance (or any other reason). How does that sound? It should at least be a start for load balancing and failover. I should have something working like this within a day or a month (Now there is a broad estimate for you). I'm happy to do this myself, but would also appreciate a keen buddy on this once the skeleton is in place. cheers Bill Burke wrote: I was looking at the way Apache+Tomcat works. They use a protocol between themselves. Apache decrypts the SSL message and extracts sticky session information as well as SSL information and passes this over a load-balanced socket over a proprietary protocol to the Tomcat instance. The message contains the HTTP request as well as all the SSL information. Tomcat unpacks the message, creates an HttpRequest, and sends the request on their local bus. I guess they call this a Tomcat Connector. --- Marc, I see no reason why you have to be separate from Apache. The issues here aren't trivial and they've already done all the work and they're open-source. Actually their license is much more liberal than LGPL. Plus for this kind of stuff you probably want something written in C for speed as Greg suggested. Jetty already leverages Jasper, why not mod_jk? --- As far as load-balancing based on client IP and AOL, it is even worse than you think. Supposedly, a client's IP can change in the middle of the session.(I said supposedly remember ;) You acutally have to stick on a mask of the IP. Sure, client IP isn't the best way, but it was the only thing us ignorant bloaks knew at the time :) -Original Message- From: Greg Wilkins [mailto:[EMAIL PROTECTED]] Sent: Thursday, March 07, 2002 6:05 AM To: [EMAIL PROTECTED] Subject: Re: [jetty-discuss] HTTP loadbalancing Chris, Your comments are correct (and user are the only ones who should post to jetty-discuss!-) There are two common solutions: As the load balancer is in your organization then you can give it the server certificate and it can terminate the SSL connection. The forwarded connection is then done using standard HTTP or some other protocol. It is not over the internet, so it is moderately secure (depending on the organizations network setup). The main problem with this setup, is that the server knows nothing about the SSL side of things.
Re: [JBoss-dev] Re: [jetty-discuss] HTTP loadbalancing
This looks like a great plan. Don't forget to include a way to transition a sticky session from http to https. If I remember correctly, Cisco supports this by having an additional ip address added to css switch for each child web server. When the session is transitioned from http to https you redirect to one of the specific ip addresses. This is another of the projects I would love to do my self, but I'm to busy on the JBossCMP stuff. Have fun, -dain Greg Wilkins wrote: While using DNS may well be a naive, I think it is more naive to think that user space code running on a general purpose computer is a good solution if you are concerned about load and/or failover. If you really want performance and reliability, then firmware is the solution for you and the 25K will do a little to help my Cisco shares :-) However, I do think we need a reasonable solution that works as a reasonable introduction to loadbalancing and failover. If my understanding of the clustering / session manager is correct, it does not really matter if sessions are not 100% sticky. The clustering stuff will cope, but it is just more efficient if it does not need to move everything to a new node. So this is the approach that I would like to take for this: + Write it in java, because we want to be 100% pure. If that is not fast enough - buy Cisco ( I will declare an interest here as Cisco are the prime sponsor of Jetty development !-) + Use java.nio as: * if you care about performance you should be using jdk1.4 anyway * if you want one machine to handle more connections than the sum of all your servers, then java.net.ServerSocket aint going to do the job! * I want to play with nio before I write the NioListener for Jetty! * Allow sticky sessions, sticky IP and sticky host options. * If sticky sessions are used, or all new connections look for jsessionid= in the first packet of the request and the response. If found, associate the ID with the current connecton allocation. * If sticky IP is set, use a hashmap of the source InetAddress to determine the connection association. * If sticky host option is set, use hashmap on InetAddress.getCanonicalHostName() to determine the connection association. * If multiple options are set, they are tried in the order they we defined until an association is found. * Stickyness applies to a listener, so you can have different settings for http and https ports. * If you have a new connection, an association to a server will be made via a pluggable balance policy. The default will be round robin or least connections. * If a connections are only re-associated as they break or fail to connect to a server. + A failing server will continue to be tried for new connections according to the balancing policy - so that it can come back into the fold. A fancy balancing policy may wish to exclude a failed server for a period of time, or until further notice. + At this stage, there will be no re-balancing of existing associations if the load becomes asymmetric due to random chance (or any other reason). How does that sound? It should at least be a start for load balancing and failover. I should have something working like this within a day or a month (Now there is a broad estimate for you). I'm happy to do this myself, but would also appreciate a keen buddy on this once the skeleton is in place. cheers Bill Burke wrote: I was looking at the way Apache+Tomcat works. They use a protocol between themselves. Apache decrypts the SSL message and extracts sticky session information as well as SSL information and passes this over a load-balanced socket over a proprietary protocol to the Tomcat instance. The message contains the HTTP request as well as all the SSL information. Tomcat unpacks the message, creates an HttpRequest, and sends the request on their local bus. I guess they call this a Tomcat Connector. --- Marc, I see no reason why you have to be separate from Apache. The issues here aren't trivial and they've already done all the work and they're open-source. Actually their license is much more liberal than LGPL. Plus for this kind of stuff you probably want something written in C for speed as Greg suggested. Jetty already leverages Jasper, why not mod_jk? --- As far as load-balancing based on client IP and AOL, it is even worse than you think. Supposedly, a client's IP can change in the middle of the session.(I said supposedly remember ;) You acutally have to stick on a mask of the IP. Sure, client IP isn't the best way, but it was the only thing us ignorant bloaks knew at the time :) -Original Message- From: Greg Wilkins [mailto:[EMAIL PROTECTED]] Sent: Thursday, March 07, 2002 6:05 AM To: [EMAIL PROTECTED] Subject: Re:
