RE: Tomcat mod_jk very slow if used with apache
Hello Henri, hello Thomas, The problem comes from the call to fdatasync in the logging code of mod_jk. I already mentioned this to Dan Milstein. It should really be checked, if that way of flushing to the physical disk (and not only to the file system cache which should be enough) is really needed. The problem becomed more important due to the fact, that the actual production release incorrectly documents the existence of a log level "warn". In 3.2.1 this does not exist. The header file for the logging declares only: #define JK_LOG_DEBUG_LEVEL 0 #define JK_LOG_INFO_LEVEL 1 #define JK_LOG_ERROR_LEVEL 2 #define JK_LOG_EMERG_LEVEL 3 #define JK_LOG_DEBUG_VERB "debug" #define JK_LOG_INFO_VERB"info" #define JK_LOG_ERROR_VERB "error" #define JK_LOG_EMERG_VERB "emerg" and if you use any undeclared log level (as e.g. warn) the code falls through to "debug". So using warn you actually produce tons of debug output, each line calling fdatasync to flush to disk. We had the same problem on the first day of a heavy load production system and had some hard hours to find out. I think the second point (incorrect log level "warn") is corrected in the next release (by changing documentation and default - not code), the first thing, throwing out fdatasync - should still be done. Greetings, Rainer Jung At 10:54 20.04.01 , you wrote: Could you be more explicit. OS, mod_jk version, tomcat version, apache version Thanks - Henri Gomez ___[_] EMAIL : [EMAIL PROTECTED](. .) PGP KEY : 697ECEDD...oOOo..(_)..oOOo... PGP Fingerprint : 9DF8 1EA8 ED53 2F39 DC9B 904A 364F 80E6 -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] Sent: Friday, April 20, 2001 10:52 AM To: [EMAIL PROTECTED] Subject: Tomcat mod_jk very slow if used with apache I encountered the following problem: Tomcat was about 20 times slower if the access went via apache, compared to a direct access. The solution was the following: I had to put the loglevel of mod_jk to error instead of warn(as proposed). httpd.conf: JkLogLevel error Mod_jk does not seem to be a quick logger. This is the logoutput for out simple request if JkLogLevel warn: [jk_uri_worker_map.c (344)]: Into jk_uri_worker_map_t::map_uri_to_worker [jk_uri_worker_map.c (406)]: jk_uri_worker_map_t::map_uri_to_worker, Found a match loadbalancer [jk_worker.c (123)]: Into wc_get_worker_for_name loadbalancer [jk_worker.c (127)]: wc_get_worker_for_name, done found a worker [jk_lb_worker.c (471)]: Into jk_worker_t::get_endpoint [jk_lb_worker.c (300)]: Into jk_endpoint_t::service [jk_ajp13_worker.c (651)]: Into jk_worker_t::get_endpoint [jk_ajp13_worker.c (536)]: Into jk_endpoint_t::service [jk_ajp13.c (346)]: Into ajp13_marshal_into_msgb [jk_ajp13.c (480)]: ajp13_marshal_into_msgb - Done [jk_connect.c (108)]: Into jk_open_socket [jk_connect.c (115)]: jk_open_socket, try to connect socket = 8 [jk_connect.c (124)]: jk_open_socket, after connect ret = 0 [jk_connect.c (132)]: jk_open_socket, set TCP_NODELAY to on [jk_connect.c (140)]: jk_open_socket, return, sd = 8 [jk_ajp13_worker.c (166)]: In jk_endpoint_t::connect_to_tomcat, connected sd = 8 [jk_ajp13.c (527)]: ajp13_unmarshal_response: status = 200 [jk_ajp13.c (534)]: ajp13_unmarshal_response: Number of headers is = 1 [jk_ajp13.c (576)]: ajp13_unmarshal_response: Header[0] [Content-Type] = [text/html] [jk_ajp13_worker.c (489)]: Into jk_endpoint_t::done [jk_lb_worker.c (378)]: Into jk_endpoint_t::done Greethings, Thomas Dreaming of a Swiss Account? Get it here: http://freemail.swissinfo.org
RE: not getting load-balancing behavior
We're using apache 1.3.33 with tomcat 5.0.16, connected via mod_jk, on Solaris. This is very outdated. You shuld update to TC 5.0.30 and mod_jk 1.2.8. What I'm observing is that load-balancing isn't working. We have a couple of machines dedicated to our XML interface, and have apache Now, in operation, machine-a is getting hammered, while machine-b gets almost no traffic at all. Machine-a typically has CPU loads in the 4.0 range, while machine-b sits idle at 0.04. My hunch is that the bulk of the traffic is coming from a single source, and so apache/mod_jk/something is deciding to give it over to the same tomcat instance each time, even though it shouldn't be. Routing decisions are made by each Apache process independently. So directly after starting each apache process will forward it's first request to the same worker, it's second to the same other worker etc. After some time, because of differing answer times you will exhibit a balanced situation. Are you using Keep-Alive for HTTP? If your are having only few Clients and you are using very long Keep-Alive-Times maybe all clients will constantly use the same worker. But I#m not sure if it's true, although it would be an easy check to just disable Keep-Alive in Apache. Another possibility (after upgrading ;) ) is to use debug log level and study the output under low load. You didn't provide the full workers.properties file though. Are you having ONLY loadbalancer3 in the workers list? - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Cluster: how to set mcast interface for dual LAN card?
Depends on the OS. Usually the system decides which interface to use depending on the routing table. E.g. on Solaris: netstat -rnv shows: Destination Mask Gateway Device Mxfrg Rtt Ref Flg Out In/Fwd --- -- - - --- --- - -- ... 224.0.0.0240.0.0.0 192.168.100.20 eri01500* 0 1 U0 0 ... The shown entry is the multicast route (network 224.0.0.0 with netmask 240.0.0.0). The traffic will go through the local interface eri0 which is the one with the local address 192.168.100.20. If you want to route multicast through one of several interfaces on a multi homed system, you have to set the correct route (on Unixes using the route command). You can't do that inside Tomcat. You need to have a good understanding of IP, networks, netmasks, routes and multicast. Regards, Rainer Joseph Lam wrote: Hi, If I have two LAN cards and I want my Tomcat to mcast through one of them, what parameter should I set? Regards, Joseph - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: 5.5.10 cluster exception
Hi Dennis, i tried with 15/conf/server.xml:tcpListenAddress=192.168.0.180 25/conf/server.xml:tcpListenAddress=192.168.0.30 and otherwise the same cluster config which you posted on tomcat-dev. No problem with membership and multicast, no exception. So there must be something more than just differently sized IP address strings. Regards, Rainer Well, I'll move this discussion over to the user thread then.. See below. Remy Maucherat wrote: Dennis wrote: I'm working with the cvs tagged 5.5.10 version of tomcat to check out some clustering fixes. When I bring a 2nd server into the pool, I get this exception repeated every time an mcast packet is received: ==CUT== java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Ljava.lang.Object;ILjava.lang.Object;II)V(Unknown Source) at org.apache.catalina.cluster.mcast.McastMember.getMember(McastMember.java:181) at org.apache.catalina.cluster.mcast.McastServiceImpl.receive(McastServiceImpl.java:209) at org.apache.catalina.cluster.mcast.McastServiceImpl$ReceiverThread.run(McastServiceImpl.java:253) ==END CUT== Here are the relevant lines of code from McastMember.java: ==CUT== byte[] domaind = new byte[dlen]; System.arraycopy(data, nlen + 24, domaind, 0, domaind.length); ==END CUT== I added some debugging to figure out the length of the data. The exception occurs because data.length is 23. Obviously 23+24 is going to throw an ArrayIndexOBE. My question is.. is there a configuration thing that is causing data to be sent to be less than the desired length? It appears the data is not coming in in the format expected. Thoughts? After more digging and logging, I've fixed the exception. Clustering does work for some people and not for others. The problem is that when receiving a DatagramPacket the byte buffer is set at a constant size. I noticed there is one receivePacket in McastServiceImpl. The packet get's re-used. Here is some output from my logging with annotation: # I added the following line when a node sends a packet. Notice that the data length is 50. This is because name length (ip address has 23 characters and there are 27 other bytes. name: tcp://192.168.1.27:4002 domain: dev addr:(len) 4 data len: 50 # the following output when I receive a packet. The other server in the node had a name length of 22, hense data length of 49 instead of 50. Received a packet of length: 49 with offset: 0 Data.length: 49 Alive: 2649783 Port: 4002 Address: 192.168.1.5 Name Length: 22 Name: tcp://192.168.1.5:4002 #notice packet from other server has 22 for length and is receive successfully. dlen: 3 # notice that this packet was receive successfully (It's from the other server ) no exception occurs. Received a packet of length: 49 with offset: 0 #here is the problem. Data.length: 49 Alive: 7794 Port: 4002 Address: 192.168.1.27 Name Length: 23 Name: tcp://192.168.1.27:4002 # this packet has a buffer of size 49 but it has length 23 for name and was sent with a 50 byte buffer dlen: 3 # exception thrown and packet not received: Jul 28, 2005 10:07:46 AM org.apache.catalina.cluster.mcast.McastServiceImpl$ReceiverThread run == cut exception == So I got rid of the exception by allocating a new DatagramPacket (receivePacket) in the receive thread (McastServiceImpl.java approx line 206). That way, the buffer had the correct number of bytes and no exception was thrown. The reason it worked for you is because both of your servers probably had the same length in their ip address. So the bug occurs when people have a cluster with ip addresses that are not the same length. I'll file a bug and possibly a patch if I come to an confident enough understanding of the architecture. -Dennis Your post is OT on this mailing list, but I thought I would play a little with the clustering. This works for me, with this config: Cluster className=org.apache.catalina.cluster.tcp.SimpleTcpCluster managerClassName=org.apache.catalina.cluster.session.DeltaManager expireSessionsOnShutdown=false useDirtyFlag=true notifyListenersOnReplication=true Membership className=org.apache.catalina.cluster.mcast.McastService mcastAddr=228.0.0.4 mcastPort=45564 mcastFrequency=500 mcastClusterDomain=dev mcastDropTime=3000/ Receiver className=org.apache.catalina.cluster.tcp.ReplicationListener tcpListenAddress=auto tcpListenPort=4002 tcpSelectorTimeout=100 tcpThreadCount=6/ Sender className=org.apache.catalina.cluster.tcp.ReplicationTransmitter replicationMode=pooled ackTimeout=15000/ Valve
Re: DeltaManager for Clustering
Hi Dennis, assuming you have a valid cluster config you don't need to start/stop the DeltaManager yourself. Session attributes are only replicated, if they are serializable and the changes to the attributes are applied via setAttribute. If your attribute e.g. is a HashMap and you change an entry via getAttribute and then access by a key, the cluster will not know about the change and not replicate the new data. On the other hand this way you can obscure changes you don't really need from the replication and save some replication load. Is replication working in general in your setup? Is there documentation on using the DeltaManager for clustering? In the javadoc there is this statement: --CUT-- Correct behavior of session storing and reloading depends upon external calls to the start() and stop() methods of this class at the correct times. --END-- The tomcat clustering howto doesn't mention this though. Are the start/stop calls referred to for the container, or does the user application have to do something? I have some objects in the session that are not being syncronized accross the cluster and am trying to figure out if I'm doing something wrong either in the config or in the code. Thanks Dennis - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: DeltaManager for Clustering
See, the thing is, we have a 3rd party library that saves information in the session. It doesn't call setAttribute every time it changes the data. DeltaManager appears to ignore the useDirtyFlag. With SimpleTcpReplicationManager, I can set useDirtyFlag to false and we get the desired behavior. With DeltaManager, the sessions get out of sync. So for the time being, I switched the managerClass back to SimpleTcpReplicationManager and things work the way they are supposed to. The issue is whether or not DeltaManager is supposed to use a useDirtyFlag or not I guess. The config option is still there in the sample cluster config that comes with server.xml as well as in the online-documentation. It has no effect though. No, DeltaManager doesn't use that flag. It would somehow not make sense, because the whole pupose of DeltaManager is to only replicate changed attributes of a session and the flag tries to replicate every session accessed. So if you would impement it with DeltaManager it would behave the same way as the SimpleTcpReplicationManager already does. Under high load that will be a problem. If that is relevant yu must do a stress test and observe CPU/bandwidth/latency. Maybe a crude workaround: at the end of the requst do a getAttribute/setAttribute to the attributes you want to replicate. I think DeltaManager does not really check for a change, it replicates all attributes accessed via setAttribute. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: HELP: Tomcat 5.5.9 with jsvc as low priviledges user on Linux fails in Boots
For some reason during startup tomcat writes (!) the file tomcat-users.xml. It does it in the way that it writes to tomcat-users.xml.new and then renames that file to tomcat-users.xml. At least that's what was in the 5.0 code. I assume that didn't change with 5.5. As a consequence the user running tomcat needs write access to the directory the tomcat-users.xml file is in. If you don't like the idea of giving that runtime user write access to the conf directory, you can configure tomcat-users.xml inside server.xml to be in some other directory - which then is the one that needs write access. As far as I know, there is no workaround for that at the moment (except for choosing another user realm). Interesting, Thanks Darryl for sharing. So you run 5.5.9 and no problem huh ? What's the access given for the tomcat structure ? I'm interested in particular on that conf folder. I can run it fine too but not as root and root has no write access to the conf folder. How is your set up ? BTW that .new extension looked strange to me too. I cannot explain it - didn't look yet in TC source code. Here's the way I call the jsvc JAVA_HOME=/usr/local/java_home CATALINA_HOME=/usr/local/tomcat/tomcat_home TOMCAT_USER=tomcat TMP_DIR=/var/tmp CATALINA_OPTS= CLASSPATH=\ $JAVA_HOME/lib/tools.jar:\ $CATALINA_HOME/bin/commons-daemon.jar:\ $CATALINA_HOME/bin/bootstrap.jar:\ $CATALINA_HOME/bin/mx4j-jmx.jar:\ $CATALINA_HOME/bin/mx4j.jar:\ $CATALINA_HOME/bin/jsvc \ -user $TOMCAT_USER \ -home $JAVA_HOME \ -Dcatalina.home=$CATALINA_HOME \ -Djava.io.tmpdir=$TMP_DIR \ -outfile $CATALINA_HOME/logs/catalina-daemon.out \ -errfile $CATALINA_HOME/logs/catalina-daemon.err \ $CATALINA_OPTS \ -cp $CLASSPATH:$CATALINA_HOME/bin/bootstrap.jar:$CATALINA_HOME/bin/commons-daemon.jar org.apache.catalina.startup.Bootstrap Did you have any issues while installing jsvc ? Thanks again, MC http://www.goodstockimages.com From: Darryl L. Miles [EMAIL PROTECTED] Reply-To: Tomcat Users List tomcat-user@jakarta.apache.org To: Tomcat Users List tomcat-user@jakarta.apache.org Subject: Re: HELP: Tomcat 5.5.9 with jsvc as low priviledges user on Linux fails in Bootstrap Date: Tue, 02 Aug 2005 08:01:36 +0100 MC Moisei wrote: java.io.FileNotFoundException: /usr/local/tomcat/tomcat_home/conf/tomcat-users.xml.new (Permission denied) at java.io.FileOutputStream.open(Native Method) This smells like its calling for write access to the DIRECTORY /usr/local/tomcat/tomcat_home/conf/ (not the file) Unless you have a left over file that is actually called conf/tomcat-users.xml.new from a previous execution of TC that did not complete the edit and rename. In which case I think you need to delete the conf/tomcat-users.xml.new file (after you've ensured you have a valid and working conf/tomcat-users.xml file itself). FYI - I run jsvc too and have not seen this problem with 5.5.9. jsvc.exec -Djava.endorsed.dirs=./common/endorsed -classpath :/opt/jakarta-tomcat-5.5.9/bin/bootstrap.jar:/opt/jakarta-tomcat-5.5.9/bin/commons-logging-api.jar -Dcatalina.base=/opt/jakarta-tomcat-5.5.9 -Dcatalina.home=/opt/jakarta-tomcat-5.5.9 -Djava.io.tmpdir=/opt/jakarta-tomcat-5.5.9/temp -outfile ./logs/catalina.out -errfile ./logs/catalina.err -pidfile ./logs/jsvc.pid -user jakarta -Xmx2048M -Xms512M -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start -- Darryl L. Miles - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: HELP: Tomcat 5.5.9 with jsvc as low priviledges user on Linux fails in Bootstrap
CVS head now includes an improvement: 1) If the directory containing tomcat-users.xml is not writeable you will get a nice warning instead of a strange exception. 2) You can configure the MemoryUserDatabase with the attribute readonly=true. Then there will be not write attempt at all. Details under http://issues.apache.org/bugzilla/show_bug.cgi?id=36020 Will be included in 5.5.11 most probably sometime during august. MC Moisei wrote: Hi, I manage to configure my tomcat with jsvc(common-daemon) and everything work great till I start to launch it as root. If I run it as tomcat user it does work great. If I try to run it as root from command prompt or from init.d I get the following exception ( see below ) Right are given as below chown -R tomcat:tomcat /usr/local/tomcat chown -R root:root /usr/local/tomcat/bin chown -R root:root /usr/local/tomcat/common This is not right - looks like the bootstrap is trying to access the Realm and there is no write access to the conf/tomcat-users.xml file. I can't believe the common-daemon not tomcat side didn't say a thing about this, I bet there are others experiencing the matter. Do i have to disable Tomcat realms ? It doesn't sounds right. There is no way I'd give others write access on that. Looking forward to hear from you if you experienced something similar. Thanks, MC Aug 1, 2005 7:23:15 PM org.apache.naming.NamingContext lookup WARNING: Unexpected exception resolving reference java.io.FileNotFoundException: /usr/local/tomcat/tomcat_home/conf/tomcat-users.xml.new (Permission denied) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.init(FileOutputStream.java:179) at java.io.FileOutputStream.init(FileOutputStream.java:131) at org.apache.catalina.users.MemoryUserDatabase.save(MemoryUserDatabase.java:462) at org.apache.catalina.users.MemoryUserDatabaseFactory.getObjectInstance(MemoryUserDatabaseFactory.java:98) at org.apache.naming.factory.ResourceFactory.getObjectInstance(ResourceFactory.java:129) at javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:301) at org.apache.naming.NamingContext.lookup(NamingContext.java:792) at org.apache.naming.NamingContext.lookup(NamingContext.java:152) at org.apache.catalina.mbeans.GlobalResourcesLifecycleListener.createMBeans(GlobalResourcesLifecycleListener.java:138) at org.apache.catalina.mbeans.GlobalResourcesLifecycleListener.createMBeans(GlobalResourcesLifecycleListener.java:108) at org.apache.catalina.mbeans.GlobalResourcesLifecycleListener.lifecycleEvent(GlobalResourcesLifecycleListener.java:80) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119) at org.apache.catalina.core.StandardServer.start(StandardServer.java:676) at org.apache.catalina.startup.Catalina.start(Catalina.java:537) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:271) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:218) Aug 1, 2005 7:23:15 PM org.apache.catalina.mbeans.GlobalResourcesLifecycleListener createMBeans SEVERE: Exception processing Global JNDI Resources - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: anonymising Tomcat
Take a look at ./org/apache/catalina/util/ServerInfo.properties in CATALINA_HOME/server/lib/catalina.jar. It contains: server.info=Apache Tomcat/5.5.10 server.number=5.5.10.0 You can put different values in there and deploy the new properties file in CATALINA_HOME/server/classes/org/apache/catalina/util/ServerInfo.properties As far as I know, there are no negative side effects. Have fun Rainer Is it possible to configure Tomcat (5.5.9) so that a moderately able hacker couldn't figure out what is serving up our web apps? Paul Singleton -- No virus found in this outgoing message. Checked by AVG Anti-Virus. Version: 7.0.338 / Virus Database: 267.10.0/63 - Release Date: 3/Aug/2005 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: anonymising Tomcat
Yes, that's how it works. I Think taht possibility already existed at least back to 4.0... The order in which the different repositories are searched for classes (and property files) is defined in CATALINA_BASE/conf/catalina.properties. The file included in the usual distribution has classes directory before lib directory in it. Paul Singleton wrote: Rainer Jung wrote: Take a look at ./org/apache/catalina/util/ServerInfo.properties in CATALINA_HOME/server/lib/catalina.jar. It contains: server.info=Apache Tomcat/5.5.10 server.number=5.5.10.0 You can put different values in there and deploy the new properties file in CATALINA_HOME/server/classes/org/apache/catalina/util/ServerInfo.properties Do you mean that I can leave catalina.jar where it is, make a skeleton of folders in server/classes with just my new ServerInfo.properties, and the properties in server/classes/... will override those in the jar? Or must I unpack catalina.jar into server/classes and then delete it from server/lib before altering the properties? Are all references to Tomcat's description and version number derived from these properties, and is this new in 5.5.10 (we use 5.5.9 in production)? cheers Paul Singleton - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: session problems when using mod_jk (1.2.14) load balancing
That should not work! The correct way to configure session stickyness is to use jvmRoute (which you already did) and then giving the workers the same names as the jvmRoute. That is instead of bl_worker_dev use dev_alexis and instead of bl_worker_noah use noah_alexis as the worker names. You should check, that the URLs produced by your application include the ;jsessionid=32Characters.jvmRoute or - in case you use cookies - the same info is in your session cookie. mod_jk then automatically strips the jvmRoute part from the session identifier and lloks for a worker of the same name. You will only need to use the domain attribute in case you have a lot of tomcat instances and some of them have the sessions replicated, others not. Then you can give all members of a replication domain the same domain name and mod_jk will know, that in case the correct worker is down, which alternatives are good. Beautiful - worked like a charm. That might take the cake as far as longest question to quickest, shortest answer goes. ha. Thanks a bunch. I might have to gripe about doucmentation in a second (nother thread).. Noah Edgar Alves wrote: Try adding these two lines to worker.properties: worker.bl_worker_dev.domain=dev_alexis worker.bl_worker_noah.domain=noah_alexis -- Edgar Alves Mott Leroy wrote: Hi - I'm unable to get mod_jk load balancing working. The usual mod_jk setup works just fine, but using a load balancing worker however, is not. [Oddly, my webserver crashed during testing of this, but that could very well be unrelated] The problem is with user sessions. The instances (nodes) do not seem to recognize an already established session with the user and are creating new sessions. It's possible that is a session-stickiness issue, but it appears like the requests are hitting the same instance, just not getting the previously established session. As a result, I can't even reliably login to my application. I created a session listener for debugging purposes and it reports -no- destroyed sessions, but plenty of newly created sessions on both instances that make up the cluster. The session IDs, I noticed, have the jvmRoute name attached to them, which should be a good sign. I have a webserver running Apache (1.3.33), mod_jk (1.2.14), and an application server running the cluster -- 2 instances tomcat (5.0.28) on different ports. I added a unique jvmRoute to both instances in the server.xml: Engine name=Catalina defaultHost=localhost jvmRoute=dev_alexis Engine name=Catalina defaultHost=localhost jvmRoute=noah_alexis My worker.properties loadbalancer settings: worker.list=load_balancer_test worker.load_balancer_test.type=lb worker.load_balancer_test.balance_workers=bl_worker_dev,bl_worker_noah worker.load_balancer_test.sticky_session=true worker.load_balancer_test.sticky_session_force=false worker.bl_worker_dev.type=ajp13 worker.bl_worker_dev.host=alexis worker.bl_worker_dev.port=9003 worker.bl_worker_dev.lbfactor=1 worker.bl_worker_dev.socket_keepalive=1 worker.bl_worker_dev.recycle_timeout=300 worker.bl_worker_noah.type=ajp13 worker.bl_worker_noah.host=alexis worker.bl_worker_noah.port=8063 worker.bl_worker_noah.lbfactor=3 worker.bl_worker_noah.socket_keepalive=1 worker.bl_worker_noah.recycle_timeout=300 Any ideas, things I could try would be much appreciated. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: session problems when using mod_jk (1.2.14) load balancing
OK, thanks Mladen: I have to correct myself. 1) Traditional use Until mod_jk 1.2.6 there was no concept of domains. You had to specify the worker name to be exactly the same as the jvmRoute to make sticky sessions work. This way of configuring stickyness is wel known to mod_jk users. 2) I introduced domains to allow mod_jk to make good failover decisions in the case where you have many tomcats, but session replication only between subset, like 2 Tomcat clusters with three instances each (T1_1, T1_2, ..., T3_3) where T1_1, T1_2 and T1_3 replicate their sessions between each other etc. Then you can configure a common domain name for T1_1, T1_2 and T1_3 (e.g. T1) and is mod_jk is not able to correctly route a sticky request (e.g. T1_1 goes down) it will choose another worker with the same domain name. Moreover in this case it will load balance betweeen all those workers (in my example between T1_2 and T1_3). 1) and 2) still work in 1.2.14. 3) Mladen improved mod_jk a lot and during that time introduced another additional way of using the domain attribute: If stickyness for sessions is required and mod_jk does not find a worker with the name of the jvmRoute contained in the session id, then it will next try to find any worker whose domain name is equal to the jvmRoute. If there are multiple such workers, mod_jk loadbalances between them. Summary: If there is a one-to-one relationship between jvmRoutes and workers, there is no functional difference between giving the worker the same name as the jvmRoute, or giving it the same domain attribute as the jvmRoute. The first way is more compatible with what people used to do since a long time and it will be marginally faster, because that check is done first. It's always good to look at the code, in fact when I first discovered the use of the jvmRoute as a domain name in the code I thought it's a bug. I now understand, that it's at least a feature :-) Hi Mladen, I've used the domain property because it seemed the more general approach (i.e., supports clusters but can be used with a single worker). What this thread got me curious about is if using the domain property in this fashion is officially supported, or on the other hand if it can only be used reliably with clusters. After reading the documentation (http://jakarta.apache.org/tomcat/connectors-doc/config/workers.html), I got the idea that using the domain property with only one worker wouldn't be a trick but another way tell mod_jk which jvmRoute to use: If sticky_session is used, then the domain name is used as session route.. Naming the worker after the intended jvmRoute (even though it used to be the only way) seems more of a trick than explicitly specifying the jvmRoute with the domain property. However, since the same documention mentions that the domain property is used for large systems with clustering, do you know of any side effects (like lower performance) of using this approach as opposed to simply naming the workers after the jvmRoute? -- Edgar Alves Mladen Turk wrote: Edgar Alves wrote: Hi, I'm using the domain property in the same situation as the one discussed in this thread. Any reason why I shouldn't use the domain property and rely on the worker names instead? Domain is supposed to be used with multiple workers sharing the same jvmRoute having session replication between them, thus forming 'cluster groups' to lower the session data replication transfer. You can use the domain, but it's a trick rather then a proper usage. Regards, Mladen. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: workers.properties directives
mod_jk 1.2.14: List of workers, attributes and defaults. ajp14 is not included. global: - worker.maintain (60) [and some obvious others] all workers: - type (ajp13) ajp12: - host (localhost) - port (8007) ajp13: - host (localhost) - port (8009) - connection_pool_size (1) [aka cachesize which is the old and deprecated name] - cache_timeout (0) - socket_timeout (-1 = disabled) - socket_buffer (8192) - socket_keepalive (false) - recycle_timeout (0) - connect_timeout (0) - reply_timeout (0) - prepost_timeout (0) - recovery_options (0) - retries (3) lb: - balanced_workers / balance_workers - sticky_session (1) - sticky_session_force (0) - method (0=BYREQUESTS) - lock (O=OPTIMISTIC) - recover_time (60, also mimimum 60) per balanced worker: - lbfactor (1) - domain (NULL) - redirect (NULL) - disabled (0) - stopped (0) I hope this helps. Viel Spaß wünscht Rainer Jung -- kippdata informationstechnologie GmbH Bornheimer Str. 33a D-53115 Bonn Hi all, recently I stumbled over the mod_jk statusworker feature coming with recent mod_jk versions and, of course, I immediately wanted to have such neat thing :-) Up to that time, I didn't saw an imperative need to have balanced wor- kers (Apache+Tomcat on the same machine; one single ajp13 type worker did his job fine for me so far). Since it turned out that the status worker will talk to lb type workers only, I simply renamed my existing ajp13 worker, then established another lb type worker of the former name which itself sits now on that ajp13 type (or node) worker. But here I'm asking myself: Where do I preferably apply those more sophisticated settings like worker.XY.recycle_timeout, worker.XY.cachesize or worker.XY.cache_timeout ?? (seems that we can assign them both to lb type as to ajp13 type workers; at least I didn't found anything that rules out doing so in the online docs) In other words, do they a better job when defined with worker MyWorker or with node1 in the following httpd.conf example (simplified view; think yourself a JkWorkerProperty in front of all these worker... directives): worker.list=MyWorker worker.MyWorker.type=lb worker.MyWorker.balance_workers=node1 worker.node1.type=ajp13 worker.node1.host=localhost worker.node1.port=8009 Location /MyContext JkMount MyWorker /Location ?? Some of that worker.XY... options sound more IP related, so I would tend to assign them to ajp13 type workers; on the other hand, worker.XY.cachesize seems to have more impact to the endpoint software layer, which implies that it unfolds it's effect better within the load balancer tier. TIA Regards Olaf Lautenschlaeger -- ANOVA Multimedia Studios GmbH Joachim-Jungius-Strasse 9 D-18059 Rostock / Germany - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: mod_jk: Hot Standby and Load Balance
I think having multiple load balancing workers for the same group of target servers is not a problem. You simply define load balancers e.g. lb1, lb2 etc. Which load balancer is chosen is determined by your JkMount directives. So if you have different apps app1, app2 etc. on your tomcats having incompatible balancing requirements you simply use JkMount /app1/* lb1 JkMount /app2/* lb2 etc. The balanced workers behind lb1, lb2 etc. are allowed to have the same name, because each load balancer has it's own list of balanced workers with associated attributes. I expect no problem from a clash of names of balanced workers in different balancing workers. So there would be no need of having multiple jvmRoute for a single tomcat instance. Well, I was thinking of using something like (truncated for clarity): # load balanced worker.lb_traditional.type=lb worker.lb_traditional.balance_workers=lb_worker1,lb_worker2 worker.lb_traditional.sticky_session=true # workers 1 and 2 are load balanced worker.lb_worker1.type=ajp13 worker.lb_worker1.host=server1 worker.lb_worker1.domain=theJRMRoute worker.lb_worker2.type=ajp13 worker.lb_worker2.host=server2 worker.lb_worker2.domain=theJRMRoute # standby setup worker.lb_standby.type=lb worker.lb_standby.balance_workers=lb_worker3,lb_worker4 worker.lb_standby.sticky_session=true # workers 4 is hot standby for worker 3 worker.lb_worker3.type=ajp13 worker.lb_worker3.host=server1 worker.lb_worker3.domain=theJRMRoute worker.lb_worker3.redirect=worker4 worker.lb_worker4.type=ajp13 worker.lb_worker4.host=server2 worker.lb_worker4.domain=theJRMRoute worker.lb_worker4.disabled=True Guernsey, Byron (GE Consumer Industrial) wrote: I believe you can specify the jvmRoute separately by using the domain attribute, but I'm not sure I see how this would help? Byron -Original Message- From: Mott Leroy [mailto:[EMAIL PROTECTED] Sent: Wednesday, August 31, 2005 11:03 AM To: Tomcat Users List Subject: mod_jk: Hot Standby and Load Balance Due to some differences in our applications, some of them can be truly load balanced, and some of them really cannot (yet). That is, some of our applications can be (and have been) truly load balanced, and others need (and only allow) simple failover support (hot standby). I noticed that workers now support both possibilities (using disabled and redirect flags to support hot standby). What I'd like to do ultimately is have a hot standby load balancer and as well as a normal load balancer, but it doesn't seem like that's possible. From what I understand, you can really only have 1 load balanced worker per tomcat instance because it must match the jvmRoute of that instance -- having one worker that's disabled and one that's not doesn't seem possible. So if I define a load balance worker as: # traditional load balance worker worker.lb_tala_build.type=ajp13 worker.lb_tala_build.host=tala worker.lb_tala_build.port=8000 worker.lb_tala_build.lbfactor=1 worker.lb_tala_build.socket_keepalive=1 worker.lb_tala_build.recycle_timeout=300 I cannot really define a second load balanced worker like below (b/c no matching jvmRoute) # a hot standby worker based on the worker above worker.lb_tala_build2.type=ajp13 worker.lb_tala_build2.host=tala worker.lb_tala_build2.port=8000 worker.lb_tala_build2.lbfactor=1 worker.lb_tala_build2.socket_keepalive=1 worker.lb_tala_build2.recycle_timeout=300 worker.lb_tala_build2.disabled=True Is anyone familiar with this setup of have any ideas how it could be achieved? (the same problem exists for what would be the Primary server, as it would need a worker that redirects and one that doesn't) Ps - Being able to specify the jvmRoute separately would solve this problem: worker.lb_tala_build2.jvmRoute=lb_tala_build - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: mod_jk: Hot Standby and Load Balance
Of course you are right (and for me it seems to be too late today). So I agree: you either find out how to use different jvmRoutes in a single instance or you try to find a workarounf with the domain attribute: If a load balancer does not find a worker with the correct name (=jvmRoute), it will next use a worker whose domain name is equal to the jvmRoute. But this will not be very efficient, because every request will first look for the correct worker and only after that check for the domain. Also I'm not sure, how this second class worker will behave, if you stopp it with respect to it's redirect etc. attributes. Sorry! Rainer Jung wrote: The balanced workers behind lb1, lb2 etc. are allowed to have the same name, because each load balancer has it's own list of balanced workers with associated attributes. I expect no problem from a clash of names of balanced workers in different balancing workers. I must be missing something obvious here. I am with you on the JKMount part, but I just don't see how the name clash isn't an issue for worker.properties. Simplifying again ... # as per your suggestion ... where worker1 and worker2 are jvmRoutes worker.lb1.balanced_workers=worker1,worker2 worker.lb2.balanced_workers=worker1,worker2 # the balanced workers ... which should they choose ... ? worker.worker1 (failover version) worker.worker1 (not failover version) worker.worker2 (standby version) worker.worker2 (non-standby version) - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: workers.properties load balancing
Hi Steve, not a bug in 1.2.6 either: You have used the attribute balance_workers: worker.router.balance_workers=worker1,worker2 Version 1.2.6 only knew about balanced_workers. See the tiny difference? In 1.2.14 you can use either of both and balance_workers take precendence. Steve Dodge wrote: JK 1.2.14 with Tomcat 5.0.28 and Apache 2.0.52 on Linux RH AS4, Tomcats are installed on different machines. I cannot get a load balancing worker to work. mod_jk forwards request to tomcat just fine as long as I don't try and use a load balancing worker in my worker.list. The mod_jk.log says did not find a worker. ==workers.properties worker.list=router # Set properties for worker1 (ajp13) worker.worker1.type=ajp13 worker.worker1.host=server.ip1 worker.worker1.port=8009 worker.worker1.lbfactor=1 worker.worker1.cachesize=10 # Set properties for worker2 (ajp13) worker.worker2.type=ajp13 worker.worker2.host=server.ip2 worker.worker2.port=8009 worker.worker2.lbfactor=1 worker.worker2.cachesize=10 worker.router.type=lb worker.router.balance_workers=worker1,worker2 #worker.router.sticky_seesion=True worker.status.type=status =mod_jk.config== JkWorkersFile /etc/httpd/conf.d/workers.properties JkLogFile /var/log/httpd/mod_jk.log JkLogLeveltrace JkLogStampFormat [%a %b %d %H:%M:%S %Y] JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories JkRequestLogFormat %w %V %T JkMount /jmx-console/*.jsp router JkMount /jkstatus/* status When I use worker.list=worker1,worker1 and replace JkMount /jmx-console/*.jsp worker1 requests are forwarded, but I loose load balancing. Its like the minute I put the load balancer worker into the list, the whole workers configuration is bad. Thanks in advance, Steve I have some clarification on my workers.properties issue. It's not an Issue! The version of JK connectors was not JK 1.2.14, it was JK 1.2.6. Sorry to leave anyone scratching their head. I told my system admin to build from source JK 1.2.14, but to save time he found a pre-packaged rpm from RedHat containing JK 1.2.6. I wasn't aware of the fact that I had debugged version 6 instead of 14. So for those who are interested, JK 1.2.6 had BUGS. -Building mod_jk On another note, compiling JK connectors went very smooth on a stock redhat box with RPM devel packages. All we had to do was install the httpd-devel-2.0.52-12.2.ent.i386.rpm and any rpm's required by it (to get a list type rpm -qp --requires httpd-devel-2.0.52-12.2.ent.i386.rpm ) Then download and extract the JK source code. Navigate to the native directory type in |*./configure --with-apxs=/usr/sbin/apxs, the corresponding Apache2.0.x mod_jk.so resulted. Instructions are found at http://jakarta.apache.org/tomcat/connectors-doc/howto/apache.html Steve *| - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: tomcat 5.0.24 crashes silently when clustering turned on
Which platform/OS? I've no experience on Win, but I never experienced a tomcat crash on unix/linux. Nevertheless five comments: 0) jk2 is no longer under development. The only active connector development for apache is mod_jk and mod_proxy (for the upcoming apache 2.1/2.2). 1) If you really want to use clustering, either choose the tomcat 5.5 line (preferably with fastasyncmode), or at least 5.0.28 (better 5.0.30). 2) Some *nixes and shells will send signals, when the user starting tomcat logs out of the system resulting in killed tomcat processes. Inj that case use nohup or any similar workaround. 3) With replication you will need more memory. Any indications for OutOfMemory? 4) The only real process crashes I experienced where fixed by updates to bug fix releases of the JVM. I am trying to set up a 2-node cluster. Each node runs 5.0.24 with apache 2.0.50 in front (with SSL). I use the JK2 v. 2.0.43 connector. What I'm seeing doesn't make sense. I get tomcat and apache up on the first node and everything works fine. The clustering section in the server.xml file is uncommented and I've configured all of the appropriate values. Once that's working I turn to node 2. On the second node I start apache and tomcat and everything looks good for a bit. I have most of the tomcat output going to the consoleappender so I capture that output into a file. After anywhere from 1 - 2 minutes tomcat exits. There is no thread dump printed in any of the logfiles (including stdout/stderr). When I look in my log files I see normal activity for the little bit that TC is running and then it simply quits. There is no indication of TC going through the shutdown process. When tomcat exits it generates a non-zero exit code. When I first saw this problem a few weeks ago it was when I was deploying clustering in my production environment. After seeing TC exit like this 3 times in a row I rolled back my deployment and have spent the time trying to figure out what went wrong. I didn't note the exit code number! I have two different test environments setup with very similar configuration and I'm not having this problem on either of them. I'm at the point now where I'm going to have to try to deploy to production again but since I've not changed anything I'm fully expecting it to fail. This time I'll note the exit code number. Have any of you seen anything like this? What things can I adjust to enable more debugging output? Thanks for any info. -Mike Noel - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: NoSuchElementException in DeltaRequest
Hi, that should be fixed in 5.0.30 and in 5.5. Compare http://cvs.apache.org/viewcvs.cgi/jakarta-tomcat-catalina/modules/cluster/src/share/org/apache/catalina/cluster/session/DeltaRequest.java?r1=1.7r2=1.7.2.1diff_format=h and http://issues.apache.org/bugzilla/show_bug.cgi?id=31328. We use a very similar fix on top of 5.0.28 without further problems. Hi all, We have just moved to using Tomcat in a clustered environment, and now on a farily regular basis we are getting the following error occur from within the clustering logic. java.util.NoSuchElementException at java.util.LinkedList.remove(LinkedList.java:579) at java.util.LinkedList.removeFirst(LinkedList.java:131) at org.apache.catalina.cluster.session.DeltaRequest.addAction(DeltaRequest.java :102) at org.apache.catalina.cluster.session.DeltaRequest.setAttribute(DeltaRequest.j ava:69) at org.apache.catalina.cluster.session.DeltaSession.setAttribute(DeltaSession.j ava:1265) at org.apache.catalina.cluster.session.DeltaSession.setAttribute(DeltaSession.j ava:1246) at org.apache.catalina.cluster.session.DeltaSessionFacade.setAttribute(DeltaSes sionFacade.java:130) at au.com.bestbets.central.command.user.BaseUserLoginCmd.innerExecute(BaseUserL oginCmd.java:111) I believe we are using tomcat 5.0.29 running under Linux. Any ideas on what is causing this one? Steve Mactaggart Best Bets - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat Manager, Session Statistics
Hi, documentation says: Display ... the number of currently active sessions that fall within ten-minute ranges of their actual timeout times. Actual timeout times does not mean from now, but instead in general. It does not relate to when the session has been used last time. Since all sessions of your webapp will usually use the same timeout value, they will all appear with the same interval, and the interval will be the 10 minute interval located around your timeout value (in your example the timeout is 30 minutes, so the interval is [30,40[). Of course it would be interesting to see, how many sessions are close to being expired, if they will not be used in the next n minutes. Feel free to provide a patch to org.apache.catalina.manager.ManagerServlet.java (you would need to replace getMaxInactiveInterval by getLastAccessedTime and change the computation a bit). I suggest that the display of 30 - 40 minutes:1 sessions be rethink. To me it looks misleading at best. The documentation is probably wrong as well. In the example, the newly created session shouldn't be counted. Jean-Pierre Pelletier - Original Message - From: Mark Thomas [EMAIL PROTECTED] To: Tomcat Users List tomcat-user@jakarta.apache.org Sent: Wednesday, October 05, 2005 3:22 PM Subject: Re: Tomcat Manager, Session Statistics Jean-Pierre Pelletier wrote: Hi, 1) When I look at sessions statistics for an application, using https://localhost/manager/html/sessions?path=/myApplication Why does Tomcat always list the number of sessions to expired within 10 minutes as equal to the number of active sessions? Looks like a bug to me. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
