RE: Strange characters seen in RR and Cd columns on jkserver-status page after upgrading modjk to .28
-Original Message- From: Rainer Jung [mailto:rainer.j...@kippdata.de] Sent: 12 December 2009 15:51 To: Tomcat Users List Cc: Gerhardus Geldenhuis (GTA-LON) Subject: Re: Strange characters seen in RR and Cd columns on jkserver- status page after upgrading modjk to .28 On 11.12.2009 18:32, gerhardus.geldenh...@gta-travel.com wrote: Hi We have upgraded our modjk from .26 to .28 using mod_jk-1.2.28-httpd-2.2.X.so obtained from http://../tomcat-connectors/jk/binaries/linux/jk-1.2.28/x86_64/ We are running Server version: Apache/2.2.3 Server built: Nov 12 2008 10:40:14 CentOS release 5.2 (Final) After upgrading we are seeing random characters in the RR an Cd columns on the jkserver-status page. eg o%2B o+ Upgrade were done by stopping httpd. Copying new modjk to /etc/httpd/modules/ Editing /etc/httpd/conf.d/modjk.conf to reflect new version. Requests are being served but the strange characters and there origin a bit troubling. Our workers.properties looks as follows: ( There is actually 12 workers but I have removed most to simplify config) # Worker list worker.list=xml-gta,jkstatus # Worker definitions # main loadbalancer worker worker.xml-gta.type=lb worker.xml-gta.max_reply_timeouts=10 #worker.xml-gta.sticky_session=false worker.xml-gta.retries=6 worker.xml-gta.method=Busyness worker.xml-gta.balance_workers=worker1, worker2 # status page worker worker.jkstatus.type=status # Worker Reference worker.reference.port=8009 worker.reference.type=ajp13 worker.reference.lbfactor=1 worker.reference.socket_timeout=0 worker.reference.socket_keepalive=true worker.reference.connect_timeout=5000 worker.reference.prepost_timeout=0 worker.reference.reply_timeout=0 worker.reference.retries=6 worker.reference.recovery_options=27 worker.reference.connection_pool_timeout=600 worker.reference.connection_pool_size=2 worker.reference.ping_mode=A worker.reference.ping_timeout=5000 worker.reference.retry_interval=60 # Balance workers worker.worker1.reference=worker.reference worker.worker1.host=10.10.10.10 worker.worker2.reference=worker.reference worker.worker2.host=10.10.10.11 Regards It's a known problem. There's an open Bugzilla for that issue. It seems you need at least around 8 worker for it to happen and I suspect it has to do with webserver restarts (apachectl graceful or apachectl restart). Unfortunately I don't yet fully understand the reasons, but noone has reported an operational problem resulting our of it, like stickyness or load balancing being broken. Of course we need to fix it ... Can you confirm it happens only after restarting? And yes, it would be nice if your could build from the sources yourself and double check the problem is still there. Regards, Rainer Thank you for all the replies. I upgraded the httpd server to the latest CentOS version which is: Name : httpd Arch : x86_64 Version: 2.2.3 Release: 31.el5.centos.2 I also did a custom compile of modjk. The problem persist despite these changes and is not related to only restarts but persists even if the httpd service is completely stopped and then started again. We do not see the problem on a similar configured httpd server with only 5 workers which seems to corroborate the 8 worker threshold. If there is any debugging scripts or log files that I can attach to the bugzilla then please let me know and I will happily do so. Regards __ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email __
Re: Strange characters seen in RR and Cd columns on jkserver-status page after upgrading modjk to .28
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Gerhardus and Rainer, On 12/14/2009 6:32 AM, gerhardus.geldenh...@gta-travel.com wrote: -Original Message- From: Rainer Jung [mailto:rainer.j...@kippdata.de] It's a known problem. There's an open Bugzilla for that issue. It seems you need at least around 8 worker for it to happen and I suspect it has to do with webserver restarts (apachectl graceful or apachectl restart). Unfortunately I don't yet fully understand the reasons, but noone has reported an operational problem resulting our of it, like stickyness or load balancing being broken. Of course we need to fix it ... Can you confirm it happens only after restarting? And yes, it would be nice if your could build from the sources yourself and double check the problem is still there. [snip] We do not see the problem on a similar configured httpd server with only 5 workers which seems to corroborate the 8 worker threshold. I have a development server with 18 workers defined. I'll enable the status page and check to see what happens. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAksmdzMACgkQ9CaO5/Lv0PD+ewCeOmGEJRI8FfLqSQP7G5rd4+Jy BSMAoK037Zp8XrFrAWBaQTZVYOLar+MR =bzRv -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Strange characters seen in RR and Cd columns on jkserver-status page after upgrading modjk to .28
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 All, On 12/14/2009 12:34 PM, Christopher Schultz wrote: I have a development server with 18 workers defined. I'll enable the status page and check to see what happens. Hmm... I don't see any RR or Cd stuff... probably because I'm not in a clustered environment. Sorry I won't be able to offer any empirical data to this discussion. Rainer, what character encoding does mod_jk attempt to use when writing the output page? The XML document for the status page says: ?xml version=1.0 encoding=ISO-8859-1? The Content-Type of the HTTP response specifies no charset, to ISO-8859-1 ought to be the default. I am not forcing any character set for this particular URI. Rainer, is this an encoding problem, or an instrumentation problem (counter out-of-range, etc.)? Can you please post a reference to the bug you mentioned earlier? Thanks, - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAksmeUEACgkQ9CaO5/Lv0PCtdgCePYu9CYoGFI1DX33AGQ3K7nHH X28AnRPo5FoEjKdshYdqAzGDsN7BO/FL =P9xJ -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Strange characters seen in RR and Cd columns on jkserver-status page after upgrading modjk to .28
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 All, On 12/14/2009 12:43 PM, Christopher Schultz wrote: Rainer, is this an encoding problem, or an instrumentation problem (counter out-of-range, etc.)? Can you please post a reference to the bug you mentioned earlier? Looks like this is https://issues.apache.org/bugzilla/show_bug.cgi?id=46503 I've added a link to this thread in the comments. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAksmeuoACgkQ9CaO5/Lv0PBFKQCgsHbS6WklF2maGxmlcrlOJHt1 31YAoLLXUYVD8kE6M5rfbkvzf/7HmsrV =QGz7 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Strange characters seen in RR and Cd columns on jkserver-status page after upgrading modjk to .28
On 14.12.2009 18:43, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 All, On 12/14/2009 12:34 PM, Christopher Schultz wrote: I have a development server with 18 workers defined. I'll enable the status page and check to see what happens. Hmm... I don't see any RR or Cd stuff... probably because I'm not in a clustered environment. Sorry I won't be able to offer any empirical data to this discussion. It seems it is not straightforward to reproduce. Clustering shouldn't be necessary and I think some (or all?) who observed the problem are not using the domain and redirect attributes. It seems you only need a loadbalancer plus some not yet understood magic sauce. Rainer, what character encoding does mod_jk attempt to use when writing the output page? The XML document for the status page says: ?xml version=1.0 encoding=ISO-8859-1? The Content-Type of the HTTP response specifies no charset, to ISO-8859-1 ought to be the default. I am not forcing any character set for this particular URI. Rainer, is this an encoding problem, or an instrumentation problem (counter out-of-range, etc.)? Can you please post a reference to the bug you mentioned earlier? I expect it's a problem with uninitialized memory. It shouldn't have to do with encoding, unfortunately, it's really garbage we see. I thought it has to do with reusing shared memory during graceful restart, but since Gerhardus also gets it immediately after start, I guess I'm wrong. I need to analyze, whther the garbage is coming from the shared memory or is only part of the in memory worker data. I'll get back with questions or whatever as sonn as I start working on it, which is definitely before 1.2.29. And thanks to already finding the issue and adding the link :) Regards, Rainer Thanks, - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAksmeUEACgkQ9CaO5/Lv0PCtdgCePYu9CYoGFI1DX33AGQ3K7nHH X28AnRPo5FoEjKdshYdqAzGDsN7BO/FL =P9xJ -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Strange characters seen in RR and Cd columns on jkserver-status page after upgrading modjk to .28
On 11.12.2009 18:32, gerhardus.geldenh...@gta-travel.com wrote: Hi We have upgraded our modjk from .26 to .28 using mod_jk-1.2.28-httpd-2.2.X.so obtained from http://../tomcat-connectors/jk/binaries/linux/jk-1.2.28/x86_64/ We are running Server version: Apache/2.2.3 Server built: Nov 12 2008 10:40:14 CentOS release 5.2 (Final) After upgrading we are seeing random characters in the RR an Cd columns on the jkserver-status page. eg o%2B o+ Upgrade were done by stopping httpd. Copying new modjk to /etc/httpd/modules/ Editing /etc/httpd/conf.d/modjk.conf to reflect new version. Requests are being served but the strange characters and there origin a bit troubling. Our workers.properties looks as follows: ( There is actually 12 workers but I have removed most to simplify config) # Worker list worker.list=xml-gta,jkstatus # Worker definitions # main loadbalancer worker worker.xml-gta.type=lb worker.xml-gta.max_reply_timeouts=10 #worker.xml-gta.sticky_session=false worker.xml-gta.retries=6 worker.xml-gta.method=Busyness worker.xml-gta.balance_workers=worker1, worker2 # status page worker worker.jkstatus.type=status # Worker Reference worker.reference.port=8009 worker.reference.type=ajp13 worker.reference.lbfactor=1 worker.reference.socket_timeout=0 worker.reference.socket_keepalive=true worker.reference.connect_timeout=5000 worker.reference.prepost_timeout=0 worker.reference.reply_timeout=0 worker.reference.retries=6 worker.reference.recovery_options=27 worker.reference.connection_pool_timeout=600 worker.reference.connection_pool_size=2 worker.reference.ping_mode=A worker.reference.ping_timeout=5000 worker.reference.retry_interval=60 # Balance workers worker.worker1.reference=worker.reference worker.worker1.host=10.10.10.10 worker.worker2.reference=worker.reference worker.worker2.host=10.10.10.11 Regards It's a known problem. There's an open Bugzilla for that issue. It seems you need at least around 8 worker for it to happen and I suspect it has to do with webserver restarts (apachectl graceful or apachectl restart). Unfortunately I don't yet fully understand the reasons, but noone has reported an operational problem resulting our of it, like stickyness or load balancing being broken. Of course we need to fix it ... Can you confirm it happens only after restarting? And yes, it would be nice if your could build from the sources yourself and double check the problem is still there. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Strange characters seen in RR and Cd columns on jkserver-status page after upgrading modjk to .28
gerhardus.geldenh...@gta-travel.com wrote: Hi We have upgraded our modjk from .26 to .28 using mod_jk-1.2.28-httpd-2.2.X.so obtained from http://../tomcat-connectors/jk/binaries/linux/jk-1.2.28/x86_64/ We are running Server version: Apache/2.2.3 Server built: Nov 12 2008 10:40:14 CentOS release 5.2 (Final) After upgrading we are seeing random characters in the RR an Cd columns on the jkserver-status page. eg o%2B o+ That kind of thing triggers an immediate suspicion regarding a mismatched character set/encoding somewhere. I am not familiar at all with JSP pages, but could it be that during the update, you also customized some JSP pages, with an editor ? Are they really random characters (e.g. noise), or would-be real characters being replaced by the cabalistic signs above ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Strange characters seen in RR and Cd columns on jkserver-status page after upgrading modjk to .28
-Original Message- From: André Warnier [mailto:a...@ice-sa.com] Sent: 11 December 2009 17:43 To: Tomcat Users List Subject: Re: Strange characters seen in RR and Cd columns on jkserver-status page after upgrading modjk to .28 gerhardus.geldenh...@gta-travel.com wrote: Hi We have upgraded our modjk from .26 to .28 using mod_jk-1.2.28-httpd-2.2.X.so obtained from http://../tomcat-connectors/jk/binaries/linux/jk-1.2.28/x86_64/ We are running Server version: Apache/2.2.3 Server built: Nov 12 2008 10:40:14 CentOS release 5.2 (Final) After upgrading we are seeing random characters in the RR an Cd columns on the jkserver-status page. eg o%2B o+ That kind of thing triggers an immediate suspicion regarding a mismatched character set/encoding somewhere. I am not familiar at all with JSP pages, but could it be that during the update, you also customized some JSP pages, with an editor ? Are they really random characters (e.g. noise), or would-be real characters being replaced by the cabalistic signs above ? It does seem to be random... i have seen a lot of different characters appearing in these fields which change occasionaly as I refresh th jkserver-status page. I am going to hunt down character encodings on Monday. It baffles me because modjk is the only change that has happend. Maybe the .28 is just better at pointing out mistakes thats been there in the past. Best Regards __ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email __
Re: Strange characters seen in RR and Cd columns on jkserver-status page after upgrading modjk to .28
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Gerhardus, On 12/11/2009 12:32 PM, gerhardus.geldenh...@gta-travel.com wrote: We have upgraded our modjk from .26 to .28 using mod_jk-1.2.28-httpd-2.2.X.so obtained from http://../tomcat-connectors/jk/binaries/linux/jk-1.2.28/x86_64/ Any chance of compiling it yourself? Binary distros are convenient, but building it yourself (simple: ./configure make) gives you a 100% compatible binary. We are running Server version: Apache/2.2.3 Server built: Nov 12 2008 10:40:14 CentOS release 5.2 (Final) Is 2.2.3 the latest version available for CentOS? After upgrading we are seeing random characters in the RR an Cd columns on the jkserver-status page. eg o%2B o+ Is this only appearing on the mod_jk status page? In think André is asking about other pages (like those generated by JSP code). The status page is served directly by mod_jk (within httpd) and so there's no Tomcat interaction at all. Upgrade were done by stopping httpd. Copying new modjk to /etc/httpd/modules/ Editing /etc/httpd/conf.d/modjk.conf to reflect new version. Requests are being served but the strange characters and there origin bit troubling. I agree. What is the Content-Type header for the jkserver-status page? Are there any META tags in the page? It's possible that your httpd is simply advertising a Content-Type encoding that is different from what jkserver-status is actually using. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAksir5QACgkQ9CaO5/Lv0PAx/QCgwpEB+Ie6GbtfN19aeSxNvMRM SQQAnRFqdsBu8BC1/gj0lq5Ad689AoaD =8KLg -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Strange characters seen in RR and Cd columns on jkserver-status page after upgrading modjk to .28
Christopher Schultz wrote: ... After upgrading we are seeing random characters in the RR an Cd columns on the jkserver-status page. eg o%2B o+ Is this only appearing on the mod_jk status page? In think André is asking about other pages (like those generated by JSP code). The status page is served directly by mod_jk (within httpd) and so there's no Tomcat interaction at all. Ooops. Pid, we're even. I should have thought about that. Duh. What is the Content-Type header for the jkserver-status page? Are there any META tags in the page? It's possible that your httpd is simply advertising a Content-Type encoding that is different from what jkserver-status is actually using. But I see we're staying within character set/encoding issues. A small consolation for me. Looks like I'm going to have to finally look at that mod_jk status page after all. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
