[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859 --- Comment #27 from JasonDo --- Dear all, In our production environment, we did see mod_jk log print log. Fri Mar 05 10:00:14.906 2021] [17221:139662211758208] [info] ajp_handle_cping_cpong::jk_ajp_common.c (979): (worker3) timeout in reply cpong after 1 ms. Socket = 13 (event=110) [Fri Mar 05 10:00:14.951 2021] [27663:139662211758208] [info] ajp_handle_cping_cpong::jk_ajp_common.c (979): (worker3) timeout in reply cpong after 1 ms. Socket = 13 (event=110) [Fri Mar 05 10:00:14.971 2021] [3615:139662211758208] [error] uri_worker_map_update::jk_uri_worker_map.c (1320): Unable to stat the /etc/httpd/conf/uriworkermap.properties (errno=13) [Fri Mar 05 10:00:14.990 2021] [3617:139662211758208] [error] uri_worker_map_update::jk_uri_worker_map.c (1320): Unable to stat the /etc/httpd/conf/uriworkermap.properties (errno=13) [Fri Mar 05 10:00:14.996 2021] [3616:139662211758208] [error] uri_worker_map_update::jk_uri_worker_map.c (1320): Unable to stat the /etc/httpd/conf/uriworkermap.properties (errno=13) [Fri Mar 05 10:00:15.006 2021] [17221:139662211758208] [info] ajp_send_request::jk_ajp_common.c (1654): (worker3) failed sending request, socket -1 prepost cping/cpong failure (errno=110) [Fri Mar 05 10:00:15.006 2021] [17221:139662211758208] [info] ajp_send_request::jk_ajp_common.c (1712): (worker3) no usable connection found, will create a new one, detected by connect check (0), cping (1), send (0). [Fri Mar 05 10:00:15.026 2021] [16683:139662211758208] [info] ajp_handle_cping_cpong::jk_ajp_common.c (979): (worker3) timeout in reply cpong after 1 ms. Socket = 15 (event=110) [Fri Mar 05 10:00:15.051 2021] [27663:139662211758208] [info] ajp_send_request::jk_ajp_common.c (1654): (worker3) failed sending request, socket -1 prepost cping/cpong failure (errno=110) [Fri Mar 05 10:00:15.051 2021] [27663:139662211758208] [info] ajp_send_request::jk_ajp_common.c (1712): (worker3) no usable connection found, will create a new one, detected by connect check (0), cping (1), send (0). [Fri Mar 05 10:00:15.126 2021] [16683:139662211758208] [info] ajp_send_request::jk_ajp_common.c (1654): (worker3) failed sending request, socket -1 prepost cping/cpong failure (errno=110) [Fri Mar 05 10:00:15.126 2021] [16683:139662211758208] [info] ajp_send_request::jk_ajp_common.c (1712): (worker3) no usable connection found, will create a new one, detected by connect check (0), cping (1), send (0). [Fri Mar 05 10:00:15.667 2021] [18182:139662211758208] [info] ajp_handle_cping_cpong::jk_ajp_common.c (979): (worker3) timeout in reply cpong after 1 ms. Socket = 13 (event=110) the configuration is as below: /etc/httpd/conf/workers.properties #Set properties for template worker.template.type=ajp13 worker.template.host=localhost worker.template.ping_timeout=1 worker.template.ping_mode=A from the tomcat official doc - https://tomcat.apache.org/tomcat-8.5-doc/changelog.html this bug has been fixed in Tomcat 8.5.48 (markt) 63859: Do not perform a blocking read after a CPING message is received by the AJP connector because, if the JK Connector is configured with ping_mode="I", the CPING message will not always be followed by the start of a request. (markt) my tomcat version is 8.5.55. it should be fixed already in 8.5.55. When we see this error log, does it means this issue was not completely fixed in some case, or maybe, because the tomcat server is too busy, and did not respond timely? it impacts production stability, is it making sense to change the ping_mode=A to ping_mode=C to eliminate this issue? Given we cannot reproduce the issue in the lab, not sure if it is good to change to ping_mode=C, if change to ping_mode=c, is there any possible negative impact? any feedback would be much appreciated? -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859 Mark Thomas changed: What|Removed |Added Status|NEEDINFO|RESOLVED Resolution|--- |FIXED --- Comment #26 from Mark Thomas --- OK. Thanks for all you help on this. I'm marking this as resolved but do feel free to re-open if the issue re-surfaces. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859 --- Comment #25 from Aurelien Pernoud --- FYI I disabled the mod_jk debug to avoid having to clean it on a daily basis, I will continue to monitor with mode "A" if the issue eventually comes back but to me I would consider this one fix now ;) Many thanks Mark -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859 --- Comment #24 from Aurelien Pernoud --- 2 weeks now, no error anymore. I reactivated also mode "A" for 4 days now, still no error. I will wait more as I'm not in hurry but honestly I think we can consider you fixed it ! -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859 --- Comment #23 from Aurelien Pernoud --- There is absolutely no hurry ! I've been working for a week almost now and no errors... I'll leave it like that in our environment for the full month, I will even try to reactivate the full mode "A" in our test environment, and if it doesn't happen anymore then I'll consider it fix ;) Thanks -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859 --- Comment #22 from Mark Thomas --- The only thing that has changed as far as I know is Tomcat no longer always does a blocking read after the CPing/CPong. I still don't see how that could have caused the error though... Could we give it until after the weekend to see? I'll be tagging the next set of releases then. Meanwhile, I'll see if I can see how the blocking read could trigger this. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859 --- Comment #21 from Aurelien Pernoud --- Hi Mark, the error is not showing for 4 days. Is it possible that you or someone actually fixed something in this patch too from version 9.0.26 ?? Or Should I just wait a bit more... -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859 Aurelien Pernoud changed: What|Removed |Added Status|NEW |NEEDINFO --- Comment #20 from Aurelien Pernoud --- Hi Mark, I deployed your version with FINE debug as before, so far the server didn't catch fire which is pretty cool, as soon as I have the error (kept only CI mode) I will provide you the logs. Definitely appreciate your help on this one. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859 --- Comment #19 from Mark Thomas --- Thanks for those logs. The Tomcat debug logs were useful. I'm now reasonably sure Tomcat is closing the connection because of an IOException. However, the logs don't show the exception. I have extended the debug logging to include this exception in 8.5.x and 9.0.x. To save you building from source (I wasn't sure whether you were set up for that) I've uploaded the latest build here: http://people.apache.org/~markt/dev/apache-tomcat-9.0.30-dev.tar.gz Usual caveats apply. This isn't an official release. Use it at your own risk and if your server catches fire it isn't our fault ;) I really do appreciate your willingness to help track this down. Thanks again. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859 --- Comment #18 from Aurelien Pernoud --- Of course I didn't mention that but the catalina logs is the one from tom_tst01_srv09 ! -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859 Aurelien Pernoud changed: What|Removed |Added Status|NEEDINFO|NEW --- Comment #17 from Aurelien Pernoud --- Hi, I've reproduced the bug with mode CI after one day... do definitely related to CI I think :) I uploaded a new zip in your dropboy with : mod_jk.log.20191119 from both nodes You0'll see both of them have some "(tom_tst01_srv09) timeout in reply cpong after 1 ms.", not at the same timing. (I limited this log to the timing of the errors between 9 and 11 UTC) to avoid sending a huge file) catalina logs in FINE mode (org.apache.coyote.ajp.level = FINE org.apache.tomcat.util.net.level = FINE) Since startup and for the day. Let me know if I can help in any way -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859 --- Comment #16 from Aurelien Pernoud --- Small update : I've been running for a week with CP, no error in the logs. Will wait until next week and switch to CI see if it triggers the error or not. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859 --- Comment #15 from Aurelien Pernoud --- Hi Mark, thanks for all the testing. "It is interesting that - in the logs we have - it is only srv09 that is seeing these errors. Is there something about those two instances that is unique?" => Yes, it's the only instance runnning Tomcat 9.0.x. Others (srv01->srv08) are running Tomcat 7.0.x and 8.0.x. I have no 8.5.x instances, so if the "bug" you found is due to a change in Tomca > 8.5.x it might be it :) Anyway, I'll make tests with : CI / CP only (see which one triggers the rror) and also activate debug on Tomcat 9 as required and will try to provide everything back when possible. Appreciate your help ! -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859 Mark Thomas changed: What|Removed |Added Status|NEW |NEEDINFO --- Comment #14 from Mark Thomas --- I am unable to reproduce this. CPing/CPong mode "A" is equivalent to mode "CPI". Clearly, mode "C" is not an issue but it would be useful to know if it is enabling P or I that triggers the issue. You can use mode "CP" and "CI" to test this. I've been looking through the mod_jk logs in more detail and it is a P test that fails and takes Tomcat off-line but I see "I" test failures as well. It is interesting that - in the logs we have - it is only srv09 that is seeing these errors. Is there something about those two instances that is unique? I have also been looking at the Tomcat code. I do see something that isn't quite right (after a CPing/CPong Tomcat always does a blocking read) but I don't see how that could be triggering the error you see. I'll fix this anyway for 9.0.x and 8.5.x. The debug logging (as previously mentioned for Tomcat) may help. If not, we'll need to ad more logging on the Tomcat side to try and get to the bottom of this. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859 --- Comment #13 from Aurelien Pernoud --- Thanks Mark, appreciate all the time spent, I also agree this looks a complex one. Reading mod_jk logs it seems the cping / cpong is not coming back but as you also see the second httpd pointing to the same tomcat gets replies... So to me the issue is definitely in the cping cpong (mode A), but why does it fail sometimes, no idea ... Also as you can see in the logs the only server failing is the one running tomcat 9, all the others works fine, so to me the bug is on tomcat too. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859 --- Comment #12 from Mark Thomas --- Thanks. I can't see anything obviously wrong in those files. The CPING is sent and the CPONG is never seen. If any Tomcat committer wants access to the logs, let me know and I'll send you a link. Debug logging on the Tomcat side might be useful. Adding the following to logging.properties: org.apache.coyote.ajp.level = FINE org.apache.tomcat.util.net.level = FINE If that generates too much data, try starting with just the first of the two lines. Meanwhile, I am going to try and reproduce this. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859 --- Comment #11 from Aurelien Pernoud --- Thanks Mark, just uploaded everything. in "logs" : mod_jk debug log from both httpd instances limited to the period where the error occured tom_tst01_srv09 is the full tomcat log in "conf" : server.xml anonymized, I also provide the setenv as I use environment variables as you will see workers.properties also provided, except of course that here I switched it back to "C" mode, the error showed only with "A" mode. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859 --- Comment #10 from Mark Thomas --- I'll send you a Google Drive link privately where you can upload the files. Please upload the mod_jk debug logs, the Tomcat logs, the access log, the mod_jk configuration files and the server.xml from the Tomcat instances (with any passwords replaced by ***). Thanks. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859 --- Comment #9 from Aurelien Pernoud --- I also collected tomcat logs even though I can't find anything in it, except that "luckily" I also see for ex that our monitoring tool "pinged" (using HTTP protocol) the server at [04/Nov/2019:09:15:02 +] using "Probe PSI quickcheck", and this call worked like a charm... There was very limited activity too according to access_log. I can also provide this if needed. I don't know if I can provide any more details, let me know, for now I'm switching back to mode C to avoid this error, I just double checked again my production environment which has much more activity and I don't have this "timeout in reply cpong after 1 ms" showing at all. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859 Aurelien Pernoud changed: What|Removed |Added Status|NEEDINFO|NEW -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859 --- Comment #8 from Aurelien Pernoud --- Hi Mark, I have the debug but can't attach it due to limit of 1MB :-( I managed to reproduce the issue, I have 2 httpd with the exact same configuration, pointing to same tomcat. In the logs of "http node 1" I see for ex : [Mon Nov 04 09:15:03 2019] [13694:140683904751360] [info] ajp_handle_cping_cpong::jk_ajp_common.c (976): (tom_tst01_srv09) timeout in reply cpong after 1 ms. Socket = 25 (event=110) [Mon Nov 04 09:15:13 2019] [13694:140683904751360] [info] ajp_handle_cping_cpong::jk_ajp_common.c (976): (tom_tst01_srv09) timeout in reply cpong after 1 ms. Socket = 14 (event=110) [Mon Nov 04 09:15:24 2019] [13694:140683904751360] [info] ajp_handle_cping_cpong::jk_ajp_common.c (976): (tom_tst01_srv09) timeout in reply cpong after 1 ms. Socket = 14 (event=110) leading progressively to : [Mon Nov 04 09:15:24 2019] [13694:140683904751360] [error] ajp_send_request::jk_ajp_common.c (1725): (tom_tst01_srv09) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=110) (Here I only searched for the error) In the logs of "http node 2", at the same timing, the same server replies correctly to ping (at least that's how I interpret the below lines) : [Mon Nov 04 09:14:52 2019] [25023:139691817195264] [debug] ajp_maintain::jk_ajp_common.c (3441): (tom_tst01_srv09) reached pool min size 13 from 25 cache slots [Mon Nov 04 09:14:52 2019] [25023:139691817195264] [debug] ajp_maintain::jk_ajp_common.c (3493): (tom_tst01_srv09) pinged 1 and recycled 0 sockets in 0 seconds from 25 pool slots [Mon Nov 04 09:15:32 2019] [25022:139691817195264] [debug] ajp_maintain::jk_ajp_common.c (3441): (tom_tst01_srv09) reached pool min size 13 from 25 cache slots [Mon Nov 04 09:15:32 2019] [25022:139691817195264] [debug] ajp_maintain::jk_ajp_common.c (3493): (tom_tst01_srv09) pinged 1 and recycled 0 sockets in 0 seconds from 25 pool slots [Mon Nov 04 09:15:32 2019] [25024:139691817195264] [debug] ajp_maintain::jk_ajp_common.c (3441): (tom_tst01_srv09) reached pool min size 13 from 25 cache slots [Mon Nov 04 09:15:32 2019] [25024:139691817195264] [debug] ajp_maintain::jk_ajp_common.c (3493): (tom_tst01_srv09) pinged 2 and recycled 0 sockets in 0 seconds from 25 pool slots [Mon Nov 04 09:15:50 2019] [26242:139691817195264] [debug] ajp_maintain::jk_ajp_common.c (3441): (tom_tst01_srv09) reached pool min size 13 from 25 cache slots [Mon Nov 04 09:15:50 2019] [26242:139691817195264] [debug] ajp_maintain::jk_ajp_common.c (3493): (tom_tst01_srv09) pinged 2 and recycled 0 sockets in 0 seconds from 25 pool slots Both server times are synecd of course. I can send the logs, once zipped both nodes we are talking about a file of 30MB, let me know how ? -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859 Mark Thomas changed: What|Removed |Added Status|NEW |NEEDINFO --- Comment #7 from Mark Thomas --- I know it is verbose but what we really need is the mod_jk debug log from when this error occurs. Based on the information provided, that looks like the best option for further investigation. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859 --- Comment #6 from Aurelien Pernoud --- Tomcat 8.0.x -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859 --- Comment #5 from Christopher Schultz --- (In reply to Aurelien Pernoud from comment #3) > Tomcat 7&8 Is this Tomcat 8.0.x or Tomcat 8.5.x? -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859 --- Comment #4 from Aurelien Pernoud --- Also I even checked while the error was showing the "total connections" from my httpd to tomcat 9 ajp port (netstat) : it was less than 10. And when I stress test the ajp port, it worked. I'm lost :D -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859 --- Comment #3 from Aurelien Pernoud --- Hi Rainer, Any info on usage characteristics during times this happens? High load (how many requests per second), with or without load balancing, workers.properties config etc.? => There is no "rule", it even happen with only a few users on my test instance. Workers.properties is the same than for my tomcat 7&8 instances, e.g. I use a servertemplate which I apply to node and a lbtemplate too. Here is an extract : #Create one common template for all workers nodes worker.servertemplate.type=ajp13 # factor of each worker is the same worker.servertemplate.lbfactor=1 # ping_mode A is the most complete worker.servertemplate.ping_mode=C # socket_timeout in seconds worker.servertemplate.socket_timeout=30 # connection_pool_timeout in seconds worker.servertemplate.connection_pool_timeout=600 # reply_timeout in milliseconds : 10 min worker.servertemplate.reply_timeout=60 # recovery_options : 3 : don't retry on error after request was sent worker.servertemplate.recovery_options=3 # Create one common template for all LB nodes worker.balancertemplate.type=lb # If we have 10 replies timing out in the same minute => worker goes in error state worker.balancertemplate.max_reply_timeouts=10 Then based on this I apply those settings to nodes / clusters. How frequently does it happen (always, sporadicaly for NNN % of requests, in spikes, ...)? => I couldn't find any "rule" sorry but it happened at least 10 times / day and since I changed the cping cpong to C it doesn't show anymore in the logs even though I have activity Can you easily reproduce? => Yes, I have a test environment which is in use and if I push back cping/cpong mode to A I'm sure the error will show up again. Do you have root privileges, so could you sniff network traffic to the AJP port? Are Apache and Tomcat on diferent machines? Any active components (Firewalls, routers) in between? => I'm not root unfortunately, but my config is that I run 2 linux servers, which both hosts httpd and tomcat instances (both servers runs same versions of httpd, tomcat), and it works fine with the ajp port of tomcat 7&8. . There is no firewall between them (hard or soft), and the error occurs even when the httpd is connecting to "himself" (even though I don't use "localhost" but the hostname) I've ran this setup for more than a year on Tomcat 7&8 (even with upgrades), and only met the issue with 9. It failes with 9.0.20 so I tried to upgrade last week to 9.0.26 but still here. Since yesterday and switching the ping_mode to "C" it stopped failing, so for now I will go like that in production but might be good to investigate. Let me know if I can be of any help. FYI I tried : - putting mod_jk in debug => way to verbose (I run 9 instances of tomcat in the end on the same server), but at the beginning the behaviour looks correct (I saw the ping pong working with my tomcat 9 instances every minute) - adding debug in tomcat "AJP" : couldn't find exactly what I could put in debug... if I put the full org.apache logger in DEBUG it's nightmare :) Let me know if I can help, and thanks for replying so quickly -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859 --- Comment #2 from Rainer Jung --- Any info on usage characteristics during times this happens? High load (how many requests per second), with or without load balancing, workers.properties config etc.? How frequently does it happen (always, sporadicaly for NNN % of requests, in spikes, ...)? Can you easily reproduce? Do you have root privileges, so could you sniff network traffic to the AJP port? Are Apache and Tomcat on diferent machines? Any active components (Firewalls, routers) in between? Thanks and regards, Rainer -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859 Aurelien Pernoud changed: What|Removed |Added CC||aurel...@pernoud.org -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859 Aurelien Pernoud changed: What|Removed |Added OS||All --- Comment #1 from Aurelien Pernoud --- I missed a sentence... We implemented a while ago cping/pong om httpd with mode A, and it works perfectly with Tomcat 7&8. +"But with Tomcat 9 we inconsistently have those errors of cping/cpong in logs" And then the logs... -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org