[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x

2021-03-06 Thread bugzilla
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859

--- Comment #27 from JasonDo  ---
Dear all, 

In our production environment, we did see mod_jk log print log. 
Fri Mar 05 10:00:14.906 2021] [17221:139662211758208] [info]
ajp_handle_cping_cpong::jk_ajp_common.c (979): (worker3) timeout in reply cpong
after 1 ms. Socket = 13 (event=110)
[Fri Mar 05 10:00:14.951 2021] [27663:139662211758208] [info]
ajp_handle_cping_cpong::jk_ajp_common.c (979): (worker3) timeout in reply cpong
after 1 ms. Socket = 13 (event=110)
[Fri Mar 05 10:00:14.971 2021] [3615:139662211758208] [error]
uri_worker_map_update::jk_uri_worker_map.c (1320): Unable to stat the
/etc/httpd/conf/uriworkermap.properties (errno=13)
[Fri Mar 05 10:00:14.990 2021] [3617:139662211758208] [error]
uri_worker_map_update::jk_uri_worker_map.c (1320): Unable to stat the
/etc/httpd/conf/uriworkermap.properties (errno=13)
[Fri Mar 05 10:00:14.996 2021] [3616:139662211758208] [error]
uri_worker_map_update::jk_uri_worker_map.c (1320): Unable to stat the
/etc/httpd/conf/uriworkermap.properties (errno=13)
[Fri Mar 05 10:00:15.006 2021] [17221:139662211758208] [info]
ajp_send_request::jk_ajp_common.c (1654): (worker3) failed sending request,
socket -1 prepost cping/cpong failure (errno=110)
[Fri Mar 05 10:00:15.006 2021] [17221:139662211758208] [info]
ajp_send_request::jk_ajp_common.c (1712): (worker3) no usable connection found,
will create a new one, detected by connect check (0), cping (1), send (0).
[Fri Mar 05 10:00:15.026 2021] [16683:139662211758208] [info]
ajp_handle_cping_cpong::jk_ajp_common.c (979): (worker3) timeout in reply cpong
after 1 ms. Socket = 15 (event=110)
[Fri Mar 05 10:00:15.051 2021] [27663:139662211758208] [info]
ajp_send_request::jk_ajp_common.c (1654): (worker3) failed sending request,
socket -1 prepost cping/cpong failure (errno=110)
[Fri Mar 05 10:00:15.051 2021] [27663:139662211758208] [info]
ajp_send_request::jk_ajp_common.c (1712): (worker3) no usable connection found,
will create a new one, detected by connect check (0), cping (1), send (0).
[Fri Mar 05 10:00:15.126 2021] [16683:139662211758208] [info]
ajp_send_request::jk_ajp_common.c (1654): (worker3) failed sending request,
socket -1 prepost cping/cpong failure (errno=110)
[Fri Mar 05 10:00:15.126 2021] [16683:139662211758208] [info]
ajp_send_request::jk_ajp_common.c (1712): (worker3) no usable connection found,
will create a new one, detected by connect check (0), cping (1), send (0).
[Fri Mar 05 10:00:15.667 2021] [18182:139662211758208] [info]
ajp_handle_cping_cpong::jk_ajp_common.c (979): (worker3) timeout in reply cpong
after 1 ms. Socket = 13 (event=110)


the configuration is as below: 
/etc/httpd/conf/workers.properties
#Set properties for template
worker.template.type=ajp13
worker.template.host=localhost
worker.template.ping_timeout=1
worker.template.ping_mode=A

from the tomcat official doc -
https://tomcat.apache.org/tomcat-8.5-doc/changelog.html

this bug has been fixed in Tomcat 8.5.48 (markt)
 63859: Do not perform a blocking read after a CPING message is received by the
AJP connector because, if the JK Connector is configured with ping_mode="I",
the CPING message will not always be followed by the start of a request.
(markt)

my tomcat version is 8.5.55. it should be fixed already in 8.5.55. 

When we see this error log, does it means this issue was not completely fixed
in some case, or maybe, because the tomcat server is too busy, and did not
respond timely? 

it impacts production stability, is it making sense to change the ping_mode=A
to ping_mode=C to eliminate this issue? 
Given we cannot reproduce the issue in the lab, not sure if it is good to
change to ping_mode=C, if change to ping_mode=c, is there any possible negative
impact?

any feedback would be much appreciated?

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org



[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x

2019-12-10 Thread bugzilla
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859

Mark Thomas  changed:

   What|Removed |Added

 Status|NEEDINFO|RESOLVED
 Resolution|--- |FIXED

--- Comment #26 from Mark Thomas  ---
OK. Thanks for all you help on this. I'm marking this as resolved but do feel
free to re-open if the issue re-surfaces.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org



[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x

2019-12-09 Thread bugzilla
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859

--- Comment #25 from Aurelien Pernoud  ---
FYI I disabled the mod_jk debug to avoid having to clean it on a daily basis, I
will continue to monitor with mode "A" if the issue eventually comes back but
to me I would consider this one fix now ;)

Many thanks Mark

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org



[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x

2019-12-05 Thread bugzilla
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859

--- Comment #24 from Aurelien Pernoud  ---
2 weeks now, no error anymore.
I reactivated also mode "A" for 4 days now, still no error.
I will wait more as I'm not in hurry but honestly I think we can consider you
fixed it !

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org



[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x

2019-11-28 Thread bugzilla
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859

--- Comment #23 from Aurelien Pernoud  ---
There is absolutely no hurry !
I've been working for a week almost now and no errors...
I'll leave it like that in our environment for the full month, I will even try
to reactivate the full mode "A" in our test environment, and if it doesn't
happen anymore then I'll consider it fix ;)
Thanks

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org



[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x

2019-11-27 Thread bugzilla
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859

--- Comment #22 from Mark Thomas  ---
The only thing that has changed as far as I know is Tomcat no longer always
does a blocking read after the CPing/CPong.

I still don't see how that could have caused the error though...

Could we give it until after the weekend to see? I'll be tagging the next set
of releases then. Meanwhile, I'll see if I can see how the blocking read could
trigger this.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org



[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x

2019-11-27 Thread bugzilla
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859

--- Comment #21 from Aurelien Pernoud  ---
Hi Mark,

the error is not showing for 4 days. Is it possible that you or someone
actually fixed something in this patch too from version 9.0.26 ?? Or Should I
just wait a bit more...

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org



[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x

2019-11-21 Thread bugzilla
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859

Aurelien Pernoud  changed:

   What|Removed |Added

 Status|NEW |NEEDINFO

--- Comment #20 from Aurelien Pernoud  ---
Hi Mark,

I deployed your version with FINE debug as before, so far the server didn't
catch fire which is pretty cool, as soon as I have the error (kept only CI
mode) I will provide you the logs.

Definitely appreciate your help on this one.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org



[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x

2019-11-19 Thread bugzilla
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859

--- Comment #19 from Mark Thomas  ---
Thanks for those logs. The Tomcat debug logs were useful. I'm now reasonably
sure Tomcat is closing the connection because of an IOException. However, the
logs don't show the exception. I have extended the debug logging to include
this exception in 8.5.x and 9.0.x.

To save you building from source (I wasn't sure whether you were set up for
that) I've uploaded the latest build here:
http://people.apache.org/~markt/dev/apache-tomcat-9.0.30-dev.tar.gz

Usual caveats apply. This isn't an official release. Use it at your own risk
and if your server catches fire it isn't our fault ;)

I really do appreciate your willingness to help track this down. Thanks again.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org



[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x

2019-11-19 Thread bugzilla
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859

--- Comment #18 from Aurelien Pernoud  ---
Of course I didn't mention that but the catalina logs is the one from
tom_tst01_srv09 !

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org



[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x

2019-11-19 Thread bugzilla
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859

Aurelien Pernoud  changed:

   What|Removed |Added

 Status|NEEDINFO|NEW

--- Comment #17 from Aurelien Pernoud  ---
Hi,

I've reproduced the bug with mode CI after one day... do definitely related to
CI I think :)

I uploaded a new zip in your dropboy with :

mod_jk.log.20191119 from both nodes
You0'll see both of them have some "(tom_tst01_srv09) timeout in reply cpong
after 1 ms.", not at the same timing.

(I limited this log to the timing of the errors between 9 and 11 UTC) to avoid
sending a huge file)


catalina logs in FINE mode 
(org.apache.coyote.ajp.level = FINE
org.apache.tomcat.util.net.level = FINE)

Since startup and for the day.

Let me know if I can help in any way

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org



[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x

2019-11-15 Thread bugzilla
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859

--- Comment #16 from Aurelien Pernoud  ---
Small update : I've been running for a week with CP, no error in the logs.
Will wait until next week and switch to CI see if it triggers the error or not.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org



[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x

2019-11-11 Thread bugzilla
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859

--- Comment #15 from Aurelien Pernoud  ---
Hi Mark,

thanks for all the testing.

"It is interesting that - in the logs we have - it is only srv09 that is seeing
these errors. Is there something about those two instances that is unique?"

=> Yes, it's the only instance runnning Tomcat 9.0.x. Others (srv01->srv08) are
running Tomcat 7.0.x and 8.0.x.
I have no 8.5.x instances, so if the "bug" you found is due to a change in
Tomca > 8.5.x it might be it :)

Anyway, I'll make tests with : CI / CP only (see which one triggers the rror)
and also activate debug on Tomcat 9 as required and will try to provide
everything back when possible.

Appreciate your help !

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org



[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x

2019-11-08 Thread bugzilla
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859

Mark Thomas  changed:

   What|Removed |Added

 Status|NEW |NEEDINFO

--- Comment #14 from Mark Thomas  ---
I am unable to reproduce this.

CPing/CPong mode "A" is equivalent to mode "CPI". Clearly, mode "C" is not an
issue but it would be useful to know if it is enabling P or I that triggers the
issue. You can use mode "CP" and "CI" to test this.

I've been looking through the mod_jk logs in more detail and it is a P test
that fails and takes Tomcat off-line but I see "I" test failures as well.

It is interesting that - in the logs we have - it is only srv09 that is seeing
these errors. Is there something about those two instances that is unique?

I have also been looking at the Tomcat code. I do see something that isn't
quite right (after a CPing/CPong Tomcat always does a blocking read) but I
don't see how that could be triggering the error you see. I'll fix this anyway
for 9.0.x and 8.5.x.

The debug logging (as previously mentioned for Tomcat) may help. If not, we'll
need to ad more logging on the Tomcat side to try and get to the bottom of
this.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org



[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x

2019-11-05 Thread bugzilla
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859

--- Comment #13 from Aurelien Pernoud  ---
Thanks Mark,

appreciate all the time spent, I also agree this looks a complex one.

Reading mod_jk logs it seems the cping / cpong is not coming back but as you
also see the second httpd pointing to the same tomcat gets replies...

So to me the issue is definitely in the cping cpong (mode A), but why does it
fail sometimes, no idea ... Also as you can see in the logs the only server
failing is the one running tomcat 9, all the others works fine, so to me the
bug is on tomcat too.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org



[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x

2019-11-05 Thread bugzilla
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859

--- Comment #12 from Mark Thomas  ---
Thanks. I can't see anything obviously wrong in those files. The CPING is sent
and the CPONG is never seen.

If any Tomcat committer wants access to the logs, let me know and I'll send you
a link.

Debug logging on the Tomcat side might be useful. Adding the following to
logging.properties:
org.apache.coyote.ajp.level = FINE
org.apache.tomcat.util.net.level = FINE

If that generates too much data, try starting with just the first of the two
lines.

Meanwhile, I am going to try and reproduce this.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org



[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x

2019-11-04 Thread bugzilla
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859

--- Comment #11 from Aurelien Pernoud  ---
Thanks Mark, just uploaded everything.

in "logs" : mod_jk debug log from both httpd instances limited to the period
where the error occured
tom_tst01_srv09 is the full tomcat log

in "conf" : server.xml anonymized, I also provide the setenv as I use
environment variables as you will see
workers.properties also provided, except of course that here I switched it back
to "C" mode, the error showed only with "A" mode.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org



[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x

2019-11-04 Thread bugzilla
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859

--- Comment #10 from Mark Thomas  ---
I'll send you a Google Drive link privately where you can upload the files.

Please upload the mod_jk debug logs, the Tomcat logs, the access log, the
mod_jk configuration files and the server.xml from the Tomcat instances (with
any passwords replaced by ***).

Thanks.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org



[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x

2019-11-04 Thread bugzilla
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859

--- Comment #9 from Aurelien Pernoud  ---
I also collected tomcat logs even though I can't find anything in it, except
that "luckily" I also see for ex that our monitoring tool "pinged" (using HTTP
protocol) the server at  [04/Nov/2019:09:15:02 +] using "Probe PSI
quickcheck", and this call worked like a charm...
There was very limited activity too according to access_log.
I can also provide this if needed.

I don't know if I can provide any more details, let me know, for now I'm
switching back to mode C to avoid this error, I just double checked again my
production environment which has much more activity and I don't have this
"timeout in reply cpong after 1 ms" showing at all.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org



[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x

2019-11-04 Thread bugzilla
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859

Aurelien Pernoud  changed:

   What|Removed |Added

 Status|NEEDINFO|NEW

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org



[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x

2019-11-04 Thread bugzilla
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859

--- Comment #8 from Aurelien Pernoud  ---
Hi Mark,

I have the debug but can't attach it due to limit of 1MB :-(

I managed to reproduce the issue, I have 2 httpd with the exact same
configuration, pointing to same tomcat.

In the logs of "http node 1" I see for ex :

[Mon Nov 04 09:15:03 2019] [13694:140683904751360] [info]
ajp_handle_cping_cpong::jk_ajp_common.c (976): (tom_tst01_srv09) timeout in
reply cpong after 1 ms. Socket = 25 (event=110)
[Mon Nov 04 09:15:13 2019] [13694:140683904751360] [info]
ajp_handle_cping_cpong::jk_ajp_common.c (976): (tom_tst01_srv09) timeout in
reply cpong after 1 ms. Socket = 14 (event=110)
[Mon Nov 04 09:15:24 2019] [13694:140683904751360] [info]
ajp_handle_cping_cpong::jk_ajp_common.c (976): (tom_tst01_srv09) timeout in
reply cpong after 1 ms. Socket = 14 (event=110)

leading progressively to :

[Mon Nov 04 09:15:24 2019] [13694:140683904751360] [error]
ajp_send_request::jk_ajp_common.c (1725): (tom_tst01_srv09) connecting to
backend failed. Tomcat is probably not started or is listening on the wrong
port (errno=110)

(Here I only searched for the error)

In the logs of "http node 2", at the same timing, the same server replies
correctly to ping (at least that's how I interpret the below lines) :

[Mon Nov 04 09:14:52 2019] [25023:139691817195264] [debug]
ajp_maintain::jk_ajp_common.c (3441): (tom_tst01_srv09) reached pool min size
13 from 25 cache slots
[Mon Nov 04 09:14:52 2019] [25023:139691817195264] [debug]
ajp_maintain::jk_ajp_common.c (3493): (tom_tst01_srv09) pinged 1 and recycled 0
sockets in 0 seconds from 25 pool slots
[Mon Nov 04 09:15:32 2019] [25022:139691817195264] [debug]
ajp_maintain::jk_ajp_common.c (3441): (tom_tst01_srv09) reached pool min size
13 from 25 cache slots
[Mon Nov 04 09:15:32 2019] [25022:139691817195264] [debug]
ajp_maintain::jk_ajp_common.c (3493): (tom_tst01_srv09) pinged 1 and recycled 0
sockets in 0 seconds from 25 pool slots
[Mon Nov 04 09:15:32 2019] [25024:139691817195264] [debug]
ajp_maintain::jk_ajp_common.c (3441): (tom_tst01_srv09) reached pool min size
13 from 25 cache slots
[Mon Nov 04 09:15:32 2019] [25024:139691817195264] [debug]
ajp_maintain::jk_ajp_common.c (3493): (tom_tst01_srv09) pinged 2 and recycled 0
sockets in 0 seconds from 25 pool slots
[Mon Nov 04 09:15:50 2019] [26242:139691817195264] [debug]
ajp_maintain::jk_ajp_common.c (3441): (tom_tst01_srv09) reached pool min size
13 from 25 cache slots
[Mon Nov 04 09:15:50 2019] [26242:139691817195264] [debug]
ajp_maintain::jk_ajp_common.c (3493): (tom_tst01_srv09) pinged 2 and recycled 0
sockets in 0 seconds from 25 pool slots

Both server times are synecd of course.

I can send the logs, once zipped both nodes we are talking about a file of
30MB, let me know how ?

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org



[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x

2019-10-28 Thread bugzilla
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859

Mark Thomas  changed:

   What|Removed |Added

 Status|NEW |NEEDINFO

--- Comment #7 from Mark Thomas  ---
I know it is verbose but what we really need is the mod_jk debug log from when
this error occurs. Based on the information provided, that looks like the best
option for further investigation.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org



[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x

2019-10-18 Thread bugzilla
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859

--- Comment #6 from Aurelien Pernoud  ---
Tomcat 8.0.x

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org



[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x

2019-10-18 Thread bugzilla
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859

--- Comment #5 from Christopher Schultz  ---
(In reply to Aurelien Pernoud from comment #3)
> Tomcat 7&8

Is this Tomcat 8.0.x or Tomcat 8.5.x?

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org



[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x

2019-10-17 Thread bugzilla
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859

--- Comment #4 from Aurelien Pernoud  ---
Also I even checked while the error was showing the "total connections" from my
httpd to tomcat 9 ajp port (netstat) : it was less than 10.
And when I stress test the ajp port, it worked. I'm lost :D

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org



[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x

2019-10-17 Thread bugzilla
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859

--- Comment #3 from Aurelien Pernoud  ---
Hi Rainer,

Any info on usage characteristics during times this happens? High load (how
many requests per second), with or without load balancing, workers.properties
config etc.?

=> There is no "rule", it even happen with only a few users on my test
instance.
Workers.properties is the same than for my tomcat 7&8 instances, e.g. I use a
servertemplate which I apply to node and a lbtemplate too.

Here is an extract :
#Create one common template for all workers nodes
worker.servertemplate.type=ajp13
# factor of each worker is the same
worker.servertemplate.lbfactor=1
# ping_mode A is the most complete
worker.servertemplate.ping_mode=C
# socket_timeout in seconds
worker.servertemplate.socket_timeout=30
# connection_pool_timeout in seconds
worker.servertemplate.connection_pool_timeout=600
# reply_timeout in milliseconds : 10 min
worker.servertemplate.reply_timeout=60
# recovery_options : 3 : don't retry on error after request was sent
worker.servertemplate.recovery_options=3

# Create one common template for all LB nodes
worker.balancertemplate.type=lb
# If we have 10 replies timing out in the same minute => worker goes in error
state
worker.balancertemplate.max_reply_timeouts=10

Then based on this I apply those settings to nodes / clusters.

How frequently does it happen (always, sporadicaly for NNN % of requests, in
spikes, ...)?

=> I couldn't find any "rule" sorry but it happened at least 10 times / day
and since I changed the cping cpong to C it doesn't show anymore in the logs
even though I have activity

Can you easily reproduce?

=> Yes, I have a test environment which is in use and if I push back
cping/cpong mode to A I'm sure the error will show up again.

Do you have root privileges, so could you sniff network traffic to the AJP
port? Are Apache and Tomcat on diferent machines? Any active components
(Firewalls, routers) in between?

=> I'm not root unfortunately, but my config is that I run 2 linux servers,
which both hosts httpd and tomcat instances (both servers runs same versions of
httpd, tomcat), and it works fine with the ajp port of tomcat 7&8.
.
There is no firewall between them (hard or soft), and the error occurs even
when the httpd is connecting to "himself" (even though I don't use "localhost"
but the hostname)

I've ran this setup for more than a year on Tomcat 7&8 (even with upgrades),
and only met the issue with 9. It failes with 9.0.20 so I tried to upgrade last
week to 9.0.26 but still here. 
Since yesterday and switching the ping_mode to "C" it stopped failing, so for
now I will go like that in production but might be good to investigate.

Let me know if I can be of any help.

FYI I tried :
- putting mod_jk in debug => way to verbose (I run 9 instances of tomcat in the
end on the same server), but at the beginning the behaviour looks correct (I
saw the ping pong working with my tomcat 9 instances every minute)
- adding debug in tomcat "AJP" : couldn't find exactly what I could put in
debug... if I put the full org.apache logger in DEBUG it's nightmare :)

Let me know if I can help, and thanks for replying so quickly

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org



[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x

2019-10-17 Thread bugzilla
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859

--- Comment #2 from Rainer Jung  ---
Any info on usage characteristics during times this happens? High load (how
many requests per second), with or without load balancing, workers.properties
config etc.?

How frequently does it happen (always, sporadicaly for NNN % of requests, in
spikes, ...)?

Can you easily reproduce?

Do you have root privileges, so could you sniff network traffic to the AJP
port? Are Apache and Tomcat on diferent machines? Any active components
(Firewalls, routers) in between?

Thanks and regards,

Rainer

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org



[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x

2019-10-17 Thread bugzilla
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859

Aurelien Pernoud  changed:

   What|Removed |Added

 CC||aurel...@pernoud.org

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org



[Bug 63859] AJP cping/cpong mode failing on Tomcat 9.x

2019-10-17 Thread bugzilla
https://bz.apache.org/bugzilla/show_bug.cgi?id=63859

Aurelien Pernoud  changed:

   What|Removed |Added

 OS||All

--- Comment #1 from Aurelien Pernoud  ---
I missed a sentence...

We implemented a while ago cping/pong om httpd with mode A, and it works
perfectly with Tomcat 7&8.
+"But with Tomcat 9 we inconsistently have those errors of cping/cpong in logs"

And then the logs...

-- 
You are receiving this mail because:
You are the assignee for the bug.
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org