Re: Zero downtime deployments

[Frederik Nosi]

Hi Christopher,

Il 23/12/2015 19:12, Christopher Schultz ha scritto:

Jason,

On 12/22/15 11:46 PM, Jason Britton wrote:

On Tue, Dec 22, 2015 at 4:01 PM, Christopher Schultz <
ch...@christopherschultz.net> wrote:

So mod_jk routes a request to the node which is going down, and then
decides to re-route because the connection times-out? Just making sure I
have that all in my head (it's an awfully long sentence).

my understanding, could be wrong, that if the connect_timeout is reached by
mod_jk that it would try a different node.

I can't remember if it will auto-retry; it might depend upon your
configuration.


This is configurable, usually i set:

worker.$WORKER_NAME.recovery_options=19

that is from memory, retry only GET and HEAD, not POST.


This because it have occured to me, that a POST timeouts to some app 
running in a tomcat backend but the tomcat side processing thread does 
not know this and it continues running. JK retries the same POST in 
another tomcat backend. In this case you get your POST request executed 
at least twice which usually is not nice.



BTW this thread reminds me about a patch i sent but never followed on 
being busy with other stuff:


https://mail-archives.apache.org/mod_mbox/tomcat-users/201404.mbox/%3c533ec184.7070...@postecom.it%3E


The problem i tried to solve was that when you have N backends and your 
site get's slashdotted so that backends are all busy, mod_jk amplifies 
the load even more, as it retries at least all N backends. With that 
patch i added an option for thelling mod_jk to retry only X backends, 
usually only once.





A different active tomcat node now receives the request
and tries to reconstitute the user's session from the same shared
JDBCStore, but what if the first tomcat node is not finished shutting

down

and has not finished writing out this particular user's session data yet?
How can we ensure that session data will be there?

Although I'm not entirely sure of the behavior of Tomcat's clustering
features in that particular case, you are mostly asking the following:
"what happens if two nodes are essentially sharing a session? how do I
make sure their view of the session is consistent?"


I should have clarified the environment I was envisioning was one with
sticky sessions, no clustering.  And the situation I'm looking to wrap my
brain around is one where the once active node is triggered to write out
session data due to the impending shut down of the particular node.

At about the same time this shutdown is occurring a request comes in for a
client that has a session cookie value routing it to the node being shut
down.  mod_jk times out trying to connect to this node and reroutes the
request to an active node.  I see a race condition where depending on how
long it takes the first tomcat being shutdown to write out session data to
the JDBCStore, that the subsequent tomcat node trying to service the
request may not find this user's session data in the shared JDBCStore.  I'd
really like to know if this is truly something to be concerned about, and
if so, how to account for it.

This scenario really isn't any different than a clustered environment
with two requests going to two different nodes (for any reason, really).
I think this answer still stands:


I think the answer is: you can't. Make your requests as idempotent as
possible and, when possible, execute the other kinds of requests such a
way that the first one to execute "wins" and the others fail gracefully.

-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Regards,
Frederik

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 8 reliability/performance on Windows 2008 R2 Server vs. RHEL/CentOS

[Frederik Nosi]

On 10/02/2015 01:05 PM, David kerber wrote:

On 10/2/2015 3:47 AM, Mark Thomas wrote:

On 02/10/2015 00:39, Jason Britton wrote:




And if Mark says there shouldn't
be problems, I think that is some implicit guarantee of free support 
from

him ;)


:)

Bugs with reproducible tests cases are always welcome.


I have a lot of convenience shell scripts here and there but nothing I
couldn't replicate in powershell I imagine.  I would miss grep and 
find and

all the other super useful Linux command line operations.  I'm not sure
I've ever been able to successfully search for text in files on a 
windows

machine server or otherwise (except when using notepad++).


I've had similar problems using the OS provided tools on Windows. I've
long since switched to this:

http://www.wingrep.com/

and never looked back.


AstroGrep!  http://astrogrep.sourceforge.net/


Actually tail and other shell commands would be another heavy miss for 
me, it was at the times i had to manage stuff on Windows years ago. I 
got used to install Cygwin as first step in those times:


https://www.cygwin.com/







-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 8 reliability/performance on Windows 2008 R2 Server vs. RHEL/CentOS

[Frederik Nosi]

Christopher,

On 10/02/2015 09:40 PM, Christopher Schultz wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Frederik,

On 10/2/15 12:29 PM, Frederik Nosi wrote:

On 10/02/2015 01:05 PM, David kerber wrote:

On 10/2/2015 3:47 AM, Mark Thomas wrote:

On 02/10/2015 00:39, Jason Britton wrote:




And if Mark says there shouldn't be problems, I think that is
some implicit guarantee of free support from him ;)

:)

Bugs with reproducible tests cases are always welcome.


I have a lot of convenience shell scripts here and there but
nothing I couldn't replicate in powershell I imagine.  I
would miss grep and find and all the other super useful Linux
command line operations.  I'm not sure I've ever been able to
successfully search for text in files on a windows machine
server or otherwise (except when using notepad++).

I've had similar problems using the OS provided tools on
Windows. I've long since switched to this:

http://www.wingrep.com/

and never looked back.

AstroGrep!  http://astrogrep.sourceforge.net/

Actually tail and other shell commands would be another heavy miss
for me, it was at the times i had to manage stuff on Windows years
ago. I got used to install Cygwin as first step in those times:

https://www.cygwin.com/

There are also the win32 builds of GNU tools, but they all seem to run
terribly slowly. I think it's because the standard C API is probably
implemented using the win32 API so there's a layer of indirection
around a lot of things.


I know i know ... horror stories ...


Cygwin is an option, but it's always ... just not the same. It also
seems sluggish and things get really bizarre with path names and
stuff. Some tools work well with the path insanity and others fail
horribly.


I do have to support java applications on unix where you can find inside:

C:\...\Documents\... \Someone

..

So i know.

Anyway, although slow at least make you actually grep / tail and such. I 
really cant understand how one can check logs under win, but maybe 
that's me.


All this though, has little to do with Window's kernel, it's the 
userspace that IMHO is not adequate.






- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org

iQIcBAEBCAAGBQJWDt2jAAoJEBzwKT+lPKRYJecP/RWW8Al03MskdIKgjNmOgYB2
xic7xS+QYHYVeqw9tAdLflxZnkJuCEanU1vuC3vgRhi4MgLYQFmEhY7oH0h55fvT
WEtrIQNhLMlbIF1V2NTGMJQwnK+zREWJTIBTbAfoPBfft3+jHlOGFz75qydqYn72
EoS2jPQvb9mmMbbf57q0mCAHfeuUSke68tuuzRyBvbo5GjOffwp/7wgAyZuo+KRE
86WFPkrJZ47QirUxJh8D2FxpQE1rBcX5qy41cY2bYhQt/qqOSnzGr08iefBj2+db
cBROH3Bc8NGyBsclD3sC5qDlMphthpdEjhYCdFcXUdee4zTbURXWcmmHN74pmWGP
AasjPrMArX7OsXn74wX3Q8xwm45JAXFZwqCjar/HUsXp96tPe2lW179NM9QVA2nV
wub3dQflJhX80g4ppWH//Hw28Pbr4K3CETcsmsDQHjR0MOinjfbBQQj7ZiWwPEGw
Ziub6P73rNQaVdt/OY8DzrcPxfTUIyObH8qGLKoY2CX8/q/FTWiu0FtpuTMOS/Q/
L+QUnS6KUCVeJY1jPx+9qu3WcgAuXYa66dbxuqzpiuh+eJ4YA1N3AWX3Hw1lAm9B
5rPH43332hXcu7iJf3c5gZKNdPFq9/2XTGP2Dsjp2w8Lmpn+nBO/fv6+hm12DLac
uZdQVFGzkdln3uDuIhKM
=amjv
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 8 reliability/performance on Windows 2008 R2 Server vs. RHEL/CentOS

[Frederik Nosi]

On 10/02/2015 10:10 PM, Frederik Nosi wrote:

Christopher,

On 10/02/2015 09:40 PM, Christopher Schultz wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Frederik,

On 10/2/15 12:29 PM, Frederik Nosi wrote:

On 10/02/2015 01:05 PM, David kerber wrote:

On 10/2/2015 3:47 AM, Mark Thomas wrote:

On 02/10/2015 00:39, Jason Britton wrote:




And if Mark says there shouldn't be problems, I think that is
some implicit guarantee of free support from him ;)

:)

Bugs with reproducible tests cases are always welcome.


I have a lot of convenience shell scripts here and there but
nothing I couldn't replicate in powershell I imagine.  I
would miss grep and find and all the other super useful Linux
command line operations.  I'm not sure I've ever been able to
successfully search for text in files on a windows machine
server or otherwise (except when using notepad++).

I've had similar problems using the OS provided tools on
Windows. I've long since switched to this:

http://www.wingrep.com/

and never looked back.

AstroGrep!  http://astrogrep.sourceforge.net/

Actually tail and other shell commands would be another heavy miss
for me, it was at the times i had to manage stuff on Windows years
ago. I got used to install Cygwin as first step in those times:

https://www.cygwin.com/

There are also the win32 builds of GNU tools, but they all seem to run
terribly slowly. I think it's because the standard C API is probably
implemented using the win32 API so there's a layer of indirection
around a lot of things.


I know i know ... horror stories ...


Cygwin is an option, but it's always ... just not the same. It also
seems sluggish and things get really bizarre with path names and
stuff. Some tools work well with the path insanity and others fail
horribly.


I do have to support java applications on unix where you can find inside:

C:\...\Documents\... \Someone

..

So i know.

Anyway, although slow at least make you actually grep / tail and such. 
I really cant understand how one can check logs under win, but maybe 
that's me.


Sorry, i was talking about Cygwin or win32 builds in case it wasn't clear



All this though, has little to do with Window's kernel, it's the 
userspace that IMHO is not adequate.






- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org

iQIcBAEBCAAGBQJWDt2jAAoJEBzwKT+lPKRYJecP/RWW8Al03MskdIKgjNmOgYB2
xic7xS+QYHYVeqw9tAdLflxZnkJuCEanU1vuC3vgRhi4MgLYQFmEhY7oH0h55fvT
WEtrIQNhLMlbIF1V2NTGMJQwnK+zREWJTIBTbAfoPBfft3+jHlOGFz75qydqYn72
EoS2jPQvb9mmMbbf57q0mCAHfeuUSke68tuuzRyBvbo5GjOffwp/7wgAyZuo+KRE
86WFPkrJZ47QirUxJh8D2FxpQE1rBcX5qy41cY2bYhQt/qqOSnzGr08iefBj2+db
cBROH3Bc8NGyBsclD3sC5qDlMphthpdEjhYCdFcXUdee4zTbURXWcmmHN74pmWGP
AasjPrMArX7OsXn74wX3Q8xwm45JAXFZwqCjar/HUsXp96tPe2lW179NM9QVA2nV
wub3dQflJhX80g4ppWH//Hw28Pbr4K3CETcsmsDQHjR0MOinjfbBQQj7ZiWwPEGw
Ziub6P73rNQaVdt/OY8DzrcPxfTUIyObH8qGLKoY2CX8/q/FTWiu0FtpuTMOS/Q/
L+QUnS6KUCVeJY1jPx+9qu3WcgAuXYa66dbxuqzpiuh+eJ4YA1N3AWX3Hw1lAm9B
5rPH43332hXcu7iJf3c5gZKNdPFq9/2XTGP2Dsjp2w8Lmpn+nBO/fv6+hm12DLac
uZdQVFGzkdln3uDuIhKM
=amjv
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org






-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Need help understanding support for Unix Domain Sockets in Tomcat 7.0.x

[Frederik Nosi]

Hi,
On 09/26/2015 02:04 AM, Christopher Schultz wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Graham,

On 9/25/15 7:23 PM, Graham Leggett wrote:

On 25 Sep 2015, at 10:33 PM, Christopher Schultz
 wrote:


While I obviously agree with the sentiment, I do feel bad for
the OP who has to fight this battle.

It is important however to clarify that this isn’t a typical
scenario, lest someone cites this thread as to why they should be
doing the same thing.


1. All the code we currently have in tcnative uses APR for
everything, and I'm not sure if APR supports AF_UNIX sockets, or
even if it would have to support them to do this.

The as-yet-unreleased v1.6 of APR does support unix domain
sockets, although the docs for it don’t appear to be very clear.


2. The plumbing required to configure an AF_UNIX socket is
non-trivial, and it's currently all wired-around using AF_INET
sockets, so it's got hostname, port, etc. I suppose we could
stuff the inode's name into the hostname and ignore the port
number or something like that, but it's fairly hacky.

Currently APR seems to accept the UDS filename where the IP
address would otherwise be provided.


So this is a non-trivial amount of work, here.

Srini, is there any chance your employer would pay someone to
write this code? Patches are always welcome, and Tomcat is
otherwise completely free…

If there was a push for unix domain sockets from Tomcat it would
definitely help working out whether the APR_UNIX implementation
does what it needs to do, and gets properly documented and v1.6
released.

I don't really see this happening.

I'm fairly sure that the widespread use of HTTP/2 is going to kill AJP
forever, leaving only mod_proxy_http(2) as a viable long-term
connector. Nobody is ever going to bother writing an AF_UNIX connector
for HTTP/2, so I think this idea is very likely to die in this thread.


Not sure on this, as AJP is quite handy. Expecialy load balancing java 
webapps and i find mod_jk quite good at this.
Out of curiosity, why do you think so? What does offer HTTP/2 that can 
be handy in a reverse proxy scenario? Compression / streams?





- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org

iQIcBAEBCAAGBQJWBeEhAAoJEBzwKT+lPKRYzU0QAIKV6imgl5CIyQW60QaTxERY
92VX4/s305D02r1Pp2Mji/kxmnylLgRq1ZQEwq7Jnygcm8NKPjHQQtOieSlSTHEO
S/OvB8p57arsY8N5JRcf2+mequYcNx5CbeWSSbqp2DS7KEli8FKyDHKpeioNVmZX
qWaWqG1mADxQBoLOgsk0opa82R18VadRQL4iKk+K28S3QLIFoIlOpi5pWOaothiV
RHBPQ282FfSJMfwhqmtirS3ZyqNu9Tve/e21kz1eCeBh/9L9sOI6E2EFKcF2Mq1Q
2PPoY3EuxSIlUeVKQgZZL+QPUy3UDcWmlsm6WNxCLQRIkpSXLfmJr2JpAwJaL2if
6Ssd9nVizP29WcnKQa8qWC50vrlbOROb9OaI/2t/zjdAWFKRdyG2FKHJQB+lolxV
+A+xiHIaATrcLTtkbbwm+dcUl6KFx/UMKmCLdK7+m19RDmTdeYeUSEvNXidZopb8
mZe4T87KbOrwMpqAjlehlqSOY3B993ZQu3bQdB+S1H0AdjXAL+C7umtzNX3qFb1C
KmJBpmt6AhIzaLYCiwHK3sYEp0BW5PYTMrM6MNJErheGvbxmfo7yPTYsVgfmTsMM
VS6kZqCXogj/FPaIna8X3UJr4BSEP5Dwx5AUZXS4qd1EcFT7gSKYD5+1vApsfUGh
cjAzQL+QFx/pl3DEtG8w
=whRf
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: logging input and output HTTP message payload

[Frederik Nosi]

Hi Christopher,

On 06/18/2015 05:55 PM, Christopher Schultz wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Frederik,

On 6/17/15 8:10 AM, Frederik Nosi wrote:

It helps only with HTTP though, no HTTPS or at least not easily.

All you need is the server's TLS key and Wireshark will look directly
at the HTTP conversation. This is a skill worth developing, especially
since it takes such little effort.


Been there done that, with mildly recent versions of openssl / https 
(PFS) you cant do that:


https://ask.wireshark.org/questions/34393/how-to-decrypt-ssl-traffic-using-wireshark





- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org

iQIcBAEBCAAGBQJVguoGAAoJEBzwKT+lPKRYp/IP/jlJ8sLXsf0hxL/NE/dZmE/h
rq4kFb3/9itszwJnL7vnaQHetml+JZNAk7fsCvk8wGx4JIX/aYVpmmyJ+A1KsQ+Q
1nZeJ2uSef6iTEtYsypMxUasabguc8LI7KG0QMRyBYSLICcbp/36SLgJmUVLCuvw
ErClKJIlFDetd7r0dUuBIvlixThQ0emSOPRAgutNauiUCb2m6CCFAnFWmmW7Z01/
yLV+AUGHp3b77Yq3jJlKGtlmNmTuJ/tdnY0N7VwYlnA644DkUpetykFZZ/pdLa5p
wYRHXz5qv/tmVT383SAeK3FgSVYIp9EnxVHs44ImZ42XwCnkoHLK9TdzjkS6Cxrs
WHQJcP6vfSWQ7WIGLNpJzUclEkCaJRCLktGcA8SE4aNIg0JvVe2Y25RVWOdYX//E
RHiZxrDJxctoK6zmTuCOpd7DFx1cxSp2s1xXfghl80lFouMryoqwL7vui8/V/8u/
PsxT6/kupsTFTGoTv1RvJdk0rPVhrQnpHneOpQcKyoH/1lJoTswO8j7T6suLACqL
/K1HCVO8E+tu49Mn5bsr9rUb0uUo/qtYUu/cPp4Pv6CkNxLxYw48T1t82HSGkQHr
g7qCt1aWltsfc6O6yx07Zfdair8Hvy/QmOifcnpWlHn01wF6phSjrXhN3t5M7pnt
ScsjQVvlQoi7ATLm+4/a
=f8p5
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: logging input and output HTTP message payload

[Frederik Nosi]

Some more info,

On 06/18/2015 06:00 PM, Frederik Nosi wrote:

Hi Christopher,

On 06/18/2015 05:55 PM, Christopher Schultz wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Frederik,

On 6/17/15 8:10 AM, Frederik Nosi wrote:

It helps only with HTTP though, no HTTPS or at least not easily.

All you need is the server's TLS key and Wireshark will look directly
at the HTTP conversation. This is a skill worth developing, especially
since it takes such little effort.


Been there done that, with mildly recent versions of openssl / https 
(PFS) you cant do that:


https://ask.wireshark.org/questions/34393/how-to-decrypt-ssl-traffic-using-wireshark 



Till some years ago I used tihs technique for troubleshooting. But after 
a webserver upgrade or such, i found that this was no more possible (i'm 
noit going into details now). So the quick and dirty way i had to use 
was with apache mod_bumpio, or strace -fe trace=network -s 1024 -p 
`pidof tomcat`, you got the idea.

With another product i had to put a HTTP/HTTPS proxy in the middle.

That's why i said it's not so quick :-)








- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org

iQIcBAEBCAAGBQJVguoGAAoJEBzwKT+lPKRYp/IP/jlJ8sLXsf0hxL/NE/dZmE/h
rq4kFb3/9itszwJnL7vnaQHetml+JZNAk7fsCvk8wGx4JIX/aYVpmmyJ+A1KsQ+Q
1nZeJ2uSef6iTEtYsypMxUasabguc8LI7KG0QMRyBYSLICcbp/36SLgJmUVLCuvw
ErClKJIlFDetd7r0dUuBIvlixThQ0emSOPRAgutNauiUCb2m6CCFAnFWmmW7Z01/
yLV+AUGHp3b77Yq3jJlKGtlmNmTuJ/tdnY0N7VwYlnA644DkUpetykFZZ/pdLa5p
wYRHXz5qv/tmVT383SAeK3FgSVYIp9EnxVHs44ImZ42XwCnkoHLK9TdzjkS6Cxrs
WHQJcP6vfSWQ7WIGLNpJzUclEkCaJRCLktGcA8SE4aNIg0JvVe2Y25RVWOdYX//E
RHiZxrDJxctoK6zmTuCOpd7DFx1cxSp2s1xXfghl80lFouMryoqwL7vui8/V/8u/
PsxT6/kupsTFTGoTv1RvJdk0rPVhrQnpHneOpQcKyoH/1lJoTswO8j7T6suLACqL
/K1HCVO8E+tu49Mn5bsr9rUb0uUo/qtYUu/cPp4Pv6CkNxLxYw48T1t82HSGkQHr
g7qCt1aWltsfc6O6yx07Zfdair8Hvy/QmOifcnpWlHn01wF6phSjrXhN3t5M7pnt
ScsjQVvlQoi7ATLm+4/a
=f8p5
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: logging input and output HTTP message payload

[Frederik Nosi]

If you have this:

1) Tomcat listening in HTTP, not HTTPS
2) ssh access to the machine
3) administrator access to the machine

and are on Linux (or Unix in general, but my command is linux specific) 
do this as root:



tcpdump -nnpi any -s0 -vvv -w /tmp/dump.pcap port  8080

After the request you're interested in are done press CTRl+C

from your client, assuming it's linux but on win you can use putty or 
whatever:


scp remote-machine:/tmp/dump.pcap .
wireshark dump.pcap


Done.

P.S.
If your tomcat's HTTP Connector isn't listening at port 8080 change the 
port accordingly in my tcpdump command




On 06/17/2015 08:04 PM, Milinda Perera wrote:

Hi all,

Thanks for suggestions,

Actually I want to log all HTTP / HTTPS requests from client applications
to my Tomcat server and the Response sent back to the client. And logging
requests and payloads are not target specific application running in the
Tomcal server, which means all the request to the Tomcat server and all the
responses from the Tomcat server

My requirement cannot be fulfilled with external tools like wireshark
because:
   1. unable to log HTTPS requests and responses
   2. And I need to get logged requests and responses in server side log
files without external tools.

by changing  *org.apache.coyote.level=FINE *in log properties logs HTTP
requests to the server, but does not log response.

Highly appreciate if you guys can share your thoughts to achieve this.

Thanks,
Milinda



On Wed, Jun 17, 2015 at 5:56 PM, André Warnier a...@ice-sa.com wrote:


Frederik Nosi wrote:


It helps only with HTTP though, no HTTPS or at least not easily.

While we are at this, are you trying to debug a SOAP / REST connection
from your application running on Tomcat to another server or a connection
coming from outside to your Tomcat?


On 06/17/2015 11:16 AM, Mark Thomas wrote:


On 17/06/2015 10:10, Milinda Perera wrote:


Hi,

I need to log HTTP payload content for debugging purposes.


Use Wireshark. That has the added benefit of not having any unwanted
side-effects on your application.

Mark



If this is for one debugging session, you could also simply use an add-on
to a browser (such as Fiddler2 for IE), and record the full exchanges there.
It has the advantage that you see the complete traffic in both directions
(headers and content), and that you can easily switch between different
presentations of the data, and save it to file if you need to.  And
depending on what you are looking for, it may be a lot easier to handle
than Wireshark.





-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org







-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: logging input and output HTTP message payload

[Frederik Nosi]

It helps only with HTTP though, no HTTPS or at least not easily.

While we are at this, are you trying to debug a SOAP / REST connection 
from your application running on Tomcat to another server or a 
connection coming from outside to your Tomcat?



On 06/17/2015 11:16 AM, Mark Thomas wrote:

On 17/06/2015 10:10, Milinda Perera wrote:

Hi,

I need to log HTTP payload content for debugging purposes.

Use Wireshark. That has the added benefit of not having any unwanted
side-effects on your application.

Mark


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: File descriptors peaks with latest stable build of Tomcat 7

[Frederik Nosi]

On 04/22/2015 05:15 PM, Christopher Schultz wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Frederik,

On 4/22/15 10:53 AM, Frederik Nosi wrote:

Hi, On 04/22/2015 04:35 PM, Christopher Schultz wrote: Neill,

On 4/22/15 9:12 AM, Neill Lima wrote:

If I am not wrong, if the application in question is
monitored in VisualVM through JMX
(https://visualvm.java.net/) you could trigger a Force GC
from its monitoring console.

You can do this, but it won't close any CLOSE_WAIT connections.
Tomcat's timeout must be reached. I suspect that the timeout(s)
are simply way too long.

You can tune the network stack's timeout using sysctl, eg:
net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_fin_timeout = 3

This won't do anything, either. As far as the OS is concerned, the
application (Tomcat) is still using that connection. Therefore it
can't be cleaned up.


Indeed you are right, tuning the network stack help with TIME_WAIT, not 
CLOSE_WAIT, my bad.




Tomcat has to actively hang up the connection, and the best way to do
that is with synchronized timeouts between the reverse proxy and Tomcat.

You can try all other kinds of tricks, but the fact of the matter is
that the application is still trying to use the socket, so no other
component can step-in and kill it.


Probably the application is slow then, or the server overloaded.



- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v2
Comment: GPGTools - http://gpgtools.org

iQIcBAEBCAAGBQJVN7sZAAoJEBzwKT+lPKRYRhkP/j0GBtPH/+/QU2YEgZxbRoJE
z2lmWxDrbFNxiYFS5332SvN4bXhG/Khog83CeBM0bg0VLciSxKYqm5J8YziMlrlo
omqk3gUiNeViyjsjO5SBW9hxT1qhC1PLdtx7uZ7xUiNmmE24wQ3Gi2edyjyvYDJ0
pzLT+bEp8BjXgm0c6aOONO0PJ+PbyZPeF56PXq6iqn426IhebEUlDP8kxuSh3RwL
LQW7tg05bg3yTuP1ZjiwH4gmBfbomJ+xpY6F+zwDkZgk7Cs4okp5/Tr0uTNhsHQM
lgGaIZc9SCoqKaMFqWila3RaAnnpqDe1cdg2N44zluIaMkcO94kDSWBuT25t5dGe
GBiFG2HGczwyo5MCrx0RgYgLtb2bQ0QZQ8nHzNis8wkNQdHWzziWsvsVQOCnCqL/
3FOkWUbbJTdmnB8lx84sRcuMsDYQ0BYOYW4W/F2WmSxzBnm7V4NixHG9dD4lZ3vJ
fhIO/d0VNOpI+wesZyQg+pwWRHInbigZ0+5A3InOLHW84rWa2qX0wvt6a7rBb0YP
gonBY4xbrPTHoXDHH7ZCs3JW+gwstA5avA/Obp45C5LessbduqRPtBvMUZizyZR5
ByPtJcrCvHlFux1fwc7Idj/9seqaYvllyvO6evvhqgYVU3jV2tekOUNuFGDJ8KRt
HmrzuiH3cmU1JpT6FSen
=XyQw
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: File descriptors peaks with latest stable build of Tomcat 7

[Frederik Nosi]

Hi,
On 04/22/2015 04:35 PM, Christopher Schultz wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Neill,

On 4/22/15 9:12 AM, Neill Lima wrote:

If I am not wrong, if the application in question is monitored in
VisualVM through JMX (https://visualvm.java.net/) you could trigger
a Force GC from its monitoring console.

You can do this, but it won't close any CLOSE_WAIT connections.
Tomcat's timeout must be reached. I suspect that the timeout(s) are
simply way too long.

You can tune the network stack's timeout using sysctl, eg:

net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_fin_timeout = 3




- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v2
Comment: GPGTools - http://gpgtools.org

iQIcBAEBCAAGBQJVN7GvAAoJEBzwKT+lPKRYPUIP+gIbeAUIJNo7budTt59iriqX
JJzpceyz8RQUPmqOKpfSSj+5xLL9wFUBe9WaNsTLgdGkTxPWk1O/UYqmOmeYjYm6
tKkL/VwI6ySghkKImIMBAOpa8up6cvjqRbziu6He0K7gMgf1d8ipcPI0GQdmGWlm
7sMM9FWgQiBtP1+WrRFyEH/7ldD3xbGfgrdzYO4RAqaqbtplvS8ept8ecXZp12RT
RUeUIZByHBE2x39mcN2piZkPtAB0htN/DPSuAAPi850bBo5cECLlbyDusEoWa4G/
LQX6i5iIe68M6u2HqRM2gGPB/5LxDnBrCbQdVpcyGBr0CbI/NcfpxKx5IQYYf7PP
fG5RV3EViqLuIuMTzlMig1b/6h0djGCmMZc8JIZWlsX1SQXf/gbHPOIwEBE2M4pD
wtfoXZjWOmPep+a8y5QbiiYGZo5wIp9dKNdZEta4KIa/WAUkIYwVT5dEQS9pZ7N0
/M4NRDngbPdL7FZNh2q4/FNm/gR9W7bg5iIpjz5wVpEwhvqpjU7kJ/rIIE1Vdh6/
VbRI72dE3P9W1qm8XeQwGFkv8uHcTCqRVxPVN5JvQTIkbWF54tUqvFFB4Dk4gHYW
s6yDRQdHIAGRnAH9nSF4xp4Bdl9vhl/zAoEW44/MUdeeuknxROMPBerHG3QwoMFw
MSjyDI5wdBDuFCcuex+s
=OINr
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Cheers,
Frederik

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: mod_jk dealy sendng request to backend tomcat

[Frederik Nosi]

Please dont top post.

On 03/02/2015 04:07 PM, Rajesh Cherukuri wrote:

Yes it is always 20 sec always the case when the MOD_jk assigns the request
to the tomcat that was down


Are you sure that there's not some other tomcat in the same load 
balanced worker that's in hang? Didnt had time to check your log file, 
just a suggestion.





using Mod_jk 1.2.37 version

On Mon, Mar 2, 2015 at 8:27 PM, Rainer Jung rainer.j...@kippdata.de wrote:


Please don't top post, it makes following the communication very hard.

Am 02.03.2015 um 12:48 schrieb Rajesh Cherukuri:


for telnet immediateconnection refused  is given


   telnet 10.xxx.xxx.xx 8911
Trying 10.xx.xx.x...
telnet: connect to address 10.xx.x.xx: Connection refused


Then I would expect that mod_jk gets exactly the same quick error. Maybe
the network situation between your mod_jk server and the Tomcat worker was
different, when the problem occurred. But ...

  here is the error log for the specifed time
the logs looks strange to me. It could be, that mod_jk did not really run
into a 10 second timeout. Error 111 on Linux is connection refused, not a
timeout. Is that 20 second delay always the case? Could it be that your
http server was overloaded? Are you using the latest mod_jk version?


  [Fri Feb 27 02:26:14.463 2015] [31713:140059770595072] [error]

ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat
failed.
[Fri Feb 27 02:27:30.458 2015] [3471:140059728635648] [error]
ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to
backend failed. Tomcat is probably not started or is listening on the
wrong
port (errno=111)
[Fri Feb 27 02:27:30.560 2015] [3471:140059728635648] [error]
ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to
backend failed. Tomcat is probably not started or is listening on the
wrong
port (errno=111)
[Fri Feb 27 02:27:30.560 2015] [3471:140059728635648] [error]
ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat
failed.
[Fri Feb 27 02:28:14.585 2015] [31713:140059550308096] [error]
ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to
backend failed. Tomcat is probably not started or is listening on the
wrong
port (errno=111)
[Fri Feb 27 02:28:14.686 2015] [31713:140059550308096] [error]
ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to
backend failed. Tomcat is probably not started or is listening on the
wrong
port (errno=111)
[Fri Feb 27 02:28:14.687 2015] [31713:140059550308096] [error]
ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat
failed.
[Fri Feb 27 02:29:14.401 2015] [31713:140059728635648] [error]
ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to
backend failed. Tomcat is probably not started or is listening on the
wrong
port (errno=111)
[Fri Feb 27 02:29:14.503 2015] [31713:140059728635648] [error]
ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to
backend failed. Tomcat is probably not started or is listening on the
wrong
port (errno=111)
[Fri Feb 27 02:29:14.503 2015] [31713:140059728635648] [error]
ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat
failed.
[Fri Feb 27 02:30:40.149 2015] [31713:140059739125504] [error]
ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to
backend failed. Tomcat is probably not started or is listening on the
wrong
port (errno=111)
[Fri Feb 27 02:30:40.250 2015] [31713:140059739125504] [error]
ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to
backend failed. Tomcat is probably not started or is listening on the
wrong
port (errno=111)
[Fri Feb 27 02:30:40.251 2015] [31713:140059739125504] [error]
ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat
failed.
[Fri Feb 27 02:31:15.442 2015] [31713:140059644716800] [error]
ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to
backend failed. Tomcat is probably not started or is listening on the
wrong
port (errno=111)
[Fri Feb 27 02:31:15.543 2015] [31713:140059644716800] [error]
ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to
backend failed. Tomcat is probably not started or is listening on




On Mon, Mar 2, 2015 at 4:28 PM, Rainer Jung rainer.j...@kippdata.de
wrote:

  Am 02.03.2015 um 11:34 schrieb Rajesh Cherukuri:

  rainer

looks like what you said is correct , but not sure why the Mod_jk has to
wait for 10 seconds when the backend tomcat servers is down



Because your network layer behaves like that. It simply hangs for (more
than) 10 seconds. You should be able to observer that yourself e.g. using

telnet tomcatserverip tomcatajpport

It should hang that long as well.

   here is the error log  i don't see that any place where it is aitng
for 20


sec



The situation you want to discuss happened at 02:28:14, the log snippet
is
from 01:28:35 to 01:33:31. So they do not match.


   error log


[Thu Feb 26 

Re: High thread count load on Tomcat8 when accessing AJP port with no request

[Frederik Nosi]

On 11/19/2014 09:27 PM, Lisa Woodring wrote:

Actually, I received a little clarification on the monitoring software
(I didn't write it).  What it's trying to test is that the AJP port
itself is actually accepting connections.  With Apache in front in a
production system, it could forward the actual request to one of
several Tomcat boxes -- but we don't know which one from the outside.
The monitoring software is trying to test -- for each Tomcat instance
-- if it is accepting connections.  It used to send an nmap request,
but now sends essentially a tcp ping -- gets a response  moves on.


In my case (homemade monitoring) i choosed to check mod_jk's log, after all
mod_jk does indeed check the state of the ajp connector in tomcat.

Hope this helps.
[... ]


Thanks for the idea.  Can you tell me what you specifically look for
in the mod_jk_log file?  Do you look for the presence of something?
or the absence of something?


grep out cping,


I only see 'negative' events in the logfile.  For example,
all endpoints are disconnected, detected by connect check(1),
cping(0), send(0)
which evidently, is when Tomcat releases a connection on its end.
(I set JkLogLevel = DEBUG, but still don't see any messages that look
like what I would want...)
Just ignore the cping part. I categorize the failure modes in two, 
client error (user closes the browser window or is slow), example:



[Thu Nov 20 10:19:36 2014] [29858:1626331456] [info] 
service::jk_lb_worker.c (1388): service failed, worker p3 is in local 
error state
[Thu Nov 20 10:19:36 2014] [29858:1626331456] [info] 
service::jk_lb_worker.c (1407): unrecoverable error 200, request failed. 
Client failed in the middle of request, we can't recover to another 
instance.
[Thu Nov 20 10:19:36 2014] [29858:1626331456] [info] 
jk_handler::mod_jk.c (2611): Aborting connection for worker=worker_p


Or server error, can be because of timeout (backend too busy):

[Thu Nov 20 10:19:54 2014] [31475:1317062976] [error] 
ajp_get_reply::jk_ajp_common.c (2020): (p7) Timeout with waiting reply 
from tomcat. Tomcat is down, stopped or network problems (errno=110)
[Thu Nov 20 10:19:54 2014] [31475:1317062976] [info] 
ajp_service::jk_ajp_common.c (2540): (p7) sending request to tomcat 
failed (recoverable), because of reply timeout (attempt=1)
[Thu Nov 20 10:19:54 2014] [31475:1317062976] [error] 
ajp_service::jk_ajp_common.c (2559): (p7) connecting to tomcat failed.


Another server error is connection refused, when the backend is extra 
busy (on linux net.ipv4.tcp_max_syn_backlog sockets waiting on the tcp 
stack) or tomcat is down. I dont have an example of this right now though



Anyway, this way you use mod_jk's logic instead of having to create an 
ad hoc one. This is at JkLogLevel notice, no need to enable debug.




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: High thread count load on Tomcat8 when accessing AJP port with no request

[Frederik Nosi]

Hi Lisa,
On 11/19/2014 07:28 PM, Lisa Woodring wrote:

On Wed, Nov 19, 2014 at 1:20 PM, Lisa Woodring lisa.woodr...@iglass.net wrote:

On Tue, Nov 18, 2014 at 2:26 PM, André Warnier a...@ice-sa.com wrote:

Lisa Woodring wrote:
...

In order to monitor
the availability of the HTTPS/AJP port (Apache--Tomcat), our
monitoring software opens a port to verify that this works -- but then
does not follow that up with an actual request.  This happens every 2
minutes.

...

This sounds like the perfect recipe for simulating a DOS attack.  Your
monitoring system is forcing Tomcat to allocate a thread to process the
request which should subsequently arrive on that connection, yet that
request never comes; so basically this thread is wasted, until the
ConnectionTimeout triggers (after 20 seconds, according to your HTTP
connector settings).

...

The thread count grows over time (goes up to 130-150 threads after 2
hours).  Setting 'connectionTimeout' (as opposed to the default of
never timing out) does seems to help some


Have you tried setting it shorter ? 2 = 2 ms = 20 seconds. That is
still quite long if you think about a legitimate browser/application making
a connection, and then sending a request on that connection.  Why would it
wait so long ? A browser would never do that : it would open a connection to
the server when it needs to send a request, and then send the request
immediately, as soon as the connection is established.

In other words : anything which opens a HTTP connection to your server, and
then waits more than 1 or 2 seconds before sending a request on that
connection, is certainly not a browser.
And it probably is either a program designed to test or attack your server,
or else a badly-designed monitoring system.. ;-)



The monitoring software is going thru Apache to AJP connector in
Tomcat.  As I described, with the default of no timeout, the # of
threads were much higher.  I currently have the AJP connectionTimeout
set to 3 seconds.


Actually, I received a little clarification on the monitoring software
(I didn't write it).  What it's trying to test is that the AJP port
itself is actually accepting connections.  With Apache in front in a
production system, it could forward the actual request to one of
several Tomcat boxes -- but we don't know which one from the outside.
The monitoring software is trying to test -- for each Tomcat instance
-- if it is accepting connections.  It used to send an nmap request,
but now sends essentially a tcp ping -- gets a response  moves on.


In my case (homemade monitoring) i choosed to check mod_jk's log, after 
all mod_jk does indeed check the state of the ajp connector in tomcat.


Hope this helps.
[... ]


Bye,
Frederik

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: [OT] Forward TLS connection information from AWS ELB - httpd - Tomcat

[Frederik Nosi]

Hi Christopher,
On 10/01/2014 04:26 PM, Christopher Schultz wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

All,

I'm interested in using AWS ELB for SSL termination but allowing the
client's TLS connection information to be forwarded all the way
through the chain to Tomcat.

The setup looks like this:

   ELB
/\
   /  \
  /\
 w0w1
/  \   / \
   t0  t1 t0  t1

(t0 and t1 are repeated because otherwise the diagram would be even
more difficult to read).

w0 and w1 are running Apache httpd, t0 and t1 are running Tomcat. The
client's connection is TLS terminated at ELB and whether the
connections between ELB/wx/tx are encrypted should be immaterial. I'm
using mod_jk from httpd - Tomcat.

ELB provides the following HTTP headers to wx:
X-Forwarded-For (client's IP)
X-Forwarded-Port443
X-Forwarded-Proto   https

Unfortunately, it looks like I can't get things like the cipher
default, etc. but I'm okay with that for the time being.

I'm wondering two things:

1. How can I get Apache httpd to trust that the connection is encrypted?
I want to be able to use RequireSSL for certain resources and have
httpd trust that the connection coming from the ELB is in fact
secure.


Maybe i'm missing something, but you can check that X-Forwarded-Proto 
header contains https? Seems a bit risky, maybe additionally adding 
another check that the incomming request comes from ELB's IP(s)?



2. How can I use that connection information to tell mod_jk that things
are to be trusted as well?


Just pass a custom header. BTW Are you encrypting the w --- t 
connections as well? BTW I recall a setup i've made times ago, where the 
SSL termination was on the apache webservers, ex:


LB (tcp)  https --- apache httpd (SSL Termination doing client 
certificate verification) / mod_jk --- AJP --- Tomcat


I was able to send client's certificate information as headers to 
tomcat. But not sure this is your situation.

For #2, I might just be able to use SetEnv to set
REMOTE_ADDR=X-Forwarded-For, but I'm not sure how to say yes, this is
encrypted. Should I set up a separate VirtualHost on a different
(non-80) port that is configured only for ELB connections and then
force SSL to on regardless of the actual incoming connections?

Maybe this can help:

RewriteEngine on
RewriteCond %{HTTP:X-Forwarded-For} ^(.*)$ [NC]
RewriteRule ^(.*)$ - [env=JK_REMOTE_ADDR:%0]

This way you send to tomcat as REMOTE_ADDR  the contents of the 
X-Forwarded-For header




That would allow me to use port 80 for regular web traffic and not
have to worry about proper checking to make sure that the connection
was in fact coming from the ELB and not directly into the web server.

Thanks,
- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org

iQIcBAEBCAAGBQJULA8qAAoJEBzwKT+lPKRYMf4P/2yONDv5rQFgHguhMeWv8BJs
jbv8bLOOK5Vf+r5idJgyEgOFEI4jbEKfGdhIvD5BasT4PZF65sn3AsOXQpav9GA4
kgomQHDipou3u5PFGi2d3xQQsDB9MjOTfAmmvQFNEnPxtisYQA+wNHGGxJDwyHIZ
tJkS4jP8mA3vxLaoCLoSiOi2MEGr2nbj5Xcfd75F2IXfci9QEhGEgsUxyFq+K+Vb
p+GVv4px55+zO9sLaIk6SiaNOGI3p86W+IX5spvoxO2Qxah+DVSoq9HRGryWd/Wn
O3ZwSGqCHYKsPI1xHECaN/58pAR7polyU5nEFmzWbxFhc31Q2hpDkZuyZ3SIY2u1
7lLY+Zx41nizjfjeYeIcMtZ4OBj0uHBSj5qzLehF7zItZoRqEhgv2b4yn8vJjIj0
GF4wpVqAqSWaIJ2F1C9ZjTnL9LhTJHZBurpt1JDSe7ALS/s4EoEQ/rbaz9kEUMNq
BBThIapN+VXCwaqsA7hQliCWRoGuP2kNFStsatgeaNaBZd5Cf8cg8iTSUcoDR4UW
Z4CHSi/4H6uD3wmcI6Jca7dfJEY+eNGM3zLsUF1hQPYP9MG6Fohy6h/UGGhlRehh
sXZ6bL0oVfGVxSM9gMCDQzB4ptb9zuqU5UgWjKEB50lbwXgMLUm7XP3/C/bY7Zgt
cXABRHoZSqoq2tPV1Lov
=g2oZ
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: [OT] Forward TLS connection information from AWS ELB - httpd - Tomcat

[Frederik Nosi]

Hi Christopher,

On 10/01/2014 06:05 PM, Christopher Schultz wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Frederik,

On 10/1/14 11:15 AM, Frederik Nosi wrote:

Hi Christopher, On 10/01/2014 04:26 PM, Christopher Schultz wrote:
All,

I'm interested in using AWS ELB for SSL termination but allowing
the client's TLS connection information to be forwarded all the
way through the chain to Tomcat.

The setup looks like this:

ELB /\ /  \ /\ w0w1 /  \   / \ t0  t1 t0  t1

(t0 and t1 are repeated because otherwise the diagram would be
even more difficult to read).

w0 and w1 are running Apache httpd, t0 and t1 are running Tomcat.
The client's connection is TLS terminated at ELB and whether the
connections between ELB/wx/tx are encrypted should be immaterial.
I'm using mod_jk from httpd - Tomcat.

ELB provides the following HTTP headers to wx: X-Forwarded-For
(client's IP) X-Forwarded-Port 443 X-Forwarded-Proto https

Unfortunately, it looks like I can't get things like the cipher
default, etc. but I'm okay with that for the time being.

I'm wondering two things:

1. How can I get Apache httpd to trust that the connection is
encrypted? I want to be able to use RequireSSL for certain
resources and have httpd trust that the connection coming from the
ELB is in fact secure.


Maybe i'm missing something, but you can check that
X-Forwarded-Proto header contains https? Seems a bit risky, maybe
additionally adding another check that the incomming request
comes from ELB's IP(s)?

Yes, I can check this. I can also ensure that the port is only
accessible from the ELB. I'm less worried about this and more worried
about getting everything else working first. Protecting the connection
itself will not be a problem.



Maybe i didn't got your question right, what you're interested first, is 
letting know to tomcat that the client is using a secure connection? If 
so you can just pass a custom header from apache to tomcat, but this 
seems too easy :-)



2. How can I use that connection information to tell mod_jk that
things are to be trusted as well?


Just pass a custom header. BTW Are you encrypting the w --- t
connections as well? BTW I recall a setup i've made times ago,
where the SSL termination was on the apache webservers, ex:
LB (tcp)  https --- apache httpd (SSL Termination doing
client certificate verification) / mod_jk --- AJP --- Tomcat
I was able to send client's certificate information as headers
to tomcat. But not sure this is your situation.

I don't need to use client certificates, but being able to support
them would be nice.

AWS ELB seems to support TCP pass-through but you can't do it for port
443. If you want to use port 443, you can either choose HTTPS/SSL or
TCP/SSL. If you choose HTTPS/SSL then you have to use either HTTP
or HTTPS as the back-end protocol. For some reason, choosing HTTPS
causes endless stalling when trying to make a connection.


I would get a tcpdump from the apache frontend, maybe you can get more 
info this way.




Using TCP/SSL - TCP/SSL (what I would call TCP pass-through) ought to
allow me to do SSL termination at the web server level, accept client
certificates, and have mod_ssk work without any modification at all. I
think in order to do this, I have to configure Apache httpd to accept
connections using the proxy protocol, and I'm not sure how to do that.


Hmm, didn't knowed about this protocol before. From some quick googling 
and reading, seems interesting, as at your endpoint the connection comes 
from ELBs'IP not from the client's IP, this protocol adds the missing 
info, real client ip.


http://blog.haproxy.com/haproxy/proxy-protocol/

So using this seems you need to add another piece to you'r infrastructure.




For #2, I might just be able to use SetEnv to set
REMOTE_ADDR=X-Forwarded-For, but I'm not sure how to say yes, this
is encrypted. Should I set up a separate VirtualHost on a
different (non-80) port that is configured only for ELB connections
and then force SSL to on regardless of the actual incoming
connections?

Maybe this can help:
RewriteEngine on RewriteCond %{HTTP:X-Forwarded-For} ^(.*)$ [NC]
RewriteRule ^(.*)$ - [env=JK_REMOTE_ADDR:%0]
This way you send to tomcat as REMOTE_ADDR  the contents of the
X-Forwarded-For header

Why use mod_rewrite (slow) when you can use mod_setenvif (fast)?

SetEnvIf X-Forwarded-For (.*) JK_REMOTE_ADDR=$1


Indeed is better your way



What I'm mainly looking for is a way to say the incoming connection
(from ELB) is HTTP and I want to pretend that the connection is HTTPS.


Then the easier solution seems using ELB for SSL termination and using 
the X-Forwarded-Proto header, passing from apache to tomcat


[...]





-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: catalina.out is 13G

[Frederik Nosi]

On 04/22/2014 04:28 PM, Randhir Singh wrote:

Thanks for your answer Leon. The space should be freed immediately on the
system, can the catalina.out be truncated on an running system?


On Uniux the classic trick is:

:  catalina.out


This command empties the file, you get free space on your filesystem but 
you'l lose all content in catalina.out



As another option, you can use cronolog for catalina.out


Bye,
Frederik



Regards

-Original Message-
From: Leon Rosenberg [mailto:rosenberg.l...@gmail.com]
Sent: Tuesday, April 22, 2014 7:55 PM
To: Tomcat Users List
Subject: Re: catalina.out is 13G

Hello Randhir,

whatever revert means.
However, if you remove the file the place will be occupied on most *'nix
system until a process restart. You will probably have to restart your
server to free this mount point's storage anyway.

regards
Leon


On Tue, Apr 22, 2014 at 4:20 PM, Randhir Singh
randhir.si...@sterlite.comwrote:


Hi,

I have a immediate concern as the mount point on which Tomcat is
placed is 99% and on checking I found that catalina.out is 13GB. I
wanted to implement a solution for this but am not sure, can I take a
backup of catalina.out and truncate catalina.out on the running
application?

Humbly requesting a revert on an immediate basis on whether I can
truncate catalina.out after taking a backup on a running tomcat
application.

Regards

--

*STL Disclaimer:*
The content of this message may be legally privileged and confidential
and are for the use of the intended recipient(s) only. It should not
be read, copied and used by anyone other than the intended
recipient(s). If you have received this message in error, please
immediately notify the sender, preserve its confidentiality and delete
it. Before opening any attachments please check them for viruses and
defects. No employee or agent is authorised to conclude any binding
agreement on behalf of Sterlite Technologies Limited with another
party by email without express written confirmation by authorised
person. Visit us at www.sterlitetechnologies.com  Please consider
environment before printing this email !








-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Performance - Java Profiler, JVM instrmentation

[Frederik Nosi]

Hi Shanti,
On 04/15/2014 09:56 PM, Shanti Suresh wrote:
[...]

I find Chris' example on writing filters to map to URL patterns for
response-time metrics relevant.  I would also like stall counts,
concurrent invocations etc.

What is a stall-count? How would you record concurrent invocations,
etc.?


So here is my understanding of these metrics:

So if a request for a servlet or JSP exceeds a given time interval, that
would be a stall.  The interval may depend upon the application.  In some
cases, 10 seconds would be considered a stall, some cases, 30 seconds would
be a stall.


This can be done enabling the access log and adding a %D on the log 
format string, here's

what i add to server.xml in tomcat 6:

!-- --
Valve className=org.apache.catalina.valves.AccessLogValve 
directory=logs

   prefix=access. suffix=.log resolveHosts=false
   pattern='%h %u %t %r %s %b %I %D'
   buffered=false /


then you get another log file, in this case access.DATE.log where the 
last entry is the time in milliseconds

it took to complete the request.

Than just do a:

cat access.DATE.log | awk '{ if ($NF  DURATION) { print $0 } }'

Hope you got the idea





Similarly, how many times a servlet is invoked in a given time period would
count as concurrent invocations.  Intervals used for the reckoning here may
be shorter - like 5 seconds - to make it more meaningful for concurrency
values.


You can use the access log for this too

[..]


Frederik

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: mod_jk - Failover behaviour and load + patch

[Frederik Nosi]

Hi Konstantin,

On 04/02/2014 12:01 PM, Konstantin Kolinko wrote:

2014-04-02 5:21 GMT+04:00 Frederik Nosi frederik.n...@postecom.it:

On 04/02/2014 12:21 AM, Christopher Schultz wrote:

On 3/26/14, 9:32 PM, Frederik Nosi wrote:

My scenario is Apache httpd + mod_jk + N Tomcat's in. The default
behaviour of load balanced workers in mod_jk in my testing is that
when a client requests a page (GET / POST / Whatever), the LB
worker tries the request to every ajp worker. This in contrast with
what i read here:


http://people.apache.org/~mturk/docs/article/ftwai.html
http://people.apache.org/%7Emturk/docs/article/ftwai.html

Expecially this part:


When having multiple nodes in a cluster you can improve your
application availability by implementing failover. The failover
means that if the particular elected node can not fulfill the
request the another node will be selected automatically. In case of
three nodes you are actually doubling your application
availability. The application response time will be slower during
failover, but none of your users will be rejected. Inside the
mod_jk configuration there is a special configuration parameter
called worker.retries that has default value of 3, but that needs
to be adjusted to the actual number of nodes in the cluster.

... worker.list=lbworker worker.lbworker.type=lb # Adjust to the
number of workers worker.retries=4
worker.lbworker.balance_workers=node1,node2,node3,node4 If you add
more then three workers to the load balancer adjust the retries
parameter to reflect that number. It will ensure that even in the
worse case scenario the request gets served if there is a single
operable node.

  From that it seems that the retries parameter in a load balancer
worker context should mean the number of real (AJP) workers to
try. (what i need indeed) but in my testing, that LB worker
parameter is the number of times that all the AJP workers that are
part of the LB worker get a round retry. In eg, having a LB worker
with 4 AJP workers, setting LB Worker's retries = 2, the behaviour
i see is that the AJP workers get called this way:

AJP1 - timeout [...] AJP4 - timeout

=== repeat again (retries == 2)

AJP1 - timeout [...] AJP4 - timeout

-- LB sends an error to the client.



Now from the online documentation the meaning of that parameter in
a load balancer worker context is'nt that clear, but from the link
i provided seems it was exactly what i needed, not the number of
retries to all AJP workers, but the number of single AJP workers to
try..

If that is not correct i can fill a bug report. If instead it's by
design, the attached patch adds a new parameter, lb_retries, that
does what i need. Of course it's a bit rough, but works.

Any comments? Am I getting stuff wrong?

I'm bumping this because I can see Rainer has fixed a bunch of things
in mod_jk over the last few days. Perhaps he's getting ready to do a
release or something.


Thanks Christopher, BTW on this issue I'm not even sure it's a documentation
bug, a bug in mod_jk or an unfullfilled expectation of mine :-)
Thing is, i had to use this cure.

I'm sure my patch is a bit faulty, the HTTP status codes returned are  500,
but i'm not sure they are in line with the protocol (503 / 504). but for now
it works for me though.


1. If you really want to submit a patch,  please attach it to an issue
in Bugzilla, so that it is not forgotten.


Okay, i will,


http://tomcat.apache.org/bugreport.html

2. You were lucky that you attachment has reached the list.  Usually
attachments are just removed by mailing list server.


Ok, sorry for that, i put that as attachment for avoiding word wrapping 
problems with my

mail client.



3. I cannot comment on the essence,  just two formal nits
1) The following line has a tab character instead of whitespaces:
+jk_log(l, JK_LOG_DEBUG, attempt %d, max attempts %d,


Ok, willl fix, thanks for pointing this.



2) An unneeded comment
+/* fredi - default */


Yep, leftover from my testing



3) Documentation =?
(xdocs/reference/workers.xml)


Ok, will do,




Noticed there were changes in mod_jk's git repo, i'm following it.


It is good that it works for you.
The official repository is subversion one.


Yes, noticed that, but as i'm not always inline i tend to use git. But 
svn is fine too


(Patches against the git repository are OK.  Maybe you want to submit
those .gitignore files, as a separate issue?)


Okay, will separate that part in case it turns useful.

Anyway, i know the patch i sent was rough, but i prefered to send it 
anyway to have

something concrete to explain what i wanted to do.



Best regards,
Konstantin Kolinko


Thanks Konstantin for your attention!


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




Frederik

-
To unsubscribe, e-mail: users

Re: AW: grab hostname from tomcat manager

[Frederik Nosi]

Hi,

On 04/02/2014 04:54 PM, bjoern.bec...@easycash.de wrote:

-Ursprüngliche Nachricht-
Von: André Warnier [mailto:a...@ice-sa.com]
Gesendet: Mittwoch, 2. April 2014 16:35
An: Tomcat Users List
Betreff: Re: grab hostname from tomcat manager

bjoern.bec...@easycash.de wrote:

Hello,

I need to grab the hostname from the tomcat manager somehow.
Unfortunately this URL manager/text/serverinfo doesn't contain the

hostname.

Is there any other smart way to receive the hostname via tomcat

manager app?
For give me for asking, but how do you access the tomcat manager if you
do not know the hostname ?

Good question :). But I got a good reason for it.

I got two servers with several tomcat instances.
In front of them is a loadbalancer with is configured to do a failover.

-LB-
 /   \
Server1:8081Server2:8081

I need to write a shell script to sync a specific directory and for each tomcat 
instance I need to know on which one the loadbalancer is targeting at the 
moment.
If tomcat 8081 on server 1 is down, the loadbalancer will point to server 2 
tomcat 8081. I can find it out through the loadbalancer address only.


I think the right source of the information you need is the load 
balancer, if you have access obviously. If not, you have the other 
indirect methods suggested from the others in this thread, jvmRoute or 
a page which shows the hostname.


Beware to the load balancing method used too, source ip, simple round 
robin or other.






-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Federik

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: [OT] Tomcat under load frontend reverse proxy timeouts

[Frederik Nosi]

Hi Christopher,

On 04/01/2014 05:50 PM, Christopher Schultz wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Ferderik,

On 3/31/14, 7:14 PM, Frederik Nosi wrote:

We've tried scaling horizontaly, and there's where i noticed the
amplification effect of mod_jk, i posted another mail on this issue
and a patch, with subject: Re: mod_jk - Failover behaviour and
load + patch but the thread went nowhere FWICS.

I still have your message as unread and want to discuss it at
ApacheCon with the core mod_jk guys (if they are actually there). If
the bug (oversight) is real, it should get patched right away. Not
sure about an official follow-up release... we just had one.


Ah, okay, that's perfect thanks! Just ask if you need any info. I'll be 
waiting.



- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJTOuBWAAoJEBzwKT+lPKRY0ncP+wSJJ7Qq0UmAS3MHPR0TlQMx
Ae7sqxXiEw5Bw+hjMagzn3I0DfJuV8k/W52gSYNHFu4aePGxRl2R6iyBb5SwEcPU
kxX/DRXJ3hAfFrEpBb+/W5Jitp8QhKxle+hylNxrJYvatshXxukMqKXuN9ietP0v
BU7hcIvUGpw8jjMMl4wBMK6BbvtPJPpIeE5xgWVF10kcfzZJgZa63Py94iMHWhPn
hxHpvUyAFDXjWfUiBr/qPUhNz04XtqleEARkQhFU9cV7RPjJYg6zr3lxlCMvtCGu
9pl4uAnj+ec++jgczBqb3cL2RXF09WUN8ugqTAA3KCYkm7BSTFX1FpgitZ/rT62R
kElDY3xG6lYb2x+CR7JCx4g0pfphcNmBhslpJQZSiLeiQ3W47mS12gmUt5BmkVJ0
pM7hHKIgygxgyU0VOl/QqpQK0DXBXYx3PT4gdh66ykV9BqlnQmFPL7+7me9/UM37
H9q9SRiekzwFp3GepqmVC8DjPHIzGbQyrYDUNMhZY2VD/bjZreD/LJsn8Ihxr7jh
11nWCSl0xpTvg+K+yXH1QCNz4wXPiuyBXoL/0KCrsCGZidCFMES7QDGexL5ySS6c
1I2NeP5sWlMykKi7toG2u0A4arxWbPLy84v1AbacYnO7XvMZ3vSucsSZ3Nu0oHiE
dpjlTNH1oZFFPI96xfq/
=jQJq
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


Frederik

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: mod_jk - Failover behaviour and load + patch

[Frederik Nosi]

Hi Christopher,

On 04/02/2014 12:21 AM, Christopher Schultz wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Frederik,

On 3/26/14, 9:32 PM, Frederik Nosi wrote:

My scenario is Apache httpd + mod_jk + N Tomcat's in. The default
behaviour of load balanced workers in mod_jk in my testing is that
when a client requests a page (GET / POST / Whatever), the LB
worker tries the request to every ajp worker. This in contrast with
what i read here:


http://people.apache.org/~mturk/docs/article/ftwai.html
http://people.apache.org/%7Emturk/docs/article/ftwai.html

Expecially this part:


When having multiple nodes in a cluster you can improve your
application availability by implementing failover. The failover
means that if the particular elected node can not fulfill the
request the another node will be selected automatically. In case of
three nodes you are actually doubling your application
availability. The application response time will be slower during
failover, but none of your users will be rejected. Inside the
mod_jk configuration there is a special configuration parameter
called worker.retries that has default value of 3, but that needs
to be adjusted to the actual number of nodes in the cluster.

... worker.list=lbworker worker.lbworker.type=lb # Adjust to the
number of workers worker.retries=4
worker.lbworker.balance_workers=node1,node2,node3,node4 If you add
more then three workers to the load balancer adjust the retries
parameter to reflect that number. It will ensure that even in the
worse case scenario the request gets served if there is a single
operable node.

 From that it seems that the retries parameter in a load balancer
worker context should mean the number of real (AJP) workers to
try. (what i need indeed) but in my testing, that LB worker
parameter is the number of times that all the AJP workers that are
part of the LB worker get a round retry. In eg, having a LB worker
with 4 AJP workers, setting LB Worker's retries = 2, the behaviour
i see is that the AJP workers get called this way:

AJP1 - timeout [...] AJP4 - timeout

=== repeat again (retries == 2)

AJP1 - timeout [...] AJP4 - timeout

-- LB sends an error to the client.



Now from the online documentation the meaning of that parameter in
a load balancer worker context is'nt that clear, but from the link
i provided seems it was exactly what i needed, not the number of
retries to all AJP workers, but the number of single AJP workers to
try..

If that is not correct i can fill a bug report. If instead it's by
design, the attached patch adds a new parameter, lb_retries, that
does what i need. Of course it's a bit rough, but works.

Any comments? Am I getting stuff wrong?

I'm bumping this because I can see Rainer has fixed a bunch of things
in mod_jk over the last few days. Perhaps he's getting ready to do a
release or something.


Thanks Christopher, BTW on this issue I'm not even sure it's a 
documentation bug, a bug in mod_jk or an unfullfilled expectation of 
mine :-)

Thing is, i had to use this cure.

I'm sure my patch is a bit faulty, the HTTP status codes returned are  
500, but i'm not sure they are in line with the protocol (503 / 504). 
but for now it works for me though.


Noticed there were changes in mod_jk's git repo, i'm following it.

Thanks for your attention and have fun at ApacheCon!
- Fredi



- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJTOzvQAAoJEBzwKT+lPKRY6uEQAISOkIa3VwSlhVEu/JPuhqDD
SHN7/hui6l/0oj2chPRr1YWs0XO8yuP8afgzlfuJhWQ5fNeVshDliYijYusKZPSx
666iVZ4NqVF8zTYo7Y6LE7gJJS8LrZUnD/nEF3ngCWGp5V6KzbQkB2haQBeFFRX2
bK2xcPFAC/hhT4QmkSkyCb8KaqyGtgIZDXToiQlA1oWRkleRip8yU+JwZEQXUz8v
MNhRtG5hUiX5z6wJh7OVTRu421mwmfZav4SUNxvjN71UnvoiRST4H3qrQGeD6fHh
2bKq4ot+olZ7T0Wo4fd6NqRzn/7fwRfp9U6OIrwe5umjaWzYd99o/iTm31hcFJFz
f6d1A0A5uqZaDT2+o8y01qSbdIWDYZFKRv7KZuX1/+6WXp6h7VHXhY9mA/R8RYf9
NpSi0bk5FprrTNJoJx9+q/LBGlFrwFZqJpYfL93FMUlkOP47z2U1Z6ihf+HC4vb9
zIS/NRBMgYWcKezbcc7E40XHYChE3SedO809AVAyaz7VASAQsfMDVrih0eu7f+w0
R5FY/32Ks/o0B8Udi3numTAovEE72YjIuuvMZQp9CTGBwNMFQUUsiGAdILSBSULl
XNgAJXDOEOyVPDMONaHXQPk4xgDH/iZAcNdgqhnV+lcmgnhwSKAV8hz9/wF/u/H8
qs1OVu0vib9AXJ6ZFFEh
=GEhg
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: RES: Configuring limits of requests/sessions/threads in Tomcat

[Frederik Nosi]

Hi Danilo,
On 04/02/2014 02:37 AM, Danilo Amaral de Oliveira wrote:

Hello Chritopher,

You make score. Exactly, I am waiting 500 users on the peak time. The archive 
is in a separated storage in our environment. So, I think that it is not a 
limiting factor. I will try some maxthreads configurations and stress 
situations to see if the server will could handle 500 users. If necessary I can 
increase the server memory and processing, we have robust cluster. Actually the 
server has 4 cores and 8gb of memory.


First, this video is a static file right? If so, just put an nginx or 
such in front of you'r Tomcat, you dont need an application server for 
that, it's just like using a tank to shoot a mosquito :P




At the same time, I am analyzing with security team if we could liberate a 
specific video in youtube in order to instead use JWPlayer the site use an 
embedded youtube video, so I take off the responsibility of tomcat to deliver 
the video and I can sleep in peace. hahha


Well, this clever, bandwidth for free.



Thank you all!



Grupo Energisa
Danilo A. Oliveira
Analista Suporte Aplicacao TI - DPTO CORP. DE INFRAESTR. TI
e-mail: danilo.olive...@energisa.com.br | tel: (32) 3429-6342 | cel: (32) 
8452-9478

Esta mensagem contém informação confidencial. Se você a recebeu por engano, não 
divulgue ou copie seu conteúdo. Por favor, avise ao remetente imediatamente e 
apague-a do computador.
Privileged and confidential. If this message has been received by mistake, do 
not disclose or copy its contents. Please notify sender and delete immediately.


-Mensagem original-
De: Christopher Schultz [mailto:ch...@christopherschultz.net]
Enviada em: terça-feira, 1 de abril de 2014 18:32
Para: Tomcat Users List
Assunto: Re: Configuring limits of requests/sessions/threads in Tomcat

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Danilo,

On 4/1/14, 2:03 PM, Danilo Amaral de Oliveira wrote:

Next week we will deploy a institutional video to all company (more
than 5k users) in a webpage in an application manges by the Tomcat
7.0.40. I have made a stress test through JMeter, simulating a lot of
simultaneous access, and when the simultaneous access reaches
100 users the application stop answering.

What is your expected peak load? If you only need to handle 500 simultaneous 
users, then you should be fine.


I would like to know if I can set a max number (80, for example) of
requests (maxsession or threads, I dunno) in order to configure the
tomcat to send a warning page advising the user that the server is
loaded and try again in few minutes if the limits reach the maximum
value.

You can change the size of Tomcat's request processing pool. That's either maxConnections on your 
Connector or, better yet, maxThreads in your Executor.

If you set maxThreads=80, then the 100th request will wait in a queue. You 
may get request timeouts at this point, which will be equivalent to the 100-user 
situation you describe above.

What you really need is a CDN so your servers don't even handle the requests at 
all. Is there a place where you can upload your file that will be more scalable 
and/or faster? Something like Amazon S3 maybe?

- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJTOzBFAAoJEBzwKT+lPKRYKjQP/0Ntz/ZT1ylSwMnFMCLFUDqC
4C0eFUXJvPXu3Rr9+EjoSPRbX7r3FN5JMrx4SmwBwfxZhsjiyXksUMNVvUlW0QeD
CUtDsy4oYu3MM8BZqa2zqHtzDXQu8Qp8Hg3a5N0PQGWUqZCR/f2FOgR1fV+Zco2S
Av7ioK67dC/u2elUoC/2oUJ1J4xXSLlVOFRKGNB1i/EmkpDLi07tyLs/L2Y9TmWQ
VfUv5RETqdIA682OshfHJ5ViR5NbZKZVFwTA3H8+oPn5NtPBJ3UgT5hv4GmVYpgz
m4S6O/8CRJz33q+tz9N35e8Fx5JnU8l4Nwn6AfOW0tgUX9JO6V0+x6W7bvCjSrlH
vbmJYJjPynoP8Y1smAxGrNg8jwjQklTP5qTIRr2FBm2DZ0chJQYMAmbETShrYZu8
n4na84JJ67gEMD2LV+ym5k7ma6gVmfCw3inT12kXtiwzC+d8vyxzdvToCT589cCJ
lRG5jhihpxPJTZtJ27db4ZxxgvG87ArQNhpvzEQCKnGa22/UkOaC56NZwzxbtfqb
64u7Ok+Jn6z8fJBMeq9lxILUVJTsbgfD3wx4WqNH+f1D2U+LsSPXrm8xmP5DS5hd
KQ4GZUNT4kEApRCCSfqlzbi+BC2dwo7mm7zmhaMIV5Otu43Wda4fUpK2z+ZAwE35
YJJvZvdzD8LkeMFiQz4G
=/hsP
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Problem in accessing link

[Frederik Nosi]

Nithun,

first, i think you have to reread this: http://tomcat.apache.org/lists.html


On 04/01/2014 10:00 PM, Bomma, Nithun wrote:

Thanks Chris!

I want to get public  private keys from WebSphere and import into Tomcat.


This is not related to the subject of your email.



We have WebSphere certificates (Signed by Verisign) until 2015 and we want to 
use the same in tomcat.

When I create a keystore (keytool -genkey -alias tomcat -keyalg RSA -keystore 
/opt/tomcat/SSL/tomcat.keystore), a keystore is getting created. But I'm unable 
to import the certificates into it.

Is there any document or documentation which might be helpful? Could you please 
let me know?


Tomcat version?

Anyhow:

http://lmgtfy.com/?q=using+https+with+tomcatl=1




Thanks.

Thanks,
Nithun Bomma
WebSphere Administrator
Amtrak - Information Technology (Operations)
AIM: nithunbomma
EMAIL: nithun.bo...@amtrak.com
Desk: 215-349-2065; ATS: 728-2065; Cell: 215-704-4981

-Original Message-
From: Christopher Schultz [mailto:ch...@christopherschultz.net]
Sent: Monday, March 31, 2014 5:18 PM
To: Tomcat Users List
Subject: Re: Problem in accessing link

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Randir,

On 3/31/14, 5:14 AM, Randhir Singh wrote:

Thanks for your answer. There are 2 applications hosted on this
Tomcat. 1 is working fine and the other only is giving problems which
we are accessing on port 10080.

Request inputs on this so that the resolution is found.

You need to request inputs on this from whoever Elite Core is. We cannot help 
you, here.


Good luck,
- -chris


-Original Message- From: Christopher Schultz
[mailto:ch...@christopherschultz.net] Sent: Saturday, March 29,
2014 6:22 PM To: Tomcat Users List Subject: Re: Problem in accessing
link

Randhir,

On 3/29/14, 8:38 AM, Randhir Singh wrote:

We are using tomcat in our production setup, the version of tomcat is
5.0.28.

It would be really good for you to investigate upgrading your version
of Tomcat. Not only has Tomcat 5.0 been retired, but Tomcat
5.5 has also been retired. Tomcat 8 is about to be released. There are
known and unknown unpatched security problems with the Tomcat 5
versions which will never be patched.


While accessing the link hosted on tomcat, the following error is
coming as below:
-
-



- --

-
-



- 

*HTTP Status 500 - * --
*type* Exception report
*message*
*description* *The server encountered an internal error () that
prevented it from fulfilling this request.*
*exception*
java.lang.NullPointerException



com.elitecore.reports.manager.server.GetCompanyInfoServlet.doAction(G
e



tCompanyInfoServlet.java:93)

This

is a problem with whatever elitecore' is. You'll have to look there.


Request you to please advice on what steps should be taken so that
this problem is resolved.

This is not a problem with Tomcat: it's a problem with the web
application deployed on Tomcat. This is something you'll have to do
yourself or engage the original developer.

-chris

-



To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org

For additional commands, e-mail: users-h...@tomcat.apache.org


-BEGIN PGP SIGNATURE-
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJTOdt4AAoJEBzwKT+lPKRYFJUP/2skKVpGkF09mj44ZJlR7731
WSKCe7C7fI6wZTD94hAVMMEqVKcyLvMBpeFih4S+zjUSvCs9Cw0kz9pjtvwA9mA9
8THXGKoNQuJs/xSRdEkJUSfQf2JAOXcklIKRL5EOH+bq/AT8n9ARHWX4rj72BKe5
UHYU50qsLIKQnjpxSJDz2cebQY9DuDe5MWn9TF16YL0ySfRT7wAmaXVeR4bgfuJr
lPlGydpsQ/1FXDOf/uU7gTjL6opVfK03RjNAKCuOdqN7HXdqNLQs2SHOA2V/jItc
uI+VXx6bj1+PuE9iNteBP5BJaDmlaBVVuBPEIRTplF3KBewSUaluDh6aOf8wnc9X
ThDDPrSWiwHLID2J+h0jCsxnfdAxEjgoTS25i2kTwDisVWlngafnlyoTBacZPOp4
hCcoKdRguU6mUYpyvu19NyZLPGmBofYGB69tNMXQOhlA12HxGUHMqXs9ndag1pgq
jhimFiMDYlQ+2FS+vtKAGQDBa8HVohgIdCYsI5r5R0sZHZwx09K5KR1nZw7RK8Vf
nE1ydf+4r2YkJKCRKlF+lQxpdmxASAdSrzlqIrWj6sszXf24g+mK5EzYuVT7twUi
Y6ycVI3fxo4ovm5AWcFlBcoDwl4wpzGWcVNc8D0hbmCS7gBELuWbwTnTsjT7Ockx
hYzYZQUInB0f9mqap9Pb
=itWA
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat under load frontend reverse proxy timeouts

[Frederik Nosi]

Hi Igor,

On 03/27/2014 10:51 PM, Igor Cicimov wrote:

On 27/03/2014 12:39 PM, Frederik Nosi frederik.n...@postecom.it wrote:

Hi all,

Having to deal with slow applications deployed under tomcat, with a

reverse proxy in front, frequently i've noticed that even when the frontend
timeouts and closes it's part of the TCP connection, the Tomcat thread
processing the request goes on and on till it finishes. Is there a way to
make the proccessing thread stop when the frontend connection get's closed?


Thanks in advance,

Frederik


What kind of application is this? What is the reason that the thread takes
longer than expected?


Unfortunately for me is legacy apps, sometimes writen from companies 
that never exist anymore, but i have to somehow make stuff work. I cant 
modify them.



Is it just the load on the app server or its waiting
on the backend database maybe?


Yes, nearly all apps talk with a backend DB, but for that part knowing 
our DBA stuff and havnig access to the database servers i'm confident it 
is not due to the load on our databases. Is mostly due to the number of 
queries per request from the application side. And another problematic 
point is that some of this applications dont even use connection pooling.


And most of this apps make calls to webservices outside my control, 
without having the possibility to configure a timeout.


Load is an issue too, usualy CPU load not memory, so not connected to 
the garbage collector.


We've tried scaling horizontaly, and there's where i noticed the 
amplification effect of mod_jk, i posted another mail on this issue and 
a patch, with subject: Re: mod_jk - Failover behaviour and load + 
patch but the thread went nowhere FWICS.




If load then for sure you can limit the
tomcat connector threads and the accept queue to reduce it and add some
more app servers. If backend connection then you can sync the client
timeout and the db connection timeout.


See above.




Thanks Igor,
Frederik

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Is it possible to send a 'keep-alive' packet back to client session every x seconds?

[Frederik Nosi]

Hi Matthew,

On 03/28/2014 01:35 AM, Matthew Turany wrote:

Thanks everyone, development tells me that their going to build the
function into the application.


You're lucky ;-)


trying to build custom kernels and then using a tcp_keepalive would be a
logistic nightmare.


I know you solved differently this issue, but in case other people is 
watching this thread, who said you have to build custom kernels?!? Just 
change a sysctl knob and be done with that. Exagerating, it's just ~100 
characers to type, no reboot required neither. Check here for the 
details, i'm sure will come handy in other situations:


http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/usingkeepalive.html




Cheers,
Matt



[...]

Frederik

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat under load frontend reverse proxy timeouts

[Frederik Nosi]

First thanks for your reply,

On 03/27/2014 10:32 AM, Mark Thomas wrote:

On 27/03/2014 01:39, Frederik Nosi wrote:

Hi all,

Having to deal with slow applications deployed under tomcat, with a
reverse proxy in front, frequently i've noticed that even when the
frontend timeouts and closes it's part of the TCP connection, the Tomcat
thread processing the request goes on and on till it finishes. Is there
a way to make the proccessing thread stop when the frontend connection
get's closed?

No.



Any hints on how to deal with situations like this?


Mark



Frederik.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat under load frontend reverse proxy timeouts

[Frederik Nosi]

Hi Andre',

On 03/27/2014 02:37 PM, André Warnier wrote:

Frederik Nosi wrote:

First thanks for your reply,

On 03/27/2014 10:32 AM, Mark Thomas wrote:

On 27/03/2014 01:39, Frederik Nosi wrote:

Hi all,

Having to deal with slow applications deployed under tomcat, with a
reverse proxy in front, frequently i've noticed that even when the
frontend timeouts and closes it's part of the TCP connection, the 
Tomcat
thread processing the request goes on and on till it finishes. Is 
there

a way to make the proccessing thread stop when the frontend connection
get's closed?

No.



Any hints on how to deal with situations like this?



This is basically an issue similar to the one that is the subject of 
the other message thread Re: Is it possible to send a 'keep-alive' 
packet back to client session every x seconds?.


Not exactly, it's the opposite scenario, the load balancer and the 
firewall / NAT part is fine. Probably my english is a bit poor. What i 
wanted to acchieve is, without having the possibility to modify the 
webapp, when the uplink request (this can be mod_jk/apache or user's 
browser, whatever calls the application) timeouts to find a way to stop 
the tomcat / webapp proccessing thread. Though from Mark's reply i 
understand that this is not possible with tomcat. So, at this point, 
having to deal with such webapps, is there some common solution?


My problem is that the failover mecchanism (mod_jk in my case) during 
peak usage amplifies the load on the application servers, it's a spiral 
of death :-)


How do you guys deal with this situations?




AFAIK, the only portable way to detect this situation, is by forcing 
the web application to regularly send some output to the client.  If 
the connection has been closed, it will then (*) get an exception, 
which it can handle.


in mod_jk cping/cpong, or the TCP stack (keepalive tcp). But is not my 
scenario unfortunately.


Anyway, thanks for your reply!



It implies that the webapp itself is not blocked waiting on some 
separate resource, and can regularly break out of its own main 
processing to send such output, resuming its main work if the result 
is ok.


(*) with possibly some delay before everything gets flushed all the 
way to the client connection, and the disconnected state makes its way 
back up the chain.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Frederik

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: mod_jk - Failover behaviour and load + patch

[Frederik Nosi]

Hi all,

My scenario is Apache httpd + mod_jk + N Tomcat's in. The default 
behaviour of load balanced workers in mod_jk in my testing is that when 
a client requests a page (GET / POST / Whatever), the LB worker tries 
the request to every ajp worker. This in contrast with what i read here:



http://people.apache.org/~mturk/docs/article/ftwai.html 
http://people.apache.org/%7Emturk/docs/article/ftwai.html


Expecially this part:


When having multiple nodes in a cluster you can improve your application 
availability by implementing failover. The failover means that if the 
particular elected node can not fulfill the request the another node 
will be selected automatically. In case of three nodes you are actually 
doubling your application availability. The application response time 
will be slower during failover, but none of your users will be rejected. 
Inside the mod_jk configuration there is a special configuration 
parameter called worker.retries that has default value of 3, but that 
needs to be adjusted to the actual number of nodes in the cluster.


...
worker.list=lbworker
worker.lbworker.type=lb
# Adjust to the number of workers
worker.retries=4
worker.lbworker.balance_workers=node1,node2,node3,node4
If you add more then three workers to the load balancer adjust the 
retries parameter to reflect that number. It will ensure that even in 
the worse case scenario the request gets served if there is a single 
operable node.


From that it seems that the retries parameter in a load balancer 
worker context should mean the number of real (AJP) workers to try. 
(what i need indeed) but in my testing, that LB worker parameter is the 
number of times that all the AJP workers that are part of the LB worker 
get a round retry. In eg, having a LB worker with 4 AJP workers, setting 
LB Worker's retries = 2, the behaviour i see is that the AJP workers get 
called this way:


AJP1 - timeout
[...]
AJP4 - timeout

=== repeat again (retries == 2)

AJP1 - timeout
[...]
AJP4 - timeout

-- LB sends an error to the client.



Now from the online documentation the meaning of that parameter in a 
load balancer worker context is'nt that clear, but from the link i 
provided seems it was exactly what i needed, not the number of retries 
to all AJP workers, but the number of single AJP workers to try..


If that is not correct i can fill a bug report. If instead it's by 
design, the attached patch adds a new parameter, lb_retries, that does 
what i need. Of course it's a bit rough, but works.


Any comments? Am I getting stuff wrong?


Thanks in advance,


Frederik


diff --git a/native/.gitignore b/native/.gitignore
new file mode 100644
index 000..1f8d345
--- /dev/null
+++ b/native/.gitignore
@@ -0,0 +1,5 @@
+Makefile.in
+aclocal.m4
+config.log
+config.nice
+configure
diff --git a/native/common/.gitignore b/native/common/.gitignore
new file mode 100644
index 000..2a9005d
--- /dev/null
+++ b/native/common/.gitignore
@@ -0,0 +1 @@
+config.h.in
diff --git a/native/common/jk_ajp_common.c b/native/common/jk_ajp_common.c
index 08bcc02..9a12a89 100644
--- a/native/common/jk_ajp_common.c
+++ b/native/common/jk_ajp_common.c
@@ -2904,6 +2904,9 @@ int ajp_init(jk_worker_t *pThis,
 p-retries =
 jk_get_worker_retries(props, p-name,
   JK_RETRIES);
+p-lb_retries =
+jk_get_worker_lb_retries(props, p-name,
+  JK_LB_RETRIES);
 
 p-max_packet_size =
 jk_get_max_packet_size(props, p-name);
diff --git a/native/common/jk_ajp_common.h b/native/common/jk_ajp_common.h
index 0c1636c..7b342d0 100644
--- a/native/common/jk_ajp_common.h
+++ b/native/common/jk_ajp_common.h
@@ -363,6 +363,13 @@ struct ajp_worker
  */
 int retries;
 
+/*
+ * Public property used in load balancer workers, meaning
+ * the maximum number of failover attempts between ajp
+ * workers of cluster.
+ */
+int lb_retries;
+
 unsigned int max_packet_size;  /*  Maximum AJP Packet size */
 
 int retry_interval;/*  Number of milliseconds to sleep before doing a retry */
diff --git a/native/common/jk_lb_worker.c b/native/common/jk_lb_worker.c
index a9894eb..d6e0251 100644
--- a/native/common/jk_lb_worker.c
+++ b/native/common/jk_lb_worker.c
@@ -1159,6 +1159,7 @@ static int JK_METHOD service(jk_endpoint_t *e,
 if (p-worker-sequence  p-worker-s-h.sequence)
 jk_lb_pull(p-worker, JK_FALSE, l);
 for (i = 0; i  num_of_workers; i++) {
+jk_log(l, JK_LOG_DEBUG, LB - num_of_workers: %d, retry: %d, lb_retries: %d, num_of_workers, i, p-worker-lb_retries);
 lb_sub_worker_t *rec = (p-worker-lb_workers[i]);
 if (rec-s-state == JK_LB_STATE_BUSY) {
 if (ajp_has_endpoint(rec-worker, l)) {
@@ -1203,7 +1204,10 @@ static int JK_METHOD service(jk_endpoint_t *e,
service sticky_session=%d id='%s',

Tomcat under load frontend reverse proxy timeouts

[Frederik Nosi]

Hi all,

Having to deal with slow applications deployed under tomcat, with a 
reverse proxy in front, frequently i've noticed that even when the 
frontend timeouts and closes it's part of the TCP connection, the Tomcat 
thread processing the request goes on and on till it finishes. Is there 
a way to make the proccessing thread stop when the frontend connection 
get's closed?



Thanks in advance,

Frederik

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Is it possible to send a 'keep-alive' packet back to client session every x seconds?

[Frederik Nosi]

Hi,
On 03/27/2014 04:08 AM, Matthew Turany wrote:

Hi,

Trying to figure out if this is possible; apache reverse-proxy sitting in
front of a server running tomcat serving a web app. Due to the amount of
data in the backend DB, when a user generated report is requested, it can
take several minutes (3-4) for the report to be presented to the browser
session.

In this particular case the client browser is sitting behind a gateway /
firewall that will drop the connection after 60 seconds of 'inactvity' e.g.
no traffic back to the browser (high-security environment).
The app presents a web pop-up stating Your report is being prepared and
session keepalive timeouts are all set accordingly, however since the
gateway doesn't see any active traffic it will close the connection forcing
the client browser to reconnect on a new connection which in affect loses
their report. (Note: that this all works fine for anyone not behind that
particular gateway)

Is it possible to configure either apache or tomcat to send a packet every
x number of seconds so that at the client end the gateway thinks the
session is still active and will keep the connection open, or is this
something best put into the actual web app?


The usual solution in this case is using Keepalive TCP. Check this, it's 
based on Linux but

the theory works on every other OS:

http://www.tldp.org/HOWTO/html_single/TCP-Keepalive-HOWTO/


Thanks,
Matt

Re: mod_jk - Failover behaviour and load

[Frederik Nosi]

Hi,

sure, the conf follows:

=

worker.list = worker_test,worker_status

worker.worker_status.type=status

worker.worker_test.type = lb
worker.worker_test.balance_workers = test1,test2,test3,test4
worker.worker_test.sticky_session = true
worker.worker_test.sticky_session_force = false
worker.worker_test.method = business
worker.worker_test.retries = 1


worker.default_params.type = ajp13
worker.default_params.host = localhost
worker.default_params.port = 8080
worker.default_params.lbfactor = 1
worker.default_params.socket_timeout = 40
worker_default_params.socket_keepalive = true
worker.default_params.connection_pool_size = 300
worker.default_params.connection_pool_minsize = 25
worker.default_params.connection_pool_timeout = 60
worker.default_params.reply_timeout = 500

worker.default_params.retries = 1
worker.default_params.recovery_options = 19

worker.test1.reference = worker.default_params
worker.test1.host = 127.0.0.1
worker.test1.port = 8009

worker.test2.reference = worker.default_params
worker.test2.host = 127.0.0.2
worker.test2.port = 8009

worker.test3.reference = worker.default_params
worker.test3.host = 127.0.0.3
worker.test3.port = 8009

worker.test4.reference = worker.default_params
worker.test4.host = 127.0.0.4
worker.test4.port = 8009

==



On 03/19/2014 02:40 PM, Christopher Schultz wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Frederik,

On 3/18/14, 10:22 PM, Frederik Nosi wrote:

Each apache httpd talks with every tomcat.

mod_jk is configured with load balancing by business, sticky
sessions, only one try for ajp worker (so if it's busy i dont add
more stuff to the poor worker) and only 1 retry for all load
balanced workers, in short, here's the interesting part of my
conf:

worker.worker_lb.balance_workers = w1, w2, w3, w4 sticky session
on, sticky session force = off ecc

Could you post the whole configuration without editing?


[...]



Now, even with this settings, on peak times i have an
amplification effect, all tomcats are busy, mod_jk reaches atimeout
and retries again and again ... in a death spiral for the poor
cats.

This behaviour brings me two problems:

1) Even if apache httpd / mod_jk timeouts (read_timeout) from his
side this does not stop the tomcat thread proccessing that request.
It goes sometimes on and on but after all finishes the request.
This seems a resource waste, is there a way to let tomcat know
that mod_jk droped the connection and stop proccessingthe dropped
request? I tried to find a way to get this behaviour without
success, so any suggestions are welcome.


2) I noticed that the default behaviour of load balanced workers
and AJP workers is as follows:

a) ajp worker reaches retry_timeout, at this point retries again
to the same worker once, thus amplifying load. This onewas easy to
solve, just add a worker.w1.retries = 1 and this does not happen
anymore

b) Load balancer worker goes to the next ajp worker.. till the
last one, than does another round of requests:

lb - w1 - timeout [...] lb- w4 - timeout

LB try 2 lb - w1 - timeout [...] request fails.

The second round to the workers IMHO is wasteful, but easy solved
with: worker.worker_lb.retries = 1

So at this point for every request all the busy tomcats get only
one request. I tryed finding a way to tell the LB worker to try
just once, only another worker for request but i didnt  find an
option for doing this. Well, as i had an each to scratch i made a
patch that adds an option to a LB worker telling how much times to
retry before giving up, i'm glad to post it if somebody is
interested. Not sure it's perfect, but works for me.


Now, my questions:

1) Am i doing something wrong? 2) Is there a way to make a tomcat
thread stop proccessing when AJP timeouts?

- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJTKZ5iAAoJEBzwKT+lPKRYeowQAJ1XufNZe0mc/DFwGHiIPYFK
4mqyVX00hjSwMQTSwxsTuFygiTMyTTgezsbg+EkoiDdMFtMKVRt3QZpXzCAWOthr
DWYSGcnnEONDcjWv4WMaPZ8cCnT4NCh/0FmAjbCAcVv+RiJBeoOEuNuIDX4DkS71
+4HmvUB9KwQGrSnAmtS17gA6Wauf4jW0QVe/yS642Kxv+1njFFp+n0Ezx8DnzJQr
VyRaKC7ITQTiAMbgHoYUOq+7Nwy1wOfRiwkfvWY8WNdJ1xOZJsicu5wnU1i4RDkp
rpUvhMvMJjvqO6waHO8HYcqoAhakOPBclTDpD24JuMlL5fJJVOMiFhD4iRp83tGr
X+FxMHZBTMlVhsz7ve+gQniy9zAvsTMJi5JMSkMO04WHRuqzY6dd4Iw7vrSm+LIQ
JfTUOS2Bv1NXv4MOwBCdwndQLftPDDmi1TVgFuoJH8BmeZPKqoLyGKbsOpIQkYFz
aZHtSViccTcNXQwQLl0pk9UVbYzGiRT4LdLFqRGEtq1yRdU7X3H9wuKkbQmRVrFe
Ou0xnx5/LF7RZ2cpeVnV5stZGTlTlnrlmkRX++78kplvoo7RRLP6J8A5Xmxk9zbN
Z8d8JWkNkoy1lbQr9tPaSR5A77qjwBChl3L6Elq7Gc3B1JV8SNJs8Pxau0uSm3dI
gSnf9kDocRm8ta6Ykgck
=PCNV
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: mod_jk - Failover behaviour and load

[Frederik Nosi]

Hi again,

sorry for the html email, top posting and for putting you in CC, got 
used to this on other mailing lists.


Thanks,
Frederik

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: mod_jk - Failover behaviour and load + patch

[Frederik Nosi]

Hi,

On 03/19/2014 05:26 PM, Christopher Schultz wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Frederik,

On 3/19/14, 10:00 AM, Frederik Nosi wrote:

worker.list = worker_test,worker_status
worker.worker_status.type=status

worker.worker_test.type = lb worker.worker_test.balance_workers =
test1,test2,test3,test4 worker.worker_test.sticky_session = true
worker.worker_test.sticky_session_force = false
worker.worker_test.method = business

I think you mean busyness. Could that be the trouble?


Oh, thanks for the catch, it was obviously a typo.
Corrected that and retested but the behaviour does not change though.
FWICS there's no option of load balanced workers for setting the total 
number of retries. I'll try to explain it bette, i have a load balancer 
with 4 ajp workers. I needed a way to tell the LB worker that in case of 
failure to retry only one other worker and if the second fails, to fail 
the request without retrying the other two workers. As i didnt found an 
option doing this i made a patch that adds this functionality and it 
works, though not sure it's perfect, i'm attaching it in case someone is 
interested, it adds a new parameter, lb_retries, which does what i need. 
Hope you find it useful, suggestions and corrections welcome obviously.


What i was asking was:

1) Is my solution correct? Or is there some other way to have the 
behaviour i needed without patching?


The second question is more related to tomcat probably, but here it is:

2) When an ajp worker fails for reply_timeout the tomcat thread that is 
busy working on the corresponding request does not stop but continues 
proccessing. Is there a way to change this behaviour? It's problematic 
in case of mod_jk failover to another ajp worker, requests get doubled.



Hope i was more clear as english is a bit rusty.

Thanks!




- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJTKcUoAAoJEBzwKT+lPKRYHEUP/3xA1yWIEqC/NwqGebZc31oS
uH+495ih7+e1gPyNdoLy4/G3wSwBl3AD/6+B7ou2YFEnHsNHQbJUQ+jJlmfVU6/w
n9viCjmyoLTM1DjsGbNNbkdGqVwmlS9qHK4GTfkaT2b4US3QepxM0JnLHMWj8Ow6
F4k3se0E0WzmH8U5VluaCy69q4QQK3IY82daZrj7XKsmXCewIys+E+Tc0xe1In9W
smjz6KB5xBsfGuZBVqXNZJ+VvGXhQl7kOlbVkpNK/BQpRSd7ewqmuekG+Nq8G5wA
L7hBqRTbACgRD9i8RNA7FnFaHbUyd/xZjOe6uwEj+zfRj36KljyQdpJmrxs2rskc
GLX/an+HkKJjFsidEB2MLoocgSCw/5satejL7u1Ky0TMmA4/8CT7EneQ57LeZgp4
QSza2rcBcE+LJIsnQX50K3hsz7yV9JkZB7K5AU+nJeXRk0RA2tC1o4x7m/U4u3U8
xkBq0I3JcGFjt99xCynu1K5JF4dL3FuM3NwomOx3d8d0F92/SQOy9jrCYRFuzjcg
XPS6GRzCQ6hi2HPvBaKWL0wql0cUJBuf4+bQF0/NgNx7OsKBhZv8VLOBKh0VSkZA
XbFrxj6bqxgzM1HD8rGudK/EN/hAFBBkVsEKyApYT5NLR+q4ZvsXXfcMxzt3HN8s
5x8gmn6D6c1EILouhiq4
=Zyt8
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



diff --git a/native/.gitignore b/native/.gitignore
new file mode 100644
index 000..1f8d345
--- /dev/null
+++ b/native/.gitignore
@@ -0,0 +1,5 @@
+Makefile.in
+aclocal.m4
+config.log
+config.nice
+configure
diff --git a/native/common/.gitignore b/native/common/.gitignore
new file mode 100644
index 000..2a9005d
--- /dev/null
+++ b/native/common/.gitignore
@@ -0,0 +1 @@
+config.h.in
diff --git a/native/common/jk_ajp_common.c b/native/common/jk_ajp_common.c
index 08bcc02..9a12a89 100644
--- a/native/common/jk_ajp_common.c
+++ b/native/common/jk_ajp_common.c
@@ -2904,6 +2904,9 @@ int ajp_init(jk_worker_t *pThis,
 p-retries =
 jk_get_worker_retries(props, p-name,
   JK_RETRIES);
+p-lb_retries =
+jk_get_worker_lb_retries(props, p-name,
+  JK_LB_RETRIES);
 
 p-max_packet_size =
 jk_get_max_packet_size(props, p-name);
diff --git a/native/common/jk_ajp_common.h b/native/common/jk_ajp_common.h
index 0c1636c..7b342d0 100644
--- a/native/common/jk_ajp_common.h
+++ b/native/common/jk_ajp_common.h
@@ -363,6 +363,13 @@ struct ajp_worker
  */
 int retries;
 
+/*
+ * Public property used in load balancer workers, meaning
+ * the maximum number of failover attempts between ajp
+ * workers of cluster.
+ */
+int lb_retries;
+
 unsigned int max_packet_size;  /*  Maximum AJP Packet size */
 
 int retry_interval;/*  Number of milliseconds to sleep before doing a retry */
diff --git a/native/common/jk_lb_worker.c b/native/common/jk_lb_worker.c
index a9894eb..d6e0251 100644
--- a/native/common/jk_lb_worker.c
+++ b/native/common/jk_lb_worker.c
@@ -1159,6 +1159,7 @@ static int JK_METHOD service(jk_endpoint_t *e,
 if (p-worker-sequence  p-worker-s-h.sequence)
 jk_lb_pull(p-worker, JK_FALSE, l);
 for (i = 0; i  num_of_workers; i++) {
+jk_log(l, JK_LOG_DEBUG, LB - num_of_workers: %d, retry: %d, lb_retries: %d

mod_jk - Failover behaviour and load

[Frederik Nosi]

Hi all,

I have this situation, an hardware loadbalancer configured with sticky 
session based
on souce IP, two httpd servers behind it doing HTTPS termination with 
mod_jk doing
load balancing vs four tomcats, i'll try to do my best in ascii art, 
hopping my mailer

does the right thing:


- {internet}-
|
  v
  [HW LB with source IP stickyness]
|   |
|   |
   [httpd1][httpd2]
   [mod_jk] [mod_jk]
  | |
==
 |  ||  |
   [TC1]  [TC2]   [TC3]  [TC4]



Each apache httpd talks with every tomcat.

mod_jk is configured with load balancing by business, sticky sessions, only
one try for ajp worker (so if it's busy i dont add more stuff to the 
poor worker)
and only 1 retry for all load balanced workers, in short, here's the 
interesting

part of my conf:


worker.worker_lb.balance_workers = w1, w2, w3, w4
sticky session on, sticky session force = off ecc
[...]



Now, even with this settings, on peak times i have an amplification 
effect, all
tomcats are busy, mod_jk reaches atimeout and retries again and again 
... in a

death spiral for the poor cats.

This behaviour brings me two problems:

1) Even if apache httpd / mod_jk timeouts (read_timeout) from his side 
this does
not stop the tomcat thread proccessing that request. It goes sometimes 
on and on

but after all finishes the request.
This seems a resource waste, is there a way to let tomcat know that 
mod_jk droped
the connection and stop proccessingthe dropped request? I tried to find 
a way to

get this behaviour without success, so any suggestions are welcome.


2) I noticed that the default behaviour of load balanced workers and AJP 
workers is

as follows:

a) ajp worker reaches retry_timeout, at this point retries again to 
the same worker

once, thus amplifying load. This onewas easy to solve, just add a
worker.w1.retries = 1
and this does not happen anymore

b) Load balancer worker goes to the next ajp worker.. till the last 
one, than does

another round of requests:

lb - w1 - timeout
[...]
lb- w4 - timeout

LB try 2
lb - w1 - timeout
[...]
request fails.

 The second round to the workers IMHO is wasteful, but easy solved 
with:

worker.worker_lb.retries = 1

So at this point for every request all the busy tomcats get only one 
request. I tryed finding a way to tell
the LB worker to try just once, only another worker for request but i 
didnt  find an option for doing this.
Well, as i had an each to scratch i made a patch that adds an option to 
a LB worker telling how much times
to retry before giving up, i'm glad to post it if somebody is 
interested. Not sure it's perfect, but works

for me.


Now, my questions:

1) Am i doing something wrong?
2) Is there a way to make a tomcat thread stop proccessing when AJP 
timeouts?



Thanks in advance,
Frederik


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org