Re: File "catalina.out" not being created/populated when using Tomcat 9.0.31 + Ubuntu 20.04, and content goes to the Ubuntu syslog instead?

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Brian,

On 6/22/20 12:43, Brian wrote:
>
> -BEGIN PGP SIGNED MESSAGE- Hash: SHA256
>
> Brian,
>
> On 6/22/20 02:04, Brian wrote:
>>  Hello,
>>
>> I have been using Tomcat for about 18 years. As far as I can
>> remember, everything that I leave on the log with
>> System.out.println() has been found inside the file
>> "catalina.out" which has been nice. However, this seems to have
>> changed. I just migrated to Ubuntu 20.04 + Tomcat 9.0.31, and
>> installed Tomcat doing a "sudo apt install tomcat9". Now I see
>> that the file "catalina.out" never gets created, and that
>> everything that had been sent there is now being sent to the
>> Ubuntu/Linux/Debian log "syslog". I don't like that, I want to
>> keep having a separate/dedicated log for Tomcat (which is
>> something I check very often) and not havoing to search for it
>> inside syslog (which is something I rarely inspect).
>>
>> I have done some research, and this is what I have discovered (if
>> I understood everything correctly):
>>
>> - Tomcat now runs as a service inside something called "systemsd"
>> - For some reason, the people at Ubuntu/Debian/Linux decided
>> that Tomcat's log should be found inside syslog, instead of
>> staying independent inside "catalina.out". Why is that? I don't
>> know and I don't like it! - The other files inside
>> /var/log/tomcat get created, including the
>> "catalina.-MM-DD.log" files. But I dont' find my
>> "System.out.println()" messages there, which has always been the
>> case. - I have found, however, that syslog tries to create and
>> populate the "catalina.out" file as well, since I have found an
>> error inside syslog that says something like "rsyslogd: file
>> '/var/log/tomcat9/catalina.out': open error: Permission denied".
>> - I have also found a file "/etc/rsyslog.d/tomcat9.conf" which,
>> indeed, seams to indicate syslog that the file
>> "/var/log/tomcat9/catalina.out" must be populated. Which makes
>> sense and should solve my needs. - Considering the "Permission
>> denied" error message, I started playing with the permissions
>> (something that I really don't have much experience with). If I
>> remember correctly, I created the file "catalina.out" manually,
>> and the modified its properties so the owners are "tomcat/adm",
>> since the "syslog" process/user seems to be inside the "adm"
>> group. Restarted everything... and the "catalina.out" file got
>> created and populated!!! So it seems that the main obstacle here
>> is a lack of correct permissions so syslog can do what has been
>> told and populate the "catalina.out" file. Did I get it right? -
>> However, when I delete all the log files (which I do every once
>> in a while), the permissions that I assigned get lost and the
>> file doesn't get created anymore.
>>
>> What are we supposed to do to deal with this problem? Can/should
>> we do something so the Tomcat log doesn't go to syslog? If not,
>> and considering that it seems that syslog is trying to populate
>> the "catalina.out" file as it has been told, what should we do
>> to correct the permissions problem?
>
> What are the permissions of the /var/log/tomcat directory?
>
> Hello Chris,
>
> I did a "ls -l /var/log/". According to that, this is what I see
> for "/var/log/tomcat9": drwxr-s---  3 tomcatadm
> 4096 Jun 22 10:51 tomcat9
>
> I'm not really an expert with Linux. It would seem to me that the
> "adm" group (to which syslog seems to belong) lacks a write
> permission, but in my old instance (Tomcat 8.5.39 + Ubuntu 18.04)
> the "catalina.out" file works perfectly (gets created and
> populated) and this is what I see there, it doesn't seem to me that
> the adm group has a write permission there either: drwxr-x---  3
> tomcat8   adm   4096 Jun 22 10:00 tomcat8
>
> Perhaps you need to chmod g+w /var/log/tomcat ? - -chris
>
> OK, I did it and the write permission was added to the adm group. I
> restarted Tomcat and it worked, the catalina.out file got
> created! However, after I restarted the whole Ubuntu, I discovered
> that the permissions went back to how there were (not write for
> adm). Why is that? In fact, now I remember that I tried this
> before, but since I restarted the whole Ubuntu instead of just
> Tomcat, I never saw any progress. I didn't notice that it would
> have worked if I just restarted Tomcat.
>
> In any case, why is this permission required in my new VPS, if the
> old one lacks it and catalina.out works perfectly?

It looks like whoever is responsible for packing Tomcat for Ubuntu has
something missing, there. I suggest you file a bug with Ubuntu.

- -chris
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl7w8lwACgkQHPApP6U8
pFicVg/7BbuJTaZVf2jTBP9WupuWfVo5YeNy2VXsCYv26xXbzVZXOrDpjrXpPQ9/
yA4oz2c3MeWyyEqtSUeft6X7c8ayKKBbwPpGGc+/20VRsKDYv0aOiNd+MfpdmhOJ

Re: File "catalina.out" not being created/populated when using Tomcat 9.0.31 + Ubuntu 20.04, and content goes to the Ubuntu syslog instead?

[Emmanuel Bourg]

Le 22/06/2020 à 18:43, Brian a écrit :

> I'm not really an expert with Linux. It would seem to me that the "adm" group 
> (to which syslog seems to belong) lacks a write permission

Indeed, rsyslog in Ubuntu runs as syslog:adm and needs special
permissions to write to /var/log/tomcat9. This issue should be reported
to Ubuntu.

The tomcat9 package in Debian isn't affected by this issue because
rsyslogd runs as root.


> OK, I did it and the write permission was added to the adm group. I restarted 
> Tomcat and it worked, the catalina.out file got created! However, after I 
> restarted the whole Ubuntu, I discovered that the permissions went back to 
> how there were (not write for adm). Why is that?

The permissions on this directory are managed by systemd-tmpfiles. The
tomcat9 package defines the expected permissions in
/usr/lib/tmpfiles.d/tomcat9.conf and the permissions are enforced when
the system starts.

It's possible to override the default settings, you have to copy the
configuration file to /etc/tmpfiles.d/ and change the permissions on
/var/log/tomcat9 from 2750 to 2760:

  cp /usr/lib/tmpfiles.d/tomcat9.conf /etc/tmpfiles.d/
  sed -i s/2750/2760/ /etc/tmpfiles.d/tomcat9.conf

The write permissions for the adm group will then be persistent.


> In any case, why is this permission required in my new VPS, if the old one 
> lacks it and catalina.out works perfectly?

I guess your older VPS had the tomcat8 package installed. The tomcat9
package is different as it leverages several systemd features to improve
the security and the reliability.

Emmanuel Bourg

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Strange crash-on-takeoff, Tomcat 7.0.104

[calder]

On Mon, Jun 22, 2020, 11:56 James H. H. Lampert 
wrote:

> On 6/20/20 8:41 AM, Mark Thomas wrote:
>
> > 7.0.105 hasn't been released yet. You can use catalina.sh from 7.0
> > 103 or the latest version from source control.
>
> Where would I find "the latest version from source control"?
>

https://gitbox.apache.org/repos/asf?p=tomcat.git


Found at
https://en.m.wikipedia.org/wiki/Apache_Tomcat

Re: File "catalina.out" not being created/populated when using Tomcat 9.0.31 + Ubuntu 20.04, and content goes to the Ubuntu syslog instead?

[Emmanuel Bourg]

Le 22/06/2020 à 10:44, calder a écrit :

> However, this is one reason we do not use "distro-specific" Tomcat
> installations (to include implementations of WebSphere and WebLogic).
> .
> For example, we grab the plain vanilla Tomcat ZIP and extract it to "/opt/"
> (as in "/opt/tomcat/") - we now have complete control over its
> configuration and runtime instantiation.

This is a valid strategy, but you miss the tight integration with the
system that comes out of the box with a distro specific package (proper
system service, started on boot, restarted on crashes, run by a non
privileged user allowed to bind to ports below 1024, sandboxed, with
logs in a consistent location). And you are still able to override some
aspects of the default configuration you might not like.

Emmanuel Bourg

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Tomcat Connector issue

[George Stanchev]

We are getting HSE_REQ_SEND_RESPONSE_HEADER failed with error=87 (0x0057) 
on a 302 redirect proxied by TC connector 1.2.46. I can see the 302 response 
come over from TC and it looks legit. Trace logs below. Anyone else running 
into a similar error or perhaps some clue as to why this can be happening?



[Thu Jun 18 09:37:28.185 2020] [6060:2544] [trace] 
jk_tcp_socket_sendfull::jk_connect.c (1042): enter
[Thu Jun 18 09:37:28.186 2020] [6060:2544] [trace] 
jk_tcp_socket_sendfull::jk_connect.c (1072): exit
[Thu Jun 18 09:37:28.187 2020] [6060:2544] [trace] 
ajp_connection_tcp_send_message::jk_ajp_common.c (1292): exit
[Thu Jun 18 09:37:28.188 2020] [6060:2544] [debug] 
ajp_send_request::jk_ajp_common.c (1768): (worker-local) Statistics about 
invalid connections: connect check (0), cping (0), send (0)
[Thu Jun 18 09:37:28.188 2020] [6060:2544] [debug] 
ajp_send_request::jk_ajp_common.c (1779): (worker-local) request body to send 0 
- request body to resend 0
[Thu Jun 18 09:37:28.189 2020] [6060:2544] [trace] 
ajp_send_request::jk_ajp_common.c (1904): exit
[Thu Jun 18 09:37:28.190 2020] [6060:2544] [trace] 
ajp_get_reply::jk_ajp_common.c (2201): enter
[Thu Jun 18 09:37:28.191 2020] [6060:2544] [trace] 
ajp_connection_tcp_get_message::jk_ajp_common.c (1325): enter
[Thu Jun 18 09:37:28.193 2020] [6060:2544] [trace] 
jk_tcp_socket_recvfull::jk_connect.c (1093): enter
[Thu Jun 18 09:37:28.199 2020] [6060:2544] [trace] 
jk_tcp_socket_recvfull::jk_connect.c (1122): exit
[Thu Jun 18 09:37:28.200 2020] [6060:2544] [trace] 
jk_tcp_socket_recvfull::jk_connect.c (1093): enter
[Thu Jun 18 09:37:28.202 2020] [6060:2544] [trace] 
jk_tcp_socket_recvfull::jk_connect.c (1122): exit
[Thu Jun 18 09:37:28.203 2020] [6060:2544] [debug] 
ajp_connection_tcp_get_message::jk_ajp_common.c (1459): received from ajp13 
pos=0 len=307 max=16384
..
[Thu Jun 18 09:37:28.223 2020] [6060:2544] [trace] 
ajp_connection_tcp_get_message::jk_ajp_common.c (1463): exit
[Thu Jun 18 09:37:28.224 2020] [6060:2544] [trace] 
ajp_process_callback::jk_ajp_common.c (1919): enter
[Thu Jun 18 09:37:28.225 2020] [6060:2544] [trace] 
ajp_unmarshal_response::jk_ajp_common.c (718): enter
[Thu Jun 18 09:37:28.226 2020] [6060:2544] [debug] 
ajp_unmarshal_response::jk_ajp_common.c (736): (worker-local) status = 302
[Thu Jun 18 09:37:28.228 2020] [6060:2544] [debug] 
ajp_unmarshal_response::jk_ajp_common.c (743): Number of headers is = 5
[Thu Jun 18 09:37:28.229 2020] [6060:2544] [debug] 
ajp_unmarshal_response::jk_ajp_common.c (799): (worker-local) Header[0] 
[Expires] = [19-Mar-1971 08:23:00 GMT]
[Thu Jun 18 09:37:28.229 2020] [6060:2544] [debug] 
ajp_unmarshal_response::jk_ajp_common.c (799): (worker-local) Header[1] 
[Cache-Control] = [no-store, no-cache, private, must-revalidate]
[Thu Jun 18 09:37:28.230 2020] [6060:2544] [debug] 
ajp_unmarshal_response::jk_ajp_common.c (799): (worker-local) Header[2] 
[Pragma] = [no-cache]
[Thu Jun 18 09:37:28.231 2020] [6060:2544] [debug] 
ajp_unmarshal_response::jk_ajp_common.c (799): (worker-local) Header[3] 
[Location] = 
[https://host/idp/login?fsid=2baf82dbd9e74657988568e2ae98ceef=https%3A%2F%host%2Fworkcenter%2Ftmtrack.dll%3Fshell%3Dswc]
[Thu Jun 18 09:37:28.232 2020] [6060:2544] [debug] 
ajp_unmarshal_response::jk_ajp_common.c (799): (worker-local) Header[4] 
[Content-Length] = [0]
[Thu Jun 18 09:37:28.233 2020] [6060:2544] [trace] 
ajp_unmarshal_response::jk_ajp_common.c (806): exit
[Thu Jun 18 09:37:28.234 2020] [6060:2544] [trace] 
start_response::jk_isapi_plugin.c (889): enter
[Thu Jun 18 09:37:28.235 2020] [6060:2544] [debug] 
start_response::jk_isapi_plugin.c (939): Starting response for URI '/idp/login' 
(protocol HTTP/1.1)
[Thu Jun 18 09:37:28.236 2020] [6060:2544] [debug] 
start_response::jk_isapi_plugin.c (1048): Not using Keep-Alive
[Thu Jun 18 09:37:28.237 2020] [6060:2544] [error] 
start_response::jk_isapi_plugin.c (1082): HSE_REQ_SEND_RESPONSE_HEADER failed 
with error=87 (0x0057)
[Thu Jun 18 09:37:28.238 2020] [6060:2544] [trace] 
start_response::jk_isapi_plugin.c (1092): exit
[Thu Jun 18 09:37:28.239 2020] [6060:2544] [trace] 
ajp_connection_tcp_get_message::jk_ajp_common.c (1325): enter
[Thu Jun 18 09:37:28.240 2020] [6060:2544] [trace] 
jk_tcp_socket_recvfull::jk_connect.c (1093): enter
[Thu Jun 18 09:37:28.241 2020] [6060:2544] [trace] 
jk_tcp_socket_recvfull::jk_connect.c (1122): exit
[Thu Jun 18 09:37:28.242 2020] [6060:2544] [trace] 
jk_tcp_socket_recvfull::jk_connect.c (1093): enter
[Thu Jun 18 09:37:28.243 2020] [6060:2544] [trace] 
jk_tcp_socket_recvfull::jk_connect.c (1122): exit
[Thu Jun 18 09:37:28.244 2020] [6060:2544] [debug] 
ajp_connection_tcp_get_message::jk_ajp_common.c (1459): received from ajp13 
pos=0 len=2 max=16384

Re: File "catalina.out" not being created/populated when using Tomcat 9.0.31 + Ubuntu 20.04, and content goes to the Ubuntu syslog instead?

[Emmanuel Bourg]

Le 22/06/2020 à 20:03, Christopher Schultz a écrit :

> It looks like whoever is responsible for packing Tomcat for Ubuntu has
> something missing, there. I suggest you file a bug with Ubuntu.

That must be me :) I'll get a look.

Emmanuel Bourg

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: File "catalina.out" not being created/populated when using Tomcat 9.0.31 + Ubuntu 20.04, and content goes to the Ubuntu syslog instead?

[Brian]

-Original Message-
From: Emmanuel Bourg 
Reply-To: Tomcat Users List 
Date: Monday, June 22, 2020 at 15:36
To: "users@tomcat.apache.org" 
Subject: Re: File "catalina.out" not being created/populated when using Tomcat 
9.0.31 + Ubuntu 20.04, and content goes to the Ubuntu syslog instead?

Le 22/06/2020 à 20:03, Christopher Schultz a écrit :

> It looks like whoever is responsible for packing Tomcat for Ubuntu has
> something missing, there. I suggest you file a bug with Ubuntu.

That must be me :) I'll get a look.

Emmanuel Bourg


Hi Emmanuel,

Please do! It definitely looks like a bug.
I just reported the bug here: 
https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1884591
 

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: broken pipe error keeps increasing open files

[Ayub Khan]

Chris,

I am using HikariCP for connection pooling. If the database is leaking
connections then I should see connection not available exception.

How do I find out which file descriptors are leaking ?  these are not files
open on disk as there is no explicit disk file I/O in this application.

I just use the below command to check for open file descriptors:

watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc -l"

Thanks and Regards
Ayub

On Sun, Jun 21, 2020 at 8:18 PM Christopher Schultz <
ch...@christopherschultz.net> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Ayub,
>
> On 6/20/20 11:51, Ayub Khan wrote:
> > Sorry we are using  8.0.32 version of tomcat.
> >
> > below is the configuration:
> >
> > Server version: Apache Tomcat/8.0.32 (Ubuntu) Server built:   Jan
> > 24 2020 16:24:30 UTC Server number:  8.0.32.0 OS Name:
> > Linux OS Version: 4.4.0-1087-aws Architecture:   amd64 JVM
> > Version:1.8.0_181-b13 JVM Vendor: Oracle Corporation
> >
> > I use the below command to check the file descriptors:
> >
> > watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc -l"
>
> So you know there is some kind of increase in file-handle use, but you
> don't know what types of file handles are increasing, right?
>
> Can you try to find out which kinds of file handles are increasing?
>
> I have a sneaking suspicion that it's your database connections and
> not actually files open on the disk.
>
> Are you using a database connection pool? If not, you should really
> use one and limit the number of connections to something sane. If you
> are using one, are you monitoring it to see how many connections are
> actually being used? Are you sure you are using proper resource
> management[1]? Even a single code-path that leaks connections can leak
> them quickly under load.
>
> > When there an issue related to broken files, this value keeps
> > increasing, the only way to bring it down is to remove vm instance
> > from AWS load balancer.> Which version of tomcat should I install
> > ?
>
> Tomcat 8.0.x hasn't been supported since its last release on 29 June
> 2018. That was 8.0.53. Your release is from 8 February 2016 and is
> dangerously out of date (unless you are using the Ubuntu-packaged
> version, in which case I hope they kept-up with security patches thee
> past 4 years).
>
> - -chris
>
> > On Sat, Jun 20, 2020 at 6:28 PM Christopher Schultz <
> > ch...@christopherschultz.net> wrote:
> >
> > Ayub,
> >
> > On 6/19/20 16:46, Ayub Khan wrote:
>  tomcat 8.5 broken pipe increases open files on ubuntu AWS
> >
> > Which exact version of Tomcat 8.5? If you aren't running the
> > latest version (8.5.56), please upgrade and re-test.
> >
>  If there is slow response from db I see this stack trace and
>  the open files goes high and the only way to open files go
>  down is to remove the instance from Amazon load balancer.
> 
>  Is there a way to keep the open files low even when Broken
>  pipe error is thrown ?
> >
> > What is your evidence that file handles are being left open?
> >
> > Which file handles are being left open?
> >
> > -chris
> >>
> >> -
> >>
> >>
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >> For additional commands, e-mail: users-h...@tomcat.apache.org
> >>
> >>
> >
> -BEGIN PGP SIGNATURE-
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl7vlj0ACgkQHPApP6U8
> pFiCqBAAqDO2+sUHpjcL8c5qwQbc0EG1XjgawI/ggjegVNSLTZ6mjdCVmxU3V10T
> IAdAmckT+o5qkZKQSVAQHpNW7i5A21U0dOUF1r1EWq0WZIN3fE1dcpQVsOlvUSqD
> qZ+gnGzg3UasSMJFyOj2hnU1+PBnhSFWvKcHwTda9gelqmgHp24rwpCvHFFFmH+2
> I/QrHoQc1W9F0bUmGOsfBy5+eC981KMUfZrIJmUBQFOhopaD19W+yofZCl0S15E7
> dxfIcqvFSF7FzZ1FraKGVd+nEUlW6PLuNqHkqpNGIstMWuQuo1RkukdXeP3FqjeA
> 36W5GOo6tlJTPGjbGaVPLnKw+hQuQa9he5Dv4E03C9vk1cbU6g+3hguKGhIfV3FM
> 2Pg4SE4LwVv3LMY1uxuuHGPgrs6CqHmMaDA6FQARhm/5iD5WJQBrXwzCYztHK/Z+
> +QILfdBa1VZl9qyujBneY9oJ9yc7HeBb1DDZAAqZFrOJctRmDrdZQrUwZ+b6vUzL
> kBr7fCSnzEqUbPzv5DiY6qfD2x2f1GfJFLY6LuJ36g4P0YSItRTaR2nX/VOzVM9D
> oZsDFZhlSZa0U7KpdaQDpi1dw6wgW83gfc7GxrbYvQsgXZPnm25JeU/UI5p1O+cQ
> q+iynFZ8poxSOIjx+2A9anwCXAiiWbyVRJ6Rmy2NwkCbXH/VhwY=
> =oNTV
> -END PGP SIGNATURE-
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

-- 

Sun Certified Enterprise Architect 1.5
Sun Certified Java Programmer 1.4
Microsoft Certified Systems Engineer 2000
http://in.linkedin.com/pub/ayub-khan/a/811/b81
mobile:+966-502674604
--
It is proved that Hard Work and kowledge will get you close but attitude
will get you there. However, it's the Love
of God that will put you over the top!!

RE: Issue found during migration of Tomcat version 6.0.35 to 8.5.5

[Lavitesh Verma]

Hi John,

We didn't use woodstox previously in our application and after we updated 
tomcat to version 8.5.5 we are getting this issue.

We were able to resolve the issue by adding "wstx-asl-2.9.1.jar" to the 
application, could you please confirm if this is the recommended fix for this 
issue.

Thanks & Regards
Lavitesh Verma
Software Engineering Associate
Amdocs Global SmartOps
[download12]+91.9810157771
[cid:image001.jpg@01D2912B.17505E90]

From: john.e.gr...@wellsfargo.com.INVALID 
Sent: Wednesday, June 17, 2020 12:48 AM
To: users@tomcat.apache.org
Subject: RE: Issue found during migration of Tomcat version 6.0.35 to 8.5.5

Sorry for the top post.  I'm having email formatting problems.

Lavitesh,

15-Jun-2020 07:34:45.122 SEVERE [http-nio-7080-exec-10] 
org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() for 
servlet [MainControllerServlet] in context with path [/porequest] threw 
exception [Servlet execution threw an exception] with root cause
javax.xml.stream.FactoryConfigurationError: Provider 
com.ctc.wstx.stax.WstxInputFactory not found
at javax.xml.stream.XMLInputFactory.newInstance(Unknown Source)

This is saying it can't find WstxInputFactory, which appears to be the Woodstox 
implementation of the XMLInputFactory abstract class.

If you want to continue using Woodstox, you need to add it to your classpath 
again.

If you don't want to use it, there are a few things to check to find out why 
it's being used.  Look for:

  *   System property 
javax.xml.stream.XMLInputFactory=com.ctc.wstx.stax.WstxInputFactory
  *   $JAVA_HOME/lib/stax.properties
  *   $JAVA_HOME/lib/jaxp.properties
  *   META-INF/services/javax.xml.stream.XMLInputFactory

That last one is actually a file of the same name as the class.  There is no 
traditional file extension.  This file might also be in a 3rd-party lib, too.  
The file contains the name of the implementation to use.

One of those things is probably telling the finder to use that Woodstox class.

If you don't want to use Woodstox anymore, you have to remove or change one of 
those sources.  Removing it just means you'll get the default.


John



From: Lavitesh Verma 
mailto:lavitesh.ve...@amdocs.com>>
Sent: Monday, June 15, 2020 8:49 AM
To: Tomcat Users List mailto:users@tomcat.apache.org>>
Subject: RE: Issue found during migration of Tomcat version 6.0.35 to 8.5.5


Hi,



PFA the complete Stack Trace for the Issue.

Thanks & Regards
Lavitesh Verma
Software Engineering Associate
Amdocs Global SmartOps
[download12]+91.9810157771
OOO - 06/16 - 06/19
[cid:image001.jpg@01D2912B.17505E90]




-Original Message-

From: Jason Wee mailto:peich...@gmail.com>>

Sent: Monday, June 15, 2020 6:49 PM

To: Tomcat Users List mailto:users@tomcat.apache.org>>

Subject: Re: Issue found during migration of Tomcat version 6.0.35 to 8.5.5



guess looks like jar not found, full stacktrace would be helpful.



On Mon, Jun 15, 2020 at 8:17 PM Lavitesh Verma 
mailto:lavitesh.ve...@amdocs.com>>

wrote:



> Hi Team,

>

>

>

> Below are the details of the system and tomcat version

>

> Old tomcat version: *Apache Tomcat/6.0.35*

>

> New tomcat version: *Apache Tomcat/8.5.5*

>

> Operating System: *SunOS *

>

> OS Version: *5.10*

>

> Architecture: *sparcv9*

>

> JVM Version: *1.8.0_101-b13*

>

> Vendor: *Oracle Corporation*

>

>

>

> We are trying to migrate Apache Tomcat version 6.0.35 to 8.5.5.

>

>

>

> We found the issue javax.xml.stream.FactoryConfigurationError:

> Provider com.ctc.wstx.stax.WstxInputFactory not found in localhost logs.

>

>

>

>

>

> Could you please assist on how we could resolve the issue.

>

>

>

> Thanks & Regards

>

> *Lavitesh Verma*

>

> Software Engineering Associate

>

> Amdocs Global SmartOps

>

> [image: download12]+91.9810157771

>

> *OOO - 06/16 - 06/19*

>

> [image: cid:image001.jpg@01D2912B.17505E90]

>

>

>

> *This email and the information contained herein is proprietary and

> confidential and subject to the Amdocs Email Terms of Service, which

> you may review at*

> *https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww

> .amdocs.com%2Fabout%2Femail-terms-of-service*data=02%7C01%7CLavit

> esh.Verma%40amdocs.com%7Cf5593255a1244392bb8a08d8113145f6%7Cc8eca3ca12

> 7646d59d9da0f2a028920f%7C0%7C0%7C637278250614631760sdata=%2B6rV%2

> Bbnayil4ZJr7yAuCsl2YyE86CZV19JWANZPz%2BIo%3Dreserved=0

>  .amdocs.com%2Fabout%2Femail-terms-of-servicedata=02%7C01%7CLavite

> sh.Verma%40amdocs.com%7Cf5593255a1244392bb8a08d8113145f6%7Cc8eca3ca127

> 646d59d9da0f2a028920f%7C0%7C0%7C637278250614631760sdata=Wtzp%2BGE

> %2BjEvBwZwExHs1jlNLsN5wxGZvYhHK7SAwfxM%3Dreserved=0>

>
This email and the information contained herein is proprietary and confidential 
and subject to the Amdocs Email Terms of Service, which you may review at 

Re: Connection Closure due to Fatal Stream with HTTP2

[Chirag Dewan]

Hi,

Update: We found that Tomcat goes OOM when a client closes and opens new
connections every second. In the memory dump, we see a lot of RequestInfo
objects that are causing the memory spike.

After a while, Tomcat goes OOM and start rejecting request(I get a
request timed out on my client). This seems like a bug to me.

For better understanding, let me explain my use case again:

I have a jetty client that sends HTTP2 requests to Tomcat. My requirement
is to close a connection after a configurable(say 5000) number of
requests/streams and open a new connection that continues to send requests.
I close a connection by sending a GoAway frame.

When I execute this use case under load, I see that after ~2hours my
requests fail and I get a series of errors like request timeouts(5seconds),
invalid window update frame, and connection close exception on my client.
On further debugging, I found that it's a Tomcat memory problem and it goes
OOM after sometime under heavy load with multiple connections being
re-established by the clients.

[image: image.png]

[image: image.png]

Is this a known issue? Or a known behavior with Tomcat?

Please let me know if you any experience with such a situation. Thanks in
advance.

On Sun, Jun 14, 2020 at 11:30 AM Chirag Dewan 
wrote:

> Hi,
>
> This is without load balancer actually. I am directly sending to Tomcat.
>
> Update:
>
> A part issue I found was to be 9.0.29. I observed that when request were
> timed out on client (2seconds), the client would send a RST frame. And the
> GoAway from Tomcat was perhaps a bug. In 9.0.36, RST frame is replied with
> a RST from Tomcat.
>
> Now the next part to troubleshoot is why after about an hour or so,
> requests are timed out at Tomcat.
>
> Could close to 100 HTTP2 connections per second cause this on Tomcat?
>
> Thanks
>
> On Sun, 14 Jun, 2020, 12:27 AM Michael Osipov, 
> wrote:
>
>> Am 2020-06-13 um 08:42 schrieb Chirag Dewan:
>> > Hi,
>> >
>> > We are observing that under high load, my clients start receiving a
>> GoAway
>> > frame with error:
>> >
>> > *Connection[{id}], Stream[{id}] an error occurred during processing that
>> > was fatal to the connection.*
>> >
>> > Background : We have implemented our clients to close connections after
>> > every 500-1000 requests (streams). This is a load balancer requirement
>> that
>> > we are working on and hence such a behavior. So with a throughput of
>> around
>> > 19k, almost 40 connections are closed and recreated every second.
>> >
>> > After we receive this frame, my clients start behaving erroneously.
>> Before
>> > this as well, my clients start sending RST_STREAM with canceled for each
>> > request. Could this be due to the number of connections we open? Is it
>> > related to the version of Tomcat? Or maybe my clients are misbehaving?
>> >
>> > Now since I only receive this under heavy load, I can't quite picture
>> > enough reasons for this to happen.
>> >
>> > Any possible clues on where I should start looking?
>> >
>> > My Stack:
>> > Server - Tomcat 9.0.29
>> > Client - Jetty 9.x
>>
>> Does the same happen w/o the load balancer?
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>>

Re: broken pipe error keeps increasing open files

[Felix Schumacher]

Am 22.06.20 um 11:41 schrieb Ayub Khan:
> Chris,
>
> I am using HikariCP for connection pooling. If the database is leaking
> connections then I should see connection not available exception.
>
> How do I find out which file descriptors are leaking ?  these are not
files
> open on disk as there is no explicit disk file I/O in this application.
>
> I just use the below command to check for open file descriptors:
>

> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc -l"

You could have a look at the name of the files in the pids proc directory.

 $ ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/

Or you could use the tool lsof to find the open file descriptors.

 $ lsof -p $(cat /var/run/tomcat8.pid)

For both calls you should first change to the uid of the tomcat user or
use sudo as in your example.

Felix

>
> Thanks and Regards
> Ayub
>
> On Sun, Jun 21, 2020 at 8:18 PM Christopher Schultz <
> ch...@christopherschultz.net> wrote:
>
> Ayub,
>
> On 6/20/20 11:51, Ayub Khan wrote:
> >>> Sorry we are using  8.0.32 version of tomcat.
> >>>
> >>> below is the configuration:
> >>>
> >>> Server version: Apache Tomcat/8.0.32 (Ubuntu) Server built:   Jan
> >>> 24 2020 16:24:30 UTC Server number:  8.0.32.0 OS Name:
> >>> Linux OS Version: 4.4.0-1087-aws Architecture:   amd64 JVM
> >>> Version:    1.8.0_181-b13 JVM Vendor: Oracle Corporation
> >>>
> >>> I use the below command to check the file descriptors:
> >>>
> >>> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc -l"
>
> So you know there is some kind of increase in file-handle use, but you
> don't know what types of file handles are increasing, right?
>
> Can you try to find out which kinds of file handles are increasing?
>
> I have a sneaking suspicion that it's your database connections and
> not actually files open on the disk.
>
> Are you using a database connection pool? If not, you should really
> use one and limit the number of connections to something sane. If you
> are using one, are you monitoring it to see how many connections are
> actually being used? Are you sure you are using proper resource
> management[1]? Even a single code-path that leaks connections can leak
> them quickly under load.
>
> >>> When there an issue related to broken files, this value keeps
> >>> increasing, the only way to bring it down is to remove vm instance
> >>> from AWS load balancer.> Which version of tomcat should I install
> >>> ?
>
> Tomcat 8.0.x hasn't been supported since its last release on 29 June
> 2018. That was 8.0.53. Your release is from 8 February 2016 and is
> dangerously out of date (unless you are using the Ubuntu-packaged
> version, in which case I hope they kept-up with security patches thee
> past 4 years).
>
> -chris
>
> >>> On Sat, Jun 20, 2020 at 6:28 PM Christopher Schultz <
> >>> ch...@christopherschultz.net> wrote:
> >>>
> >>> Ayub,
> >>>
> >>> On 6/19/20 16:46, Ayub Khan wrote:
> >> tomcat 8.5 broken pipe increases open files on ubuntu AWS
> >>>
> >>> Which exact version of Tomcat 8.5? If you aren't running the
> >>> latest version (8.5.56), please upgrade and re-test.
> >>>
> >> If there is slow response from db I see this stack trace and
> >> the open files goes high and the only way to open files go
> >> down is to remove the instance from Amazon load balancer.
> >>
> >> Is there a way to keep the open files low even when Broken
> >> pipe error is thrown ?
> >>>
> >>> What is your evidence that file handles are being left open?
> >>>
> >>> Which file handles are being left open?
> >>>
> >>> -chris
> 
>  -
> 
> 
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>  For additional commands, e-mail: users-h...@tomcat.apache.org
> 
> 
> >>>
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>>
>


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: broken pipe error keeps increasing open files

[Ayub Khan]

Felix,

I executed ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/ and  from the output
I see majority of them are related to sockets as shown below, some of them
point to the jar file of tomcat and others to the log file which is created.

 socket:[2084570754]
 socket:[2084579487]
 socket:[2084578478]
socket:[2084570167]

On Mon, Jun 22, 2020 at 1:28 PM Felix Schumacher <
felix.schumac...@internetallee.de> wrote:

>
> Am 22.06.20 um 11:41 schrieb Ayub Khan:
> > Chris,
> >
> > I am using HikariCP for connection pooling. If the database is leaking
> > connections then I should see connection not available exception.
> >
> > How do I find out which file descriptors are leaking ?  these are not
> files
> > open on disk as there is no explicit disk file I/O in this application.
> >
> > I just use the below command to check for open file descriptors:
> >
>
> > watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc -l"
>
> You could have a look at the name of the files in the pids proc directory.
>
>  $ ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/
>
> Or you could use the tool lsof to find the open file descriptors.
>
>  $ lsof -p $(cat /var/run/tomcat8.pid)
>
> For both calls you should first change to the uid of the tomcat user or
> use sudo as in your example.
>
> Felix
>
> >
> > Thanks and Regards
> > Ayub
> >
> > On Sun, Jun 21, 2020 at 8:18 PM Christopher Schultz <
> > ch...@christopherschultz.net> wrote:
> >
> > Ayub,
> >
> > On 6/20/20 11:51, Ayub Khan wrote:
> > >>> Sorry we are using  8.0.32 version of tomcat.
> > >>>
> > >>> below is the configuration:
> > >>>
> > >>> Server version: Apache Tomcat/8.0.32 (Ubuntu) Server built:   Jan
> > >>> 24 2020 16:24:30 UTC Server number:  8.0.32.0 OS Name:
> > >>> Linux OS Version: 4.4.0-1087-aws Architecture:   amd64 JVM
> > >>> Version:1.8.0_181-b13 JVM Vendor: Oracle Corporation
> > >>>
> > >>> I use the below command to check the file descriptors:
> > >>>
> > >>> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc -l"
> >
> > So you know there is some kind of increase in file-handle use, but you
> > don't know what types of file handles are increasing, right?
> >
> > Can you try to find out which kinds of file handles are increasing?
> >
> > I have a sneaking suspicion that it's your database connections and
> > not actually files open on the disk.
> >
> > Are you using a database connection pool? If not, you should really
> > use one and limit the number of connections to something sane. If you
> > are using one, are you monitoring it to see how many connections are
> > actually being used? Are you sure you are using proper resource
> > management[1]? Even a single code-path that leaks connections can leak
> > them quickly under load.
> >
> > >>> When there an issue related to broken files, this value keeps
> > >>> increasing, the only way to bring it down is to remove vm instance
> > >>> from AWS load balancer.> Which version of tomcat should I install
> > >>> ?
> >
> > Tomcat 8.0.x hasn't been supported since its last release on 29 June
> > 2018. That was 8.0.53. Your release is from 8 February 2016 and is
> > dangerously out of date (unless you are using the Ubuntu-packaged
> > version, in which case I hope they kept-up with security patches thee
> > past 4 years).
> >
> > -chris
> >
> > >>> On Sat, Jun 20, 2020 at 6:28 PM Christopher Schultz <
> > >>> ch...@christopherschultz.net> wrote:
> > >>>
> > >>> Ayub,
> > >>>
> > >>> On 6/19/20 16:46, Ayub Khan wrote:
> > >> tomcat 8.5 broken pipe increases open files on ubuntu AWS
> > >>>
> > >>> Which exact version of Tomcat 8.5? If you aren't running the
> > >>> latest version (8.5.56), please upgrade and re-test.
> > >>>
> > >> If there is slow response from db I see this stack trace and
> > >> the open files goes high and the only way to open files go
> > >> down is to remove the instance from Amazon load balancer.
> > >>
> > >> Is there a way to keep the open files low even when Broken
> > >> pipe error is thrown ?
> > >>>
> > >>> What is your evidence that file handles are being left open?
> > >>>
> > >>> Which file handles are being left open?
> > >>>
> > >>> -chris
> > 
> > 
> -
> > 
> > 
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >  For additional commands, e-mail: users-h...@tomcat.apache.org
> > 
> > 
> > >>>
> >>
> >> -
> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >> For additional commands, e-mail: users-h...@tomcat.apache.org
> >>
> >>
> >
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

-- 

Sun Certified Enterprise Architect 

Re: File "catalina.out" not being created/populated when using Tomcat 9.0.31 + Ubuntu 20.04, and content goes to the Ubuntu syslog instead?

[calder]

On Mon, Jun 22, 2020, 01:04 Brian  wrote

[ snip ]

- For some reason, the people at Ubuntu/Debian/Linux decided that Tomcat's
> log should be found inside syslog, instead of staying independent inside
> "catalina.out". Why is that? I don't know and I don't like it!

[ snip ]
.
Sorry - don't have a specific answer for your Ubuntu implementation.
.
However, this is one reason we do not use "distro-specific" Tomcat
installations (to include implementations of WebSphere and WebLogic).
.
For example, we grab the plain vanilla Tomcat ZIP and extract it to "/opt/"
(as in "/opt/tomcat/") - we now have complete control over its
configuration and runtime instantiation.

File "catalina.out" not being created/populated when using Tomcat 9.0.31 + Ubuntu 20.04, and content goes to the Ubuntu syslog instead?

[Brian]


Hello,
 
I have been using Tomcat for about 18 years. As far as I can remember, 
everything that I leave on the log with System.out.println() has been found 
inside the file "catalina.out" which has been nice. However, this seems to have 
changed.
I just migrated to Ubuntu 20.04 + Tomcat 9.0.31, and installed Tomcat doing a 
"sudo apt install tomcat9". Now I see that the file "catalina.out" never gets 
created, and that everything that had been sent there is now being sent to the 
Ubuntu/Linux/Debian log "syslog". I don't like that, I want to keep having a 
separate/dedicated log for Tomcat (which is something I check very often) and 
not havoing to search for it inside syslog (which is something I rarely 
inspect).
 
I have done some research, and this is what I have discovered (if I understood 
everything correctly):
 
- Tomcat now runs as a service inside something called "systemsd"
- For some reason, the people at Ubuntu/Debian/Linux decided that Tomcat's log 
should be found inside syslog, instead of staying independent inside 
"catalina.out". Why is that? I don't know and I don't like it!
- The other files inside /var/log/tomcat get created, including the 
"catalina.-MM-DD.log" files. But I dont' find my "System.out.println()" 
messages there, which has always been the case.
- I have found, however, that syslog tries to create and populate the 
"catalina.out" file as well, since I have found an error inside syslog that 
says something like "rsyslogd: file '/var/log/tomcat9/catalina.out': open 
error: Permission denied".
- I have also found a file "/etc/rsyslog.d/tomcat9.conf" which, indeed, seams 
to indicate syslog that the file "/var/log/tomcat9/catalina.out" must be 
populated. Which makes sense and should solve my needs.
- Considering the "Permission denied" error message, I started playing with the 
permissions (something that I really don't have much experience with). If I 
remember correctly, I created the file "catalina.out" manually, and the 
modified its properties so the owners are "tomcat/adm", since the "syslog" 
process/user seems to be inside the "adm" group. Restarted everything... and 
the "catalina.out" file got created and populated!!! So it seems that the main 
obstacle here is a lack of correct permissions so syslog can do what has been 
told and populate the "catalina.out" file. Did I get it right?
- However, when I delete all the log files (which I do every once in a while), 
the permissions that I assigned get lost and the file doesn't get created 
anymore.
 
What are we supposed to do to deal with this problem? Can/should we do 
something so the Tomcat log doesn't go to syslog? If not, and considering that 
it seems that syslog is trying to populate the "catalina.out" file as it has 
been told, what should we do to correct the permissions problem?
 
Thanks in advance!
 
Brian

Re: broken pipe error keeps increasing open files

[Felix Schumacher]

Am 22.06.20 um 13:22 schrieb Ayub Khan:
> Felix,
>
> I executed ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/ and  from the output
> I see majority of them are related to sockets as shown below, some of them
> point to the jar file of tomcat and others to the log file which is created.
>
>  socket:[2084570754]
>  socket:[2084579487]
>  socket:[2084578478]
> socket:[2084570167]

Can you try the other command (lsof -p $(cat ...tomcat.pid))? It should
give a bit more details on the used sockets that the proc directory.

Felix

>
> On Mon, Jun 22, 2020 at 1:28 PM Felix Schumacher <
> felix.schumac...@internetallee.de> wrote:
>
>> Am 22.06.20 um 11:41 schrieb Ayub Khan:
>>> Chris,
>>>
>>> I am using HikariCP for connection pooling. If the database is leaking
>>> connections then I should see connection not available exception.
>>>
>>> How do I find out which file descriptors are leaking ?  these are not
>> files
>>> open on disk as there is no explicit disk file I/O in this application.
>>>
>>> I just use the below command to check for open file descriptors:
>>>
>>> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc -l"
>> You could have a look at the name of the files in the pids proc directory.
>>
>>  $ ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/
>>
>> Or you could use the tool lsof to find the open file descriptors.
>>
>>  $ lsof -p $(cat /var/run/tomcat8.pid)
>>
>> For both calls you should first change to the uid of the tomcat user or
>> use sudo as in your example.
>>
>> Felix
>>
>>> Thanks and Regards
>>> Ayub
>>>
>>> On Sun, Jun 21, 2020 at 8:18 PM Christopher Schultz <
>>> ch...@christopherschultz.net> wrote:
>>>
>>> Ayub,
>>>
>>> On 6/20/20 11:51, Ayub Khan wrote:
>> Sorry we are using  8.0.32 version of tomcat.
>>
>> below is the configuration:
>>
>> Server version: Apache Tomcat/8.0.32 (Ubuntu) Server built:   Jan
>> 24 2020 16:24:30 UTC Server number:  8.0.32.0 OS Name:
>> Linux OS Version: 4.4.0-1087-aws Architecture:   amd64 JVM
>> Version:1.8.0_181-b13 JVM Vendor: Oracle Corporation
>>
>> I use the below command to check the file descriptors:
>>
>> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc -l"
>>> So you know there is some kind of increase in file-handle use, but you
>>> don't know what types of file handles are increasing, right?
>>>
>>> Can you try to find out which kinds of file handles are increasing?
>>>
>>> I have a sneaking suspicion that it's your database connections and
>>> not actually files open on the disk.
>>>
>>> Are you using a database connection pool? If not, you should really
>>> use one and limit the number of connections to something sane. If you
>>> are using one, are you monitoring it to see how many connections are
>>> actually being used? Are you sure you are using proper resource
>>> management[1]? Even a single code-path that leaks connections can leak
>>> them quickly under load.
>>>
>> When there an issue related to broken files, this value keeps
>> increasing, the only way to bring it down is to remove vm instance
>> from AWS load balancer.> Which version of tomcat should I install
>> ?
>>> Tomcat 8.0.x hasn't been supported since its last release on 29 June
>>> 2018. That was 8.0.53. Your release is from 8 February 2016 and is
>>> dangerously out of date (unless you are using the Ubuntu-packaged
>>> version, in which case I hope they kept-up with security patches thee
>>> past 4 years).
>>>
>>> -chris
>>>
>> On Sat, Jun 20, 2020 at 6:28 PM Christopher Schultz <
>> ch...@christopherschultz.net> wrote:
>>
>> Ayub,
>>
>> On 6/19/20 16:46, Ayub Khan wrote:
> tomcat 8.5 broken pipe increases open files on ubuntu AWS
>> Which exact version of Tomcat 8.5? If you aren't running the
>> latest version (8.5.56), please upgrade and re-test.
>>
> If there is slow response from db I see this stack trace and
> the open files goes high and the only way to open files go
> down is to remove the instance from Amazon load balancer.
>
> Is there a way to keep the open files low even when Broken
> pipe error is thrown ?
>> What is your evidence that file handles are being left open?
>>
>> Which file handles are being left open?
>>
>> -chris
>>>
>> -
>>>
>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>>
>>>
 -
 To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: users-h...@tomcat.apache.org


>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: 

Re: File "catalina.out" not being created/populated when using Tomcat 9.0.31 + Ubuntu 20.04, and content goes to the Ubuntu syslog instead?

[Chris Cheshire]

> On Jun 22, 2020, at 4:44 AM, calder  wrote:
> 
> On Mon, Jun 22, 2020, 01:04 Brian  wrote
> 
> [ snip ]
> 
> - For some reason, the people at Ubuntu/Debian/Linux decided that Tomcat's
>> log should be found inside syslog, instead of staying independent inside
>> "catalina.out". Why is that? I don't know and I don't like it!
> 
> [ snip ]
> .
> Sorry - don't have a specific answer for your Ubuntu implementation.
> .
> However, this is one reason we do not use "distro-specific" Tomcat
> installations (to include implementations of WebSphere and WebLogic).
> .
> For example, we grab the plain vanilla Tomcat ZIP and extract it to "/opt/"
> (as in "/opt/tomcat/") - we now have complete control over its
> configuration and runtime instantiation.

+1

Also it is far better to use a logging framework than writing directly to 
System.out as you have more control over when and where the logs are written. 
You can use java.util.logging (juli) directly or even a facade like SLF4j and 
then use whatever implementation you like behind it. The advantage of the 
facade is that the implementation (logback, log4j, juli) can be changed with 
configuration and a new jar instead of code changes everywhere you write to the 
logs.

With either method you can write logs at different levels (trace, debug, info, 
warn error - juli does use different names though, same idea) and turn them 
on/off in configuration and even dynamically.

Chris
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: users Digest 22 Jun 2020 10:06:54 -0000 Issue 13885

[Richard Huntrods]

Brian & Calder,

On 6/22/2020 3:06 AM, 
users-digest-h...@tomcat.apache.org 
wrote:

On Mon, Jun 22, 2020, 01:04 Brian 
 wrote

[ snip ]

- For some reason, the people at Ubuntu/Debian/Linux decided that Tomcat's


log should be found inside syslog, instead of staying independent inside
"catalina.out". Why is that? I don't know and I don't like it!


[ snip ]
.
Sorry - don't have a specific answer for your Ubuntu implementation.
.
However, this is one reason we do not use "distro-specific" Tomcat
installations (to include implementations of WebSphere and WebLogic).
.
For example, we grab the plain vanilla Tomcat ZIP and extract it to "/opt/"
(as in "/opt/tomcat/") - we now have complete control over its
configuration and runtime instantiation.

I agree completely. I started with Tomcat in 2000 on Sun Sparc servers running 
Solaris 8. Over the years I gradually updated to Solaris 10 on Sparc then Intel 
servers. Finally a couple of years ago I switched to Ubuntu (currently 
18.04LTS) for simplicity in engaging the community.

Because my development server was a windows/intel machine (now perma-set to 
Win7) I needed a generic way to operate Tomcat on any OS.

As mentioned above, I always grabbed the zip and/or the tar.gz and simply 
extracted the file to an appropriate working directory. For Ubuntu, I use 
/opt/tomcat as well. This allows me to keep things under my control, or at 
least know where everything is and that it will remain relatively constant. Any 
variations or changes are well documented in the Tomcat changelogs and 
assistance can be rendered by the community.

Likewise I try and keep my Tomcat current and my Java versions consistent with 
the current Tomcat. I've found that although the upgrade process can sometimes 
be painful, it is at least well documented.

Cheers,

-Richard




[https://ipmcdn.avast.com/images/icons/icon-envelope-tick-round-orange-animated-no-repeat-v1.gif]
  Virus-free. 
www.avast.com
--
This communication is intended for the use of the recipient to whom it is 
addressed, and may contain confidential, personal, and or privileged 
information. Please contact us immediately if you are not the intended 
recipient of this communication, and do not copy, distribute, or take action 
relying on it. Any communications received in error, or subsequent reply, 
should be deleted or destroyed.
---

Re: File "catalina.out" not being created/populated when using Tomcat 9.0.31 + Ubuntu 20.04, and content goes to the Ubuntu syslog instead?

[Brian]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Brian,

On 6/22/20 02:04, Brian wrote:
>  Hello,
>
> I have been using Tomcat for about 18 years. As far as I can
> remember, everything that I leave on the log with
> System.out.println() has been found inside the file "catalina.out"
> which has been nice. However, this seems to have changed. I just
> migrated to Ubuntu 20.04 + Tomcat 9.0.31, and installed Tomcat
> doing a "sudo apt install tomcat9". Now I see that the file
> "catalina.out" never gets created, and that everything that had
> been sent there is now being sent to the Ubuntu/Linux/Debian log
> "syslog". I don't like that, I want to keep having a
> separate/dedicated log for Tomcat (which is something I check very
> often) and not havoing to search for it inside syslog (which is
> something I rarely inspect).
>
> I have done some research, and this is what I have discovered (if I
> understood everything correctly):
>
> - Tomcat now runs as a service inside something called "systemsd" -
> For some reason, the people at Ubuntu/Debian/Linux decided that
> Tomcat's log should be found inside syslog, instead of staying
> independent inside "catalina.out". Why is that? I don't know and I
> don't like it! - The other files inside /var/log/tomcat get
> created, including the "catalina.-MM-DD.log" files. But I dont'
> find my "System.out.println()" messages there, which has always
> been the case. - I have found, however, that syslog tries to create
> and populate the "catalina.out" file as well, since I have found an
> error inside syslog that says something like "rsyslogd: file
> '/var/log/tomcat9/catalina.out': open error: Permission denied". -
> I have also found a file "/etc/rsyslog.d/tomcat9.conf" which,
> indeed, seams to indicate syslog that the file
> "/var/log/tomcat9/catalina.out" must be populated. Which makes
> sense and should solve my needs. - Considering the "Permission
> denied" error message, I started playing with the permissions
> (something that I really don't have much experience with). If I
> remember correctly, I created the file "catalina.out" manually, and
> the modified its properties so the owners are "tomcat/adm", since
> the "syslog" process/user seems to be inside the "adm" group.
> Restarted everything... and the "catalina.out" file got created and
> populated!!! So it seems that the main obstacle here is a lack of
> correct permissions so syslog can do what has been told and
> populate the "catalina.out" file. Did I get it right? - However,
> when I delete all the log files (which I do every once in a while),
> the permissions that I assigned get lost and the file doesn't get
> created anymore.
>
> What are we supposed to do to deal with this problem? Can/should we
> do something so the Tomcat log doesn't go to syslog? If not, and
> considering that it seems that syslog is trying to populate the
> "catalina.out" file as it has been told, what should we do to
> correct the permissions problem?

What are the permissions of the /var/log/tomcat directory?

Hello Chris,

I did a "ls -l /var/log/". According to that, this is what I see for 
"/var/log/tomcat9":
drwxr-s---  3 tomcatadm4096 Jun 22 10:51 tomcat9

I'm not really an expert with Linux. It would seem to me that the "adm" group 
(to which syslog seems to belong) lacks a write permission, but in my old 
instance (Tomcat 8.5.39 + Ubuntu 18.04) the "catalina.out" file works perfectly 
(gets created and populated) and this is what I see there, it doesn't seem to 
me that the adm group has a write permission there either:
drwxr-x---  3 tomcat8   adm   4096 Jun 22 10:00 tomcat8

Perhaps you need to chmod g+w /var/log/tomcat ?
- -chris

OK, I did it and the write permission was added to the adm group. I restarted 
Tomcat and it worked, the catalina.out file got created! However, after I 
restarted the whole Ubuntu, I discovered that the permissions went back to how 
there were (not write for adm). Why is that?
In fact, now I remember that I tried this before, but since I restarted the 
whole Ubuntu instead of just Tomcat, I never saw any progress. I didn't notice 
that it would have worked if I just restarted Tomcat.

In any case, why is this permission required in my new VPS, if the old one 
lacks it and catalina.out works perfectly? 

Re: Strange crash-on-takeoff, Tomcat 7.0.104

[James H. H. Lampert]

On 6/20/20 8:41 AM, Mark Thomas wrote:


7.0.105 hasn't been released yet. You can use catalina.sh from 7.0
103 or the latest version from source control.


Where would I find "the latest version from source control"?

--
JHHL

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Issue found during migration of Tomcat version 6.0.35 to 8.5.5

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Lavitesh,

On 6/22/20 07:12, Lavitesh Verma wrote:
> We didn’t use woodstox previously in our application and after we
> updated tomcat to version 8.5.5 we are getting this issue.
>
> We were able to resolve the issue by adding “wstx-asl-2.9.1.jar” to
> the application, could you please confirm if this is the
> recommended fix for this issue.

Tomcat doesn't use any particular XML API. It just uses JAXP which
will use whatever XML parser your environment has configured. It's
possible that your application (or a library you are using) or
configuration is specifying the use of this particular XML parser
which has some kind of missing dependency.

But this really has nothing to do with Tomcat.

Your stack trace shows that Tomcat is calling your code which,
ultimately, is using the Woodstox XML parser. It's up to you to decide
how best to fix your application.

- -chris
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl7w0KsACgkQHPApP6U8
pFiXyw/+IBqW3IFBee+JjHU4nSLJuBHZOdWMrJjpbRtJgieLYkkOWToWhqlQ++HT
C9Tq5duINP0aAVZlXnE+2rzFXJrC4pESQSZOgbx1fYQ+6J8G4BABANtYbCpKFIbn
RolwfhsETOTdI68ROr7PgUBerVhMjEF8Gfh+i0J1f4/3pLQ4+bFbu1DVCseViaSP
JmcEmdWp/ODVAHq5fCD5I//zYfVKxjjsLHJnusIqUtMR9wTLOMb1m7t5VOUR27iJ
RyLfeq0dhm+61n9Z2VE3hqP3rKKOqIUuOG7wbOmZrAhRqnbWncvCR0xE+xEr9olg
VyBnWHAOcPZwBi533D8JbbV+IF9gkcWGe0x4Fm0rB3hvX6OMJwykDmqhXP1k8yha
1nBe8uadfB5A1o4H/e8VVwsRJx/WYdgx27V3IIbaZSwzT9x0bY8XOMzeBcH1pVMY
+qL5TTFzbVLDAsZVO1ggH2O61ehjpzz+siwO3K57mmncFFmmentdWEmhHtJpuqo6
gOk5zw6TWz6HeO9xZLsoQ3JqBHmj+jUyJ89nbJJaM271Sp/t2JgiGt1OvUvob69t
8i54NxXf1lszfqmLRr1qSHY9YBABLzP0vEhvVjNM8Nf180Ng9OGkMyO+htScl9ni
9GJMSH82PFlIO73wSpKKkNcn4cfJ8J1NI0ELs3+re3nHPsSgHfM=
=mfTw
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: broken pipe error keeps increasing open files

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256


Ayub,

On 6/22/20 05:41, Ayub Khan wrote:
> I am using HikariCP for connection pooling. If the database is
> leaking connections then I should see connection not available
> exception.

That probably depends upon your connection pool configuration. It's
definitely possible to configure a connection pool to leak connections
without complaint.

> How do I find out which file descriptors are leaking ?  these are
> not files open on disk as there is no explicit disk file I/O in
> this application.
You might have to use a profiler.

> I just use the below command to check for open file descriptors:
>
> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc -l"

Felix's suggestions for how to check the types of file handles are
good. You should be able to get host:port information from the sockets
you are seeing and, I suspect, you'll find that they all point to your
database.

- -chris

> On Sun, Jun 21, 2020 at 8:18 PM Christopher Schultz <
> ch...@christopherschultz.net> wrote:
>
> Ayub,
>
> On 6/20/20 11:51, Ayub Khan wrote:
 Sorry we are using  8.0.32 version of tomcat.

 below is the configuration:

 Server version: Apache Tomcat/8.0.32 (Ubuntu) Server built:
 Jan 24 2020 16:24:30 UTC Server number:  8.0.32.0 OS Name:
 Linux OS Version: 4.4.0-1087-aws Architecture:   amd64
 JVM Version:1.8.0_181-b13 JVM Vendor: Oracle
 Corporation

 I use the below command to check the file descriptors:

 watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc -l"
>
> So you know there is some kind of increase in file-handle use, but
> you don't know what types of file handles are increasing, right?
>
> Can you try to find out which kinds of file handles are
> increasing?
>
> I have a sneaking suspicion that it's your database connections
> and not actually files open on the disk.
>
> Are you using a database connection pool? If not, you should
> really use one and limit the number of connections to something
> sane. If you are using one, are you monitoring it to see how many
> connections are actually being used? Are you sure you are using
> proper resource management[1]? Even a single code-path that leaks
> connections can leak them quickly under load.
>
 When there an issue related to broken files, this value
 keeps increasing, the only way to bring it down is to remove
 vm instance from AWS load balancer.> Which version of tomcat
 should I install ?
>
> Tomcat 8.0.x hasn't been supported since its last release on 29
> June 2018. That was 8.0.53. Your release is from 8 February 2016
> and is dangerously out of date (unless you are using the
> Ubuntu-packaged version, in which case I hope they kept-up with
> security patches thee past 4 years).
>
> -chris
>
 On Sat, Jun 20, 2020 at 6:28 PM Christopher Schultz <
 ch...@christopherschultz.net> wrote:

 Ayub,

 On 6/19/20 16:46, Ayub Khan wrote:
>>> tomcat 8.5 broken pipe increases open files on ubuntu
>>> AWS

 Which exact version of Tomcat 8.5? If you aren't running the
 latest version (8.5.56), please upgrade and re-test.

>>> If there is slow response from db I see this stack
>>> trace and the open files goes high and the only way to
>>> open files go down is to remove the instance from
>>> Amazon load balancer.
>>>
>>> Is there a way to keep the open files low even when
>>> Broken pipe error is thrown ?

 What is your evidence that file handles are being left open?

 Which file handles are being left open?

 -chris
>
> --
- ---
>
>
>
>
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail:
> users-h...@tomcat.apache.org
>
>

>>
>> -
>>
>>
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>>
>
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl7w0U0ACgkQHPApP6U8
pFgnCw/9GxKtFIdNS71zYPAYgEY0O6bh4+SyvD7cWsNRwAEtN4jwwmvmbk+9ldGa
5u8jalMeWjBaYtVev+T7XQ0EOJN9ZuunyXHzKR8LeIqPCzka/34inQKBhlwOFyR+
lQUhod8VsDrQJ4lb2ZxC1yS/mnUFdKeV2u2Px9mIr5WdkfVAxYWCOuKTW9DrW3HE
nbPG5t3oiLS8Rx+xXssyaTii5DAaQm0X0FYkkPp1GQ8Tj0/fylsE17nV8LrxLTb5
qhv0l6avR2/IsXaD6DC8MjdUO6kz5hkWNLtO/WcKrIFI+xPcYs2+Hog7Hrajetwp
QR8f7R7ry0W0HLy3p3R10jMwDQzOWjn2RNxQqL2qfNGmOqMbTi2XskQhtkKPoM9Y
LqZJsG0Ax2kNr6N4YeYpotRVcGqCJDgLwQ3HMjkHW/Y1M4O2NicphW9iFd5L+X9Z
fh4KFz5XbAUM4py7IuMtj1SH/FrOMyfBtKl6ECJ98VbtoBPU0I6Kmc0lgbf3h9wD
yphZ+3pW6a9Oeroq7aRTd35XdFqLdi9eg4Ze19/KpAr0hw3pgPvQqZiqq5qhpOrv
HhNWfHhvzeXmmab0DeLB9On+FDp30svWv9hmAR1ZB8ViBeCDoFjaUxv1UltqS6NQ
m9UsvtNR3035EpiSw9p5Jh5+eiG6swSnJXQH4yZxkVNAsHthyBA=
=KsIE
-END 

Re: File "catalina.out" not being created/populated when using Tomcat 9.0.31 + Ubuntu 20.04, and content goes to the Ubuntu syslog instead?

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Brian,

On 6/22/20 02:04, Brian wrote:
>  Hello,
>
> I have been using Tomcat for about 18 years. As far as I can
> remember, everything that I leave on the log with
> System.out.println() has been found inside the file "catalina.out"
> which has been nice. However, this seems to have changed. I just
> migrated to Ubuntu 20.04 + Tomcat 9.0.31, and installed Tomcat
> doing a "sudo apt install tomcat9". Now I see that the file
> "catalina.out" never gets created, and that everything that had
> been sent there is now being sent to the Ubuntu/Linux/Debian log
> "syslog". I don't like that, I want to keep having a
> separate/dedicated log for Tomcat (which is something I check very
> often) and not havoing to search for it inside syslog (which is
> something I rarely inspect).
>
> I have done some research, and this is what I have discovered (if I
> understood everything correctly):
>
> - Tomcat now runs as a service inside something called "systemsd" -
> For some reason, the people at Ubuntu/Debian/Linux decided that
> Tomcat's log should be found inside syslog, instead of staying
> independent inside "catalina.out". Why is that? I don't know and I
> don't like it! - The other files inside /var/log/tomcat get
> created, including the "catalina.-MM-DD.log" files. But I dont'
> find my "System.out.println()" messages there, which has always
> been the case. - I have found, however, that syslog tries to create
> and populate the "catalina.out" file as well, since I have found an
> error inside syslog that says something like "rsyslogd: file
> '/var/log/tomcat9/catalina.out': open error: Permission denied". -
> I have also found a file "/etc/rsyslog.d/tomcat9.conf" which,
> indeed, seams to indicate syslog that the file
> "/var/log/tomcat9/catalina.out" must be populated. Which makes
> sense and should solve my needs. - Considering the "Permission
> denied" error message, I started playing with the permissions
> (something that I really don't have much experience with). If I
> remember correctly, I created the file "catalina.out" manually, and
> the modified its properties so the owners are "tomcat/adm", since
> the "syslog" process/user seems to be inside the "adm" group.
> Restarted everything... and the "catalina.out" file got created and
> populated!!! So it seems that the main obstacle here is a lack of
> correct permissions so syslog can do what has been told and
> populate the "catalina.out" file. Did I get it right? - However,
> when I delete all the log files (which I do every once in a while),
> the permissions that I assigned get lost and the file doesn't get
> created anymore.
>
> What are we supposed to do to deal with this problem? Can/should we
> do something so the Tomcat log doesn't go to syslog? If not, and
> considering that it seems that syslog is trying to populate the
> "catalina.out" file as it has been told, what should we do to
> correct the permissions problem?

What are the permissions of the /var/log/tomcat directory?

Perhaps you need to chmod g+w /var/log/tomcat ?

- -chris
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl7w0hUACgkQHPApP6U8
pFgABA/7BwxNOI313uSbRMSFPpdilpxhTkmujrvcit+A8lpre0hSNOUw2Q6i38L6
uT536ff2TwrFoizfEz/ydEW8lZQoEOWejlXPU8eO3/ehi7Gpat645+a559rCjr3m
S5vfKw1UqrjKQXkeSlFGH1SFaKJfEnSKyjeH+/5yWVTL0z4G6Zatt9iZyUdHbPKh
cuTrSpgfuOlsPdqKvO/7Y7gWoLqb52+17gawN6rrviRHs6U1S9EIgTWk0of1JYe/
K2l2ob9DfuVJCW+byL5FPE0AEwzyWMfh1W1yEMRGJKsUHFv+5BaXXRyc/yh9/E/l
vqf+7OJPJ2V5Kp9KasX6NxBiyCGxH910lfyooeEIRAYmiWx61FD6OTcrklm5boEN
fT7KyeuujIhnQSIq+c+0rKX+t2DtyQCuqP/e5SFiUt4WL4xH85e5FaoExk2ZlOoP
E5phuijWPJakSTiXBG4cKRQyFqzLVqHSl6VbCHEzbj6rDHaiqOW0Wqs+Z2OV5q1k
c7vMiHbawiqlo+5NzB/kVG+HISMQvK0lmuLsZOUaGFv2E5L85UuQhp2KLD01j0AF
iEizODrxbs3AVs3+jx/LM6mXhqb6oCq93DLbWwuV0mhOTX6Dj1VxqkSUUiyyElD5
bCFl5dfyc0gc+daEAj32gshL9MBDPCkkyvBKO/N3Wdy2YnLP4GM=
=ltDw
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org