Performance issue 8.5.20 (metaspace related?)

[Ing. Andrea Vettori]

Hello,
W're running several instances of Tomcat 8.5.20 / JDK8.144 / CentOs7 on our 
company for various web sites in many hosts. Recently I’m trying to understand 
a performance problem we’re having on our e-commerce web site.
The configuration is the following

HAProxy   <—> 2x Tomcat 8.5.20 <—>  JBoss 5.1 EJB <—> Postgres 9.6

Tomcat runs a web site built with Struts / Freemarker that does call JBoss EJBs 
with RMI.
Monitoring a specific task (putting a product on the cart) I see the following :

- with a fresh started tomcat instance, the time it takes is around 0,8 
seconds. Most of the time is spent on the two RMI calls the task does.
- with an instance that is running from some time, the time can reach 2/3 
seconds; occasionally 5/6 seconds. Most time is still spent on RMI calls. I.e. 
what slows down are the RMI calls.
- restarting the jvm fixes the issue
- ***it seems*** but I’m still testing this since it seems there’s no 
‘meatspace gc trigger command available', that when Metaspace is garbage 
collected, tomcat then performs like a fresh instance.

Since we’re using more than one tomcat instance (2 in production for this 
website, 1 for development) I can see that the issue is isolated to Tomcat or 
the JVM/Host where it runs because other Tomcat instances behave well at the 
same time one is slow. The same JBoss/Postgres backend is used by java batches 
and fat clients and it does work well with consistent times.

To clarify: the moment one production tomcat that is running from some time 
finishes the task in 3 seconds, the development tomcat or a fresh started 
production tomcat instance does the same task in less that one second. Note 
that repeating the task gives always consistent results, i.e. the instance is 
running from some time is always slow,  the fresh running instance is always 
fast.

Tomcat is running with these VM options: 

-Xms20480m -Xmx20480m -Dsun.rmi.dgc.client.gcInterval=360 
-Dsun.rmi.dgc.server.gcInterval=360 -Xloggc:/tmp/gc.txt 
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9012 
-Dcom.sun.management.jmxremote.ssl=false 
-Dcom.sun.management.jmxremote.authenticate=false -XX:+PrintGCDetails 
-XX:+PrintGCTimeStamps -XX:+UseG1GC -XX:ReservedCodeCacheSize=1g 
-XX:InitialCodeCacheSize=256m -XX:+UseHugeTLBFS -XX:MetaspaceSize=1g 
-XX:MaxMetaspaceSize=2g

Some of the options have been recently added (for example the increase in code 
cache  size) but it seems they had no impact on the issue.

Metaspace goes up to 1,6GB before being collected. Value after garbage collect 
is around 200MB. Heap usage is variable, it usually stays under 10G and is 
around 1G after garbage collect.
CPU usage rarely goes over 10%. Loaded classes between 20k and 40k. Active 
sessions around 100/120 for each instance.

Any help or direction to understand what’s causing this is greatly appreciated.
Thank you
—
Ing. Andrea Vettori

TomcatCon London 2017

[Mark Thomas]

All,

TomcatCon London 2017 is now 4 weeks away.

Tickets sales have been stronger than expected so if you haven't yet
booked your tickets, now would be a great time to do so to avoid
disappointment:
https://www.eventbrite.com/e/tomcatcon-london-2017-tickets-36683639754

TomcatCon London will be held on Tuesday 26th September in
central London and features talks from:
- Mark Thomas (markt)
- Rémy Maucherat (remm)
- Jean-Frederic Clere (jfclere)

Full details, including the schedule is available on the website:
http://tomcat.apache.org/conference.html

We hope to see you there!

Mark
on behalf of the Apache Tomcat PMC


This event is sponsored by Liferay, the robust java-based open source
platform (5 million downloads) used to build sophisticated transactional
websites and deliver them to web browsers and native mobile clients.
Liferay, Inc provides enterprise support for the commercial release
which is used by organisations such as Allianz, Volkswagen Group and the
NHS.

https://web.liferay.com/community/liferay-projects/liferay-portal/overview


Thanks to c2b2 for providing sponsorship for speaker travel and
associated expenses.

https://www.c2b2.co.uk/


Thanks to Red Hat and Pivotal for donating the speakers' time.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Performance issue 8.5.20 (metaspace related?)

[Suvendu Sekhar Mondal]

Hello,

On Tue, Aug 29, 2017 at 2:54 PM, Ing. Andrea Vettori
 wrote:
> Hello,
> W're running several instances of Tomcat 8.5.20 / JDK8.144 / CentOs7 on our 
> company for various web sites in many hosts. Recently I’m trying to 
> understand a performance problem we’re having on our e-commerce web site.
> The configuration is the following
>
> HAProxy   <—> 2x Tomcat 8.5.20 <—>  JBoss 5.1 EJB <—> Postgres 9.6
>
> Tomcat runs a web site built with Struts / Freemarker that does call JBoss 
> EJBs with RMI.
> Monitoring a specific task (putting a product on the cart) I see the 
> following :
>
> - with a fresh started tomcat instance, the time it takes is around 0,8 
> seconds. Most of the time is spent on the two RMI calls the task does.
> - with an instance that is running from some time, the time can reach 2/3 
> seconds; occasionally 5/6 seconds. Most time is still spent on RMI calls. 
> I.e. what slows down are the RMI calls.

Sill question, do you mean RMI calls generating from Tomcat is getting
slower with time? or, JBoss is taking time to return the response?

> - restarting the jvm fixes the issue
> - ***it seems*** but I’m still testing this since it seems there’s no 
> ‘meatspace gc trigger command available', that when Metaspace is garbage 
> collected, tomcat then performs like a fresh instance.
>

Yes, no command to only garbage collect Metaspace area. Without
recycling Tomcat, you can trigger a Full GC either using VisualVM or
jcmd's GC.run command.

> Since we’re using more than one tomcat instance (2 in production for this 
> website, 1 for development) I can see that the issue is isolated to Tomcat or 
> the JVM/Host where it runs because other Tomcat instances behave well at the 
> same time one is slow. The same JBoss/Postgres backend is used by java 
> batches and fat clients and it does work well with consistent times.
>
> To clarify: the moment one production tomcat that is running from some time 
> finishes the task in 3 seconds, the development tomcat or a fresh started 
> production tomcat instance does the same task in less that one second. Note 
> that repeating the task gives always consistent results, i.e. the instance is 
> running from some time is always slow,  the fresh running instance is always 
> fast.
>
> Tomcat is running with these VM options:
>
> -Xms20480m -Xmx20480m -Dsun.rmi.dgc.client.gcInterval=360 
> -Dsun.rmi.dgc.server.gcInterval=360 -Xloggc:/tmp/gc.txt 
> -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9012 
> -Dcom.sun.management.jmxremote.ssl=false 
> -Dcom.sun.management.jmxremote.authenticate=false -XX:+PrintGCDetails 
> -XX:+PrintGCTimeStamps -XX:+UseG1GC -XX:ReservedCodeCacheSize=1g 
> -XX:InitialCodeCacheSize=256m -XX:+UseHugeTLBFS -XX:MetaspaceSize=1g 
> -XX:MaxMetaspaceSize=2g
>
> Some of the options have been recently added (for example the increase in 
> code cache  size) but it seems they had no impact on the issue.
>

Is there any specific reason to trigger Full GC every
60min(-Dsun.rmi.dgc.client.gcInterval=360)? I think that's
overkill.


> Metaspace goes up to 1,6GB before being collected. Value after garbage 
> collect is around 200MB. Heap usage is variable, it usually stays under 10G 
> and is around 1G after garbage collect.
> CPU usage rarely goes over 10%. Loaded classes between 20k and 40k. Active 
> sessions around 100/120 for each instance.
>

That sounds like a healthy behavior to me. That means GC is occurring
and Metaspace is getting cleaned up. Why do you think Metaspace is the
problem?
>From the given data, my first impression is JVM is not having any
memory pressure. If you can post your gc.txt file to this group, I
would be happy to check it.

Another thing, is there any way you can measure the processing time in
JBoss for each RMI calls?

> Any help or direction to understand what’s causing this is greatly 
> appreciated.
> Thank you
> —
> Ing. Andrea Vettori
>

Thanks!
Suvendu

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 8.5.16 cuts end of body after 10485761 bytes

[Felix Schumacher]

Am 29.08.2017 um 04:44 schrieb Simon De Uvarow:

Hi, I have to respond a big json file:

return Response.status(Status.OK).entity(new
SimpleDataMessageResponse("TASK_FINISH",
str)).build();

The tomcat logs correctly the size in the access log file:

127.0.0.1 - - [29/Aug/2017:02:26:07 +] "GET /./interactiveTask/
1076dde0-b199-4043-9047-e897050eb7fa HTTP/1.1" 200 *12815716 *748 .

*But the browser receives only 10485761 bytes (10 MB).*

I tested with Chrome, Firefox and finally with JMeter, and it's the same in
all cases:
The following is the result of the JMeter:

Size in bytes: *12815830*
Sent bytes:564
Headers size in bytes: 114
Body size in bytes: *12815716<- but if I run the JMeter script , copy
the response and check the size, it's **10485761 bytes, not 12815716 bytes*
Data type ("text"|"bin"|""): text
Response code: 200
Response message:

Response headers:
HTTP/1.1 200
Content-Type: application/json
Transfer-Encoding: chunked
Date: Tue, 29 Aug 2017 02:26:07 GMT

GET http://... /interactiveTask/1076dde0-b199-4043-9047-e897050eb7fa

Request Headers:
Connection: keep-alive
Referer: -
Accept-Language: es-ES,es;q=0.8,en;q=0.6,gl;q=0.4,en-US;q=0.2
Accept-Encoding: gzip, deflate, br
Accept: application/json, text/plain, */*
User-Agent: 
Host: localhost:8080



Any idea of what could be happening ?
Is there any max size to configure and fix this?


Well, JMeter has a default max size for storing body contents. It is 10 
MB and can be configured with a property in bin/jmeter.properties:



# Max size of bytes stored in memory per SampleResult
# Ensure you don't exceed max capacity of a Java Array and remember.
# that the higher it is, the higher JMeter will consume heap
# Defaults to 10MB
#httpsampler.max_bytes_to_store_per_request=10485760

Do you have any other signs for the truncation apart from JMeter? How 
did you measure the size in the browsers?


Regards,
 Felix



thanks !


"No olvides, no traiciones, lo que llevas bien dentro de ti. No olvides, no
traiciones, lo que siempre te ha hecho vivir."




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: applicatio shared path is not working on tomcat8 version

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Naga,

On 8/28/17 12:38 PM, Naga Ramesh wrote:
> Chris, We don’t have the context file on app/WEB-INF path

You need to create a new one.

> , that’s why we have done on /tomcat/conf/context.xml file.

You have modified the defaults for the whole server. If you deploy
another web application, it will also have the JarScanner disabled,
even if you need JAR scanning enabled. Please fix your global defaults
file and configure your application correctly.

> After that issue is resolved, but how do I resolve the issue, 
> please guide me
Which issue? You said that disabling JAR scanning has fixed your
original report (which doesn't actually make any sense to me). so what
problem are you now facing?

- -chris

> -Original Message- From: Christopher Schultz 
> [mailto:ch...@christopherschultz.net] Sent: Monday, August 28,
> 2017 9:09 PM To: users@tomcat.apache.org Subject: Re: applicatio
> shared path is not working on tomcat8 version
> 
> Naga,
> 
> On 8/28/17 11:23 AM, Naga Ramesh wrote:
>> Thanks Chris.. issues is resolved..
> 
>> We have enabled the below thing on context file and it's working 
>> fine.
> 
> 
>> tomcat8/conf/context.xml:  added the below line and restarted
>> the service.
> 
>> > scanAllDirectories="false"/>
> 
> STOP. Undo what you have just done.
> 
> You only want to disable the JAR scanner for your application.
> What you have done is disable the JAR scanner for the entire
> server.
> 
> The file you want is $yourapplication/WEB-INF/context.xml not 
> $CATALINA_BASE/conf/context.xml
> 
> -chris
> 
>> -Original Message- From: Christopher Schultz 
>> [mailto:ch...@christopherschultz.net] Sent: Monday, August 28, 
>> 2017 8:11 PM To: users@tomcat.apache.org Subject: Re: applicatio 
>> shared path is not working on tomcat8 version
> 
>> Naga,
> 
>> On 8/27/17 12:24 PM, Naga Ramesh wrote:
>>> Christopher,
> 
>>> We have done the load balancer configuration and one of the 
>>> properties file need to give the shared path and that shared 
>>> path need to give the tomcat end for some functionality 
>>> purpose.
> 
>>> So we have set the class path on tomcat7 and it's working fine 
>>> but same thing we have tried on tomcat8 version, here throwing 
>>> the below mentioned errors.
> 
>>> Please suggest any alternative solution for this.
> 
>> Please post your configuration showing how you are setting the 
>> CLASSPATH .
> 
>> -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJZpVc9AAoJEBzwKT+lPKRY3IcP/jXCNSf0CJ/WXyyBR9ErO742
Jp9N2JN2hEi3JzZU3PttQwpmosYQpxlyIpMn56Hii1keb18YYuGRkEp9dxO73bhd
ikv6DLgj9hAE57Q8IT1y06uiLE6QvJ4QrQNcsMuZSWSS+0jBOM/+mhPt0zila/3+
VGCU3WbCDAVRNVuKyr/JEqkw4YEtFLniyanrFKroTjnCEvleifCEamcv9Xscq4IC
4jGzfFKgiLoyfijOTGgP+Kxgtrih6KOjrW3qfeH+WWGPZAdAogSPphag6HBiiLjL
Jhegmuq2ACcQZTPpwxPakb48qUMwEZFy2TfqqmEYxMTtJqh8yqYDUCdGFRj9d67n
vHByFzuS0FsOo5vGPqW8tuQzW7SS8rZWwT68YnIgKNvJsc95+PZbtXx6j9g9uEh8
V1DR4DD0DL0MNyBTsPEcf3PSRnkdbuXDrAbCpbLEhnTRK9Lj2tkjeMC39rcTnuKk
gcbNWMBa9I65iJ6v9UDyRmpEcnt0FUyg74mN1MADHXCFwszjXJOOtq2UHkKX1qWE
0Vl43Og2qDZ+QNto8CSCYmU5AEJw6KG6BIlV1p+xqv5WBTKAZUut5R8vy7zAInIN
QWJErq4YXuXkYZqBoyU8n66sExH6NlKnvbU+SnyThXF4S8TPpHtEcI4AUkfwmSyJ
wwquS99TaJStG5hFHEcp
=jtZZ
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Performance issue 8.5.20 (metaspace related?)

[Ing. Andrea Vettori]

> On 29 Aug 2017, at 12:29, Suvendu Sekhar Mondal  wrote:
> 
> On Tue, Aug 29, 2017 at 2:54 PM, Ing. Andrea Vettori
>  wrote:
>> - with a fresh started tomcat instance, the time it takes is around 0,8 
>> seconds. Most of the time is spent on the two RMI calls the task does.
>> - with an instance that is running from some time, the time can reach 2/3 
>> seconds; occasionally 5/6 seconds. Most time is still spent on RMI calls. 
>> I.e. what slows down are the RMI calls.
> 
> Sill question, do you mean RMI calls generating from Tomcat is getting
> slower with time? or, JBoss is taking time to return the response?

Thanks for your help.
What I see is that the http requests are slower. Looking at the details of that 
specific request (putting a product in the cart) using System.nanotime I can 
see that most of the time is spent during the RMI calls. I’m pretty sure it’s 
not jboss that is slower because doing other calls at the same time with fresh 
tomcats or with a java client they're not slow.

> 
>> - restarting the jvm fixes the issue
>> - ***it seems*** but I’m still testing this since it seems there’s no 
>> ‘meatspace gc trigger command available', that when Metaspace is garbage 
>> collected, tomcat then performs like a fresh instance.
>> 
> 
> Yes, no command to only garbage collect Metaspace area. Without
> recycling Tomcat, you can trigger a Full GC either using VisualVM or
> jcmd's GC.run command.

I think that explicit GC is not doing a metaspace GC… that’s what I observed 
starting the GC with the java console.

> 
>> Since we’re using more than one tomcat instance (2 in production for this 
>> website, 1 for development) I can see that the issue is isolated to Tomcat 
>> or the JVM/Host where it runs because other Tomcat instances behave well at 
>> the same time one is slow. The same JBoss/Postgres backend is used by java 
>> batches and fat clients and it does work well with consistent times.
>> 
>> To clarify: the moment one production tomcat that is running from some time 
>> finishes the task in 3 seconds, the development tomcat or a fresh started 
>> production tomcat instance does the same task in less that one second. Note 
>> that repeating the task gives always consistent results, i.e. the instance 
>> is running from some time is always slow,  the fresh running instance is 
>> always fast.
>> 
>> Tomcat is running with these VM options:
>> 
>> -Xms20480m -Xmx20480m -Dsun.rmi.dgc.client.gcInterval=360 
>> -Dsun.rmi.dgc.server.gcInterval=360 -Xloggc:/tmp/gc.txt 
>> -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9012 
>> -Dcom.sun.management.jmxremote.ssl=false 
>> -Dcom.sun.management.jmxremote.authenticate=false -XX:+PrintGCDetails 
>> -XX:+PrintGCTimeStamps -XX:+UseG1GC -XX:ReservedCodeCacheSize=1g 
>> -XX:InitialCodeCacheSize=256m -XX:+UseHugeTLBFS -XX:MetaspaceSize=1g 
>> -XX:MaxMetaspaceSize=2g
>> 
>> Some of the options have been recently added (for example the increase in 
>> code cache  size) but it seems they had no impact on the issue.
>> 
> 
> Is there any specific reason to trigger Full GC every
> 60min(-Dsun.rmi.dgc.client.gcInterval=360)? I think that's
> overkill.

That’s a value I saw “everywhere” as suggested value (the default is too 
short). Note that on the GC logs I don’t see full GCs every hour so maybe that 
option is not triggering full gc ?


>> Metaspace goes up to 1,6GB before being collected. Value after garbage 
>> collect is around 200MB. Heap usage is variable, it usually stays under 10G 
>> and is around 1G after garbage collect.
>> CPU usage rarely goes over 10%. Loaded classes between 20k and 40k. Active 
>> sessions around 100/120 for each instance.
>> 
> 
> That sounds like a healthy behavior to me. That means GC is occurring
> and Metaspace is getting cleaned up. Why do you think Metaspace is the
> problem?

Because I can observe that when the metaspace is collected the requests become 
fast. I observer that a few hours ago, looking at the metaspace graph of the 
java console and doing requests just after the collect. 

> From the given data, my first impression is JVM is not having any
> memory pressure. If you can post your gc.txt file to this group, I
> would be happy to check it.

Thanks, here’s a link for gceasy report: 
http://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTcvMDgvMjkvLS1nY3N0YXRzLnR4dC5nei0tMTEtNDktMzQ=
 

and the txt log file: http://ge.tt/1Un0DLm2
Please consider that I triggered a Full GC manually two times on this server.


> 
> Another thing, is there any way you can measure the processing time in
> JBoss for each RMI calls?
> 

At the moment I’m not measuring the time on the jboss side but I’m considering 
using a telemetry product in the future. I’m measuring the time on the client 
side (tomcat or fat client).

Thank you!
—
Andrea Vettori

Re: Performance issue 8.5.20 (metaspace related?)

[Mark Thomas]

On 29/08/17 13:09, Ing. Andrea Vettori wrote:
>> On 29 Aug 2017, at 12:29, Suvendu Sekhar Mondal  wrote:
>>
>> On Tue, Aug 29, 2017 at 2:54 PM, Ing. Andrea Vettori
>>  wrote:
>>> - with a fresh started tomcat instance, the time it takes is around 0,8 
>>> seconds. Most of the time is spent on the two RMI calls the task does.
>>> - with an instance that is running from some time, the time can reach 2/3 
>>> seconds; occasionally 5/6 seconds. Most time is still spent on RMI calls. 
>>> I.e. what slows down are the RMI calls.
>>
>> Sill question, do you mean RMI calls generating from Tomcat is getting
>> slower with time? or, JBoss is taking time to return the response?
> 
> Thanks for your help.
> What I see is that the http requests are slower. Looking at the details of 
> that specific request (putting a product in the cart) using System.nanotime I 
> can see that most of the time is spent during the RMI calls. I’m pretty sure 
> it’s not jboss that is slower because doing other calls at the same time with 
> fresh tomcats or with a java client they're not slow.

I'd try profiling the problematic code. Given the magnitude of the
times, sampling should be sufficient which means you could do this on
production when the problem happens. You'll probably need to configure
it to log JVM internal calls.

I use YourKit (because they give me a free license to use to develop
Tomcat). Other profilers are available.

Mark


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 8.5.16 cuts end of body after 10485761 bytes

[Simon De Uvarow]

Hi,

"No olvides, no traiciones, lo que llevas bien dentro de ti. No olvides, no
traiciones, lo que siempre te ha hecho vivir."

On Tue, Aug 29, 2017 at 11:42 AM, Felix Schumacher <
felix.schumac...@internetallee.de> wrote:

> Am 29.08.2017 um 04:44 schrieb Simon De Uvarow:
>
>> Hi, I have to respond a big json file:
>>
>> return Response.status(Status.OK).entity(new
>> SimpleDataMessageResponse("TASK_FINISH",
>> str)).build();
>>
>> The tomcat logs correctly the size in the access log file:
>>
>> 127.0.0.1 - - [29/Aug/2017:02:26:07 +] "GET /./interactiveTask/
>> 1076dde0-b199-4043-9047-e897050eb7fa HTTP/1.1" 200 *12815716 *748 .
>>
>> *But the browser receives only 10485761 bytes (10 MB).*
>>
>> I tested with Chrome, Firefox and finally with JMeter, and it's the same
>> in
>> all cases:
>> The following is the result of the JMeter:
>>
>> Size in bytes: *12815830*
>> Sent bytes:564
>> Headers size in bytes: 114
>> Body size in bytes: *12815716<- but if I run the JMeter script , copy
>> the response and check the size, it's **10485761 bytes, not 12815716
>> bytes*
>> Data type ("text"|"bin"|""): text
>> Response code: 200
>> Response message:
>>
>> Response headers:
>> HTTP/1.1 200
>> Content-Type: application/json
>> Transfer-Encoding: chunked
>> Date: Tue, 29 Aug 2017 02:26:07 GMT
>>
>> GET http://... /interactiveTask/1076dde0-b199-4043-9047-e897050eb7fa
>>
>> Request Headers:
>> Connection: keep-alive
>> Referer: -
>> Accept-Language: es-ES,es;q=0.8,en;q=0.6,gl;q=0.4,en-US;q=0.2
>> Accept-Encoding: gzip, deflate, br
>> Accept: application/json, text/plain, */*
>> User-Agent: 
>> Host: localhost:8080
>>
>>
>>
>> Any idea of what could be happening ?
>> Is there any max size to configure and fix this?
>>
>
> Well, JMeter has a default max size for storing body contents. It is 10 MB
> and can be configured with a property in bin/jmeter.properties:
>
>
> # Max size of bytes stored in memory per SampleResult
> # Ensure you don't exceed max capacity of a Java Array and remember.
> # that the higher it is, the higher JMeter will consume heap
> # Defaults to 10MB
> #httpsampler.max_bytes_to_store_per_request=10485760
>
> Do you have any other signs for the truncation apart from JMeter? How did
> you measure the size in the browsers?
>
> Regards,
>  Felix
>
>
>> thanks !
>>
>>
>> "No olvides, no traiciones, lo que llevas bien dentro de ti. No olvides,
>> no
>> traiciones, lo que siempre te ha hecho vivir."
>>
>>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
Ok, thanks, didn't know that JMeter configuration.

I also update the Jersey version to 2.25.1 (I had 2.23.1)

[image: Inline image 1]

Now I see whole content in JMeter. So, looks like the clients (browsers)
are dropping the content.

So, is not a question for this mail list.
Thanks for the help !

AuthenticatorBase.getJaspicProvider taking most time

[Venkata Pavan Kumar Sannisetty]

Hi,

Our Application is running on Tomcat 8.5.11 with Java Security Manager. We
see a huge difference in performance dip with tomcat when Java security
manager is enabled. The test uses Jmeter and measures the throughput. Using
JVisualVM provided few bottlenecks in our application as well as with
Tomcat. Our application is around 5 times slow. Jvisualvm shows in each
thread the below call is taking around 70 percent of cpu time. I think this
is because the synchronized call made to AuthConfigFactory.getFactory()
method.

When security manager is enabled tomcat most of the time spends
at AuthenticatorBase.getJaspicProvider() call. Looking at this piece of
code in Tomcat Github


private AuthConfigProvider getJaspicProvider() {
AuthConfigProvider provider = jaspicProvider;
if (provider == null) {
provider = findJaspicProvider();
}
if (provider == NO_PROVIDER_AVAILABLE) {
return null;
}
return provider;
}

private AuthConfigProvider findJaspicProvider() {
AuthConfigFactory factory = AuthConfigFactory.getFactory()
;//bottleneck
AuthConfigProvider provider = null;
if (factory != null) {
provider = factory.getConfigProvider("HttpServlet",
jaspicAppContextID, this);
}
if (provider == null) {
provider = NO_PROVIDER_AVAILABLE;
}
jaspicProvider = provider;
return provider;
}

The jaspicProvider variable is an instance variable i am assuming may be it
is getting created in each and every request (just my assumption).

Is it possible cache the AuthConfigFactory instance in the code
findJaspicProvider to improve the performance of tomcat with java security
manager.

Please let me know your inputs.

Thanks,
Pavan.

RE: applicatio shared path is not working on tomcat8 version

[Naga Ramesh]

Chris,

At present application configured on one of app server and jars located on 
different server.

So we have shared the jars folder and given the shared path of classpath on 
tomcat end.

But in tomcat 7 version is working fine but same shared path of jars classpath 
it's not working on tomcat 8 version and observed the below errors.

So we have enabled changes on context file end and it's working.

Can you please validate and let me know if any proper solution for this.


25-Aug-2017 23:58:40.282 WARNING [localhost-startStop-1] 
org.apache.tomcat.util.scan.StandardJarScanner.scan Failed to scan 
[file://msbpo120/config/workflow/] from classloader hierarchy
java.io.IOException
at 
org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:339)
at 
org.apache.tomcat.util.scan.StandardJarScanner.scan(StandardJarScanner.java:245)
at 
org.apache.catalina.startup.ContextConfig.processJarsForWebFragments(ContextConfig.java:1887)
at 
org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1127)
at 
org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:779)
at 
org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:306)
at 
org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:95)
at 
org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
at 
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5150)
at 
org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
at 
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:725)
at 
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:701)
at 
org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717)
at 
org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:587)
at 
org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1798)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: URI has an authority component
at java.io.File.(File.java:423)
at 
org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:318)
... 19 more


Regards,
Naga
 
-Original Message-
From: Christopher Schultz [mailto:ch...@christopherschultz.net] 
Sent: Tuesday, August 29, 2017 5:30 PM
To: users@tomcat.apache.org
Subject: Re: applicatio shared path is not working on tomcat8 version

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Naga,

On 8/28/17 12:38 PM, Naga Ramesh wrote:
> Chris, We don’t have the context file on app/WEB-INF path

You need to create a new one.

> , that’s why we have done on /tomcat/conf/context.xml file.

You have modified the defaults for the whole server. If you deploy another web 
application, it will also have the JarScanner disabled, even if you need JAR 
scanning enabled. Please fix your global defaults file and configure your 
application correctly.

> After that issue is resolved, but how do I resolve the issue, please 
> guide me
Which issue? You said that disabling JAR scanning has fixed your original 
report (which doesn't actually make any sense to me). so what problem are you 
now facing?

- -chris

> -Original Message- From: Christopher Schultz 
> [mailto:ch...@christopherschultz.net] Sent: Monday, August 28,
> 2017 9:09 PM To: users@tomcat.apache.org Subject: Re: applicatio 
> shared path is not working on tomcat8 version
> 
> Naga,
> 
> On 8/28/17 11:23 AM, Naga Ramesh wrote:
>> Thanks Chris.. issues is resolved..
> 
>> We have enabled the below thing on context file and it's working 
>> fine.
> 
> 
>> tomcat8/conf/context.xml:  added the below line and restarted the 
>> service.
> 
>> > scanAllDirectories="false"/>
> 
> STOP. Undo what you have just done.
> 
> You only want to disable the JAR scanner for your application.
> What you have done is disable the JAR scanner for the entire server.
> 
> The file you want is $yourapplication/WEB-INF/context.xml not 
> $CATALINA_BASE/conf/context.xml
> 
> -chris
> 
>> -Original Message- From: Christopher Schultz 
>> [mailto:ch...@christopherschultz.net] Sent: Monday, August 28,
>> 2017 8:11 PM To: users@tomcat.apache.org Subject: Re: applicatio 
>> shared path is not working on tomcat8 version
> 
>> Naga,
> 
>> On 8/27/17 12:24 

Re: AuthenticatorBase.getJaspicProvider taking most time

[Mark Thomas]

On 29/08/17 17:27, Venkata Pavan Kumar Sannisetty wrote:
> Hi,
> 
> Our Application is running on Tomcat 8.5.11 with Java Security Manager. We
> see a huge difference in performance dip with tomcat when Java security
> manager is enabled. The test uses Jmeter and measures the throughput. Using
> JVisualVM provided few bottlenecks in our application as well as with
> Tomcat. Our application is around 5 times slow. Jvisualvm shows in each
> thread the below call is taking around 70 percent of cpu time. I think this
> is because the synchronized call made to AuthConfigFactory.getFactory()
> method.
> 
> When security manager is enabled tomcat most of the time spends
> at AuthenticatorBase.getJaspicProvider() call. Looking at this piece of
> code in Tomcat Github
> 
> 
> private AuthConfigProvider getJaspicProvider() {
> AuthConfigProvider provider = jaspicProvider;
> if (provider == null) {
> provider = findJaspicProvider();
> }
> if (provider == NO_PROVIDER_AVAILABLE) {
> return null;
> }
> return provider;
> }
> 
> private AuthConfigProvider findJaspicProvider() {
> AuthConfigFactory factory = AuthConfigFactory.getFactory()
> ;//bottleneck
> AuthConfigProvider provider = null;
> if (factory != null) {
> provider = factory.getConfigProvider("HttpServlet",
> jaspicAppContextID, this);
> }
> if (provider == null) {
> provider = NO_PROVIDER_AVAILABLE;
> }
> jaspicProvider = provider;
> return provider;
> }
> 
> The jaspicProvider variable is an instance variable i am assuming may be it
> is getting created in each and every request (just my assumption).
> 
> Is it possible cache the AuthConfigFactory instance in the code
> findJaspicProvider to improve the performance of tomcat with java security
> manager.
> 
> Please let me know your inputs.

And if you test with a more recent version than 8.5.11 ?

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: AuthenticatorBase.getJaspicProvider taking most time

[Venkata Pavan Kumar Sannisetty]

I have tried with 8.5.15 still experiencing the same issue. I have seen
this particular piece of code getting changed from 8.5.11 to 8.5.15. But
hitting the same issue,

On 29 August 2017 at 23:11, Mark Thomas  wrote:

> On 29/08/17 17:27, Venkata Pavan Kumar Sannisetty wrote:
> > Hi,
> >
> > Our Application is running on Tomcat 8.5.11 with Java Security Manager.
> We
> > see a huge difference in performance dip with tomcat when Java security
> > manager is enabled. The test uses Jmeter and measures the throughput.
> Using
> > JVisualVM provided few bottlenecks in our application as well as with
> > Tomcat. Our application is around 5 times slow. Jvisualvm shows in each
> > thread the below call is taking around 70 percent of cpu time. I think
> this
> > is because the synchronized call made to AuthConfigFactory.getFactory()
> > method.
> >
> > When security manager is enabled tomcat most of the time spends
> > at AuthenticatorBase.getJaspicProvider() call. Looking at this piece of
> > code in Tomcat Github
> >
> >
> > private AuthConfigProvider getJaspicProvider() {
> > AuthConfigProvider provider = jaspicProvider;
> > if (provider == null) {
> > provider = findJaspicProvider();
> > }
> > if (provider == NO_PROVIDER_AVAILABLE) {
> > return null;
> > }
> > return provider;
> > }
> >
> > private AuthConfigProvider findJaspicProvider() {
> > AuthConfigFactory factory = AuthConfigFactory.getFactory()
> > ;//bottleneck
> > AuthConfigProvider provider = null;
> > if (factory != null) {
> > provider = factory.getConfigProvider("HttpServlet",
> > jaspicAppContextID, this);
> > }
> > if (provider == null) {
> > provider = NO_PROVIDER_AVAILABLE;
> > }
> > jaspicProvider = provider;
> > return provider;
> > }
> >
> > The jaspicProvider variable is an instance variable i am assuming may be
> it
> > is getting created in each and every request (just my assumption).
> >
> > Is it possible cache the AuthConfigFactory instance in the code
> > findJaspicProvider to improve the performance of tomcat with java
> security
> > manager.
> >
> > Please let me know your inputs.
>
> And if you test with a more recent version than 8.5.11 ?
>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: applicatio shared path is not working on tomcat8 version

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Naga,

On 8/29/17 1:21 PM, Naga Ramesh wrote:
> At present application configured on one of app server and jars 
> located on different server.
> 
> So we have shared the jars folder and given the shared path of 
> classpath on tomcat end.

Last chance: please post your configuration that shows how you have
modified your classpath.

> But in tomcat 7 version is working fine but same shared path of
> jars classpath it's not working on tomcat 8 version and observed
> the below errors.
> 
> So we have enabled changes on context file end and it's working.

So: it's working? I'm confused.

> Can you please validate and let me know if any proper solution for
> this.

Solution for what? You just said it's working.

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJZpbHgAAoJEBzwKT+lPKRYDaoQAMk5yBrhH64zA7PG19uGeY7o
2Y5yFvEKRADo95ZvCF5DEtvcJv8TlxNpP3oqRxOjjOFq8NWD+GzXMSlH2yxi7GLA
ytcUunxH3chROs+30+y7I/aVwjmav9CspZ3zfHRK1dg47FQAm2EeVy/sc7UZ2Geu
V5k4kJRTelXv2eEKuy0XnaXDTtTbMR+arPAxwbBejFxUqK1zLBR3iRqbtoT89pgf
3+uI5E4p3KuxRDbryziZb92S5bIl2TWJHARYHaTj1aIBckj4K36t6lM6MRkBlKZt
+7cEKN5DbQoBLMAevc46wcwxCb058mkPPjBddWBpGIWnWmsPp+6dQRjBoYYFYOqS
mm/I2FJOjSURz7+9cB8thLpMMzdiRQpUMBv4NyoBZSArxVTgnqcGGCwdDF0k/0fc
z6NlEIz3mI4ClykpXYThHA9l3F+Ajp2GS9tGWM+shAzYAaGep+Y5QXpQ/N1BVdk2
1u4QdGez2KntFKjzeQVa1qqlmekNXmnJ5HDnJdz0qdldoT9B41SXeyTqhGrmUgF8
EmEH5/dm5bfSVhuXErsziqI6jYJtai17G3XJyattUAkEMww9G98wGPLWdNOCpMr4
9KStjTg94M9FkQzTVVKANc7IJPNz1E1xTWqitrvZpgIB4Cos9dNzV+YwzMLiwVxL
l9KcNCObxEljsQEQv/ok
=DSpW
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Performance issue 8.5.20 (metaspace related?)

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Andrea,

On 8/29/17 8:09 AM, Ing. Andrea Vettori wrote:
>> On 29 Aug 2017, at 12:29, Suvendu Sekhar Mondal
>>  wrote:
>> 
>> On Tue, Aug 29, 2017 at 2:54 PM, Ing. Andrea Vettori 
>>  wrote:
>>> - with a fresh started tomcat instance, the time it takes is
>>> around 0,8 seconds. Most of the time is spent on the two RMI
>>> calls the task does. - with an instance that is running from
>>> some time, the time can reach 2/3 seconds; occasionally 5/6
>>> seconds. Most time is still spent on RMI calls. I.e. what slows
>>> down are the RMI calls.
>> 
>> Sill question, do you mean RMI calls generating from Tomcat is
>> getting slower with time? or, JBoss is taking time to return the
>> response?
> 
> Thanks for your help. What I see is that the http requests are
> slower. Looking at the details of that specific request (putting a
> product in the cart) using System.nanotime I can see that most of
> the time is spent during the RMI calls. I’m pretty sure it’s not
> jboss that is slower because doing other calls at the same time
> with fresh tomcats or with a java client they're not slow.
> 
>> 
>>> - restarting the jvm fixes the issue - ***it seems*** but I’m
>>> still testing this since it seems there’s no ‘meatspace gc
>>> trigger command available', that when Metaspace is garbage
>>> collected, tomcat then performs like a fresh instance.
>>> 
>> 
>> Yes, no command to only garbage collect Metaspace area. Without 
>> recycling Tomcat, you can trigger a Full GC either using VisualVM
>> or jcmd's GC.run command.
> 
> I think that explicit GC is not doing a metaspace GC… that’s what I
> observed starting the GC with the java console.
> 
>> 
>>> Since we’re using more than one tomcat instance (2 in
>>> production for this website, 1 for development) I can see that
>>> the issue is isolated to Tomcat or the JVM/Host where it runs
>>> because other Tomcat instances behave well at the same time one
>>> is slow. The same JBoss/Postgres backend is used by java
>>> batches and fat clients and it does work well with consistent
>>> times.
>>> 
>>> To clarify: the moment one production tomcat that is running
>>> from some time finishes the task in 3 seconds, the development
>>> tomcat or a fresh started production tomcat instance does the
>>> same task in less that one second. Note that repeating the task
>>> gives always consistent results, i.e. the instance is running
>>> from some time is always slow,  the fresh running instance is
>>> always fast.
>>> 
>>> Tomcat is running with these VM options:
>>> 
>>> -Xms20480m -Xmx20480m -Dsun.rmi.dgc.client.gcInterval=360
>>> -Dsun.rmi.dgc.server.gcInterval=360 -Xloggc:/tmp/gc.txt
>>> -Dcom.sun.management.jmxremote
>>> -Dcom.sun.management.jmxremote.port=9012
>>> -Dcom.sun.management.jmxremote.ssl=false
>>> -Dcom.sun.management.jmxremote.authenticate=false
>>> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseG1GC
>>> -XX:ReservedCodeCacheSize=1g -XX:InitialCodeCacheSize=256m
>>> -XX:+UseHugeTLBFS -XX:MetaspaceSize=1g -XX:MaxMetaspaceSize=2g
>>> 
>>> Some of the options have been recently added (for example the
>>> increase in code cache  size) but it seems they had no impact
>>> on the issue.
>>> 
>> 
>> Is there any specific reason to trigger Full GC every 
>> 60min(-Dsun.rmi.dgc.client.gcInterval=360)? I think that's 
>> overkill.
> 
> That’s a value I saw “everywhere” as suggested value (the default
> is too short). Note that on the GC logs I don’t see full GCs every
> hour so maybe that option is not triggering full gc ?
> 
> 
>>> Metaspace goes up to 1,6GB before being collected. Value after
>>> garbage collect is around 200MB. Heap usage is variable, it
>>> usually stays under 10G and is around 1G after garbage
>>> collect. CPU usage rarely goes over 10%. Loaded classes between
>>> 20k and 40k. Active sessions around 100/120 for each instance.
>>> 
>> 
>> That sounds like a healthy behavior to me. That means GC is
>> occurring and Metaspace is getting cleaned up. Why do you think
>> Metaspace is the problem?
> 
> Because I can observe that when the metaspace is collected the
> requests become fast. I observer that a few hours ago, looking at
> the metaspace graph of the java console and doing requests just
> after the collect.
> 
>> From the given data, my first impression is JVM is not having
>> any memory pressure. If you can post your gc.txt file to this
>> group, I would be happy to check it.
> 
> Thanks, here’s a link for gceasy report:
> http://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTcvMDgvMjkvLS1nY3N0YX
RzLnR4dC5nei0tMTEtNDktMzQ=
> 
>
> 
and the txt log file: http://ge.tt/1Un0DLm2
> Please consider that I triggered a Full GC manually two times on
> this server.
> 
> 
>> 
>> Another thing, is there any way you can measure the processing
>> time in JBoss for each RMI calls?
>> 
> 
> At the moment I’m not measu

RE: Performance issue 8.5.20 (metaspace related?)

[Caldarale, Charles R]

> From: Christopher Schultz [mailto:ch...@christopherschultz.net] 
> Subject: Re: Performance issue 8.5.20 (metaspace related?)
 
> > > That sounds like a healthy behavior to me. That means GC is
> > > occurring and Metaspace is getting cleaned up. Why do you think
> > > Metaspace is the problem?

> > Because I can observe that when the metaspace is collected the
> > requests become fast. I observer that a few hours ago, looking at
> > the metaspace graph of the java console and doing requests just
> > after the collect.

> RMI is known for flagrantly wasting permgen/metaspace because of all
> the Proxy objects and special throw-away Class objects it uses, etc.
> But I'm not sure why the metaspace filling up would have such a
> dramatic effect on performance.

> At any rate, this is not a problem with Tomcat itself: this is likely
> entirely JVM-related.

Is it possible that the system is getting into swapping?  The heap has been
set to 20 GiB, but I didn't see any mention of how much actual memory the
system has.  Do you really need a 20 GiB heap?  Sometimes smaller is better.
Might also want to try turning off UseHugeTLBFS.

I wonder if there are heap objects tied up due to dead but not collected
metaspace items; when metaspace is GC'd, the heap usage might well shrink
also.  (This might be a G1GC bug, but that seems unlikely.)  Perhaps the RMI
mechanism (which I haven't looked at in many years) is searching an
ever-growing set of soft/weak references that are removed when metaspace is
collected.  A smaller heap might help there.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you received
this in error, please contact the sender and delete the e-mail and its
attachments from all computers.



smime.p7s
Description: S/MIME cryptographic signature