RE: Monitoring resources comsumption at context level

[Jäkel , Guido]

Hi,

that's a question that is not specific to Zabbix or Tomcat but to all kind of 
cases that use *one* JVM to run more than one "Application". And that's also an 
open point to me since nearly decades. 

Some years ago I googled something that is called "Classmexer" (as a 
subproject(?) of something that is called "Javamex"

https://www.javamex.com/classmexer/

This seems to provide a Package that may calculate the memory consumption of an 
java object (tree?). Maybe, by help of this, on may be able to calculate what's 
the memory attatched to a certain classloader or servlet container. Maybe some 
Java guru here can take a look at this, it even seem to provide a JMX 
instrumentation interface, yet.

If it is possible to use this an if it's not a real expensive operation, then 
somebody may develop a kind of adapter to be used for Tomcat or other 
Application Servers.


@Fellipe: There are some tools to examine a JVM heap dump, but -- as me -- 
that's not you're looking for. Dumping the heap on a running JVM is a 
stop-world-event that will take significant time in case of the usecase here, 
where the heap of the Tomcat JVM will be sized to many GBs to serve a couple of 
applications. From this, in a production environment one may pull a heap dump 
for offline debugging purposes in case of already ongoing trouble. But it's not 
appropriate for continuous monitoring.

Guido



>-Original Message-
>From: Christopher Schultz [mailto:ch...@christopherschultz.net]
>Sent: Friday, April 05, 2019 12:35 AM
>To: users@tomcat.apache.org
>Subject: Re: Monitoring resources comsumption at context level
>
>-BEGIN PGP SIGNED MESSAGE-
>Hash: SHA256
>
>Fellipe,
>
>On 4/4/19 16:51, Fellipe Theophilo wrote:
>> Hi Cris, thank you for reply. So, I've talked to the developer team
>> about this and they told me that they want to be able to know at
>> OBJECT level or at CLASS level what is consuming more resources, to
>> make easier to debug the code. For example, suppose that some JVM
>> is having high resource comsumption, they want to know what CLASS,
>> or what METHOD is the responsible for the issue. Is there any way
>> to do it?
>
>Your developer team should already know that the only way to do that
>would be to take a heap dump or attach a debugger to a running JVM.
>What they want is simply not possible via JMX.
>
>Taking a heap dump can cause your JVM to stall for an indeterminate
>amount of time.

RE: Monitoring resources comsumption at context level

[Jäkel , Guido]

>Some years ago I googled something that is called "Classmexer" (as a 
>subproject(?) of something that is called "Javamex"
>
>   https://www.javamex.com/classmexer/
>

I just noticed, that I also have bookmarked an post called "Instrumentation 
Memory Counter"

https://www.javaspecialists.eu/archive/Issue142.html

to deal with related things ...

Re: Monitoring resources comsumption at context level

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Fellipe,

On 4/4/19 16:51, Fellipe Theophilo wrote:
> Hi Cris, thank you for reply. So, I've talked to the developer team
> about this and they told me that they want to be able to know at
> OBJECT level or at CLASS level what is consuming more resources, to
> make easier to debug the code. For example, suppose that some JVM
> is having high resource comsumption, they want to know what CLASS,
> or what METHOD is the responsible for the issue. Is there any way
> to do it?

Your developer team should already know that the only way to do that
would be to take a heap dump or attach a debugger to a running JVM.
What they want is simply not possible via JMX.

Taking a heap dump can cause your JVM to stall for an indeterminate
amount of time.

Attaching a debugger can significantly degrade performance.

For realtime monitoring, you may want to build-in some application
self-monitoring features and then keep an eye on those from e.g. Zabbix.

- -chris

> Em ter, 2 de abr de 2019 às 20:01, Christopher Schultz < 
> ch...@christopherschultz.net> escreveu:
> 
> Fellipe,
> 
> On 4/2/19 17:04, Fellipe Theophilo wrote:
 Hi everyone, I'm trying to find a way to monitor metrics of 
 resources comsumption at context level. I've opened this
 thread: 
 https://stackoverflow.com/questions/55070370/monitoring-multiple-ja
va-
>
 
applications-at-once-with-one-zabbix-java-gateway
> 


> 
But no one knows a solution. By using jConsole and Zabbix I can
 check the value of some objects at context level, but none of
 them is any metric. So I tried the Spring Actuator, which
 expose many metrics through REST , returning a JSON. However,
 the call to 
 http://:8080//metrics give
 metrics information of the JVM as a whole, even specifying
 the context name. To prove this I ran an AB(Apache Benchmark)
 to do a stress test and I've saw that memory usage grew
 together across the two contexts I have for testing. So the
 conclusion is that the Spring Actuator is not exposing
 metrics at context level. Do anyone know if is there a way to
 get metrics at context level?
> 
> The reason that no tool provides per-context memory consumption 
> metrics is because it's not practical to actually measure that kind
> of thing.
> 
> You'd basically need to walk the object tree from GC roots (like
> the GC does whenever it collects garbage) after determining what
> the list of "GC roots" is for each context -- such as the
> WebappClassLoader.
> 
> The JVM treats memory as a shared resource. There is no memory 
> isolation between contexts within an application server.
> 
> If you really want to monitor your web applications separately,
> run them each in isolated JVMs.
> 
> -chris
>> 
>> -
>>
>> 
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>> 
>> 
> 
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlymhnYACgkQHPApP6U8
pFi1lg//bvCIPS2VAnJ8wT+YiUCldhk0IrOROWHngppYSX7hSLnhrlswu1srP0tO
fhOpkXBXBbsp2wkX6xGugWhTqfmeRFplSokcWhKGsAi9822JeTqEk2LkhWsUA88R
HGbKT1jk4Qw0OG+T94gJJPXhcQQYalNIE4+z68Jq/qA5HjTjDckM/7ORt2Tx049w
rowge/AtBpZ3/E5GEtmh2LwOBA4oEUUJm6u3abnPU+0TlhZd1uVCR2MttNTaK4iG
nYnn12bk+Ho03tXBxjdhd5fELt+VtpYidvlDA3wAqPwLA5xDKiRJJPKMvI0zgElv
ShWZpE+2pGknROChEl+RCHNO3sSeOru4Ri5N0IHBfMcns6KpbM7xODoP7IrwFZUp
zRD4+ffVan4qGGnLVsVZXyY5HA2K3Z51wz445lWK9cuSXJDDd4u+5vAW8cF7KIhH
mXDypdS3L1RNatK6rDcRhDvkRgNiCzdTP2/qaAp5FOEkvGEshyX0AyL3WBD6Fqpb
NcuR11jH80SKxeSUFatxjRMi/qx8zoO7WEHesfL1cvA3O1MBXYb/FO1d23+tK/f9
EXETpC90KQE9QjHXi6tZeKUBKkcU2YQyCOtARhlKAWgo/CoXekNbJ7wE8YAKhJp6
IjHDBcOOGHgaNexg+847Ds6+KqP1JjFmol6sjIVQQM8Xij1Y/jw=
=N9dt
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Monitoring resources comsumption at context level

[Fellipe Theophilo]

Hi Cris, thank you for reply. So, I've talked to the developer team about
this and they told me that they want to be able to know at OBJECT level or
at CLASS level what is consuming more resources, to make easier to debug
the code. For example, suppose that some JVM is having high resource
comsumption, they want to know what CLASS, or what METHOD is the
responsible for the issue. Is there any way to do it?
-
Fellipe


Em ter, 2 de abr de 2019 às 20:01, Christopher Schultz <
ch...@christopherschultz.net> escreveu:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Fellipe,
>
> On 4/2/19 17:04, Fellipe Theophilo wrote:
> > Hi everyone, I'm trying to find a way to monitor metrics of
> > resources comsumption at context level. I've opened this thread:
> > https://stackoverflow.com/questions/55070370/monitoring-multiple-java-
> applications-at-once-with-one-zabbix-java-gateway
> 
> >
> >  But no one knows a solution. By using jConsole and Zabbix I can
> > check the value of some objects at context level, but none of them
> > is any metric. So I tried the Spring Actuator, which expose many
> > metrics through REST , returning a JSON. However,  the call to
> > http://:8080//metrics give metrics
> > information of the JVM as a whole, even specifying the context
> > name. To prove this I ran an AB(Apache Benchmark) to do a stress
> > test and I've saw that memory usage grew together across the two
> > contexts I have for testing. So the conclusion is that the Spring
> > Actuator is not exposing metrics at context level. Do anyone know
> > if is there a way to get metrics at context level?
>
> The reason that no tool provides per-context memory consumption
> metrics is because it's not practical to actually measure that kind of
> thing.
>
> You'd basically need to walk the object tree from GC roots (like the
> GC does whenever it collects garbage) after determining what the list
> of "GC roots" is for each context -- such as the WebappClassLoader.
>
> The JVM treats memory as a shared resource. There is no memory
> isolation between contexts within an application server.
>
> If you really want to monitor your web applications separately, run
> them each in isolated JVMs.
>
> - -chris
> -BEGIN PGP SIGNATURE-
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlyj6cMACgkQHPApP6U8
> pFhcKBAAiAUUS9pchJB0JIjTrldg4VOAuLxyTOMQG7qiaffDq/eQCac7OAH4uoaf
> Z7gCV44QK+rIpRmKqSdHWEEccGZ0b0ywPnLM7GUUptynL2ZYB7qTH7/wjnzsEKPV
> pMdDp06d5eo5dAFyKiKSkT8bT2h6+wmbP/xrWkgL4DtyUPQzbPOIgRdjDPfihaxp
> 0l0xI8H/HYaqyNhxj0xDmEy7IbO2buNt5kUzrC2HmS+TC+rl4rlUDEUAISK9vz6a
> WTGaYnGCbbbDmAJc2NtRMSwCMrzt4Oz3yLJR1+HzFCidmKYasCVTOp3elBtctvzE
> BEUuKOgNIo9f1rte2HKj1nAoxrKOw4hlWfdqqxfIm3yBk3ThtGEjMAoa0enfKXnt
> kWObLMuL64c+XEmmqDMh4Q1AwpvzFeIQ1KgcxBGSGcgsmYZQQRY2LIE7TMI20Xp/
> b4Xv19+0HoI6VqHVL8pXt8BWznjM+ygJb2mtUuO4nFRGuCYSLw6s2IqrL59p4Pld
> DtsBa5U5eehtIrYOlHi9qndQP5G9BtbSHe4HI5U3mkqC5i2f3oywzHKcE5/roTc0
> 19zMzkJL8ISDcnZ7EBUgV5Qx13q+nEQeu9gZfjQjQCMTTIcmWnyrvWcRjIUUZwb7
> 2cCG80EfB5iny3fR0gtozj/SDmx5TOxU3MMnMunBQVZOoqf7uyc=
> =uO0P
> -END PGP SIGNATURE-
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Monitoring resources comsumption at context level

[John Dale]

I was interested in doing this at one point in the past.  A solution
that I found that was useful...

If you have control of the code for your application, review it.
Where in the code are you allocating resources of significance?

For my purposes, I decided to disregard thread/request/connection
handling since the values are essentially the same across contexts.

My main memory driver was a managed cache of objects .. images, text,
and encoded audio.  Based on some database metadata, I was able to
count the bytes of data in any object read from the database
automatically in few lines of code, giving me a really nice
representation of the bytes used by the cache at any given time.
Database objects cached was my memory driver, so I measured that
specifically (granted, using some techniques that I've developed over
the years to automate database heuristics, inspired by, but not
limited to, OMG.org).  Then, I can access the statistics on a
classloader/instance/context basis, segmented by deployment URL for my
most important memory driver.

Nothing better than just taking a weekend to make it through critical
sections in code to understand the answers to questions like this.
Can't stress that enough if you aren't already doing that. :)

Be the John Henry of software code,

John


On 4/2/19, Christopher Schultz  wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Fellipe,
>
> On 4/2/19 17:04, Fellipe Theophilo wrote:
>> Hi everyone, I'm trying to find a way to monitor metrics of
>> resources comsumption at context level. I've opened this thread:
>> https://stackoverflow.com/questions/55070370/monitoring-multiple-java-
> applications-at-once-with-one-zabbix-java-gateway
>>
>>  But no one knows a solution. By using jConsole and Zabbix I can
>> check the value of some objects at context level, but none of them
>> is any metric. So I tried the Spring Actuator, which expose many
>> metrics through REST , returning a JSON. However,  the call to
>> http://:8080//metrics give metrics
>> information of the JVM as a whole, even specifying the context
>> name. To prove this I ran an AB(Apache Benchmark) to do a stress
>> test and I've saw that memory usage grew together across the two
>> contexts I have for testing. So the conclusion is that the Spring
>> Actuator is not exposing metrics at context level. Do anyone know
>> if is there a way to get metrics at context level?
>
> The reason that no tool provides per-context memory consumption
> metrics is because it's not practical to actually measure that kind of
> thing.
>
> You'd basically need to walk the object tree from GC roots (like the
> GC does whenever it collects garbage) after determining what the list
> of "GC roots" is for each context -- such as the WebappClassLoader.
>
> The JVM treats memory as a shared resource. There is no memory
> isolation between contexts within an application server.
>
> If you really want to monitor your web applications separately, run
> them each in isolated JVMs.
>
> - -chris
> -BEGIN PGP SIGNATURE-
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlyj6cMACgkQHPApP6U8
> pFhcKBAAiAUUS9pchJB0JIjTrldg4VOAuLxyTOMQG7qiaffDq/eQCac7OAH4uoaf
> Z7gCV44QK+rIpRmKqSdHWEEccGZ0b0ywPnLM7GUUptynL2ZYB7qTH7/wjnzsEKPV
> pMdDp06d5eo5dAFyKiKSkT8bT2h6+wmbP/xrWkgL4DtyUPQzbPOIgRdjDPfihaxp
> 0l0xI8H/HYaqyNhxj0xDmEy7IbO2buNt5kUzrC2HmS+TC+rl4rlUDEUAISK9vz6a
> WTGaYnGCbbbDmAJc2NtRMSwCMrzt4Oz3yLJR1+HzFCidmKYasCVTOp3elBtctvzE
> BEUuKOgNIo9f1rte2HKj1nAoxrKOw4hlWfdqqxfIm3yBk3ThtGEjMAoa0enfKXnt
> kWObLMuL64c+XEmmqDMh4Q1AwpvzFeIQ1KgcxBGSGcgsmYZQQRY2LIE7TMI20Xp/
> b4Xv19+0HoI6VqHVL8pXt8BWznjM+ygJb2mtUuO4nFRGuCYSLw6s2IqrL59p4Pld
> DtsBa5U5eehtIrYOlHi9qndQP5G9BtbSHe4HI5U3mkqC5i2f3oywzHKcE5/roTc0
> 19zMzkJL8ISDcnZ7EBUgV5Qx13q+nEQeu9gZfjQjQCMTTIcmWnyrvWcRjIUUZwb7
> 2cCG80EfB5iny3fR0gtozj/SDmx5TOxU3MMnMunBQVZOoqf7uyc=
> =uO0P
> -END PGP SIGNATURE-
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Monitoring resources comsumption at context level

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Fellipe,

On 4/2/19 17:04, Fellipe Theophilo wrote:
> Hi everyone, I'm trying to find a way to monitor metrics of
> resources comsumption at context level. I've opened this thread: 
> https://stackoverflow.com/questions/55070370/monitoring-multiple-java-
applications-at-once-with-one-zabbix-java-gateway
>
>  But no one knows a solution. By using jConsole and Zabbix I can
> check the value of some objects at context level, but none of them
> is any metric. So I tried the Spring Actuator, which expose many
> metrics through REST , returning a JSON. However,  the call to 
> http://:8080//metrics give metrics
> information of the JVM as a whole, even specifying the context 
> name. To prove this I ran an AB(Apache Benchmark) to do a stress
> test and I've saw that memory usage grew together across the two
> contexts I have for testing. So the conclusion is that the Spring
> Actuator is not exposing metrics at context level. Do anyone know
> if is there a way to get metrics at context level?

The reason that no tool provides per-context memory consumption
metrics is because it's not practical to actually measure that kind of
thing.

You'd basically need to walk the object tree from GC roots (like the
GC does whenever it collects garbage) after determining what the list
of "GC roots" is for each context -- such as the WebappClassLoader.

The JVM treats memory as a shared resource. There is no memory
isolation between contexts within an application server.

If you really want to monitor your web applications separately, run
them each in isolated JVMs.

- -chris
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlyj6cMACgkQHPApP6U8
pFhcKBAAiAUUS9pchJB0JIjTrldg4VOAuLxyTOMQG7qiaffDq/eQCac7OAH4uoaf
Z7gCV44QK+rIpRmKqSdHWEEccGZ0b0ywPnLM7GUUptynL2ZYB7qTH7/wjnzsEKPV
pMdDp06d5eo5dAFyKiKSkT8bT2h6+wmbP/xrWkgL4DtyUPQzbPOIgRdjDPfihaxp
0l0xI8H/HYaqyNhxj0xDmEy7IbO2buNt5kUzrC2HmS+TC+rl4rlUDEUAISK9vz6a
WTGaYnGCbbbDmAJc2NtRMSwCMrzt4Oz3yLJR1+HzFCidmKYasCVTOp3elBtctvzE
BEUuKOgNIo9f1rte2HKj1nAoxrKOw4hlWfdqqxfIm3yBk3ThtGEjMAoa0enfKXnt
kWObLMuL64c+XEmmqDMh4Q1AwpvzFeIQ1KgcxBGSGcgsmYZQQRY2LIE7TMI20Xp/
b4Xv19+0HoI6VqHVL8pXt8BWznjM+ygJb2mtUuO4nFRGuCYSLw6s2IqrL59p4Pld
DtsBa5U5eehtIrYOlHi9qndQP5G9BtbSHe4HI5U3mkqC5i2f3oywzHKcE5/roTc0
19zMzkJL8ISDcnZ7EBUgV5Qx13q+nEQeu9gZfjQjQCMTTIcmWnyrvWcRjIUUZwb7
2cCG80EfB5iny3fR0gtozj/SDmx5TOxU3MMnMunBQVZOoqf7uyc=
=uO0P
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Monitoring resources comsumption at context level

[Fellipe Theophilo]

Hi everyone, I'm trying to find a way to monitor metrics of resources
comsumption at context level.
I've opened this thread:
https://stackoverflow.com/questions/55070370/monitoring-multiple-java-applications-at-once-with-one-zabbix-java-gateway

But no one knows a solution. By using jConsole and Zabbix I can check the
value of some objects at context level, but none of them is any metric. So
I tried the Spring Actuator, which expose many metrics through REST ,
returning a JSON. However,  the call to
http://:8080//metrics
give metrics information of the JVM as a whole, even specifying the context
name. To prove this I ran an AB(Apache Benchmark) to do a stress test and
I've saw that memory usage grew together across the two contexts I have for
testing. So the conclusion is that the Spring Actuator is not exposing
metrics at context level.
Do anyone know if is there a way to get metrics at context level?
-
Fellipe