Re: Tomcat not scaling under load - Tsung v Apache Bench test

[Szymon Czaja]

I will run a test in the meantime to check but this is very much unlikely,
I can see on the Amazon console that the load balancer CPU is hardly doing
anything with peaks at 2%. Let's assume load balancer is not an issue.

Szymon

On 28 November 2016 at 12:29, André Warnier (tomcat)  wrote:

> On 28.11.2016 12:52, Szymon Czaja wrote:
>
>> Hi,
>> I have updated my question on SO. I have noticed that the number of
>> ESTABLISHED connections goes up on the client after few minutes. I
>> expected
>> the same on the server which does not seem to be the case. Any ideas?
>>
>
> Just a guess without looking very deep into your data : you say somewhere
> that the requests go through a proxy. Maybe the client "established" TCP
> connections are with that proxy, while the ones you see on the tomcat
> server are the connections from the proxy ?
> (In other words, it is the proxy that is the bottleneck ?)
>
>
>
>
>> Szymon
>>
>> On 28 November 2016 at 10:05, Mark Thomas  wrote:
>>
>> On 28/11/2016 09:53, Szymon Czaja wrote:
>>>
 Hi,
 I have asked the question on StackOverflow but I am not getting much
 response:

 http://stackoverflow.com/questions/40793335/why-tomcat-
 does-not-pass-a-tsung-performance-test-at-40-requests-per-second

 Could anyone help me understand why is Tomcat unable to keep up with the
 processing when running Tsung yet Apache Bench tests do not relveal any
 scalability issues?

>>>
>>> I'd recommend taking some thread dumps and looking at netstat output to
>>> try and figure out what is going on.
>>>
>>> Mark
>>>
>>>
>>> -
>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>>
>>>
>>>
>>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Tomcat not scaling under load - Tsung v Apache Bench test

[Szymon Czaja]

Completely agree. I am in the process of setting up a direct connection to
Tomcat bypassing load balancer. Will give an update soon.

Szymon

On 28 November 2016 at 13:40, André Warnier (tomcat)  wrote:

> On 28.11.2016 14:10, Szymon Czaja wrote:
>
>> I will run a test in the meantime to check but this is very much unlikely,
>> I can see on the Amazon console that the load balancer CPU is hardly doing
>> anything with peaks at 2%. Let's assume load balancer is not an issue.
>>
>
> I am not saying that the proxy cannot handle it. But maybe it is limiting
> the number of connections that it forwards, if they are all coming from the
> same client IP ?
> (Avoid DoS attacks, that kind of thing..)
> Still a guess. But netstat on both sides may be telling you more.
>
> If it is tomcat that could not handle the load, then you'd probably still
> see the same number of connections on the tomcat server side, at the TCP
> level.
> That tomcat would be able to "service" these connections is another
> matter, but the connections themselves should be there.
>
>
>
>> Szymon
>>
>> On 28 November 2016 at 12:29, André Warnier (tomcat) 
>> wrote:
>>
>> On 28.11.2016 12:52, Szymon Czaja wrote:
>>>
>>> Hi,
 I have updated my question on SO. I have noticed that the number of
 ESTABLISHED connections goes up on the client after few minutes. I
 expected
 the same on the server which does not seem to be the case. Any ideas?


>>> Just a guess without looking very deep into your data : you say somewhere
>>> that the requests go through a proxy. Maybe the client "established" TCP
>>> connections are with that proxy, while the ones you see on the tomcat
>>> server are the connections from the proxy ?
>>> (In other words, it is the proxy that is the bottleneck ?)
>>>
>>>
>>>
>>>
>>> Szymon

 On 28 November 2016 at 10:05, Mark Thomas  wrote:

 On 28/11/2016 09:53, Szymon Czaja wrote:

>
> Hi,
>> I have asked the question on StackOverflow but I am not getting much
>> response:
>>
>> http://stackoverflow.com/questions/40793335/why-tomcat-
>> does-not-pass-a-tsung-performance-test-at-40-requests-per-second
>>
>> Could anyone help me understand why is Tomcat unable to keep up with
>> the
>> processing when running Tsung yet Apache Bench tests do not relveal
>> any
>> scalability issues?
>>
>>
> I'd recommend taking some thread dumps and looking at netstat output to
> try and figure out what is going on.
>
> Mark
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
>
>

>>> -
>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>>
>>>
>>>
>>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Tomcat not scaling under load - Tsung v Apache Bench test

[Szymon Czaja]

Hi,
I have updated my question on SO. I have noticed that the number of
ESTABLISHED connections goes up on the client after few minutes. I expected
the same on the server which does not seem to be the case. Any ideas?

Szymon

On 28 November 2016 at 10:05, Mark Thomas  wrote:

> On 28/11/2016 09:53, Szymon Czaja wrote:
> > Hi,
> > I have asked the question on StackOverflow but I am not getting much
> > response:
> >
> > http://stackoverflow.com/questions/40793335/why-tomcat-
> > does-not-pass-a-tsung-performance-test-at-40-requests-per-second
> >
> > Could anyone help me understand why is Tomcat unable to keep up with the
> > processing when running Tsung yet Apache Bench tests do not relveal any
> > scalability issues?
>
> I'd recommend taking some thread dumps and looking at netstat output to
> try and figure out what is going on.
>
> Mark
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Tomcat not scaling under load - Tsung v Apache Bench test

[tomcat]

On 28.11.2016 14:10, Szymon Czaja wrote:

I will run a test in the meantime to check but this is very much unlikely,
I can see on the Amazon console that the load balancer CPU is hardly doing
anything with peaks at 2%. Let's assume load balancer is not an issue.


I am not saying that the proxy cannot handle it. But maybe it is limiting the number of 
connections that it forwards, if they are all coming from the same client IP ?

(Avoid DoS attacks, that kind of thing..)
Still a guess. But netstat on both sides may be telling you more.

If it is tomcat that could not handle the load, then you'd probably still see the same 
number of connections on the tomcat server side, at the TCP level.
That tomcat would be able to "service" these connections is another matter, but the 
connections themselves should be there.




Szymon

On 28 November 2016 at 12:29, André Warnier (tomcat)  wrote:


On 28.11.2016 12:52, Szymon Czaja wrote:


Hi,
I have updated my question on SO. I have noticed that the number of
ESTABLISHED connections goes up on the client after few minutes. I
expected
the same on the server which does not seem to be the case. Any ideas?



Just a guess without looking very deep into your data : you say somewhere
that the requests go through a proxy. Maybe the client "established" TCP
connections are with that proxy, while the ones you see on the tomcat
server are the connections from the proxy ?
(In other words, it is the proxy that is the bottleneck ?)





Szymon

On 28 November 2016 at 10:05, Mark Thomas  wrote:

On 28/11/2016 09:53, Szymon Czaja wrote:



Hi,
I have asked the question on StackOverflow but I am not getting much
response:

http://stackoverflow.com/questions/40793335/why-tomcat-
does-not-pass-a-tsung-performance-test-at-40-requests-per-second

Could anyone help me understand why is Tomcat unable to keep up with the
processing when running Tsung yet Apache Bench tests do not relveal any
scalability issues?



I'd recommend taking some thread dumps and looking at netstat output to
try and figure out what is going on.

Mark


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org







-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org







-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat not scaling under load - Tsung v Apache Bench test

[tomcat]

On 28.11.2016 12:52, Szymon Czaja wrote:

Hi,
I have updated my question on SO. I have noticed that the number of
ESTABLISHED connections goes up on the client after few minutes. I expected
the same on the server which does not seem to be the case. Any ideas?


Just a guess without looking very deep into your data : you say somewhere that the 
requests go through a proxy. Maybe the client "established" TCP connections are with that 
proxy, while the ones you see on the tomcat server are the connections from the proxy ?

(In other words, it is the proxy that is the bottleneck ?)




Szymon

On 28 November 2016 at 10:05, Mark Thomas  wrote:


On 28/11/2016 09:53, Szymon Czaja wrote:

Hi,
I have asked the question on StackOverflow but I am not getting much
response:

http://stackoverflow.com/questions/40793335/why-tomcat-
does-not-pass-a-tsung-performance-test-at-40-requests-per-second

Could anyone help me understand why is Tomcat unable to keep up with the
processing when running Tsung yet Apache Bench tests do not relveal any
scalability issues?


I'd recommend taking some thread dumps and looking at netstat output to
try and figure out what is going on.

Mark


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org







-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

redirect 8445 to 445

[Vivek Patil]

Hello All,

I have setup Tomcat8 with SSL on AWS. Suppose my url is 
"https://mydomain.com:8443;
I want to do, when I hit "mydomain.com" it should move to https without 
any port no.


Please help me out.


--
Thanks & Regrads
Vivek Patil
Sr IT Engineer.
Spring Computing Technologies Pvt. Ltd.
Contact- +91-95792 16049.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Thread-safety of javax.servlet.Servlet#getServletConfig()

[Péter Gergely Horváth]

Hi Andy,

Thank you very much for your feedback, but I think there is a slight
misunderstanding here: I know the order in which a servlet is initialized
and put into service, and this question is not related to that. It's
related to the _visibility_ of the field.

The question is about the thread safety of the field access: if you
carefully check the details of javax.servlet.GenericServlet#config and
compare the implementation to the sample "NoVisibility" from the book Java
Concurrency in Practice, then it is seems to follow the same pattern, which
is "not thread-safe because the value field is accessed from both get and
set without synchronization. Among other hazards, it is susceptible to
stale values: if one thread calls set, other threads calling get may or may
not see that update." [1].

Personally, I would like to understand why this implementation is not (if
not) susceptible to the stale values phenomenon [2]; there might be
someone, who can point me to the right direction.

Thanks,
Peter


[1] Java Concurrency in Practice: Chapter 3. Sharing Objects 3.1 Visibility
[2]
https://www.javacodegeeks.com/2014/08/java-concurrency-tutorial-visibility-between-threads.html



On Mon, Nov 28, 2016 at 6:08 AM, andreas  wrote:

>  On Fri, 25 Nov 2016 23:02:00 +0930 Péter Gergely Horváth  wrote 
> >Hi All,
> >
> >I am wondering why calling javax.servlet.Servlet#getServletConfig() is
> >thread safe: if you check the implementation in
> > javax.servlet.GenericServlet from servlet API 3.0.1, you see the
> following:
> >
> >package javax.servlet;
> >
> >// lines omitted
> >
> >public abstract class GenericServlet
> > implements Servlet, ServletConfig, java.io.Serializable
> >{
> > // lines omitted
> >
> > private transient ServletConfig config;
> >
> > // lines omitted
> >
> > public void init(ServletConfig config) throws ServletException {
> > this.config = config;
> > this.init();
> > }
> >
> > // lines omitted
> >
> > public ServletConfig getServletConfig() {
> > return config;
> > }
> > // lines omitted
> >}
> >
> >
> >The field config is written in init(ServletConfig) without any
> >synchronization, locking or config being volatile, while
> getServletConfig()
> >could be called anytime from any later worker thread of the servlet
> >container. I took a quick look into Tomcat/Catalina code base, however I
> >see no indication of the servlet container performing any magic, which
> >would guarantee thread safe access to field config through
> >getServletConfig() from e.g.
> >javax.servlet.GenericServlet#service(ServletRequest, ServletResponse) and
> >the corresponding methods in javax.servlet.http.HttpServlet.
> >
> >Am I missing something here? What will guarantee that if Thread A is used
> >to init(ServletConfig) then Thread B calling getServletConfig() will see
> >the correct state of the field config (Java "happens-before"), and not
> e.g.
> >null?
> >
> >I am asking this because I have seen some legacy code, where a servlet
> >stored something into a field inside the init() method, which was then
> used
> >from within a javax.servlet.http.HttpServlet#doGet or
> >javax.servlet.http.HttpServlet#doPost method, without synchronization of
> >any kind like this:
> >
> >public class FoobarServlet extends HttpServlet {
> >
> > private FoobarService foobarService;
> >
> > @Override
> > public void init() throws ServletException {
> >this.foobarService = // ... acquire foobarService
> > }
> > protected void doGet(HttpServletRequest request, HttpServletResponse
> >response) throws ServletException, IOException {
> > List foobars = this.foobarService.getFoobars(); // read the
> >field WITHOUT synchronization
> > // ...
> > }
> > // ...
> >Is this approach (having no synchronization, locking or the field being
> >volatile) correct? I assumed it is not, seeing something like that in the
> >servlet API is quite confusing.
> >
> >
> >What do you think?
> >
> >Thanks,
> >Peter
>
>
> A Servlet will process requests only if it is fully initialized, i.e. init
> has been executed. The init method gets called only once and the servlet
> config won't change afterwards. I don't think there is need for
> synchronization. The same is probably valid for your own objects. Problems
> arise when individual requests change the state of these objects.
>
> Andy
>
>
>
>
>
>
>
>
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: redirect 8445 to 445

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Vivek,

On 11/28/16 9:55 AM, Vivek Patil wrote:
> Hello All,
> 
> I have setup Tomcat8 with SSL on AWS. Suppose my url is 
> "https://mydomain.com:8443; I want to do, when I hit "mydomain.com"
> it should move to https without any port no.

Then you need to configure your service to use port 443 instead of 8443.

https://wiki.apache.org/tomcat/HowTo#How_to_run_Tomcat_without_root_priv
ileges.3F

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJYPG3pAAoJEBzwKT+lPKRYBoQQAJQWhH/w108KknyN8h085beG
x+3HyE8HjLhCNpQvQG+UpPvihLmytdHKi+OFVTXta6/iIaBeDWMsJpoMaI9kszHN
DOADJXpQSRWfVrTC5LWUIwE0QCazXlODkMkr6DCbgIkTr8RY/RJ/5Ic+InasXdsb
k+AO00tc5o68xIyrMV/MJ51SWrO4oSkm4A81NHKDI/334rQrQMdZJ4uyq6sTkHOz
Kd9jtT8V6Xv1KMhHsEMl7by7FsilZ8AZIZ3TkCkYl1wKvhZ0mkwMfb4dQOSrqzgs
6Q6/SH8W5yAQ3JKUkoLGPdI4z3sVTEl5WIixr/dvr8IvHTBq9ng3utLBjPewAuRk
R3UxNlqco2ZddSg9MTW1hfelps/UP/OwyRPK1ICfkzhbEkYQhBUEZsydP1HbxUSm
ATBp7wBKkGpIFfIY115bGPe5FEZIRBLK6XRYUd6kTvlotfr5LjN9MBge/iwmM+l3
YQl5/2Y7J4sTsguyFcDB9x/CmqZYehs2m/bIhveUQV54oQmJR/qKFE86ynRGwSp4
PgFBF/NXvBMRZPCTisFO/f1iUcLgeDimUDX57J7txVDDxTpvalT6WfStc70r5WL2
Yg/7LHvHfs1RTlHvuX8qvXEQCSYi29PY+fnZ13mq00dGmZT6JOL4iKiPM2HZjjMc
FA7L162L+mAI/SLafVwV
=Rgos
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Apache/Tomcat vulnerability

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Jaaz,

On 11/27/16 1:03 PM, Jaaz Portal wrote:
> Then they exploited some well know vulnerability in mod_proxy. We
> have updated apache to the latest but again they has exploited it,
> so we have switched to mod_jk. And then guess what. They exploited
> it too so i decided to write to this list looking for help before
> trying jetty.

I'm unaware of any vulnerabilities in mod_jk, "well-known" or
otherwise. What vulnerability are you talking about?

Note that switching to Jetty won't solve anything if you think the
"well-known vulnerability" is in mod_jk.

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJYPIFQAAoJEBzwKT+lPKRYUPkP/iV5vbBj/v3bDe4Yp9qHO0dA
5p740/FRYFicM6fZf+SZ8dqsnPKZ30Ij6PTFfwNRFkGHiszErY1RZ94s3ZdXYJRF
QxNiSvToQq750ujwkDL4iZv8h4Ash0TRA/E6e0SvZ9W5yjT/FBSfxsSKe8uixN43
6xcXaJcTuipjoyNHnVlUFkZFcmMykg7hktcMeaGp9OI2j/4x+OkkWsTjxvEucDXx
9ediRNaldrnU+bvmMTKnI9FZSHmxg1hzkpky3gNOv+CF7gOCLo8/ixj8ZELb+Lwt
AREr30gtBdupfmGLO0pdFJ2EYHfmfW3Rlf6xwCzSBHZtdX8ewR1dw28SCJDlq/um
vRArMdG+s3OoLQ04GHRm+shom+X3KuhRf/NjZsF//we6XIpgxWv686HaVXPNa0Ys
6TomDIDM1N0o5l0mQ1799P2+/9t+hxUoGUcUIOUadn4QHtuGFONe266qYiguhuFj
sDVsi7R6jK4VodTA2HznKrdUZf1WqE2bw5zLSi7AOuoSu85jDfN+uPRTD/Q3fw3a
4vYiyKSYijvHYlS0mi/mtuCMBIgUXHRKYT4SkR7W6SQlwy10QiRZUGxofnzUicO1
eUDTNa1/pEecci7RXJ3C54aih0nrUZI5mQZdaFBBX1D7h3DAnTHKcnqzeQ0u/Wun
3Gf4MfVVYOfn9yyYmr7Q
=bdyD
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Exception in thread "http-bio-8080-exec-120" java.lang.OutOfMemoryError: PermGen space

[Kaushal Shriyan]

Hi,

I am seeing the below PermGen space issue in tomcat catalina.out file.
Currently i am running Apache Tomcat Version 7.0.59 on CentOS release 6.8
(Final)
java -version
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)

Exception in thread "http-bio-8080-exec-120" java.lang.OutOfMemoryError:
PermGen space
Exception in thread "http-bio-8080-exec-165" java.lang.OutOfMemoryError:
PermGen space

2016-11-28 13:40:30.543 ERROR 19011 --- [o-8080-exec-110]
o.s.boot.context.web.ErrorPageFilter : Forwarding to error page from
request [/push-notifications] due to exception [PermGen space]
java.lang.OutOfMemoryError: PermGen space


Is there a way to debug PermGen space issue and are there any tools to
debug the same. Currently i have set -XX:MaxPermSize=1024m. How do i
determine what value should be set for MaxPermSize?

Any help will be highly appreciable. Please do let me know if you need any
additional details.

Thanks in Advance

Regards,

Kaushal

Re: Apache/Tomcat vulnerability

[tomcat]

On 28.11.2016 20:34, Jaaz Portal wrote:

hi mark,
yes, i understand now what slowloris attack is.
maybe it was this maybe *this one based on * * mod_proxy denial of service *
CVE-2014-0117 


You keep on saying this, but the description of that vulnerability of *Apache httpd*, and 
the symptoms which you described, *do not match*.
You described the problem as ocurring in Apache tomcat, which in your case is sitting as a 
back-end, *behind* Apache httpd. And restarting tomcat cured the problem.


The CVE above applies to Apache httpd, and describes how an attacker could attack Apache 
httpd and cause *its children* processes to crash (the children processes of Apache 
httpd), by leading them to consume a lot of memory and crash with an out-of-memory error.
Granted, the problem occurred in the mod_proxy module of Apache httpd; but it was httpd 
which crashed, not tomcat.
And tomcat processes are not "Apache httpd children processes" in any understanding of the 
term.


As far as I remember, you never mentioned Apache httpd crashing. You mentioned "the pool" 
getting full or satured or something like that, without ever describing properly what you 
meant by "the pool".


As far as I am concerned, according to all the relatively unspecific information which you 
have previously provided :
1) the attack - if that is what it is/was - is definitely NOT related to the CVE which you 
have repeatedly mentioned
2) it is apparently not a "classical" DoS or "slowloris DoS" directed at your front-end 
Apache. Instead, it seems that the requests are properly received by Apache, properly 
decoded by Apache, and then whatever Apache proxy module you are using (mod_proxy_http, 
mod_proxy_ajp or mod_jk) is properly forwarding these requests to a back-end tomcat; and 
it is at the level of that back-end tomcat that the requests never seem to end, and in the 
end paralyse your tomcat server (and later on maybe your Apache httpd server too, because 
it is also waiting for tomcat to respond).


So your very way of describing the problem, in terms of "first we used this proxy module, 
and then they exploited the vulnerability so and so; then we changed the proxy module, and 
they exploited that too; etc.."
seems to not have anything to do with the problem per se, and (I believe) confuses 
everyone, including yourself.


It is not that "they" exploited different vulnerabilities of various httpd proxy modules, 
one after the other. Each of these proxy modules was doing its job properly, and 
forwarding valid HTTP requests properly to tomcat. When you changed from one proxy module 
to another, you did not really change anything in that respect, because any proxy module 
would do the same.


But in all cases, what did not change, was the tomcat behind the front-end, and the 
application running on that tomcat.  So the presumed attackers did not have to change 
anything, they just kept on sending the same requests, because they were really targeting 
your back-end tomcat or the tomcat application in it, no matter /how/ you were forwarding 
requests from Apache httpd to tomcat.


So either it is tomcat itself, which has a problem with some request URLs which do not 
bother Apache httpd (possible, but statistically unlikely), or it is the application which 
runs in tomcat that has such a problem (statistically, much more likely).



we do not know yet

we have setup more logging and are waiting for them to attack once again


Yes, that is the right thing to do.  Before deciding what the problem may be, and what you 
can do about it, the first thing you need is *data*.  You need to know

- which request URL(s?) cause that problem
- which IPs these requests come from (always the same ? multiple IPs that change all the 
time ? how many ? can these IPs be valid/expected clients or not ? do these IPs look like 
some "coordinated group" ?)

- how many such requests there may be during some period of time (10, 100, 
1000, more ?)
- if these URLs result in passing the request to tomcat
- what tomcat application (if any) they are directed to
- if so, when that application receives such a request, what is it supposed to do ? does 
it do it properly ? how long does it need, to respond to such a request ?


You also need to ask yourself a question : is the application which you run inside tomcat 
something that you designed yourself (and which hackers are unlikely to know well-enough 
to find such a URL which paralyses your server) ? or is it some well-known third-party 
java application which you are running (and for which would-be attackers would be much 
more likely to know exactly such a bug) ?




anyway, thank you for all informations, it was very useful and educational
reading for all of us

best wishes,
artur

2016-11-28 19:46 GMT+01:00 Mark Eggers :


Jaaz,

On 11/27/2016 2:46 PM, André Warnier (tomcat) wrote:

On 27.11.2016 19:03, Jaaz Portal 

Re: Apache/Tomcat vulnerability

[Jaaz Portal]

hi Andre,
you are wrong. This vulnerability is not only causing memory leaks, it
makes also apache workers to hang, making it easy to exhaust the pool.
what i have in my log files. But it is true also that such exhaustion can
be made by other forms of dos attacks described in this thread.

regarding you suggestion on our application, it does not dos bind server
nether does not scan for various vulnerabilities in apache, what i have
also in the logs

kindly regards,
artur

2016-11-28 21:33 GMT+01:00 André Warnier (tomcat) :

> On 28.11.2016 20:34, Jaaz Portal wrote:
>
>> hi mark,
>> yes, i understand now what slowloris attack is.
>> maybe it was this maybe *this one based on * * mod_proxy denial of
>> service *
>> CVE-2014-0117 > cvename.cgi?name=CVE-2014-0117>
>>
>
> You keep on saying this, but the description of that vulnerability of
> *Apache httpd*, and the symptoms which you described, *do not match*.
> You described the problem as ocurring in Apache tomcat, which in your case
> is sitting as a back-end, *behind* Apache httpd. And restarting tomcat
> cured the problem.
>
> The CVE above applies to Apache httpd, and describes how an attacker could
> attack Apache httpd and cause *its children* processes to crash (the
> children processes of Apache httpd), by leading them to consume a lot of
> memory and crash with an out-of-memory error.
> Granted, the problem occurred in the mod_proxy module of Apache httpd; but
> it was httpd which crashed, not tomcat.
> And tomcat processes are not "Apache httpd children processes" in any
> understanding of the term.
>
> As far as I remember, you never mentioned Apache httpd crashing. You
> mentioned "the pool" getting full or satured or something like that,
> without ever describing properly what you meant by "the pool".
>
> As far as I am concerned, according to all the relatively unspecific
> information which you have previously provided :
> 1) the attack - if that is what it is/was - is definitely NOT related to
> the CVE which you have repeatedly mentioned
> 2) it is apparently not a "classical" DoS or "slowloris DoS" directed at
> your front-end Apache. Instead, it seems that the requests are properly
> received by Apache, properly decoded by Apache, and then whatever Apache
> proxy module you are using (mod_proxy_http, mod_proxy_ajp or mod_jk) is
> properly forwarding these requests to a back-end tomcat; and it is at the
> level of that back-end tomcat that the requests never seem to end, and in
> the end paralyse your tomcat server (and later on maybe your Apache httpd
> server too, because it is also waiting for tomcat to respond).
>
> So your very way of describing the problem, in terms of "first we used
> this proxy module, and then they exploited the vulnerability so and so;
> then we changed the proxy module, and they exploited that too; etc.."
> seems to not have anything to do with the problem per se, and (I believe)
> confuses everyone, including yourself.
>
> It is not that "they" exploited different vulnerabilities of various httpd
> proxy modules, one after the other. Each of these proxy modules was doing
> its job properly, and forwarding valid HTTP requests properly to tomcat.
> When you changed from one proxy module to another, you did not really
> change anything in that respect, because any proxy module would do the same.
>
> But in all cases, what did not change, was the tomcat behind the
> front-end, and the application running on that tomcat.  So the presumed
> attackers did not have to change anything, they just kept on sending the
> same requests, because they were really targeting your back-end tomcat or
> the tomcat application in it, no matter /how/ you were forwarding requests
> from Apache httpd to tomcat.
>
> So either it is tomcat itself, which has a problem with some request URLs
> which do not bother Apache httpd (possible, but statistically unlikely), or
> it is the application which runs in tomcat that has such a problem
> (statistically, much more likely).
>
> we do not know yet
>>
>> we have setup more logging and are waiting for them to attack once again
>>
>
> Yes, that is the right thing to do.  Before deciding what the problem may
> be, and what you can do about it, the first thing you need is *data*.  You
> need to know
> - which request URL(s?) cause that problem
> - which IPs these requests come from (always the same ? multiple IPs that
> change all the time ? how many ? can these IPs be valid/expected clients or
> not ? do these IPs look like some "coordinated group" ?)
> - how many such requests there may be during some period of time (10, 100,
> 1000, more ?)
> - if these URLs result in passing the request to tomcat
> - what tomcat application (if any) they are directed to
> - if so, when that application receives such a request, what is it
> supposed to do ? does it do it properly ? how long does it need, to respond
> to such a request ?
>
> You also need to ask 

Re: Thread-safety of javax.servlet.Servlet#getServletConfig()

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Péter,

On 11/28/16 11:01 AM, Péter Gergely Horváth wrote:
> Thank you very much for your feedback, but I think there is a
> slight misunderstanding here: I know the order in which a servlet
> is initialized and put into service, and this question is not
> related to that. It's related to the _visibility_ of the field.
> 
> The question is about the thread safety of the field access: if
> you carefully check the details of
> javax.servlet.GenericServlet#config and compare the implementation
> to the sample "NoVisibility" from the book Java Concurrency in
> Practice, then it is seems to follow the same pattern, which is
> "not thread-safe because the value field is accessed from both get
> and set without synchronization. Among other hazards, it is
> susceptible to stale values: if one thread calls set, other threads
> calling get may or may not see that update." [1].
> 
> Personally, I would like to understand why this implementation is
> not (if not) susceptible to the stale values phenomenon [2]; there
> might be someone, who can point me to the right direction.

I think you are correct that there is a theoretical multi-threaded
race-condition, here. The container may initialize the servlet in one
thread and service e.g. the first request in another thread, and the
ServletContext reference might not be written to main memory and then
read-back by the request-processing thread.

Potential fixes for this would be either to define the ServletContext
member to be volatile or to use synchronized methods.'

Adding synchronization to the init() method would not be a problem at
all: there is very little monitor contention at that stage and the
container would only expect to call that method a single time. Adding
synchronization to the getServletContext method, though, might
represent a reduction in performance, since getServletContext gets
called many times during the lifetime of a Servlet, and from many thread
s.

Likewise, marking the ServletContext member as "volatile" would
necessarily require a slow main-memory read every time
getServletContext is called.

I suspect simple analysis above is the reason for no multithreaded
protection being placed on the ServletContext member in question.

Péter, is this an academic discussion, or have you in fact seen a case
where a servlet has been initialized and yet the first thread to
process a request receives a null value when calling getServletContext?

- -chris

> On Mon, Nov 28, 2016 at 6:08 AM, andreas 
> wrote:
> 
>>  On Fri, 25 Nov 2016 23:02:00 +0930 Péter Gergely Horváth
>> wrote 
>>> Hi All,
>>> 
>>> I am wondering why calling
>>> javax.servlet.Servlet#getServletConfig() is thread safe: if you
>>> check the implementation in javax.servlet.GenericServlet from
>>> servlet API 3.0.1, you see the
>> following:
>>> 
>>> package javax.servlet;
>>> 
>>> // lines omitted
>>> 
>>> public abstract class GenericServlet implements Servlet,
>>> ServletConfig, java.io.Serializable { // lines omitted
>>> 
>>> private transient ServletConfig config;
>>> 
>>> // lines omitted
>>> 
>>> public void init(ServletConfig config) throws ServletException
>>> { this.config = config; this.init(); }
>>> 
>>> // lines omitted
>>> 
>>> public ServletConfig getServletConfig() { return config; } //
>>> lines omitted }
>>> 
>>> 
>>> The field config is written in init(ServletConfig) without any 
>>> synchronization, locking or config being volatile, while
>> getServletConfig()
>>> could be called anytime from any later worker thread of the
>>> servlet container. I took a quick look into Tomcat/Catalina
>>> code base, however I see no indication of the servlet container
>>> performing any magic, which would guarantee thread safe access
>>> to field config through getServletConfig() from e.g. 
>>> javax.servlet.GenericServlet#service(ServletRequest,
>>> ServletResponse) and the corresponding methods in
>>> javax.servlet.http.HttpServlet.
>>> 
>>> Am I missing something here? What will guarantee that if Thread
>>> A is used to init(ServletConfig) then Thread B calling
>>> getServletConfig() will see the correct state of the field
>>> config (Java "happens-before"), and not
>> e.g.
>>> null?
>>> 
>>> I am asking this because I have seen some legacy code, where a
>>> servlet stored something into a field inside the init() method,
>>> which was then
>> used
>>> from within a javax.servlet.http.HttpServlet#doGet or 
>>> javax.servlet.http.HttpServlet#doPost method, without
>>> synchronization of any kind like this:
>>> 
>>> public class FoobarServlet extends HttpServlet {
>>> 
>>> private FoobarService foobarService;
>>> 
>>> @Override public void init() throws ServletException { 
>>> this.foobarService = // ... acquire foobarService } protected
>>> void doGet(HttpServletRequest request, HttpServletResponse 
>>> response) throws ServletException, IOException { List
>>> foobars = this.foobarService.getFoobars(); // read 

Re: Apache/Tomcat vulnerability

[Jaaz Portal]

hi mark,
yes, i understand now what slowloris attack is.
maybe it was this maybe *this one based on * * mod_proxy denial of service *
CVE-2014-0117 
we do not know yet

we have setup more logging and are waiting for them to attack once again

anyway, thank you for all informations, it was very useful and educational
reading for all of us

best wishes,
artur

2016-11-28 19:46 GMT+01:00 Mark Eggers :

> Jaaz,
>
> On 11/27/2016 2:46 PM, André Warnier (tomcat) wrote:
> > On 27.11.2016 19:03, Jaaz Portal wrote:
> >> 2016-11-27 18:30 GMT+01:00 André Warnier (tomcat) :
> >>
> >>> On 27.11.2016 14:26, Jaaz Portal wrote:
> >>>
>  hi,
>  everything i know so far is just this single log line that appeared in
>  apache error.log
> 
>  [Fri Nov 25 13:08:00.647835 2016] [mpm_event:error] [pid 13385:tid
>  1397934896385
>  92] AH00484: server reached MaxRequestWorkers setting, consider
> raising
>  the
>  MaxR
>  equestWorkers setting
> 
>  there was nothing else, just this strange line
> 
> >>>
> >>> This is not a "strange" line. It is telling you something.
> >>> One problem is that you seem convinced in advance, without serious
> >>> proof,
> >>> that there is a "bug" or a vulnerability in httpd or tomcat.
> >>> Read the explanation of the httpd parameter, here :
> >>> http://httpd.apache.org/docs/2.4/mod/mpm_common.html#maxrequestworkers
> >>> and try to understand it.
> >>>
> >>
> >> I understand it very well. We are serving no more that 500 clients per
> >> day
> >> so there was no other option that some kind of attack.
> >>
> >>
> >> About the "bug" or "vulnerability" : a webserver is supposed to serve
> >> HTTP
> >>> requests from clients.  That is what you expect of it. It has no
> >>> choice but
> >>> to accept any client connection and request, up to the maximum it can
> >>> handle considering its configuration and the capacity of the system on
> >>> which it runs. That is not a bug, it is a feature.
> >>>
> >>>
> >> We have some weeks ago come under attack from some Polish group. It was
> >> first bind that was DoS'ed, we was running on stable Debian so i updated
> >> bind to latest version. It did not helped. They has dos'ed it so we
> >> switched to other dns provider. That has helped.
> >>
> >> Then they exploited some well know vulnerability in mod_proxy. We have
> >> updated apache to the latest but again they has exploited it, so we have
> >> switched to mod_jk. And then guess what. They exploited it too so i
> >> decided
> >> to write to this list looking for help before trying jetty.
> >>
> >>
> >>>
> >>> The normal Apache httpd access log, will log a request only when it is
> >>> finished.  If the request never finishes, it will not get logged.
> >>> That may be why you do not see these requests in the log.
> >>> But have a look at this Apache httpd module :
> >>> http://httpd.apache.org/docs/2.4/mod/mod_log_forensic.html
> >>>
> >>
> >> ok, thanks, will take care
> >>
> >> Note : that is also why I was telling you to enable the mod_jk log,
> >> and to
> >>> examine it.
> >>> Because mod_jk will also log information before the request produces a
> >>> response.
> >>>
> >>>
> >>> and server hanged.
> >>>
> >>> Again, /what/ is "hanged" ? Apache httpd, or tomcat ?
> >>>
> >>
> >> Apache was accepting connection but not processed it. After restart of
> >> tomcat server it worked again.
> >>
> >>
> >>> Also in
> >>>
>  access logs there are no clues that it was under any heavy load.
> 
>  after around hour after discovering that our server hanged-out we have
>  restarted tomcat server
>  and it worked again
> 
> >>>
> >>> Yes, because that will close all connections between Apache httpd and
> >>> tomcat, and abort all requests that are in the process of being
> >>> processed
> >>> by tomcat. So mod_jk will get an error from tomcat, and will report an
> >>> error to httpd, and httpd will communicate that error to the clients,
> >>> and
> >>> close their connection.
> >>> It still does not tell you what the problem was.
> >>> The only thing that it suggests, is that the "bad" requests actually
> >>> make
> >>> it all the way to tomcat.
> >>>
> >>
> >> correct
> >>
> >> i will enable logs that you has pointed out and we will look what i will
> >> catch
> >> however i think we have only one chance, as if the solution we has found
> >> out (connection_timeout + mod_bn)
> >> will work they will stop exploiting it
> >>
> >> thank you very much for all the help and explanations
> >> i will report to the list new facts once they will attack us again
> >>
> >> best regards,
> >> artur
> >>
> >
> > Ok, but also read this e.g. :
> > https://www.corero.com/blog/695-going-after-the-people-
> behind-ddos-attacks.html
> >
> > Attempts to bring down a site by DoS attacks is a crime, in most places.
> > You can report it, while at the same 

Re: Apache/Tomcat vulnerability

[Mark Eggers]

Jaaz,

On 11/27/2016 2:46 PM, André Warnier (tomcat) wrote:
> On 27.11.2016 19:03, Jaaz Portal wrote:
>> 2016-11-27 18:30 GMT+01:00 André Warnier (tomcat) :
>>
>>> On 27.11.2016 14:26, Jaaz Portal wrote:
>>>
 hi,
 everything i know so far is just this single log line that appeared in
 apache error.log

 [Fri Nov 25 13:08:00.647835 2016] [mpm_event:error] [pid 13385:tid
 1397934896385
 92] AH00484: server reached MaxRequestWorkers setting, consider raising
 the
 MaxR
 equestWorkers setting

 there was nothing else, just this strange line

>>>
>>> This is not a "strange" line. It is telling you something.
>>> One problem is that you seem convinced in advance, without serious
>>> proof,
>>> that there is a "bug" or a vulnerability in httpd or tomcat.
>>> Read the explanation of the httpd parameter, here :
>>> http://httpd.apache.org/docs/2.4/mod/mpm_common.html#maxrequestworkers
>>> and try to understand it.
>>>
>>
>> I understand it very well. We are serving no more that 500 clients per
>> day
>> so there was no other option that some kind of attack.
>>
>>
>> About the "bug" or "vulnerability" : a webserver is supposed to serve
>> HTTP
>>> requests from clients.  That is what you expect of it. It has no
>>> choice but
>>> to accept any client connection and request, up to the maximum it can
>>> handle considering its configuration and the capacity of the system on
>>> which it runs. That is not a bug, it is a feature.
>>>
>>>
>> We have some weeks ago come under attack from some Polish group. It was
>> first bind that was DoS'ed, we was running on stable Debian so i updated
>> bind to latest version. It did not helped. They has dos'ed it so we
>> switched to other dns provider. That has helped.
>>
>> Then they exploited some well know vulnerability in mod_proxy. We have
>> updated apache to the latest but again they has exploited it, so we have
>> switched to mod_jk. And then guess what. They exploited it too so i
>> decided
>> to write to this list looking for help before trying jetty.
>>
>>
>>>
>>> The normal Apache httpd access log, will log a request only when it is
>>> finished.  If the request never finishes, it will not get logged.
>>> That may be why you do not see these requests in the log.
>>> But have a look at this Apache httpd module :
>>> http://httpd.apache.org/docs/2.4/mod/mod_log_forensic.html
>>>
>>
>> ok, thanks, will take care
>>
>> Note : that is also why I was telling you to enable the mod_jk log,
>> and to
>>> examine it.
>>> Because mod_jk will also log information before the request produces a
>>> response.
>>>
>>>
>>> and server hanged.
>>>
>>> Again, /what/ is "hanged" ? Apache httpd, or tomcat ?
>>>
>>
>> Apache was accepting connection but not processed it. After restart of
>> tomcat server it worked again.
>>
>>
>>> Also in
>>>
 access logs there are no clues that it was under any heavy load.

 after around hour after discovering that our server hanged-out we have
 restarted tomcat server
 and it worked again

>>>
>>> Yes, because that will close all connections between Apache httpd and
>>> tomcat, and abort all requests that are in the process of being
>>> processed
>>> by tomcat. So mod_jk will get an error from tomcat, and will report an
>>> error to httpd, and httpd will communicate that error to the clients,
>>> and
>>> close their connection.
>>> It still does not tell you what the problem was.
>>> The only thing that it suggests, is that the "bad" requests actually
>>> make
>>> it all the way to tomcat.
>>>
>>
>> correct
>>
>> i will enable logs that you has pointed out and we will look what i will
>> catch
>> however i think we have only one chance, as if the solution we has found
>> out (connection_timeout + mod_bn)
>> will work they will stop exploiting it
>>
>> thank you very much for all the help and explanations
>> i will report to the list new facts once they will attack us again
>>
>> best regards,
>> artur
>>
> 
> Ok, but also read this e.g. :
> https://www.corero.com/blog/695-going-after-the-people-behind-ddos-attacks.html
> 
> Attempts to bring down a site by DoS attacks is a crime, in most places.
> You can report it, while at the same time trying to defend yourself
> against it.
> 
> It is also relatively easy, and quite inexpensive in terms of system
> resources, to run a small shell script which takes a list every few
> seconds of the connections to the port of your webserver, and which IPs
> they are coming *from*.
> E.g.
> First try the netstat command, to see what it lists, like :
> # netstat -n --tcp | more
> 
> Then you will want to filter this a bit, to only consider established
> connections to your webserver, for example :
> # netstat -n --tcp | grep ":80" | grep "ESTABLISHED"
> 
> Then you will want to send this to a logfile, regularly, like this :
> 
> # date >> some_file.log
> # netstat -n --tcp | grep ":80" | grep "ESTABLISHED" >> some_file.log
> 

Re: Apache/Tomcat vulnerability

[Jaaz Portal]

hi,
i written "exploited some well know vulnerability in mod_proxy" not mod_jk.

This one:
*moderate: * * mod_proxy denial of service * CVE-2014-0117


A flaw was found in mod_proxy in httpd versions 2.4.6 to 2.4.9. A remote
attacker could send a carefully crafted request to a server configured as a
reverse proxy, and cause the child process to crash. This could lead to a
denial of service against a threaded MPM.
we have updated apache and this module but they exploited it again with
newest version (sorry, no time to investigate this)
we switched to mod_jk but they exploited it too, rest you can read on list

anyway we have configured apache to mitigate such kind of attack (whatever
flavor it is)

best,
artur

2016-11-28 20:11 GMT+01:00 Christopher Schultz :

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Jaaz,
>
> On 11/27/16 1:03 PM, Jaaz Portal wrote:
> > Then they exploited some well know vulnerability in mod_proxy. We
> > have updated apache to the latest but again they has exploited it,
> > so we have switched to mod_jk. And then guess what. They exploited
> > it too so i decided to write to this list looking for help before
> > trying jetty.
>
> I'm unaware of any vulnerabilities in mod_jk, "well-known" or
> otherwise. What vulnerability are you talking about?
>
> Note that switching to Jetty won't solve anything if you think the
> "well-known vulnerability" is in mod_jk.
>
> - -chris
> -BEGIN PGP SIGNATURE-
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBCAAGBQJYPIFQAAoJEBzwKT+lPKRYUPkP/iV5vbBj/v3bDe4Yp9qHO0dA
> 5p740/FRYFicM6fZf+SZ8dqsnPKZ30Ij6PTFfwNRFkGHiszErY1RZ94s3ZdXYJRF
> QxNiSvToQq750ujwkDL4iZv8h4Ash0TRA/E6e0SvZ9W5yjT/FBSfxsSKe8uixN43
> 6xcXaJcTuipjoyNHnVlUFkZFcmMykg7hktcMeaGp9OI2j/4x+OkkWsTjxvEucDXx
> 9ediRNaldrnU+bvmMTKnI9FZSHmxg1hzkpky3gNOv+CF7gOCLo8/ixj8ZELb+Lwt
> AREr30gtBdupfmGLO0pdFJ2EYHfmfW3Rlf6xwCzSBHZtdX8ewR1dw28SCJDlq/um
> vRArMdG+s3OoLQ04GHRm+shom+X3KuhRf/NjZsF//we6XIpgxWv686HaVXPNa0Ys
> 6TomDIDM1N0o5l0mQ1799P2+/9t+hxUoGUcUIOUadn4QHtuGFONe266qYiguhuFj
> sDVsi7R6jK4VodTA2HznKrdUZf1WqE2bw5zLSi7AOuoSu85jDfN+uPRTD/Q3fw3a
> 4vYiyKSYijvHYlS0mi/mtuCMBIgUXHRKYT4SkR7W6SQlwy10QiRZUGxofnzUicO1
> eUDTNa1/pEecci7RXJ3C54aih0nrUZI5mQZdaFBBX1D7h3DAnTHKcnqzeQ0u/Wun
> 3Gf4MfVVYOfn9yyYmr7Q
> =bdyD
> -END PGP SIGNATURE-
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Apache/Tomcat vulnerability

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Jaaz,

On 11/28/16 2:24 PM, Jaaz Portal wrote:
> hi, i written "exploited some well know vulnerability in mod_proxy"
> not mod_jk.

Yes but then you implied that mod_jk had the same problem:

On 11/27/16 1:03 PM, Jaaz Portal wrote:
> Then they exploited some well know vulnerability in mod_proxy. We 
> have updated apache to the latest but again they has exploited it,
> so we have switched to mod_jk. *And then guess what. They exploited
> it too* so i decided to write to this list looking for help before
>  trying jetty.

(emphasis mine).

> we switched to mod_jk but they exploited it too, rest you can read
> on list

I have read all your posts, and I see no evidence of an exploit.

You have posted a single line from your log file which tells us next
to nothing:

> [Fri Nov 25 13:08:00.647835 2016] [mpm_event:error] [pid 13385:tid
>  139793489638592] AH00484: server reached MaxRequestWorkers
> setting, consider raising the MaxRequestWorkers setting

Okay, you ran out of workers, meaning that you have a large number of
incoming requests. This indicates neither a vulnerability nor an exploit
.

> anyway we have configured apache to mitigate such kind of attack
> (whatever flavor it is)

Great. How did you do that? This is a community of users who support
each other, not a Help Desk. So please help the community by
explaining what the problem was and what the solution was as well.

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJYPLZoAAoJEBzwKT+lPKRYSdkP/2wqolHXAjHGdYiY3qlQVwt5
0ND8fSnZQmSJROFwd1foGAiiAlSHsQV30QML6AgLzlhpTnyb7rOEE1VohZl5QMKv
Z3No4N3LXH/c9OtoYsf3A18h5GyvXzG0abRaI9dbIYLjqrv1XAR0J3cDrvmmirqW
Nh22fC0aYNJk68MbREgO8t6D2TIWEPqLyXAJAxc6nRtCaWaMJ22BnGH4rPmPcV8h
KzuJ6oxjSzSzewJGRDuZsdpLqJl+TvOGRzuv4PTKWnXmUXs1DeBLaUJHOmNeuGKB
WW18GlIwOOhVxdPY8qatiIuInotO6w0v1nL1r9nqV+d/3gA9Rl/myH/Vi5tWa6Io
Ca30NBKEk4A3xXmuJyTwryqJlK4QHP2JjjLCGQvYu8/fsNLzJPBViHRKQvgM+Yjt
YT46Kels0mXGtgLE8D4rhgA+Zu7wrIAtHKWh7pXZ7lBm3RQB31e6elkwfGDaSk2v
yEbXKWDWkfxPbIG9e/fxbZT/pgfE6hOMCL94l7oUGvxLxjEzVBzkuTjHBYV7zGpA
klxWOHrOJWtWn0KDmaU21jeuVzWZjKr/HVDnMGyAo2p9svNTuB/yLx+aToXFe2Sq
hrpZZoa61Jcdq+tU0maDq01PvUfeLW/fxcSjTXChZpuEFv5IUo6NKP5GoxqC3wrv
wDNqKX5IyyKJXVupnsB4
=q1fb
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Exception in thread "http-bio-8080-exec-120" java.lang.OutOfMemoryError: PermGen space

[Román Valoria]

Use Java VisualVM to see PermGen memory allocation and troubleshoot. It
would also let you validate that your sizing parameters are properly set.

On Tue, Nov 29, 2016 at 2:09 AM, Kaushal Shriyan 
wrote:

> Hi,
>
> I am seeing the below PermGen space issue in tomcat catalina.out file.
> Currently i am running Apache Tomcat Version 7.0.59 on CentOS release 6.8
> (Final)
> java -version
> java version "1.7.0_80"
> Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
> Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
>
> Exception in thread "http-bio-8080-exec-120" java.lang.OutOfMemoryError:
> PermGen space
> Exception in thread "http-bio-8080-exec-165" java.lang.OutOfMemoryError:
> PermGen space
>
> 2016-11-28 13:40:30.543 ERROR 19011 --- [o-8080-exec-110]
> o.s.boot.context.web.ErrorPageFilter : Forwarding to error page from
> request [/push-notifications] due to exception [PermGen space]
> java.lang.OutOfMemoryError: PermGen space
>
>
> Is there a way to debug PermGen space issue and are there any tools to
> debug the same. Currently i have set -XX:MaxPermSize=1024m. How do i
> determine what value should be set for MaxPermSize?
>
> Any help will be highly appreciable. Please do let me know if you need any
> additional details.
>
> Thanks in Advance
>
> Regards,
>
> Kaushal
>

Re: Exception in thread "http-bio-8080-exec-120" java.lang.OutOfMemoryError: PermGen space

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Kaushal,

On 11/28/16 1:09 PM, Kaushal Shriyan wrote:
> Hi,
> 
> I am seeing the below PermGen space issue in tomcat catalina.out 
> file. Currently i am running Apache Tomcat Version 7.0.59 on
> CentOS release 6.8 (Final) java -version java version "1.7.0_80"
> Java(TM) SE Runtime Environment (build 1.7.0_80-b15) Java
> HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
> 
> Exception in thread "http-bio-8080-exec-120" 
> java.lang.OutOfMemoryError: PermGen space Exception in thread 
> "http-bio-8080-exec-165" java.lang.OutOfMemoryError: PermGen space
> 
> 2016-11-28 13:40:30.543 ERROR 19011 --- [o-8080-exec-110] 
> o.s.boot.context.web.ErrorPageFilter : Forwarding to error
> page from request [/push-notifications] due to exception [PermGen 
> space] java.lang.OutOfMemoryError: PermGen space
> 
> 
> Is there a way to debug PermGen space issue and are there any
> tools to debug the same. Currently i have set
> -XX:MaxPermSize=1024m. How do i determine what value should be set
> for MaxPermSize?
> 
> Any help will be highly appreciable. Please do let me know if you 
> need any additional details.

Do you do hot re-deployments? If you do and your application does not
unload cleanly, then you can get ClassLoader-pinning leaks.

In a dev environment, run your app for a while and then perform a
hot-reload, then use Tomcat's manager application to "check for
leaks". If your application is still hanging around, it will tell you.

If you don't use hot-deployment or you don't find any leaks when you
do hot re-deployments, then your problem lies in either of two places:

1. Heavy use of RMI to load classes across the network
(java.lang.Class objects for proxies, etc. can lead to PermGen exhaustio
n)

2. You just need more PermGen. Frameworks such as Spring are known to
require surprisingly-high amounts of PermGen space because of the many
java.lang.Class objects loaded by them.

If you are under situation #2, my recommendation is simply to increase
the PermGen space by a factor of 2 until you stop getting OOME:PermGen
errors, then observe the PermGen space after your webapp has been
running for a while (maybe a week or two). The current usage of
PermGen at that point will be roughly your target. Always remember to
leave a little extra space "just in case".

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJYPPdAAAoJEBzwKT+lPKRYLc8P/ieD4bGjJoBrIMJs3Frdjh9W
r3lPfltFgzYEWfu0MYhfs26bXG0Ivk23w614WZbw70otNB+x25bqdxp+yRsHBVmg
jQxPt4+olXJqsHRQU2rQbLSkHbN+UWp5UYlbMUzM8uGFl6kGkK6vFeOMPmd9a50Z
WE6LGVkqAJSEJivYPqkAmldiT8r3QoR3cjB5sSwHaPs8Hjp2uQCZVMATScC5Daz7
/3lLu89a6VG2sKctYqYeUeS01FfivXZ6KuHE0df46stTBDYz6KlrDF2yKz2G22Rb
dwckx7yYm+lnzrxlUpMv1N3vqZTAOwtqehZ3WS5TiZ4fCCHFwzBYy2/hxWU3hamq
Vv0wc9npkJDDQXnCJsOm1vjcekAQwAeJ6KjBVgqpCScv1AK5QH1WYkoDOlC3ovYm
BFNKVtIhhhrKhpA2OeyVvEafZ5ZpMpD0u02NJGYr4NXUWFsdsrDxVOEg4zxHUb02
qpM/Vm4U7M3d0aC4g4yVBtutEToBjSAVtXDqB8b8CVILlGzZjhbDlwizqeymcaAM
zasObN1q49U8CbBFlSDbNE6T88G14ScxY+IvYJDG9PZNfK1m64cMppbnxRaK60Tn
ej1wKGogzMIVkHDLwbLse9BgrkLWOYL9jvXAY17W9Hg0745vmTZluPtmAPJBVylf
Xt9lNhwX/qWKTrqKV/Ut
=e27x
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Exception in thread "http-bio-8080-exec-120" java.lang.OutOfMemoryError: PermGen space

[Kaushal Shriyan]

On Tue, Nov 29, 2016 at 7:40 AM, Román Valoria 
wrote:

> Use Java VisualVM to see PermGen memory allocation and troubleshoot. It
> would also let you validate that your sizing parameters are properly set.
>
>
Hi Roman,

Thanks for your reply. Are you referring to https://visualvm.github.io/?
Please comment.

Regards,

Kaushal

Re: Exception in thread "http-bio-8080-exec-120" java.lang.OutOfMemoryError: PermGen space

[Kaushal Shriyan]

Thanks Roman and Chris for the detailed explanation. Is there a way to find
out what all java classes are loaded during runtime?

Thanks in Advance.

Regards,

Kaushal

On Tue, Nov 29, 2016 at 10:47 AM, Román Valoria 
wrote:

> Sun, or should I say Oracle now, seems to be including the exact same tool
> on the JDK. It is already pre-packaged, so it may be customized by Oracle.
> It has the same look and feel as in the screenshot.
>
> On Tue, Nov 29, 2016 at 11:17 AM, Kaushal Shriyan <
> kaushalshri...@gmail.com>
> wrote:
>
> > On Tue, Nov 29, 2016 at 7:40 AM, Román Valoria 
> > wrote:
> >
> > > Use Java VisualVM to see PermGen memory allocation and troubleshoot. It
> > > would also let you validate that your sizing parameters are properly
> set.
> > >
> > >
> > Hi Roman,
> >
> > Thanks for your reply. Are you referring to https://visualvm.github.io/?
> > Please comment.
> >
> > Regards,
> >
> > Kaushal
> >
>

Re: Exception in thread "http-bio-8080-exec-120" java.lang.OutOfMemoryError: PermGen space

[Román Valoria]

Get the JDK launch the tool and look it up yourself. It has tons of issues,
am I sure it has that.

On Tue, Nov 29, 2016 at 2:56 PM, Kaushal Shriyan 
wrote:

> Thanks Roman and Chris for the detailed explanation. Is there a way to find
> out what all java classes are loaded during runtime?
>
> Thanks in Advance.
>
> Regards,
>
> Kaushal
>
> On Tue, Nov 29, 2016 at 10:47 AM, Román Valoria 
> wrote:
>
> > Sun, or should I say Oracle now, seems to be including the exact same
> tool
> > on the JDK. It is already pre-packaged, so it may be customized by
> Oracle.
> > It has the same look and feel as in the screenshot.
> >
> > On Tue, Nov 29, 2016 at 11:17 AM, Kaushal Shriyan <
> > kaushalshri...@gmail.com>
> > wrote:
> >
> > > On Tue, Nov 29, 2016 at 7:40 AM, Román Valoria  >
> > > wrote:
> > >
> > > > Use Java VisualVM to see PermGen memory allocation and troubleshoot.
> It
> > > > would also let you validate that your sizing parameters are properly
> > set.
> > > >
> > > >
> > > Hi Roman,
> > >
> > > Thanks for your reply. Are you referring to
> https://visualvm.github.io/?
> > > Please comment.
> > >
> > > Regards,
> > >
> > > Kaushal
> > >
> >
>

Re: Exception in thread "http-bio-8080-exec-120" java.lang.OutOfMemoryError: PermGen space

[Román Valoria]

Sun, or should I say Oracle now, seems to be including the exact same tool
on the JDK. It is already pre-packaged, so it may be customized by Oracle.
It has the same look and feel as in the screenshot.

On Tue, Nov 29, 2016 at 11:17 AM, Kaushal Shriyan 
wrote:

> On Tue, Nov 29, 2016 at 7:40 AM, Román Valoria 
> wrote:
>
> > Use Java VisualVM to see PermGen memory allocation and troubleshoot. It
> > would also let you validate that your sizing parameters are properly set.
> >
> >
> Hi Roman,
>
> Thanks for your reply. Are you referring to https://visualvm.github.io/?
> Please comment.
>
> Regards,
>
> Kaushal
>

RE: Issues with accessing Apache Httpd Modules

[kruthika.krishnan]

Hi Torsten,

This has been tried from various networks , and its still not accessible.

https://modules.apache.org/

The above is the correct url used.

Thanks and Regards,
Kruthika Krishnan

-Original Message-
From: Torsten Krah [mailto:krah...@gmail.com] 
Sent: Monday, November 28, 2016 1:41 PM
To: users@tomcat.apache.org
Subject: Re: Issues with accessing Apache Httpd Modules

This is the tomcat list, not the httpd one ... but your favorite search engine 
would easily have answered your question:

https://urldefense.proofpoint.com/v2/url?u=http-3A__httpd.apache.org_docs_2.4_mod_=DgICaQ=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU=mdDSftlpf-8PG7e1HDL1_xvgL8zGPczesb7TcBZsNcg=SHnSxCR9jWEPOB1SQ5MFKM9Uk6Hd6VywMlr0Mm2I5FY=6iBxLBdEjWK7BJxW_QWYZPkbR-Pbq2JrZ0vPqkEDrc4=
 

Cheers

Torsten

Am Montag, den 28.11.2016, 07:18 + schrieb
kruthika.krish...@accenture.com:
> Hi ,
> I am trying to access the apache HTTPD modules for Apache version 2.4, 
> through the URL 
> https://urldefense.proofpoint.com/v2/url?u=http-3A__modules.apache.org
> _=DgICaQ=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU=mdDSftlpf-8
> PG7e1HDL1_xvgL8zGPczesb7TcBZsNcg=SHnSxCR9jWEPOB1SQ5MFKM9Uk6Hd6VywMlr
> 0Mm2I5FY=MZ-6cZ-Zm1e6YdyqzaAI9EsdQeMa7peRXmi_5JQTJQ8=
> It shows an error:
> Service Unavailable
> The server is temporarily unable to service your request due to maintenance 
> downtime or capacity problems. Please try again later.
> Is there any other way to access the Apache modlues the modules I want to 
> access are : mod_dav and mod_dav_fs.
> 
> Thanks and Regards,
> Kruthika Krishnan
> 
> 
> 
> This message is for the designated recipient only and may contain privileged, 
> proprietary, or otherwise confidential information. If you have received it 
> in error, please notify the sender immediately and delete the original. Any 
> other use of the e-mail by you is prohibited. Where allowed by local law, 
> electronic communications with Accenture and its affiliates, including e-mail 
> and instant messaging (including content), may be scanned by our systems for 
> the purposes of information security and assessment of internal compliance 
> with Accenture policy.
> __
> 
> 
> www.accenture.com



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Issues with accessing Apache Httpd Modules

[Torsten Krah]

This is the tomcat list, not the httpd one ... but your favorite search
engine would easily have answered your question:

http://httpd.apache.org/docs/2.4/mod/

Cheers

Torsten

Am Montag, den 28.11.2016, 07:18 + schrieb
kruthika.krish...@accenture.com:
> Hi ,
> I am trying to access the apache HTTPD modules for Apache version 2.4, 
> through the URL
> http://modules.apache.org/
> It shows an error:
> Service Unavailable
> The server is temporarily unable to service your request due to maintenance 
> downtime or capacity problems. Please try again later.
> Is there any other way to access the Apache modlues the modules I want to 
> access are : mod_dav and mod_dav_fs.
> 
> Thanks and Regards,
> Kruthika Krishnan
> 
> 
> 
> This message is for the designated recipient only and may contain privileged, 
> proprietary, or otherwise confidential information. If you have received it 
> in error, please notify the sender immediately and delete the original. Any 
> other use of the e-mail by you is prohibited. Where allowed by local law, 
> electronic communications with Accenture and its affiliates, including e-mail 
> and instant messaging (including content), may be scanned by our systems for 
> the purposes of information security and assessment of internal compliance 
> with Accenture policy.
> __
> 
> www.accenture.com



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Tomcat not scaling under load - Tsung v Apache Bench test

[Szymon Czaja]

Hi,
I have asked the question on StackOverflow but I am not getting much
response:

http://stackoverflow.com/questions/40793335/why-tomcat-
does-not-pass-a-tsung-performance-test-at-40-requests-per-second

Could anyone help me understand why is Tomcat unable to keep up with the
processing when running Tsung yet Apache Bench tests do not relveal any
scalability issues?

Thank you,
Szymon

Re: Tomcat not scaling under load - Tsung v Apache Bench test

[Mark Thomas]

On 28/11/2016 09:53, Szymon Czaja wrote:
> Hi,
> I have asked the question on StackOverflow but I am not getting much
> response:
> 
> http://stackoverflow.com/questions/40793335/why-tomcat-
> does-not-pass-a-tsung-performance-test-at-40-requests-per-second
> 
> Could anyone help me understand why is Tomcat unable to keep up with the
> processing when running Tsung yet Apache Bench tests do not relveal any
> scalability issues?

I'd recommend taking some thread dumps and looking at netstat output to
try and figure out what is going on.

Mark


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org