Re: ALv2 Tomcat Training material

2018-01-05 Thread Don Flinn 
Hi Mark,

I think this is an excellent and useful task. The first step is to define
the audiences of which I would like to suggest five.

1. Experienced System administrators with experience in security and SSL
2. Experienced System administrators with no or little experience in
security and SSL
3  Non system administrators with  experience in security and SSL
4. Non system administrators with  no or little experience in security and
SSL
5. Overarching each of the above is the different CAs that might be used

A question would be are audiences 3 and 4 populated.  I for one fall
somewhere between 3 and 4, but maybe I'm the only one.  It seems to me that
small companies would fall into 3 and 4.  A big further complication is
item 5, which applies to each of the others.

I wrote a short write-up to this site about a month age addressing
audiences 2, 3 and 4, mostly 4 and was told that this was not what was
wanted, that all the information was somewhere on the web.  That is true
but trying to find it, absorb and apply it is difficult.

Don


On Thu, Jan 4, 2018 at 5:16 AM, Mark Thomas  wrote:

> Hi,
>
> One of the things on my TODO list is to put together some Tomcat
> training material licensed under the Apache License (version 2). i.e.
> material that would be made freely available for folks to use.
>
> I'd also like to make the training material available on YouTube as well
> as run some training courses (for a small fee) to deliver the material
> face to face.
>
> The structure I have in mind is a series of modules (say 30 mins in
> length) that can be organised in different ways to suit different needs.
> e.g. put the introductory modules for each area together to provide an
> 'Introduction to Tomcat course', put all the TLS modules together to
> provide an in depth 'Tomcat and TLS' course etc.
>
> I think a lot of the raw content is already available. We have the
> various Tomcat presentations that have been given over the years and my
> employer has agreed to let me make use of the material from our (now
> possibly a little dated) Tomcat training courses.
>
> I can't do this alone. Not in any reasonable time frame anyway. So I am
> reaching out to the community for help.
>
> The first step is to come with:
> - a list of modules
> - potential courses formed from combinations of modules
>
> I am asking for your ideas for modules, courses and combinations of
> modules that could make up those courses.
>
> We have a blank wiki page to host this:
> https://cwiki.apache.org/confluence/display/TOMCAT/Tomcat+Training+Course
>
> Feel free to ask for edit access to that page (you'll need to create an
> account and let us know the user name) so you can add ideas directly or
> add ideas to this thread and I'll add them to the wiki page.
>
> The second step is to start populating the modules with actual content.
> As a motivator to get this done, I'd like to run a public Tomcat
> training course in late March / early April using this material. My
> current thinking is that the course would cost ~£100 plus food per
> person for the full day. Possible locations for this course are:
> - Cardiff
> - Birmingham
> - Manchester
> - Glasgow
> (all in the UK - if successful we can expand to mainland Europe and beyond)
>
> My second request is for feedback on which location(s) are preferable
> and what content would you like to see in the training course. I'll take
> this feedback, put together a course and then make it available to book.
>
> I look forward to all your ideas.
>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Subscription to tomcat-users

2018-01-05 Thread Coty Sutherland 
Is there some reason why this user can't subscribe to the users@ list?
They found me in freenode and seemingly get no response from the list
emails (users@ and users-help@).

Cheers,

On Fri, Jan 5, 2018 at 3:40 PM, Alex  wrote:
> Hello,
>
> This is my address that doesn't get into the tomcat-users mailing list.
>  Can I please get subscribed?
>
> Thank you!
>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: [EXTERNAL]Re: org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRun Handshake failed

2018-01-05 Thread Rémy Maucherat 
On Fri, Jan 5, 2018 at 3:43 PM, Michael Peng <
michael.p...@entrustdatacard.com> wrote:

> Hi, Remy,
>
> Is  the fix available for 8.5.16? if not, could you please share the code
> changes?
>

This is the fix as the code seemed incorrect when thinking about it some
more.

Rémy

Re: internalProxies regex

2018-01-05 Thread Felix Schumacher 

Am 05.01.2018 um 15:43 schrieb Harrie Robins:

All clear.
I apologize, I was in fact not masking the backslashes, I did a wrong copy
paste from the pattern I was using in my test

I tested the following 2 patterns:

^103\.21\.(2(4[4-7]))\.([0-
9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|^103\.22
\.(2(0[0-3]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$

103\.21\.(2(4[4-7]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))|103\.22\.(2(0[0-3]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))


The regex can be "simplified" to

103\.21\.24[4-7]\.(1?[1-9]?[0-9]|2([0-4][0-9]|5[0-5]))|103\.22\.20[0-3]\.(1?[1-9]?[0-9]|2([0-4][0-9]|5[0-5]))

or even

103\.(21\.24[4-7]|22\.20[0-3])\.(1?[1-9]?[0-9]|2([0-4][0-9]|5[0-5]))

But it looks OK, if you want to match IPs from 103.21.244.x-103.21.247.x 
and 103.22.200.x-103.22.203.x


Have you enabled debug-logs for the RemoteIpValve? It should print out 
the IP it tries to match.


Regards,
 Felix



Regards,

Harrie



On 5 January 2018 at 14:46, Felix Schumacher <
felix.schumac...@internetallee.de> wrote:


Am 05.01.2018 um 09:47 schrieb Harrie Robins:


Hi Mark,

our tomcat application server are fronted by 1. cloudflare, and 2. amazon
load balancer.
In apache there is mod_remote IP and I can simply put in CIDR range:
https://www.cloudflare.com/ips/ that will swallow all those IP and will
get
the correct IP to tomcat.

In Tomcat I need
https://tomcat.apache.org/tomcat-8.5-doc/api/org/apache/cata
lina/valves/RemoteIpValve.html
which does not accept CIDR range however. I wrote a regex to match all the
addresses and it works, it's matching way to many addresses however so I
rewrote the pattern. My new pattern is not functioning however, so I
tested
then pattern in a small application.

In my test I made a list of all addresses  in this range:
^103\\.21\\.(2(4[4-7]))\\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2(
[0-4][0-9]|5[0-5]))$|^103\\.22\\.(2(0[0-3]))\\.([0-9]|[1-
9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$


If you configure the valve through the internalProxies attribute, you are
using 'real' strings and don't need to mask the backslashes as you would
have to do with java strings.

When you look at the documentation, you will find no double backslashes
there.

And  regarding the usage of the anchors '^' and '$'. They are not needed,
either. Tomcat will use match instead of find and thus they are implicitly
added.

Regards,
  Felix

||


I matched all these addresses and it works. When I set in tomcat however
it
does not, I have no understanding why not?

Hope you understand what I am trying to do.

thanks





On 2 January 2018 at 19:33, Mark Thomas  wrote:

On 02/01/18 09:50, Harrie Robins wrote:

I'm still having problems with matching my pattern.

Right now I'm feeding the following to internalProxies:

^103\\.21\\.(2(4[4-7]))\\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2(


[0-4][0-9]|5[0-5]))$|^103\\.22\\.(2(0[0-3]))\\.([0-9]|[1-
9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$


I created a list of all involved IP addresses and matched those IP


addresses:


java.util.regex.Matcher / java.util.regex.Pattern, please see


https://pastebin.com/Lija7n9k


All addresses from the list I created are matching, just not in tomcat.


What is the value of the remote IP address that is failing to match? You
might want to look at writing a short custom Valve to log that and
insert it into the Pipeline ahead of the RemoteIpValve.

Another option would be to simply remove the RemoteIpValve and write a
simple servlet that logs the remote IP.

Mark

Regards,

Harrie

-Oorspronkelijk bericht-
Van: Harrie Robins [mailto:har...@eyequestion.nl]
Verzonden: 21 December 2017 09:55
Aan: 'Tomcat Users List' 
Onderwerp: RE: internalProxies regex

This makes perfect sense.
I tested my regex, just against wrong engine.

Thanks for pointing me in the right direction

-Oorspronkelijk bericht-
Van: Konstantin Kolinko [mailto:knst.koli...@gmail.com]
Verzonden: 20 December 2017 15:19
Aan: Tomcat Users List 
Onderwerp: Re: internalProxies regex

2017-12-20 11:37 GMT+03:00 Harrie Robins :


Hello everyone,



I have a question about the remoteipvalve in tomcat 8.5:
https://tomcat.apache.org/tomcat-8.5-doc/api/org/apache/catalina/valve
s/Remo
teIpValve.html




internalProxies

Regular expression that matches the IP addresses of internal proxies.
If they appear in the remoteIpHeader value, they will be trusted and
will not appear in the proxiesHeader value

RemoteIPInternalProxy

Regular expression (in the syntax supported by java.util.regex)

10\.\d{1,3}\.\d{1,3}\.\d{1,3}|192\.168\.\d{1,3}\.\d{1,3}|
169\.254\.\d{1,3}\.\d{1,3}|127\.\d{1,3}\.\d{1,3}\.\d{1,3}|
172\.1[6-9]{1}\.\d{1,3}\.\d{1,3}|172\.2[0-9]{1}\.\d{1,3}\.\d{1,3}|
172\.3[0-1]{1}\.\d{1,3}\.\d{1,3}
By default, 10/8, 192.168/16, 169.254/16, 127/8 and 172.16/12 are


allowed.

I need to convert some CIDR ranges to regex:

Re: internalProxies regex

2018-01-05 Thread Harrie Robins 
All clear.
I apologize, I was in fact not masking the backslashes, I did a wrong copy
paste from the pattern I was using in my test

I tested the following 2 patterns:

^103\.21\.(2(4[4-7]))\.([0-
9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|^103\.22
\.(2(0[0-3]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$

103\.21\.(2(4[4-7]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))|103\.22\.(2(0[0-3]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))

Regards,

Harrie



On 5 January 2018 at 14:46, Felix Schumacher <
felix.schumac...@internetallee.de> wrote:

> Am 05.01.2018 um 09:47 schrieb Harrie Robins:
>
>> Hi Mark,
>>
>> our tomcat application server are fronted by 1. cloudflare, and 2. amazon
>> load balancer.
>> In apache there is mod_remote IP and I can simply put in CIDR range:
>> https://www.cloudflare.com/ips/ that will swallow all those IP and will
>> get
>> the correct IP to tomcat.
>>
>> In Tomcat I need
>> https://tomcat.apache.org/tomcat-8.5-doc/api/org/apache/cata
>> lina/valves/RemoteIpValve.html
>> which does not accept CIDR range however. I wrote a regex to match all the
>> addresses and it works, it's matching way to many addresses however so I
>> rewrote the pattern. My new pattern is not functioning however, so I
>> tested
>> then pattern in a small application.
>>
>> In my test I made a list of all addresses  in this range:
>> ^103\\.21\\.(2(4[4-7]))\\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2(
>> [0-4][0-9]|5[0-5]))$|^103\\.22\\.(2(0[0-3]))\\.([0-9]|[1-
>> 9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$
>>
>
> If you configure the valve through the internalProxies attribute, you are
> using 'real' strings and don't need to mask the backslashes as you would
> have to do with java strings.
>
> When you look at the documentation, you will find no double backslashes
> there.
>
> And  regarding the usage of the anchors '^' and '$'. They are not needed,
> either. Tomcat will use match instead of find and thus they are implicitly
> added.
>
> Regards,
>  Felix
>
> ||
>
>> I matched all these addresses and it works. When I set in tomcat however
>> it
>> does not, I have no understanding why not?
>>
>> Hope you understand what I am trying to do.
>>
>> thanks
>>
>>
>>
>>
>>
>> On 2 January 2018 at 19:33, Mark Thomas  wrote:
>>
>> On 02/01/18 09:50, Harrie Robins wrote:
>>>
 I'm still having problems with matching my pattern.

 Right now I'm feeding the following to internalProxies:

 ^103\\.21\\.(2(4[4-7]))\\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2(

>>> [0-4][0-9]|5[0-5]))$|^103\\.22\\.(2(0[0-3]))\\.([0-9]|[1-
>>> 9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$
>>>
 I created a list of all involved IP addresses and matched those IP

>>> addresses:
>>>
 java.util.regex.Matcher / java.util.regex.Pattern, please see

>>> https://pastebin.com/Lija7n9k
>>>
 All addresses from the list I created are matching, just not in tomcat.

>>> What is the value of the remote IP address that is failing to match? You
>>> might want to look at writing a short custom Valve to log that and
>>> insert it into the Pipeline ahead of the RemoteIpValve.
>>>
>>> Another option would be to simply remove the RemoteIpValve and write a
>>> simple servlet that logs the remote IP.
>>>
>>> Mark
>>>
>>> Regards,

 Harrie

 -Oorspronkelijk bericht-
 Van: Harrie Robins [mailto:har...@eyequestion.nl]
 Verzonden: 21 December 2017 09:55
 Aan: 'Tomcat Users List' 
 Onderwerp: RE: internalProxies regex

 This makes perfect sense.
 I tested my regex, just against wrong engine.

 Thanks for pointing me in the right direction

 -Oorspronkelijk bericht-
 Van: Konstantin Kolinko [mailto:knst.koli...@gmail.com]
 Verzonden: 20 December 2017 15:19
 Aan: Tomcat Users List 
 Onderwerp: Re: internalProxies regex

 2017-12-20 11:37 GMT+03:00 Harrie Robins :

> Hello everyone,
>
>
>
> I have a question about the remoteipvalve in tomcat 8.5:
> https://tomcat.apache.org/tomcat-8.5-doc/api/org/apache/catalina/valve
> s/Remo
> teIpValve.html
>
>
>
>
> internalProxies
>
> Regular expression that matches the IP addresses of internal proxies.
> If they appear in the remoteIpHeader value, they will be trusted and
> will not appear in the proxiesHeader value
>
> RemoteIPInternalProxy
>
> Regular expression (in the syntax supported by java.util.regex)
>
> 10\.\d{1,3}\.\d{1,3}\.\d{1,3}|192\.168\.\d{1,3}\.\d{1,3}|
> 169\.254\.\d{1,3}\.\d{1,3}|127\.\d{1,3}\.\d{1,3}\.\d{1,3}|
> 172\.1[6-9]{1}\.\d{1,3}\.\d{1,3}|172\.2[0-9]{1}\.\d{1,3}\.\d{1,3}|
> 172\.3[0-1]{1}\.\d{1,3}\.\d{1,3}
> By default, 10/8, 192.168/16, 169.254/16, 127/8 and 172.16/12 are
>
 allowed.
>>>

>
> I need to convert

RE: [EXTERNAL]Re: org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRun Handshake failed

2018-01-05 Thread Michael Peng 
Hi, Remy, 

Is  the fix available for 8.5.16? if not, could you please share the code 
changes?

Thanks,

Michael

-Original Message-
From: Rémy Maucherat [mailto:r...@apache.org] 
Sent: Friday, January 5, 2018 5:58 AM
To: Tomcat Users List 
Subject: [EXTERNAL]Re: 
org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRun Handshake failed

On Fri, Jan 5, 2018 at 8:38 AM, Rémy Maucherat  wrote:

> On Fri, Jan 5, 2018 at 12:25 AM, Michael Peng < 
> michael.p...@entrustdatacard.com> wrote:
>
>> Do the changes make sense, and what would be the side effect ? In our 
>> case, the "netInBuffer" could be full, i.e., postion = limit for 
>> large data. Maybe the "netInBuffer" should not be cleared since 
>> "compact" would reset the "netInBuffer", should it?
>>
>> The buffer is flipped after that and the NIO code is the same anyway, 
>> so
> the change doesn't make sense indeed as is.
>
> Looking at it made it seem not right compared to NIO, so I made 
> another
change and hopefully fixed it.

Rémy

Re: internalProxies regex

2018-01-05 Thread Felix Schumacher 

Am 05.01.2018 um 09:47 schrieb Harrie Robins:

Hi Mark,

our tomcat application server are fronted by 1. cloudflare, and 2. amazon
load balancer.
In apache there is mod_remote IP and I can simply put in CIDR range:
https://www.cloudflare.com/ips/ that will swallow all those IP and will get
the correct IP to tomcat.

In Tomcat I need
https://tomcat.apache.org/tomcat-8.5-doc/api/org/apache/catalina/valves/RemoteIpValve.html
which does not accept CIDR range however. I wrote a regex to match all the
addresses and it works, it's matching way to many addresses however so I
rewrote the pattern. My new pattern is not functioning however, so I tested
then pattern in a small application.

In my test I made a list of all addresses  in this range:
^103\\.21\\.(2(4[4-7]))\\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2(
[0-4][0-9]|5[0-5]))$|^103\\.22\\.(2(0[0-3]))\\.([0-9]|[1-
9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$


If you configure the valve through the internalProxies attribute, you 
are using 'real' strings and don't need to mask the backslashes as you 
would have to do with java strings.


When you look at the documentation, you will find no double backslashes 
there.


And  regarding the usage of the anchors '^' and '$'. They are not 
needed, either. Tomcat will use match instead of find and thus they are 
implicitly added.


Regards,
 Felix
||

I matched all these addresses and it works. When I set in tomcat however it
does not, I have no understanding why not?

Hope you understand what I am trying to do.

thanks





On 2 January 2018 at 19:33, Mark Thomas  wrote:


On 02/01/18 09:50, Harrie Robins wrote:

I'm still having problems with matching my pattern.

Right now I'm feeding the following to internalProxies:

^103\\.21\\.(2(4[4-7]))\\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2(

[0-4][0-9]|5[0-5]))$|^103\\.22\\.(2(0[0-3]))\\.([0-9]|[1-
9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$

I created a list of all involved IP addresses and matched those IP

addresses:

java.util.regex.Matcher / java.util.regex.Pattern, please see

https://pastebin.com/Lija7n9k

All addresses from the list I created are matching, just not in tomcat.

What is the value of the remote IP address that is failing to match? You
might want to look at writing a short custom Valve to log that and
insert it into the Pipeline ahead of the RemoteIpValve.

Another option would be to simply remove the RemoteIpValve and write a
simple servlet that logs the remote IP.

Mark


Regards,

Harrie

-Oorspronkelijk bericht-
Van: Harrie Robins [mailto:har...@eyequestion.nl]
Verzonden: 21 December 2017 09:55
Aan: 'Tomcat Users List' 
Onderwerp: RE: internalProxies regex

This makes perfect sense.
I tested my regex, just against wrong engine.

Thanks for pointing me in the right direction

-Oorspronkelijk bericht-
Van: Konstantin Kolinko [mailto:knst.koli...@gmail.com]
Verzonden: 20 December 2017 15:19
Aan: Tomcat Users List 
Onderwerp: Re: internalProxies regex

2017-12-20 11:37 GMT+03:00 Harrie Robins :

Hello everyone,



I have a question about the remoteipvalve in tomcat 8.5:
https://tomcat.apache.org/tomcat-8.5-doc/api/org/apache/catalina/valve
s/Remo
teIpValve.html




internalProxies

Regular expression that matches the IP addresses of internal proxies.
If they appear in the remoteIpHeader value, they will be trusted and
will not appear in the proxiesHeader value

RemoteIPInternalProxy

Regular expression (in the syntax supported by java.util.regex)

10\.\d{1,3}\.\d{1,3}\.\d{1,3}|192\.168\.\d{1,3}\.\d{1,3}|
169\.254\.\d{1,3}\.\d{1,3}|127\.\d{1,3}\.\d{1,3}\.\d{1,3}|
172\.1[6-9]{1}\.\d{1,3}\.\d{1,3}|172\.2[0-9]{1}\.\d{1,3}\.\d{1,3}|
172\.3[0-1]{1}\.\d{1,3}\.\d{1,3}
By default, 10/8, 192.168/16, 169.254/16, 127/8 and 172.16/12 are

allowed.



I need to convert some CIDR ranges to regex:


my concern is that /d{1,3} wil match too many (non exist) addresses

103\.21\.24\d[4-7]\.\d[0-9]\d{1,3}|103\.22\.20\d[0-3]\.\d[0-9]\d{1,3}|
103\.3
1\.\d[4-7]\.\d[0-9]\d{1,3}



So I re-wrote using capture groups, below does not function however,
and I assume it is due to OR (|) which tomcat will affectively see as a

new entry?

So I tried escaping, but I cannot get it to work:

103\.21\.(2(4[4-7]))\.([0-9]\|[1-9][0-9]\|1([0-9][0-9])\|2([0-4][0-9]\
|5[0-5
]))|103\.22\.(2(0[0-3]))\.([0-9]\|[1-9][0-9]\|1([0-9][0-9])\|2([0-4][0
-9]\|5
[0-5]))

Your assumption that "tomcat will affectively see as a new entry" is

wrong.

The string is used as whole to initialize a java.util.regex.Pattern().
Tomcat does not split it.

You may write a simple program / junit test to test how
java.util.regex.Pattern() processes your value.  Or you may run Tomcat

with debugger,

https://wiki.apache.org/tomcat/FAQ/Developing#Debugging
https://wiki.apache.org/tomcat/FAQ/Troubleshooting_

and_Diagnostics#Common_Troubleshooting_Scenario

AFAIK, '\|' in a regular expression will be interpreted as

Re: org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRun Handshake failed

2018-01-05 Thread Rémy Maucherat 
On Fri, Jan 5, 2018 at 8:38 AM, Rémy Maucherat  wrote:

> On Fri, Jan 5, 2018 at 12:25 AM, Michael Peng <
> michael.p...@entrustdatacard.com> wrote:
>
>> Do the changes make sense, and what would be the side effect ? In our
>> case, the "netInBuffer" could be full, i.e., postion = limit for large
>> data. Maybe the "netInBuffer" should not be cleared since "compact" would
>> reset the "netInBuffer", should it?
>>
>> The buffer is flipped after that and the NIO code is the same anyway, so
> the change doesn't make sense indeed as is.
>
> Looking at it made it seem not right compared to NIO, so I made another
change and hopefully fixed it.

Rémy

Re: GC allocation failure

2018-01-05 Thread Suvendu Sekhar Mondal 
On Jan 4, 2018 11:14 PM, "Rainer Jung"  wrote:

Am 04.01.2018 um 18:20 schrieb Christopher Schultz:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Ambica,
>
> On 1/4/18 11:17 AM, Sanka, Ambica wrote:
>
>> I am seeing below highlighted errors in native_err logs in all my
>> tomcat applications. I also increased memory for the VM from 4GB to
>> 8GB. Still seeing those. When do we get that errors? I am reading
>> online that when program asks for memory and java cannot give,
>> that's when we see them. Please suggest. Java HotSpot(TM) 64-Bit
>> Server VM (25.20-b23) for linux-amd64 JRE (1.8.0_20-b26), built on
>> Jul 30 2014 13:13:52 by "java_re" with gcc 4.3.0 20080428 (Red Hat
>> 4.3.0-8) Memory: 4k page, physical 8061572k(2564740k free), swap
>> 4063228k(4063228k free)
>>
>> CommandLine flags: -XX:+HeapDumpOnOutOfMemoryError
>> -XX:HeapDumpPath=/opt/apache/ancillariesmonitoring/logs/
>> -XX:InitialHeapSize=128985152 -XX:MaxHeapSize=268435456
>> -XX:+PrintGC -XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers
>> -XX:+UseCompressedOops -XX:+UseParallelGC
>>
>
> Others have commented on those messages you received, but nobody
> mentioned your heap configuration. In the above command-line
> arguments, you have specified both the minimum and maximum heap
> memory. You have expressed those values in bytes which makes it
> somewhat hard to read what they actually are, but this is what you
>

I *think* the JVM top line in GC output always shows bytes, even if you
were using other units in the original switches.


I agree.


have in readable units:
>
> - -XX:InitialHeapSize=128M -XX:MaxHeapSize=256M
>

but yes, that is a valid point!


So you aren't using an 8GiB heap. You aren't even using a 4GiB heap.
> You are using a 256 *megabyte* heap. If you really want an 8GiB heap,
> you'll need to set it properly in your command-line arguments.
>
> Note that setting the initial heap size to anything other than the
> maximum heap size just makes the JVM take longer to get the heap
> generations sized appropriately. For a long-running server process, I
> think it never makes any sense to set initial < max heap size. Always
> set them to the same value so that the heap itself does not have to be
> expanded/resized during heap allocations.


Christopher,

I really never found any explanation behind this "initial=max" heap size
theory until I saw your mail; although I see this type of configuration in
most of the places. It will be awesome if you can tell more about benefits
of this configuration.

I usually do not set initial and max heap size to same value because
garbage collection is delayed until the heap is full. Therefore, the first
time that the GC runs, the process can take longer. Also, the heap is more
likely to be fragmented and require a heap compaction. To avoid that, till
now my strategy is to:
- Start application with the minimum heap size that application requires
- When the GC starts up, it runs frequently and efficiently because the
heap is small
- When the heap is full of live objects, the GC compacts the heap. If
sufficient garbage is still not recovered or any of the other conditions
for heap expansion are met, the GC expands the heap.

Another thing, what if I know the server load varies a lot(from 10s in
night time to 1s during day time) during different time frame, does
"initial=max heap" apply for that situation also?

Please let me know what you think about it.

Thanks!
Suvendu

Re: internalProxies regex

2018-01-05 Thread Harrie Robins 
Hi Mark,

our tomcat application server are fronted by 1. cloudflare, and 2. amazon
load balancer.
In apache there is mod_remote IP and I can simply put in CIDR range:
https://www.cloudflare.com/ips/ that will swallow all those IP and will get
the correct IP to tomcat.

In Tomcat I need
https://tomcat.apache.org/tomcat-8.5-doc/api/org/apache/catalina/valves/RemoteIpValve.html
which does not accept CIDR range however. I wrote a regex to match all the
addresses and it works, it's matching way to many addresses however so I
rewrote the pattern. My new pattern is not functioning however, so I tested
then pattern in a small application.

In my test I made a list of all addresses  in this range:
^103\\.21\\.(2(4[4-7]))\\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2(
[0-4][0-9]|5[0-5]))$|^103\\.22\\.(2(0[0-3]))\\.([0-9]|[1-
9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$
I matched all these addresses and it works. When I set in tomcat however it
does not, I have no understanding why not?

Hope you understand what I am trying to do.

thanks





On 2 January 2018 at 19:33, Mark Thomas  wrote:

> On 02/01/18 09:50, Harrie Robins wrote:
> > I'm still having problems with matching my pattern.
> >
> > Right now I'm feeding the following to internalProxies:
> >
> > ^103\\.21\\.(2(4[4-7]))\\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2(
> [0-4][0-9]|5[0-5]))$|^103\\.22\\.(2(0[0-3]))\\.([0-9]|[1-
> 9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$
> > I created a list of all involved IP addresses and matched those IP
> addresses:
> >
> > java.util.regex.Matcher / java.util.regex.Pattern, please see
> https://pastebin.com/Lija7n9k
> >
> > All addresses from the list I created are matching, just not in tomcat.
>
> What is the value of the remote IP address that is failing to match? You
> might want to look at writing a short custom Valve to log that and
> insert it into the Pipeline ahead of the RemoteIpValve.
>
> Another option would be to simply remove the RemoteIpValve and write a
> simple servlet that logs the remote IP.
>
> Mark
>
> >
> > Regards,
> >
> > Harrie
> >
> > -Oorspronkelijk bericht-
> > Van: Harrie Robins [mailto:har...@eyequestion.nl]
> > Verzonden: 21 December 2017 09:55
> > Aan: 'Tomcat Users List' 
> > Onderwerp: RE: internalProxies regex
> >
> > This makes perfect sense.
> > I tested my regex, just against wrong engine.
> >
> > Thanks for pointing me in the right direction
> >
> > -Oorspronkelijk bericht-
> > Van: Konstantin Kolinko [mailto:knst.koli...@gmail.com]
> > Verzonden: 20 December 2017 15:19
> > Aan: Tomcat Users List 
> > Onderwerp: Re: internalProxies regex
> >
> > 2017-12-20 11:37 GMT+03:00 Harrie Robins :
> >> Hello everyone,
> >>
> >>
> >>
> >> I have a question about the remoteipvalve in tomcat 8.5:
> >> https://tomcat.apache.org/tomcat-8.5-doc/api/org/apache/catalina/valve
> >> s/Remo
> >> teIpValve.html
> >>
> >>
> >>
> >>
> >> internalProxies
> >>
> >> Regular expression that matches the IP addresses of internal proxies.
> >> If they appear in the remoteIpHeader value, they will be trusted and
> >> will not appear in the proxiesHeader value
> >>
> >> RemoteIPInternalProxy
> >>
> >> Regular expression (in the syntax supported by java.util.regex)
> >>
> >> 10\.\d{1,3}\.\d{1,3}\.\d{1,3}|192\.168\.\d{1,3}\.\d{1,3}|
> >> 169\.254\.\d{1,3}\.\d{1,3}|127\.\d{1,3}\.\d{1,3}\.\d{1,3}|
> >> 172\.1[6-9]{1}\.\d{1,3}\.\d{1,3}|172\.2[0-9]{1}\.\d{1,3}\.\d{1,3}|
> >> 172\.3[0-1]{1}\.\d{1,3}\.\d{1,3}
> >> By default, 10/8, 192.168/16, 169.254/16, 127/8 and 172.16/12 are
> allowed.
> >>
> >>
> >>
> >> I need to convert some CIDR ranges to regex:
> >>
> >>
> >> my concern is that /d{1,3} wil match too many (non exist) addresses
> >>
> >> 103\.21\.24\d[4-7]\.\d[0-9]\d{1,3}|103\.22\.20\d[0-3]\.\d[0-9]\d{1,3}|
> >> 103\.3
> >> 1\.\d[4-7]\.\d[0-9]\d{1,3}
> >>
> >>
> >>
> >> So I re-wrote using capture groups, below does not function however,
> >> and I assume it is due to OR (|) which tomcat will affectively see as a
> new entry?
> >> So I tried escaping, but I cannot get it to work:
> >>
> >> 103\.21\.(2(4[4-7]))\.([0-9]\|[1-9][0-9]\|1([0-9][0-9])\|2([0-4][0-9]\
> >> |5[0-5
> >> ]))|103\.22\.(2(0[0-3]))\.([0-9]\|[1-9][0-9]\|1([0-9][0-9])\|2([0-4][0
> >> -9]\|5
> >> [0-5]))
> >
> > Your assumption that "tomcat will affectively see as a new entry" is
> wrong.
> > The string is used as whole to initialize a java.util.regex.Pattern().
> > Tomcat does not split it.
> >
> > You may write a simple program / junit test to test how
> > java.util.regex.Pattern() processes your value.  Or you may run Tomcat
> with debugger,
> >
> > https://wiki.apache.org/tomcat/FAQ/Developing#Debugging
> > https://wiki.apache.org/tomcat/FAQ/Troubleshooting_
> and_Diagnostics#Common_Troubleshooting_Scenario
> >
> > AFAIK, '\|' in a regular expression will be interpreted as expecting
> literal '|' character in the matched string.  No IP address has this
>