Re: Using CsrfPreventionFilter with GET-based submissions
Chris,
> Am 13.11.2019 um 02:35 schrieb Christopher Schultz
> :
>
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Peter,
>
>> On 11/10/19 19:05, Peter Kreuser wrote:
>> Chris,
>>
>>>
>>> Am 09.11.2019 um 03:58 schrieb Christopher Schultz
>>> :
>>>
>>> -BEGIN PGP SIGNED MESSAGE- Hash: SHA256
>>>
>>> All,
>>>
>>> I'm playing with the CsrfPreventionFilter and things are working
>>> well in the following situations:
>>>
>>> link text
>>>
>>> and
>>>
>>> ...
>>>
>>> As long as the URL has been passed through request.encodeURL().
>>>
>>> However, this one is causing me a problem:
>>>
>>> ...
>>>
>>> This builds a form like this:
>>>
>>> >> action="https://host/path?org.apache.catalina.filters.CSRF_NONCE=[...
> ]">
>>>
>>>
> ...
>>>
>>>
>>> Neither Firefox nor Chrome will send the query string present in
>>> a action attribute if the method="GET". The method must be
>>> "POST" in order for this to be sent. This is due to the HTML
>>> standard[1].
>>>
>>> Short of changing all methods to "POST", is there any way
>>> around this?
>>>
>>> I have read the code for CsrfPreventionFilter and it does not
>>> appear that the nonce if stored anywhere except in the
>>> CsrfResponseWrapper for the request (and the session's nonce
>>> cache, but that isn't request-specific).
>>>
>>> Would it be inappropriate to add the CSRF_NONCE to the request
>>> attributes so that application code could use it directly if
>>> necessary? Something like this:
>>>
>>> ... >> name="org.apache.catalina.filters.CSRF_NONCE" value="<%=
>>> request.getAttribute("CSRF_NONCE") %>" />
>>
>> If i remember correctly, this is the way struts handles CSRF
>> Tokens.
>
> I'm not sure what Struts has to do with this. I'm using Tomcat's CSRF
> filter which apparently cannot work with GET-based forms. I'm not
> saying that a GET-based form is a good idea, but we have a bunch of
> them so I'm looking into how they can be effectively used with this
> implementation of a CSRF filter.
I just wanted to point out that struts implements the CSRF protection only with
a hidden field. That’s a bit of a hassle, plus you have to handle this per
form. Wether it is a POST or a GET.
> I'm really surprised this hasn't come up, yet. Maybe nobody actually
> implements CSRF protection, or maybe nobody uses Tomcat's filter to do
> it, or maybe nobody uses GET-based HTML s. But I can't believe
> that I'm the only person in the world who is trying to use all three
> at once.
>
>> However there the nonce comes directly from the session . Not
>> request.
>
> The nonces are stored in the session, otherwise this wouldn't work.
> But each request generates a new nonce, and that one would be the
> "request's nonce".
That’s another difference to struts and a bit of a nuisance, the nonce is only
created once (and there is no cache). Once you remove it or generate a new
nonce, all requests from “old” forms Will fail. But that is another story.
But nevertheless the GET problem would be interesting to figure out. I will try
this once I’m back from vacation.
Peter
> - -chris
> -BEGIN PGP SIGNATURE-
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3K7UkACgkQHPApP6U8
> pFhZBRAAkrD4pr+agtuxblW/UA8ylVm6pceqTmlz8ki39I7k7T5fjgm+Yg1mjG9R
> VgQNI/v0y94rQ3cCjIerUxTTNHTzgHUi2uRk9sSsnTsRXC4W+8wdRSojDSq9j1AB
> z4ZQI3t5Z8+e9RWBDrYOd2TkvW93aGiMzOpcnashvJkhUSDR5102RJtvoz1yAK7r
> Rwzv8feSJRy3/rxjiqQvHYdBH9DeRXVGH0CTP6KZ/+e/icFw0nnH3e+Jrh3+k5du
> fIUi/97JPT0hxkkJbNkKsOJ3P/D4kqzKnbo6WqFr4UwYMCiJizNNTuu+3pG6LAjn
> qTow+EL6/sG3Dtt/VCexyhC7jXdGjcrMDpxcXZx6NwiFxppK2kGWVMi7zKz4qm3X
> ZLR1zSsfzRhUnVPmdjYUAtDonhCbWW+FdQmBtGhGhH7+3wOeXrGpBeWcAq7jjGoD
> rgKQJMKUEN0PMH8j63tgp86Te6zhJCG23/ttBqFTLvP6XWbfHn6KoMFrwSMz8Spz
> EyFDBJihspuFncOMjEJ5kPLmJzs2x811VVkMOBA3BPqIrN2qteTIja9+t3ismo4+
> iBf0sV0q74HL24wvyAfONArae52vFmg4wJLiN38tztNLdoblJpTjqRs04gQ+Q0+5
> u4KKouD6Oz37Zo1Z7IS/HezmfGwYRrZn96CgPzfam4ZQp4Hldi8=
> =mYsp
> -END PGP SIGNATURE-
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: tomcat thread incurring CPU load
HI Mark, following my previous reply, we have now confirmed that it's indeed 8.5.45 with APR 1.2.23 that's causing such high JVM CPU usage. We used took out 2 out of 50 servers from the load balancer config, reverted tomcat, and redeployed. With near to identical user traffic, the two servers are responding normally without/without traffic with 8.5.41. The JVM dump looks a lot better with 8.5.41. We do think that the recent changes in APR and some other tomcat jar may have caused compatibility issue on Windows server 2016 (64-bit) platform. But unfortunately, we cannot pinpoint exactly what change may have caused this (i.e. actual OS vs Security Updates). With this in mind, we are also being wary to move to 8.5.47 as we don't know if the same issue will occur again. Since 8.5.41 has been packaged with previously accepted application installer, we are more comfortable rolling back. I would appreciate if this can be looked into. On Tue, 12 Nov 2019 at 11:27, M. Manna wrote: > Hey Mark (appreciate your response in US holiday time) > > On Tue, 12 Nov 2019 at 07:51, Mark Thomas wrote: > >> On November 12, 2019 12:54:53 AM UTC, "M. Manna" >> wrote: >> >Just to give an update again: >> > >> >1) We reverted the APR to 1.2.21 - but observed no difference. >> >2) We took 3 thread dumps over 1 min interval (without any user >> >sessions) - >> >All threads are tomcat's internal pool threads. >> > >> >When we checked the thread details (using fasthread.io) - we didn't see >> >any >> >of our application stack. Since there is no user traffic, this is >> >coming >> >from tomcat internally. At this stage, we cannot really figure out >> >what's >> >the root cause. >> > >> >Any help is appreciated. >> >> Migrated from what (full version info please)? >> > from 8.5.41 to 8.5.45 (we migrate 3 times a year, last was in June) > >> >> Operating system exact version? >> > Microsoft Windows Server 2016 DataCentre (64-bit) > >> >> JRE vendor and exact version? >> > C:\jdk1.8.0\bin>java.exe -version > java version "1.8.0_162" > Java(TM) SE Runtime Environment (build 1.8.0_162-b12) > Java HotSpot(TM) 64-Bit Server VM (build 25.162-b12, mixed mode) > > >> Do you see the same behavior with the latest 8.5.x and latest Tomcat >> Native? >> > We are using APR 1.2.23 which I can also see in latest tomcat. Due to > production due diligence we cannot roll to a different version that easily. > Normally, we lag behind by 2 monthly releases of tomcat. We also reverted > the APR to 1.2.21 (but no difference). > >> >> What triggers this behaviour? >> > That is quite strange. Due to US holidays, we had a low traffic on our > servers, and nothing has crept in to suggest that it's application-driven. > We took one tomcat instance out of 50 instances and removed all user > sessions (i.e. no application activities or threads). Upon restart of > tomcat, the CPU spike lingered past the initial servlet startup period. We > monitored that over 1-2 hours but there was no difference. > >> >> How often do you see this behaviour? >> > We took 2 sets of data > 1) 3 Jstack dump based on 10 seconds interval. > 2) 3 jstack dump based on 1 min interval. > > Both the above reveals that all background threads (http, pool etc.) were > from tomcat. We didn't have any application threads lingered in those 3 > samples. So yes we see this almost all the time if we take samples. > However, when we compared with pre-production instances (with Windows > server R2 x64 bit), we don't see such abnormal spike. In fact, the > application instance doesn't incur such a big CPU spike. Whilst composing > this email, I am now thinking if the APR is indeed incompatible with > WIndows Server R2 (or the presence of any Windows Updates) which blocks the > native poll() call longer than usual. > > An example is that on Windows Server 2012 - APR poll() call takes about > 30% CPU time - but with Windows Server 2016 it's almost always 95%. > > >> >> And anything else you think might be relevant. >> > > We are using end-2-end encryption using APR (with Certificate and > SSLConfig resource setup in server.xml). But it's survived past 3 tomcat > upgrades without any issue. > Except OS we don't have any obvious culprit identified at the moment. > > Thanks, > >> >> Mark >> >> > >> >Thanks, >> > >> >On Mon, 11 Nov 2019 at 20:57, M. Manna wrote: >> > >> >> Hello All, >> >> >> >> Any thoughts regarding this? Slightly clueless at this point, so any >> >> direction will be appreciated. >> >> >> >> We are seeing the poll taking all the CPU time. We are using >> >> OperatingSystemMXBean.getProcessCpuLoad() and >> >> OperatingSystemMXBean.getSystemCpuLoad() to get our metrics (then >> >x100 to >> >> get the pct). >> >> >> >> Thanks, >> >> >> >> >> >> On Mon, 11 Nov 2019 at 17:46, M. Manna wrote: >> >> >> >>> Hello, >> >>> >> >>> after migrating to 8.5.45, we are seeing a lot of cpu load by >> >following >> >>> JVM thread dump: >> >>> >> >>> "https-openssl-apr-0.0.0.0-8443-Poller" : 102 :
Re: Using CsrfPreventionFilter with GET-based submissions
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Peter,
On 11/10/19 19:05, Peter Kreuser wrote:
> Chris,
>
>>
>> Am 09.11.2019 um 03:58 schrieb Christopher Schultz
>> :
>>
>> -BEGIN PGP SIGNED MESSAGE- Hash: SHA256
>>
>> All,
>>
>> I'm playing with the CsrfPreventionFilter and things are working
>> well in the following situations:
>>
>> link text
>>
>> and
>>
>> ...
>>
>> As long as the URL has been passed through request.encodeURL().
>>
>> However, this one is causing me a problem:
>>
>> ...
>>
>> This builds a form like this:
>>
>> > action="https://host/path?org.apache.catalina.filters.CSRF_NONCE=[...
]">
>>
>>
...
>>
>>
>> Neither Firefox nor Chrome will send the query string present in
>> a action attribute if the method="GET". The method must be
>> "POST" in order for this to be sent. This is due to the HTML
>> standard[1].
>>
>> Short of changing all methods to "POST", is there any way
>> around this?
>>
>> I have read the code for CsrfPreventionFilter and it does not
>> appear that the nonce if stored anywhere except in the
>> CsrfResponseWrapper for the request (and the session's nonce
>> cache, but that isn't request-specific).
>>
>> Would it be inappropriate to add the CSRF_NONCE to the request
>> attributes so that application code could use it directly if
>> necessary? Something like this:
>>
>> ... > name="org.apache.catalina.filters.CSRF_NONCE" value="<%=
>> request.getAttribute("CSRF_NONCE") %>" />
>
> If i remember correctly, this is the way struts handles CSRF
> Tokens.
I'm not sure what Struts has to do with this. I'm using Tomcat's CSRF
filter which apparently cannot work with GET-based forms. I'm not
saying that a GET-based form is a good idea, but we have a bunch of
them so I'm looking into how they can be effectively used with this
implementation of a CSRF filter.
I'm really surprised this hasn't come up, yet. Maybe nobody actually
implements CSRF protection, or maybe nobody uses Tomcat's filter to do
it, or maybe nobody uses GET-based HTML s. But I can't believe
that I'm the only person in the world who is trying to use all three
at once.
> However there the nonce comes directly from the session . Not
> request.
The nonces are stored in the session, otherwise this wouldn't work.
But each request generates a new nonce, and that one would be the
"request's nonce".
- -chris
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3K7UkACgkQHPApP6U8
pFhZBRAAkrD4pr+agtuxblW/UA8ylVm6pceqTmlz8ki39I7k7T5fjgm+Yg1mjG9R
VgQNI/v0y94rQ3cCjIerUxTTNHTzgHUi2uRk9sSsnTsRXC4W+8wdRSojDSq9j1AB
z4ZQI3t5Z8+e9RWBDrYOd2TkvW93aGiMzOpcnashvJkhUSDR5102RJtvoz1yAK7r
Rwzv8feSJRy3/rxjiqQvHYdBH9DeRXVGH0CTP6KZ/+e/icFw0nnH3e+Jrh3+k5du
fIUi/97JPT0hxkkJbNkKsOJ3P/D4kqzKnbo6WqFr4UwYMCiJizNNTuu+3pG6LAjn
qTow+EL6/sG3Dtt/VCexyhC7jXdGjcrMDpxcXZx6NwiFxppK2kGWVMi7zKz4qm3X
ZLR1zSsfzRhUnVPmdjYUAtDonhCbWW+FdQmBtGhGhH7+3wOeXrGpBeWcAq7jjGoD
rgKQJMKUEN0PMH8j63tgp86Te6zhJCG23/ttBqFTLvP6XWbfHn6KoMFrwSMz8Spz
EyFDBJihspuFncOMjEJ5kPLmJzs2x811VVkMOBA3BPqIrN2qteTIja9+t3ismo4+
iBf0sV0q74HL24wvyAfONArae52vFmg4wJLiN38tztNLdoblJpTjqRs04gQ+Q0+5
u4KKouD6Oz37Zo1Z7IS/HezmfGwYRrZn96CgPzfam4ZQp4Hldi8=
=mYsp
-END PGP SIGNATURE-
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat Loader putting my modules to module-path as unnamed module
Man! I really didn't expect this answer! On Tue, Nov 12, 2019 at 3:27 PM Mark Thomas wrote: > On 12/11/2019 14:13, Nedim Kulovac wrote: > > Hi there. I had an issue with starting a modular Java 11 app. Basically, > I > > was wondering why my module-info. java does not require to use 'open' > > directive when using Spring Framework. I posted the question here > > > https://stackoverflow.com/questions/58737469/do-i-have-to-add-open-statement-in-module-info-java-to-open-module-for-reflectio?noredirect=1#comment103882923_58737469 > > and > > user Holger came with diagnosing that Tomcat loading my code as well as > > Spring through a custom class loader and that is why my modules and > Spring > > jars are not listed when I use --show-module-resolution, neither they > > appear in the classpath. To inspect is custom class loading done modular > or > > old style, he recommended putting > > System.out.println(SomeClass.class.getModule()); somewhere in code in the > > module. I did it and as a result, I got printed out 'unnamed module > > @595b34e5'. So the big question is: why is this happening and how can I > > resolve this? I want Tomcat to put my modules to module path normally, > not > > as an unnamed module. > > You can't. > > The Servlet API (nor any part of Java EE / Jakarta EE) does not make use > of the module system. > > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > -- *Nedim Kulovac*
Re: Tomcat Loader putting my modules to module-path as unnamed module
On 12/11/2019 14:13, Nedim Kulovac wrote: > Hi there. I had an issue with starting a modular Java 11 app. Basically, I > was wondering why my module-info. java does not require to use 'open' > directive when using Spring Framework. I posted the question here > https://stackoverflow.com/questions/58737469/do-i-have-to-add-open-statement-in-module-info-java-to-open-module-for-reflectio?noredirect=1#comment103882923_58737469 > and > user Holger came with diagnosing that Tomcat loading my code as well as > Spring through a custom class loader and that is why my modules and Spring > jars are not listed when I use --show-module-resolution, neither they > appear in the classpath. To inspect is custom class loading done modular or > old style, he recommended putting > System.out.println(SomeClass.class.getModule()); somewhere in code in the > module. I did it and as a result, I got printed out 'unnamed module > @595b34e5'. So the big question is: why is this happening and how can I > resolve this? I want Tomcat to put my modules to module path normally, not > as an unnamed module. You can't. The Servlet API (nor any part of Java EE / Jakarta EE) does not make use of the module system. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat Loader putting my modules to module-path as unnamed module
Hi there. I had an issue with starting a modular Java 11 app. Basically, I was wondering why my module-info. java does not require to use 'open' directive when using Spring Framework. I posted the question here https://stackoverflow.com/questions/58737469/do-i-have-to-add-open-statement-in-module-info-java-to-open-module-for-reflectio?noredirect=1#comment103882923_58737469 and user Holger came with diagnosing that Tomcat loading my code as well as Spring through a custom class loader and that is why my modules and Spring jars are not listed when I use --show-module-resolution, neither they appear in the classpath. To inspect is custom class loading done modular or old style, he recommended putting System.out.println(SomeClass.class.getModule()); somewhere in code in the module. I did it and as a result, I got printed out 'unnamed module @595b34e5'. So the big question is: why is this happening and how can I resolve this? I want Tomcat to put my modules to module path normally, not as an unnamed module.
Re: tomcat thread incurring CPU load
Hey Mark (appreciate your response in US holiday time) On Tue, 12 Nov 2019 at 07:51, Mark Thomas wrote: > On November 12, 2019 12:54:53 AM UTC, "M. Manna" > wrote: > >Just to give an update again: > > > >1) We reverted the APR to 1.2.21 - but observed no difference. > >2) We took 3 thread dumps over 1 min interval (without any user > >sessions) - > >All threads are tomcat's internal pool threads. > > > >When we checked the thread details (using fasthread.io) - we didn't see > >any > >of our application stack. Since there is no user traffic, this is > >coming > >from tomcat internally. At this stage, we cannot really figure out > >what's > >the root cause. > > > >Any help is appreciated. > > Migrated from what (full version info please)? > from 8.5.41 to 8.5.45 (we migrate 3 times a year, last was in June) > > Operating system exact version? > Microsoft Windows Server 2016 DataCentre (64-bit) > > JRE vendor and exact version? > C:\jdk1.8.0\bin>java.exe -version java version "1.8.0_162" Java(TM) SE Runtime Environment (build 1.8.0_162-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.162-b12, mixed mode) > Do you see the same behavior with the latest 8.5.x and latest Tomcat > Native? > We are using APR 1.2.23 which I can also see in latest tomcat. Due to production due diligence we cannot roll to a different version that easily. Normally, we lag behind by 2 monthly releases of tomcat. We also reverted the APR to 1.2.21 (but no difference). > > What triggers this behaviour? > That is quite strange. Due to US holidays, we had a low traffic on our servers, and nothing has crept in to suggest that it's application-driven. We took one tomcat instance out of 50 instances and removed all user sessions (i.e. no application activities or threads). Upon restart of tomcat, the CPU spike lingered past the initial servlet startup period. We monitored that over 1-2 hours but there was no difference. > > How often do you see this behaviour? > We took 2 sets of data 1) 3 Jstack dump based on 10 seconds interval. 2) 3 jstack dump based on 1 min interval. Both the above reveals that all background threads (http, pool etc.) were from tomcat. We didn't have any application threads lingered in those 3 samples. So yes we see this almost all the time if we take samples. However, when we compared with pre-production instances (with Windows server R2 x64 bit), we don't see such abnormal spike. In fact, the application instance doesn't incur such a big CPU spike. Whilst composing this email, I am now thinking if the APR is indeed incompatible with WIndows Server R2 (or the presence of any Windows Updates) which blocks the native poll() call longer than usual. An example is that on Windows Server 2012 - APR poll() call takes about 30% CPU time - but with Windows Server 2016 it's almost always 95%. > > And anything else you think might be relevant. > We are using end-2-end encryption using APR (with Certificate and SSLConfig resource setup in server.xml). But it's survived past 3 tomcat upgrades without any issue. Except OS we don't have any obvious culprit identified at the moment. Thanks, > > Mark > > > > >Thanks, > > > >On Mon, 11 Nov 2019 at 20:57, M. Manna wrote: > > > >> Hello All, > >> > >> Any thoughts regarding this? Slightly clueless at this point, so any > >> direction will be appreciated. > >> > >> We are seeing the poll taking all the CPU time. We are using > >> OperatingSystemMXBean.getProcessCpuLoad() and > >> OperatingSystemMXBean.getSystemCpuLoad() to get our metrics (then > >x100 to > >> get the pct). > >> > >> Thanks, > >> > >> > >> On Mon, 11 Nov 2019 at 17:46, M. Manna wrote: > >> > >>> Hello, > >>> > >>> after migrating to 8.5.45, we are seeing a lot of cpu load by > >following > >>> JVM thread dump: > >>> > >>> "https-openssl-apr-0.0.0.0-8443-Poller" : 102 : RUNNABLE : > >>> cpu=172902703125000 : cpuLoad= 74.181015 > >>> > >>> BlockedCount:8464 BlockedTime:0 LockName:null LockOwnerID:-1 > >>> LockOwnerName:null > >>> > >>> WaitedCount:5397 WaitedTime:0 InNative:false IsSuspended:false at > >>> org.apache.tomcat.jni.Poll.poll(Poll.java:-2) > >>> > >>> at > >>> > >org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1547) > >>> > >>> at java.lang.Thread.run(Thread.java:748) > >>> > >>> > >>> These are coming after 2-3 successful jvm dump. Is this something > >>> familiar to anybody? > >>> > >>> Thanks, > >>> > >> > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: TLS key management
On 11/11/2019 23:40, George Stanchev wrote: > Currently, (in most cases) Tomcat creates an in-memory keystore and > initializes kmf as follows: > KeyManagementFactory.getInstance(algo).init(keystore, kspass). The in-memory > keystore has the key, the certificate and the chain and nothing else. This > works fine in most cases but we've ran into a situation where this is not > sufficient. I am running TC with BC as JSSE provider in FIPS-approved only > mode and in certain use cases we're running into issues with RSA key reuse. > FIPS states that an RSA key should be used for encryption/decryption or for > signature/verification but not for both. So when one browser (in our case it > was FF) selects TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, BC's key manager marks > and remembers the key usage, then another browser (Chrome) settles for > TLS_RSA_WITH_AES_128_GCM_SHA256 BC throws an key-reuse exception since the > latter suite uses RSA for key exchange and the former for authentication. The > BC key manager has the ability to select a different key based on KeyUsage > extension, so it is possible to have multiple RSA keys in memory that would > be used according to their certificates KeyUsage policy. However TC feeds > only one certificate to the KM. > > Here is a thread [1] that I ran into that shows someone else running into the > issue and response from BC developer. > > To be fair, BCFIPS does have a -D override for the key usage override for RSA > keys in approved-only mode but according to this thread [2], the property is > there for completely different purpose and running it to get around the TC > issue is not FIPS compliant. > > So having looked at the code in SSLUtilBase#getKeyManagers(), is it worth > opening a BZ request to have some solution to this issue - perhaps if alias > is omitted in configuration and the keystore is of transferrable type (not > ms, hardware, etc) then transfer all entries to the KM and let it do the > selection? Absolutely. If you can provide everything we'll need to reproduce this on a clean 9.0.x build (server.xml changes, sample keystores, links to libs we need to download etc.) it should be possible to address this. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
