Re: tomcat 8.5.23 dbcp not honoring autocommit = false?

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Chris,

Sorry, I had the autocommit true/false values mixed up in my mind.

When you say you have "autocommit disabled in mysql config" what do
you mean?

On 10/13/17 10:17 AM, Chris Cheshire wrote:
> 
> 
> As a further test I just took out my explicit rollback in my 
> DAOFactory close() method, and swapped back to commons dbcp. Added
> an update that wasn't explicitly committed, and it correctly did
> not get committed when the connection was closed. Swapped back to
> tomcat dbcp and repeated, it got committed without an explicit
> commit statement.
> 
> I'm really puzzled as to why *I* have to explicitly rollback on
> close if autocommit is not enabled, instead of tomcat dbcp handling
> that when commons dbcp appears to do it.

No connection pool can read your mind. If you begin a transaction (or
never start one), you must either commit or rollback. Merely calling
close() does not explicitly cause either of those to be called.

> If I do
> 
> daoFactory = new MySQLDAOFactoryImpl(getDataSource());
> 
> // update #1 daoFactory.commit()
> 
> // update #2 daoFactory.close();
> 
> then update #2 is being committed.

I'm curious why you are doing "update #2" without either COMMIT or
ROLLBACK. That seems like ... a mistake.

- From the Connection.close() javadoc:

"
It is strongly recommended that an application explicitly commits or
rolls back an active transaction prior to calling the close method. If
the close method is called and there is an active transaction, the
results are implementation-defined.
"

There *is* an implicit COMMIT executed if the autocommit flag is
flipped for any reason, either true->false or false->true.

If you have autocommit=false in your  configuration (which
you do), then calling setAutoCommit(false) shouldn't do anything.

> If I put in this in the close() method of my DAO Factory
> 
> if (!this.dbConn.getAutoCommit()) { this.dbConn.rollback(); }
> 
> before the close() call, then update #2 is correctly not getting
> committed.

This is probably the wrong approach: your close() method doesn't know
whether it's better to call commit() or rollback(), so it should do
neither.

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlnhKXEACgkQHPApP6U8
pFhZmw/9FN376aEpeGwzLRnpWy0jIo7EezNXeV4G5Jtj1slDeqbFMOYcm/3bdSEt
sa+9FwHbe9gKMyeLWp3TJbDZ8F0RI+e2CIYV37XAchrDvpb+f80M+SjzPyuovO6a
9wRtbekKd8mbIfpGTfokjW+pNIJBbAJvZfh0UGFJP+VRX5XoE6MLzw60OnEMlmLP
7IFAdvM0t816Nuh7yJIg63I3eHYB3P7cx01ETEhWyI1oK2LQ50cODfgLoxT3iC+j
f8HAyn8wubZSkC3PKJ/oY8TGaLczSt8M/ANAucZ0mw91j7m93OB+3KGwVczGRUiD
lodMS9RZBsGmNxxzKcCUCmxydat2PRYJSmP/hRR0nQL7xRPiClAi4KJTWFCdn0hc
SNZp8shT9Z5EVBNHJ9z2ippW7K5Xr+U58bYrN+kEmq8jN5UTUcQlDoahwnRbRWV5
CCBX+9P+fd44yWuK5IGkFcuKr68LmZYHarZlL5OTxfretKB7QX9B4o4x6J8PhhFj
vnfJPlQCkKkuRyuSGTneLqM+f+CrQq7nquVxUmgUPF2btqNe1d21/g8AN8+FSdKS
YgiZ/OstBdoDHVxnfxDJQcwY8IG9qg/didN7oaNAdelulcdIFx8Ob7PEdEmXZlrV
o3vs2ntI7dCjfr0vfbvkzqrwXGgEMIPvukggB+DTgRxpYKJfCRM=
=/fEQ
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: AJP connection pool issue bug?

[TurboChargedDad .]

I missed some of these messages before.. I apologize.

Can I send these to you privately.

On Wed, Oct 4, 2017 at 4:01 PM, Christopher Schultz <
ch...@christopherschultz.net> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> TCD,
>
> On 10/4/17 3:45 PM, TurboChargedDad . wrote:
> > Perhaps I am not wording my question correctly.
>
> Can you confirm that the connection-pool exhaustion appears to be
> happening on the AJP client (httpd/mod_proxy_ajp) and NOT on the
> server (Tomcat/AJP)?
>
> If so, the problem will likely not improve by switching-over to an
> NIO-based connector on the Tomcat side.
>
> Having said that, the real problem is likely to be simple arithmetic.
> Remember this expression:
>
> Ctc = Nhttpd * Cworkers
>
> Ctc = Connections Tomcat should be prepared to accept (e.g. Connector
> maxConnections)
>
> Nhttpd = # of httpd servers
> Cworkers = total # of connections in httpd connection pool for all
> workers(!!)
>
> Imagine the following scenario:
>
> Nhttpd = 2
> Cworker = 200
> Ntomcat = 2
>
> On httpd server A, we have a connection pool with 200 connections. If
> Tomcat A goes down, all 200 connections will go to Tomcat B. If that
> happens to both proxies (Tomcat A stops responding), then both proxies
> will send all 200 connections to Tomcat B. That means that Tomcat B
> needs to be able to support 400 connections, not 200.
>
> Let's say you now have 5 workers (1 for each application). Each worker
> gets its own connection pool, and each connection pool has 200 workers
> in it. Now, we have a situation where each httpd instance actually has
> 1000 (potential) connections in the connection pool, and if Tomcat A
> goes down, Tomcat B must be able to handle 2000 connections (1000 from
> httpd A and 1000 from httpd B).
>
> At some point, you can't provision enough threads to handle all of
> those connections.
>
> The solution (bringing this back around again) is to use NIO, because
> you can handle a LOT more connections with a lower number of threads.
> NIO doesn't allow you to handle more *concurrent* traffic (in fact, it
> makes performance a tiny bit worse than BIO), but it will allow you to
> have TONS of idle connections that aren't "wasting" request-processing
> threads that are just waiting for another actual request to come
> across the wire.
>
> > As a test I changed the following line in one of the many tomcat
> > instances running on the server and bounced it. Old  New  > protocol="org.apache.coyote.ajp.AjpNioProtocol" redirectPort="8443"
> > maxThreads="300" />
>
> Yep, that's how to do it.
>
> > As the docs state I am able to verify that it did in fact switch
> > over to NIO.
> >
> > INFO: Starting ProtocolHandler ["ajp-nio-9335"]
>
> Good. Now you can handle many idle connections with the same number of
> threads.
>
> > Will running NIO and BIO on the same box have a negative impact?
>
> No.
>
> > I am thinking they should all be switched to NIO, this was just a
> > test to see if I was understanding what I was reading.
> I would recommend NIO in all cases.
>
> > That being said I suspect there are going to be far more tweaks
> > that needs to be applied as there are none to date.
>
> Hopefully not. A recent Tomcat (which you don't actually have) with a
> stock configuration should be fairly well-configured to handle a great
> deal of traffic without falling-over.
>
> > I also know that the HTTPD server is running in prefork mode.
> That will pose some other issues for you, mostly the ability to handle
> bursts of high concurrency from your clients. You can consider it
> out-of-scope for this discussion, though. What we want to do for you
> is stop httpd+Tomcat from freaking out and getting stopped-up with
> even a small number of users.
>
> > Which I think leaves me with no control over how many connections
> > can be handed back from apache on a site by site basis.
>
> Probably not on a site-by-site basis, but you can adjust the
> connection-pool size on a per-worker basis. For prefork it MUST BE
> connection_pool_size=1 (the default for prefork httpd) and for
> "worker" and similarly-threaded MPMs the default should be fine to use.
>
> > Really having hard time explaining to others how BIO could have
> > caused the connection pool for another use to become exhausted.
> Well...
>
> If one of your Tomcats locks-up (database is dead; might want to check
> to see how the application is accessing that... infinite timeouts can
> be a real killer, here), it can tie-up connections from
> mod_proxy_ajp's connection pool. But those connections should be
> per-worker and shouldn't interfere with each other. Unless you have an
> uber-worker that handles everything for all those various Tomcats.
>
> Can you give us a peek at your worker configuration? You explained it
> a bit in your first post, but it might be time for some more details...
>
> - -chris
> -BEGIN PGP SIGNATURE-
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http

Re: URL-encoding and "#"

[Mark Thomas]

On 13/10/17 18:42, André Warnier (tomcat) wrote:
> On 13.10.2017 19:29, Mark Thomas wrote:
>> On 13/10/2017 18:15, André Warnier (tomcat) wrote:
>>> On 13.10.2017 18:17, Mark Thomas wrote:
 On 13/10/2017 17:09, James H. H. Lampert wrote:
> Thanks to all of you who responded.
>
> I found a web page that explains it in ways that I can wrap my
> 55-year-old brain around, and has an easy-to-read reference chart.
>
> https://perishablepress.com/stop-using-unsafe-characters-in-urls/
>
> Question: the problem first showed up on a web service that takes a
> "bodyless" POST operation, and I assume it also applies to GET
> operations, and to the URL portion of a POST with a body.
>
> But what about the body of a POST?

   From an HTTP specification point of view, anything goes.
>>>
>>> With respect, I believe that "anything goes" is a bit imprecise here.
>>
>> Nope.
>>
>> You can POST anything. You are talking specifically about form data.
> 
> Mmm. You are being a bit casuistic here. (Granted, not that I wasn't.)

Yeah, sorry about that. I tend to read "With respect..." as meaning
pretty much exactly the opposite.

> In the real world, I would expect that 99% of what is ever POSTed, /is/
> form data.
> Not you ?

For Tomcat I don't have a clue what the split is but my guess is that is
it a lot less than 99% these days.

>  In
>> that case, as I said, the body has to conform to what the component
>> processing it expects.
> 
> And that component would be .. ?

https://svn.apache.org/viewvc/tomcat/trunk/java/org/apache/tomcat/util/http/Parameters.java?view=annotate

> I don't really know, but I would guess that in most webservers, the
> component parsing the body of a POST with Content-type =
> application/x-www-form-urlencoded, may be the same as the one which is
> parsing the query-string of a URI, no ?
> Considering the similarity of these two things, it would seem that the
> temptation would be hard to resist.

Tomcat uses exactly the same code - with a little wrapping to get the
data into the same format before it starts.

Mark


>> And yes, unicode in form data is 'interesting'...
>>
>> Mark
>>
>>
>>> See e.g. https://www.w3.org/TR/html401/interact/forms.html#h-17.13.4
>>>
>>> There are 2 ways for a user agent to send the content of a HTTP POST :
>>> 1) with Content-type header = application/x-www-form-urlencoded
>>> or
>>> 2) with Content-type header = multipart/form-data
>>>
>>> and while it is true that in the case (2), any submitted key=value pair
>>> would be sent separately 'as is', this would not necessarily be so in
>>> case (1), because then all key=value pairs would be concatenated into
>>> one long string, in which the different key=value pairs would be
>>> separated by (unescaped) "&" signs.
>>> (Apart from other required encodings, see the page above)
>>> So if the client is not a browser, and "composes" itself the POST body
>>> before sending it, and sends it with a Content-type (1), it had better
>>> encode the individual parameter pairs as described, before concatenating
>>> them, because that is what the server would expect.
>>>
>>> As an additional note, if it so happened that the data in the client
>>> could contain Unicode text, do not forget that this is (still) not the
>>> standard in HTTP (and URI's, and thus query-string-like things), and
>>> make sure that you use the proper method to encode any printable
>>> characters which are not purely US-ASCII.  Again, browsers generally do
>>> this correctly, but custom clients not necessarily. (And a "custom
>>> client" in this case, could even be a bit of javascript which is
>>> embedded in one of your own pages, but does its own calls to the server
>>> on the side).
>>>
>>> I just recently got bitten by this, even in a quite recent browser,
>>> where some javascript function was composing a POST to a server (using
>>> type (1) above), and was NOT doing it correctly, even though the page
>>> containing and calling this function was itself declared as
>>> Unicode/UTF-8.
>>> (that was with (and I am too sorely tempted to add "of course" to resist
>>> it) some revision of IE-11 - although other revisions of the same
>>> browser did not exhibit that same issue).
>>>
>>> [...]
>>>
>>>
>>> -
>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>>
>>
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
> 
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 


-
To unsubscribe, e-mail

Re: URL-encoding and "#"

[James H. H. Lampert]

On 10/13/17, 10:50 AM, Igal @ Lucee.org wrote:

On 10/13/2017 10:42 AM, André Warnier (tomcat) wrote:

Mmm. You are being a bit casuistic here. (Granted, not that I wasn't.)
In the real world, I would expect that 99% of what is ever POSTed,
/is/ form data.
Not you ?


10 years ago I would have agreed, but with REST services there are many
APIs that expect POSTed data that does not originate in web forms.


Exactly. And this whole discussion has been about RESTful web services 
(specifically, RESTful web services implemented with Swagger), so form 
data isn't even a consideration (and I'm pretty sure I plugged any 
Swagger-specific or JSON-specific holes in the body encoding months ago).



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: URL-encoding and "#"

[Igal @ Lucee.org]

On 10/13/2017 10:42 AM, André Warnier (tomcat) wrote:

Mmm. You are being a bit casuistic here. (Granted, not that I wasn't.)
In the real world, I would expect that 99% of what is ever POSTed, 
/is/ form data.

Not you ?


10 years ago I would have agreed, but with REST services there are many 
APIs that expect POSTed data that does not originate in web forms.


Respectfully,

Igal Sapir
Lucee Core Developer
Lucee.org 

Re: URL-encoding and "#"

[tomcat]

On 13.10.2017 19:29, Mark Thomas wrote:

On 13/10/2017 18:15, André Warnier (tomcat) wrote:

On 13.10.2017 18:17, Mark Thomas wrote:

On 13/10/2017 17:09, James H. H. Lampert wrote:

Thanks to all of you who responded.

I found a web page that explains it in ways that I can wrap my
55-year-old brain around, and has an easy-to-read reference chart.

https://perishablepress.com/stop-using-unsafe-characters-in-urls/

Question: the problem first showed up on a web service that takes a
"bodyless" POST operation, and I assume it also applies to GET
operations, and to the URL portion of a POST with a body.

But what about the body of a POST?


  From an HTTP specification point of view, anything goes.


With respect, I believe that "anything goes" is a bit imprecise here.


Nope.

You can POST anything. You are talking specifically about form data.


Mmm. You are being a bit casuistic here. (Granted, not that I wasn't.)
In the real world, I would expect that 99% of what is ever POSTed, /is/ form 
data.
Not you ?

 In

that case, as I said, the body has to conform to what the component
processing it expects.


And that component would be .. ?
I don't really know, but I would guess that in most webservers, the component parsing the 
body of a POST with Content-type = application/x-www-form-urlencoded, may be the same as 
the one which is parsing the query-string of a URI, no ?
Considering the similarity of these two things, it would seem that the temptation would be 
hard to resist.




And yes, unicode in form data is 'interesting'...

Mark



See e.g. https://www.w3.org/TR/html401/interact/forms.html#h-17.13.4

There are 2 ways for a user agent to send the content of a HTTP POST :
1) with Content-type header = application/x-www-form-urlencoded
or
2) with Content-type header = multipart/form-data

and while it is true that in the case (2), any submitted key=value pair
would be sent separately 'as is', this would not necessarily be so in
case (1), because then all key=value pairs would be concatenated into
one long string, in which the different key=value pairs would be
separated by (unescaped) "&" signs.
(Apart from other required encodings, see the page above)
So if the client is not a browser, and "composes" itself the POST body
before sending it, and sends it with a Content-type (1), it had better
encode the individual parameter pairs as described, before concatenating
them, because that is what the server would expect.

As an additional note, if it so happened that the data in the client
could contain Unicode text, do not forget that this is (still) not the
standard in HTTP (and URI's, and thus query-string-like things), and
make sure that you use the proper method to encode any printable
characters which are not purely US-ASCII.  Again, browsers generally do
this correctly, but custom clients not necessarily. (And a "custom
client" in this case, could even be a bit of javascript which is
embedded in one of your own pages, but does its own calls to the server
on the side).

I just recently got bitten by this, even in a quite recent browser,
where some javascript function was composing a POST to a server (using
type (1) above), and was NOT doing it correctly, even though the page
containing and calling this function was itself declared as Unicode/UTF-8.
(that was with (and I am too sorely tempted to add "of course" to resist
it) some revision of IE-11 - although other revisions of the same
browser did not exhibit that same issue).

[...]


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: URL-encoding and "#"

[Mark Thomas]

On 13/10/2017 18:15, André Warnier (tomcat) wrote:
> On 13.10.2017 18:17, Mark Thomas wrote:
>> On 13/10/2017 17:09, James H. H. Lampert wrote:
>>> Thanks to all of you who responded.
>>>
>>> I found a web page that explains it in ways that I can wrap my
>>> 55-year-old brain around, and has an easy-to-read reference chart.
>>>
>>> https://perishablepress.com/stop-using-unsafe-characters-in-urls/
>>>
>>> Question: the problem first showed up on a web service that takes a
>>> "bodyless" POST operation, and I assume it also applies to GET
>>> operations, and to the URL portion of a POST with a body.
>>>
>>> But what about the body of a POST?
>>
>>  From an HTTP specification point of view, anything goes.
> 
> With respect, I believe that "anything goes" is a bit imprecise here.

Nope.

You can POST anything. You are talking specifically about form data. In
that case, as I said, the body has to conform to what the component
processing it expects.

And yes, unicode in form data is 'interesting'...

Mark


> See e.g. https://www.w3.org/TR/html401/interact/forms.html#h-17.13.4
> 
> There are 2 ways for a user agent to send the content of a HTTP POST :
> 1) with Content-type header = application/x-www-form-urlencoded
> or
> 2) with Content-type header = multipart/form-data
> 
> and while it is true that in the case (2), any submitted key=value pair
> would be sent separately 'as is', this would not necessarily be so in
> case (1), because then all key=value pairs would be concatenated into
> one long string, in which the different key=value pairs would be
> separated by (unescaped) "&" signs.
> (Apart from other required encodings, see the page above)
> So if the client is not a browser, and "composes" itself the POST body
> before sending it, and sends it with a Content-type (1), it had better
> encode the individual parameter pairs as described, before concatenating
> them, because that is what the server would expect.
> 
> As an additional note, if it so happened that the data in the client
> could contain Unicode text, do not forget that this is (still) not the
> standard in HTTP (and URI's, and thus query-string-like things), and
> make sure that you use the proper method to encode any printable
> characters which are not purely US-ASCII.  Again, browsers generally do
> this correctly, but custom clients not necessarily. (And a "custom
> client" in this case, could even be a bit of javascript which is
> embedded in one of your own pages, but does its own calls to the server
> on the side).
> 
> I just recently got bitten by this, even in a quite recent browser,
> where some javascript function was composing a POST to a server (using
> type (1) above), and was NOT doing it correctly, even though the page
> containing and calling this function was itself declared as Unicode/UTF-8.
> (that was with (and I am too sorely tempted to add "of course" to resist
> it) some revision of IE-11 - although other revisions of the same
> browser did not exhibit that same issue).
> 
> [...]
> 
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: URL-encoding and "#"

[tomcat]

On 13.10.2017 18:17, Mark Thomas wrote:

On 13/10/2017 17:09, James H. H. Lampert wrote:

Thanks to all of you who responded.

I found a web page that explains it in ways that I can wrap my
55-year-old brain around, and has an easy-to-read reference chart.

https://perishablepress.com/stop-using-unsafe-characters-in-urls/

Question: the problem first showed up on a web service that takes a
"bodyless" POST operation, and I assume it also applies to GET
operations, and to the URL portion of a POST with a body.

But what about the body of a POST?


 From an HTTP specification point of view, anything goes.


With respect, I believe that "anything goes" is a bit imprecise here.

See e.g. https://www.w3.org/TR/html401/interact/forms.html#h-17.13.4

There are 2 ways for a user agent to send the content of a HTTP POST :
1) with Content-type header = application/x-www-form-urlencoded
or
2) with Content-type header = multipart/form-data

and while it is true that in the case (2), any submitted key=value pair would be sent 
separately 'as is', this would not necessarily be so in case (1), because then all 
key=value pairs would be concatenated into one long string, in which the different 
key=value pairs would be separated by (unescaped) "&" signs.

(Apart from other required encodings, see the page above)
So if the client is not a browser, and "composes" itself the POST body before sending it, 
and sends it with a Content-type (1), it had better encode the individual parameter pairs 
as described, before concatenating them, because that is what the server would expect.


As an additional note, if it so happened that the data in the client could contain Unicode 
text, do not forget that this is (still) not the standard in HTTP (and URI's, and thus 
query-string-like things), and make sure that you use the proper method to encode any 
printable characters which are not purely US-ASCII.  Again, browsers generally do this 
correctly, but custom clients not necessarily. (And a "custom client" in this case, could 
even be a bit of javascript which is embedded in one of your own pages, but does its own 
calls to the server on the side).


I just recently got bitten by this, even in a quite recent browser, where some javascript 
function was composing a POST to a server (using type (1) above), and was NOT doing it 
correctly, even though the page containing and calling this function was itself declared 
as Unicode/UTF-8.
(that was with (and I am too sorely tempted to add "of course" to resist it) some revision 
of IE-11 - although other revisions of the same browser did not exhibit that same issue).


[...]


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Enforcing server preference for cipher suites

[Harish Krishnan]

Hi Chris, thanks for sharing your opinion.
Just my last comment here to close this thread.
BSAFE is anyways EOL now (or will be soon). We are already working on a 
replacement. Currently we are using the latest and greatest version of BSAFE 
with extended support.
Once again, thank you all for the great support.
I have another query (different topic) coming shortly...:-)

Sent from my iPhone

> On Oct 12, 2017, at 7:59 PM, Christopher Schultz 
>  wrote:
> 
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
> 
> Harish,
> 
>> On 10/12/17 10:55 AM, Harish Krishnan wrote:
>> Thank you all for the help and responses. We figured out what the
>> problem was. What I did was correct in terms of the attribute
>> setting, the tomcat version used and the JRE version used. However,
>> I did not realize our JRE is running in FIPs mode using RSA BSAFE
>> as the crypto provider.
> FIPS strikes again!
> 
> In this case, it's not really FIPS's fault, it's RSA's BSAFE. Anyone
> using RSA's BSAFE these days ought to lose their job. Plow that thing
> under with salt and use a trusted crypto provider (lol, Oracle, I guess)
> .
> 
>> When I tested and ran under standard JRE, then the server cipher 
>> suite order was preferred.
> You are probably using an ancient version of BSAFE. Your random
> numbers are probably all ones. Seriously, you need to dump BSAFE.
> 
>> Now I will have to look into what RSA library is doing here.
> 
> Leaking like a sieve, probably.
> 
>> Probably they are setting that Java API too which could be 
>> overwriting our setting in tomcat.
> 
> If that crypto provider is in use, then it'll likely affect the whole
> JVM. It just occurred to me that Tomcat doesn't have a setting for the
> crypto provider to use for TLS itself... only for the various
> "stores", etc. We probably ought to add that, and then you could
> choose "JSSE" as your provider and avoid BSAFE.
> 
> - -chris
> -BEGIN PGP SIGNATURE-
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
> 
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlngLCgACgkQHPApP6U8
> pFjanhAAkTNcGk5/X6b9aK2gYcSDdTjkE879XA77KGYwWDF2L01jtSdF7ejnCcuN
> 4lfivY/V5TaiKv0EZrU1YVC2psBZVK5CjfsCIfUZe5gOmqRRtxm8vRARULOY31oQ
> tm4Hf3PHVXuKa/ZBQutLFOolJo7IhaYP3CtBqE+i7OWSlyy0dsqdqO40z9+vzt2n
> DBiMRXl0Y2HGCeRsm0owdsFFDqA/j0xcCTBjgckgR6TcnRPc926FZVmr+q53DEQ1
> rYVo3Kfum7AnLP3y4rVT0SsxavjI48aXqCLKcM9RzRJ//D+p9teOeiHiUtu4CzHY
> aQmkV22N6LC3M5uBwNNU1xXr62SNiarqY7euurPhPcOkbQSi4ckfknh48JzenQ41
> Ws7XvuLGOmTcLOv+rsKYjBd5s6IxuBH/+k5MfttPQaZ8mHAieMjEnVszmjZon2rE
> Mqqcd+C5Z0q2/X9wUAwNAD3muQTzx2A8C3uucJHVygvwNy76UCUCoyLakQ98/8WL
> 3SKN2l3EddObdi4OUrfga80ZTLf0AnBoflmKz+2UAbP3Xit++XHBs5dBgvN51Tji
> d6IdBRJpSq/njZmnSGQYJ/4o07v31YgLjh+xZTS+8wxm5H3C4V6/IuWlsnYPZWi5
> YQRe0GPZw54IuLs9WZG6AbNcAzhGOW+OBIMGbzSKQukeLAVpjws=
> =KUgn
> -END PGP SIGNATURE-
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: URL-encoding and "#"

[Mark Thomas]

On 13/10/2017 17:09, James H. H. Lampert wrote:
> Thanks to all of you who responded.
> 
> I found a web page that explains it in ways that I can wrap my
> 55-year-old brain around, and has an easy-to-read reference chart.
> 
> https://perishablepress.com/stop-using-unsafe-characters-in-urls/
> 
> Question: the problem first showed up on a web service that takes a
> "bodyless" POST operation, and I assume it also applies to GET
> operations, and to the URL portion of a POST with a body.
> 
> But what about the body of a POST?

>From an HTTP specification point of view, anything goes. You are only
limited by whatever rules are imposed by the component that processes
that data.

Mark


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: URL-encoding and "#"

[James H. H. Lampert]

Thanks to all of you who responded.

I found a web page that explains it in ways that I can wrap my 
55-year-old brain around, and has an easy-to-read reference chart.


https://perishablepress.com/stop-using-unsafe-characters-in-urls/

Question: the problem first showed up on a web service that takes a 
"bodyless" POST operation, and I assume it also applies to GET 
operations, and to the URL portion of a POST with a body.


But what about the body of a POST?

--
JHHL

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: tomcat 8.5.23 dbcp not honoring autocommit = false?

[Chris Cheshire]

As a further test I just took out my explicit rollback in my
DAOFactory close() method, and swapped back to commons dbcp. Added an
update that wasn't explicitly committed, and it correctly did not get
committed when the connection was closed. Swapped back to tomcat dbcp
and repeated, it got committed without an explicit commit statement.

I'm really puzzled as to why *I* have to explicitly rollback on close
if autocommit is not enabled, instead of tomcat dbcp handling that
when commons dbcp appears to do it.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: tomcat 8.5.23 dbcp not honoring autocommit = false?

[Chris Cheshire]

On Thu, Oct 12, 2017 at 11:16 PM, Christopher Schultz
 wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Chris,
>
> On 10/11/17 5:21 PM, Chris Cheshire wrote:
>> Working on a migration from 7 to 8.5, and in it I am now using the
>>  tomcat dbcp, instead of apache commons dbcp.> I have found that
>> with no other changes to the db code (except the factory param for
>> the resource), it is working fine other than there is an implicit
>> commit happening when I close a connection, even with autocommit
>> turned off in mysql config, resource config AND in my code.
> Your complaint is very close to my heart, here. <3
>
> Back in 2003 or so, I posted roughly this exact question to this
> mailing list with a little less ... diplomacy, shall we say?
>
>> try { this.dbConn = this.dataSource.getConnection();
>> this.dbConn.setAutoCommit(false);
>> this.dbConn.setTransactionIsolation(Connection.TRANSACTION_READ_COMMIT
> TED);
>>
>>
> }
>> catch (SQLException ex) { throw new DAOException("unable to get
>> database connection", ex); }
>
> I'll bet you've had this problem for a really long time, but just
> didn't notice it until now.
>

Nope, only since swapping from commons dbcp (tomcat 7.x) to tomcat
dbcp in development.
I started with 8.5.20 and upgraded yesterday to 8.5.23 and it still
exhibits this behaviour.

> The core problem is that you have autocommit=false in your
> configuration and autocommit=true in your code. If an exception occurs
> and you don't rollback the transaction, the connection pool will reset
> all of the settings to your configured settings (including
> autocommit=true). Setting autocommit=true when autocommit=false
> commits the transaction, which is SUPER surprising to anyone who
> hasn't read the Javadoc[1]
>

I *don't* have autocommit=true in code, unless

this.dbConn.setAutoCommit(false);

doesn't mean what I think it means. You even have it in your example!


> Technically, this happens whether you encounter an exception or not,
> but it's fairly rare to have code that intentionally does this:
>
> conn.setAutoCommit(false);
> // UPDATE ...;
> conn.close();
>
> So, given that this is usually an "exceptional" situation, it's your
> exceptions you need to carefully handle. In fact, you need to do more
> than you are used to doing.
>
> Have a look at this post I did years later when related questions kept
> coming up on the list:
> http://blog.christopherschultz.net/index.php/2009/03/16/properly-handlin
> g-pooled-jdbc-connections/
>

I have autocommit set to false in 3 ways :

1) /etc/my.conf : autocommit=0
2) context.xml  resource def : defaultAutoCommit=false
3) in code : dbConn.setAutoCommit(false)

When I query autocommit on the connection it returns false, yet
transactions are being committed when I issue a close() on the
connection after making changes and not explicitly committing.

Color me very, very confused.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: URL-encoding and "#"

[i...@flyingfischer.ch]

Am 13.10.2017 um 12:48 schrieb Alex O'Ree:
> Well that explains a lot. Similar issue for me. With url encoding,  tomcat
> is dropping back slash and the plus symbol.

While I think it is perfectly eligible to strive for a most perfect
alignement with standards and specs, I think Tomcat should allow a
reasonnable set of characters to be optionally allowed (as they already
are in Tomcat up to 8.5).

I am aware that these options may be a security issue and that the
documentation should state that clearly. However it is not always
possible to correct the environment to be "standard" compatible and the
educational approach by not allowing these options is understandable but
may be not appropriate in many situations.

Best regards
Markus

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: URL-encoding and "#"

[Alex O'Ree]

Well that explains a lot. Similar issue for me. With url encoding,  tomcat
is dropping back slash and the plus symbol.

On Oct 13, 2017 3:01 AM, "Mark Thomas"  wrote:

> On 13/10/2017 07:38, Peter Kreuser wrote:
> > Chris,
> >
> >
> >
> >
> > Peter Kreuser
> >> Am 13.10.2017 um 04:29 schrieb Christopher Schultz <
> w...@christopherschultz.net>:
> >>
> > James,
> >
>  On 10/12/17 8:44 PM, James H. H. Lampert wrote:
>  Question:
> 
>  The application we're developing has a suite of web services
>  (RESTful, Swagger-based), and at least one of them can accept a
>  pound sign ("#") as a URL parameter.
> 
>  Several months ago, with the application and all of its services
>  running on Tomcat 7, it was accepting a plain, naked # in the URL.
>  Now, running on Tomcat 8.5, it's returning an error message
>  ("HTTP/1.1 400").
> >
> > No client should ever send a naked # to a server. It's a violation of
> > the spec, full stop. That isn't to say that Tomcat should fail in any
> > particular way, but Tomcat is well within its rights to say "a # is
> > not allowed in a URL, so this is a bad request".
> >
> >
> >> Nevertheless there is AFAIR a commandline switch to set TC 8.5 to the
> old behavior.
>
> From memory, # isn't one of the allowed exceptions.
>
> The full list of invalid characters in the request line that Tomcat
> started to check for is:
> ' ', '\"', '#', '<', '>', '\\', '^', '`', '{', '|', '}'
>
> The allowed exceptions are (currently) '{', '|', '}'
>
> Mark
>
> >> James, please browse the mail archives.
> >> From a quick look this seems to help, for a short term solution:
> >
> >> https://marc.info/?l=tomcat-user&m=150183715500537&w=2
> >
> >> Please nevertheless fix the client, for a better world as Chris pointed
> out ;-P.
> >
> >> Best regards
> >
> >> Peter
> >
>  The developer (in a different time zone) has explained about
>  URL-encoding, but hasn't said whether there was anything in his
>  code to make it stop tolerating the naked # sign.
> 
>  Did the change from Tomcat 7 to Tomcat 8.5 have anything to do
>  with this?
> >
> > Each version of Tomcat gets more and more strict about the garbage it
> > will accept from clients. This is done to improve the world as a
> > whole, and also improve security when it comes to things like
> > converting URL paths into filesystem paths, etc. Strictly speaking,
> > everything should *always* be safe, but it helps to stop The Badness
> > at the earliest opportunity.
> >
>  And if so, are there any other common ASCII characters that used
>  to be accepted as characters, but now have to be URL-encoded?
> > Anything in the URL spec that is allowed should be allowed. Clients
> > should expect that anything not mentioned in the spec would be
> > rejected by a compliant server.
> >
> > -chris
> >>
> >> -
> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >> For additional commands, e-mail: users-h...@tomcat.apache.org
> >>
> >
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: URL-encoding and "#"

[i...@flyingfischer.ch]

Am 13.10.2017 um 09:01 schrieb Mark Thomas:
> From memory, # isn't one of the allowed exceptions.
>
> The full list of invalid characters in the request line that Tomcat
> started to check for is:
> ' ', '\"', '#', '<', '>', '\\', '^', '`', '{', '|', '}'
>
> The allowed exceptions are (currently) '{', '|', '}'
>
> Mark
By the way:

While fully agreeing, that the '#' character should not be sent by a
client to the server, it still would be desirable to have those three by
optional configuration allowed characters, also be made a configurable
exception in Tomcat 9.

As fas as I know, this option has been allowed up to and including
Tomcat 8.5 only.

While it is a good thing to save the world, real world scenarios may differ.

Thanks for considering.

Markus

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: URL-encoding and "#"

[Mark Thomas]

On 13/10/2017 07:38, Peter Kreuser wrote:
> Chris,
> 
> 
> 
> 
> Peter Kreuser
>> Am 13.10.2017 um 04:29 schrieb Christopher Schultz 
>> :
>>
> James,
> 
 On 10/12/17 8:44 PM, James H. H. Lampert wrote:
 Question:

 The application we're developing has a suite of web services
 (RESTful, Swagger-based), and at least one of them can accept a
 pound sign ("#") as a URL parameter.

 Several months ago, with the application and all of its services
 running on Tomcat 7, it was accepting a plain, naked # in the URL.
 Now, running on Tomcat 8.5, it's returning an error message
 ("HTTP/1.1 400").
> 
> No client should ever send a naked # to a server. It's a violation of
> the spec, full stop. That isn't to say that Tomcat should fail in any
> particular way, but Tomcat is well within its rights to say "a # is
> not allowed in a URL, so this is a bad request".
> 
> 
>> Nevertheless there is AFAIR a commandline switch to set TC 8.5 to the old 
>> behavior.

>From memory, # isn't one of the allowed exceptions.

The full list of invalid characters in the request line that Tomcat
started to check for is:
' ', '\"', '#', '<', '>', '\\', '^', '`', '{', '|', '}'

The allowed exceptions are (currently) '{', '|', '}'

Mark

>> James, please browse the mail archives.
>> From a quick look this seems to help, for a short term solution:
> 
>> https://marc.info/?l=tomcat-user&m=150183715500537&w=2
> 
>> Please nevertheless fix the client, for a better world as Chris pointed out 
>> ;-P.
> 
>> Best regards
> 
>> Peter
> 
 The developer (in a different time zone) has explained about 
 URL-encoding, but hasn't said whether there was anything in his
 code to make it stop tolerating the naked # sign.

 Did the change from Tomcat 7 to Tomcat 8.5 have anything to do
 with this?
> 
> Each version of Tomcat gets more and more strict about the garbage it
> will accept from clients. This is done to improve the world as a
> whole, and also improve security when it comes to things like
> converting URL paths into filesystem paths, etc. Strictly speaking,
> everything should *always* be safe, but it helps to stop The Badness
> at the earliest opportunity.
> 
 And if so, are there any other common ASCII characters that used
 to be accepted as characters, but now have to be URL-encoded?
> Anything in the URL spec that is allowed should be allowed. Clients
> should expect that anything not mentioned in the spec would be
> rejected by a compliant server.
> 
> -chris
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org