Re: Tomcat closes connections on unexpected status codes
Chris,
On Fri, Apr 19, 2024 at 4:40 AM Christopher Schultz
wrote:
>
> Pawel,
>
> On 4/18/24 20:21, Pawel Veselov wrote:
> >> On 18/04/2024 15:18, Stefan Ansing wrote:
> >>> Hi Rémy, Mark,
> >>> I just want to make sure that we’re understanding each other. I can see
> >>> that the connection needs to be closed in certain conditions to prevent
> >>> request smuggling attacks. I certainly don’t want to change that
> >>> behaviour.
> >>> However, I’m facing a scenario where an application is responding to a
> >>> valid request (from HTTP perspective), with a valid response using these
> >>> status codes (more specifically status codes 400 and 500).
> >> If the request is a valid HTTP request then a 400 status doesn't seem
> >> appropriate to me.
> >
> > It's by now, however, a de-facto standard. Every time I try to
> > determine "which HTTP response should I send back in case of issues
> > with the data", I find myself scrolling through the list of defined
> > codes and not finding anything that would otherwise fit. The HTTP
> > spec states what should the server do in case of HTTP protocol errors
> > (respond with an appropriate 4xx), but that's all that the spec
> > covers for the protocol, and it doesn't prohibit use of 400 for
> > application-level errors. Out of the entire 4xx codes, 400 is (maybe
> > also 414?) the only one that is used for protocol problems, others
> > are for application level errors, but they are very specific and
> > limiting (IMHO).
> When you say "protocol problems", what protocol are you referring to?
In this instance, I meant the actual problems with the HTTP protocol contents.
400 is normally returned by the container because it couldn't
understand the HTTP.
414 is probably as well (because container ran out of maximum
available space to read
in the URI). The rest of the error codes are expected to be produced by the
application, but they were devised for that application being a web server.
> Because if the request is readable and syntactically correct, there is
> no protocol problem. Everything else is ... something else. If you are
> establishing a protocol ON TOP OF HTTP then it's a violation of whatever
> protocol THAT is, not HTTP. So it's better to return { "error" : "Foo
> Protocol violation" } with an appropriate HTTP status code, possibly
> even *200*.
Yes. But (again, assuming web-service-like implementation), the application can
return other HTTP codes for other data (on top of HTTP) issues, i.e., when an
endpoint is not found, 404, when data type is wrong - 415; there are also those
exuberant codes like 409 and 410. So, on the surface, it looks like
the application
can use 4xx codes to signal an error, so one can lay down a contract that says
"2xx means OK response from the application's perspective, and if there is an
error, we'll return some 4xx code", but there *was* (see below) no good 4xx
code for saying "Hey, you forgot a JSON property in your input".
You're right, it probably should have been 2xx, but it's also cringy
to say things like
"there are some errors that we will return 4xx for, but for some it will be 2xx,
and here is how you differentiate successful 2xx from error response 2xx".
I did do that in at least one w/s implementation, and had the customer
yell at me for my trouble. This path is also not well supported by standards
like OAS/RAML (last time I checked, at least). The description of the responses
becomes significantly more complicated in this case.
And, again, there is AWS that is a rather large supplier of web services,
and they did decide to use 400 for this. The web-service client still needs
to distinguish between "400 because the header is mangled", or "400 because
a JSON property was missing". The RFC doesn't explicitly limit the use of
400 to HTTP protocol problems, even though all the examples that are listed
in the spec *are* such.
I did look again at the specs just now, and lo and behold, there is now a
code 422. Which, at least on the surface, looks like exactly the code to use
for responding with application-related content problems. It wasn't there in
RFC#7231 (and neither in #2616 or #2068), but #9110 does have it (yet
the language around 400 is still overly broad), which makes it about 2
years old.
Now, of course, existing w/s contracts aren't likely to switch to using that,
but for any new ones I'll certainly consider it.
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat closes connections on unexpected status codes
On Thu, Apr 18, 2024 at 9:40 AM Adwait Kumar Singh wrote: > > I'm not (yet) convinced distinguishing between those scenarios is always > > going to be possible. > I have a Tomcat patch which we use at work to do this, i.e always close the > connection if HTTP parsing fails but not if it's a user set status. I can > create a PR for feedback. It can't be that straightforward though. The HTTP parsing can very well fail well past the point the application was invoked, and returned a status, during chunked request parsing. The server can't respond with another error though, as the headers are/can be committed, it has no recourse but to try to send them if possible, and then shutdown the connection. And handling incomplete body reads from the application is also going to be a nuisance that has to be dealt with. Tomcat will have to consume the remainder of the body without application doing so, which may not be desirable in all cases, and will require either an API change to be supported as being signalled by the application (here is my status, but PLEASE don't break the HTTP connection), or requiring the application to diligently consume the body till the end before returning service back to Tomcat, or having that as a configuration parameter. I actually don't know how Tomcat deals with bodies that aren't fully read by the app right now. I would very much like to know for sure. > > On Thu, Apr 18, 2024 at 8:42 AM Mark Thomas wrote: > > > On 18/04/2024 15:18, Stefan Ansing wrote: > > > Hi Rémy, Mark, > > > > > > > > > > > > I just want to make sure that we’re understanding each other. I can see > > > that the connection needs to be closed in certain conditions to prevent > > > request smuggling attacks. I certainly don’t want to change that > > behaviour. > > > > > > However, I’m facing a scenario where an application is responding to a > > > valid request (from HTTP perspective), with a valid response using these > > > status codes (more specifically status codes 400 and 500). > > > > If the request is a valid HTTP request then a 400 status doesn't seem > > appropriate to me. > > > > If the server is correctly handling that request to generate the > > response, a 500 status doesn't seem right either. > > > > > > > > I don’t think that in this scenario a request smuggling attack could be > > > executed, or am I missing something? > > > > The main issue is if the original request is invalid HTTP there is no > > way to determine where the next HTTP request starts. > > > > If there is a proxy in the mix then the risks of something going wrong > > tend to go up. > > > > Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat closes connections on unexpected status codes
> On 18/04/2024 15:18, Stefan Ansing wrote: > > Hi Rémy, Mark, > > I just want to make sure that we’re understanding each other. I can see > > that the connection needs to be closed in certain conditions to prevent > > request smuggling attacks. I certainly don’t want to change that behaviour. > > However, I’m facing a scenario where an application is responding to a > > valid request (from HTTP perspective), with a valid response using these > > status codes (more specifically status codes 400 and 500). > If the request is a valid HTTP request then a 400 status doesn't seem > appropriate to me. It's by now, however, a de-facto standard. Every time I try to determine "which HTTP response should I send back in case of issues with the data", I find myself scrolling through the list of defined codes and not finding anything that would otherwise fit. The HTTP spec states what should the server do in case of HTTP protocol errors (respond with an appropriate 4xx), but that's all that the spec covers for the protocol, and it doesn't prohibit use of 400 for application-level errors. Out of the entire 4xx codes, 400 is (maybe also 414?) the only one that is used for protocol problems, others are for application level errors, but they are very specific and limiting (IMHO). Then, how do I return error paths, when implementing things like a web service? Either returning it with 200 and custom headers that indicate it's an error response, or a 400 with the same. In which case, the difference on the client side is - does it have to logically differentiate valid response from an error based just on these custom headers/mime type, or differentiate protocol-related 400 from application-related 400, and can always treat 200 as proper. Then I look at "what do others do", and I see AWS that uses 400 for all client errors, and the way a client knows it's an application error is because there is content that wouldn't be typically returned if there is a protocol problem. So I follow suit. > If the server is correctly handling that request to generate the > response, a 500 status doesn't seem right either. Right. 5xx - my fault, 4xx - your fault. > > I don’t think that in this scenario a request smuggling attack could be > > executed, or am I missing something? > The main issue is if the original request is invalid HTTP there is no > way to determine where the next HTTP request starts. I'd imagine it reasonable that as long as Tomcat doesn't find any issues with the HTTP protocol itself - closing connections wouldn't serve any helpful purpose in that case, except for not having to read the remainder of the message. I'd like to know how the connection can be mucked up otherwise, whether a proxy is involved or not. > > If there is a proxy in the mix then the risks of something going wrong > tend to go up. > > Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Cookie "expires" recent format change vs. Apache HTTP client 4.x
Hello. Ran into an interesting problem yesterday. After updating Tomcat to 9.0.69 to mitigate a disclosed CVE, some of our tests started failing. Investigation uncovered that the HTTP client stopped accepting "set-cookie" headers because of "expires" attribute values. There was a change to the format of the "expires" cookie attribute, to replace dashes with spaces between date/month/year : https://github.com/apache/tomcat/commit/d641a43b8a1a0a09862e101c304d2874b0771c6e Interestingly, RFC-6265, section 5.1.1 allows for these delimiters between any of the date components: %x09 / %x20-2F / %x3B-40 / %x5B-60 / %x7B-7E So, dash (0x2d), was a legitimate delimiter. Apache HTTP client 4.x, however, by default, applies "default" cookie policy, which has a JavaDoc description of: > This policy provides a higher degree of compatibility with common cookie > management of popular HTTP agents for non-standard (Netscape style) cookies. However, what happens in reality, is that if the cookie has "expires" attribute, it's always treated as a "Netspace" cookie, which has the only supported date format of EEE, dd-MMM-yy HH:mm:ss z, see sources of org.apache.http.impl.cookie.DefaultCookieSpec (I have a gnawing feeling I'm missing something here, because format wants 2-digit year, and Tomcat was sending 4 digits. But the outcome is the same - if the date has spaces between the date components, cookies are not accepted, with the "default" cookie policy). It seems that cookie parsing is rather haphazard in the Apache HTTP client. I'm sure this change is breaking things on the client sides here and there, as a result, and I haven't seen any topics on this in the list, so thought somebody might find this useful. SUMMARY: Changing something THAT old was bound to create problems for some clients, I'm not sure how useful this change was in general. Even if the change remains compliant to the relevant specs. Thank you, Pawel. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat 9x NIO connector socket timeout
Hello. I was wondering what exact value does Tomcat 9x use for NIO connector socket timeouts? I.e., when the following exception occurs: org.apache.catalina.connector.ClientAbortException java.net.SocketTimeoutException at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:353) at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:783) at org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:688) at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:388) at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:366) at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:96) , it means that the underlying socket's .setSoTimeout() has been set to a positive value. Reading https://tomcat.apache.org/tomcat-9.0-doc/config/http.html, I see: socket.soTimeout - "This is equivalent to standard attribute connectionTimeout". Does it mean that setting this property is the same as setting connectionTimeout, or that in the absence of socket.soTimeout being explicitly set, the value of connectionTimeout is used? "connectionTimeout" says that it's only about waiting for a request line after a TCP connection is accepted. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat in distroless image
Christopher, Stephan, On Tue, Jul 5, 2022 at 11:18 PM Christopher Schultz wrote: > > Stefan, > > On 7/2/22 09:45, Stefan Mayr wrote: > > Hi, > > > > Am 01.07.2022 um 17:10 schrieb Christopher Schultz: > >> Thomas, > >> > >> On 6/30/22 13:52, Thomas Meyer wrote: > >>> Sadly currently Tomcat startup relies on shell script to bootstrap > >>> JVM process. > >>> > >>> In the light of distroless images (e.g. > >>> https://blog.chainguard.dev/introducing-apko-bringing-distroless-nirvana-to-alpine-linux/) > >>> > >>> > >>> What are you thoughts on packaging tomcat in distroless base OCI > >>> images that doesn't even contain any shell anymore? > >>> > >>> Would it be possible to provide an alternative start mechanism which > >>> directly starts JVM process which setup/prepare env like the current > >>> catalina.sh shell script does? > >>> > >>> What are your thoughts on above topic? > >> > >> Speaking for myself, here, of course. > >> > >> The Tomcat team doesn't package anything other than the "standard > >> distributions" such as the source and the pre-built Java binaries, plus > >> the Windows binaries for various things. We don't package anything > >> specific like Docker base containers, etc. and I don't think we want to > >> get into that business. > >> > >> If someone in the community wants to do something like that: go for it. > >> But we have enough to do already and don't want to play favorites. > >> > >> The only reason catalina.sh exists is to figure out what's going on with > >> the hosting environment. If you want to build a launch-process that is > >> integrated into a specific environment, then you don't need all that > >> tooling to figure out what is what: you can just write one big command > >> line and launch the JVM with all the necessary arguments. > > > > I agree with that. In my opinion this part of environment detection is > > not necessary in a container because a container is immutable. So it > > should be enough to run Tomcat the way you want to have it and copy & > > paste the java command line into our container image generation manifest > > (e.g. Dockerfile). > > > > To use Tomcat in a more container friendly way you should consider also > > some other things too: > > - because containers are static there is no need to explode WAR files or > > scan the directory for changes. The extracted application can be copied > > into the image and server.xml can be tuned to disable those features. > > - you should avoid logging into files and tune the logging configuration > > to log everything to stdout and stderr Logging to stdout and stderr is a sure way to get oneself into a serious bottleneck. > This is something that I really don't understand about containerized > applications running like this. Maybe I'm too old-skool and think that > application logs go into application-log files and access-logs go into > access-log files. > > Assuming an ideal case, where should all of these things go if your > choices are "stdout" or "stderr"? > > 1. HTTP Access logs > 2. HTTP access logs from multiple virtual hosts (or is the idea that one > container ~= 1 virtual host) > 3. Server logs (e.g. DEBUG/INFO/WARN coming from the app server) > 4. Application logs (e.g. DEBUG/INFO/WARN coming from the application) > 5. Application logs for applications other than the primary (e.g. Tomcat > Manager) They don't go to files. They get sent to a logging router instead (e.g., Fluentd), which send them to logging aggregators (e.g., Graylog) > > -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat + Safari WebSocket issue
Florian, On Tue, May 3, 2022 at 3:12 PM Hagenauer, Florian wrote: > since Apple released Safari 15 (both iOS and macOS) I am running into a > strange issue related to Apache Tomcat, Safari/WebKit and certain sequences > of messages received via a WebSocket. When the browser receives messages in > this order, the connection gets closed. > > The following sequence triggers the issue: > 1. Connect from Safari to a Tomcat WebSocket server and use the > 'permessage-deflate' extension > 2. Receive a text message from the WebSocket server > 3. Receive a large binary message from the server > 4. The WebSocket connection is closed with the close code PROTOCOL_ERROR. Try capturing the traffic (with Wireshark, etc) for the broken connection against Tomcat, and then also to the "working" Python server, at least what are the differences on the wire. That would probably help identify the problem a lot faster. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: OpenSSL issues with Tomcat 9.0 on Corretto
Hello. On Fri, Jul 2, 2021 at 1:04 AM Pawel Veselov wrote: > > Hello. > > We've been using Tomcat 9 OpenJDK(8) images for a while, but are now > trying to switch to Corretto. I sincerely apologize. I didn't realize that Tomcat images weren't maintained by the Tomcat group. I probably need to take this here: https://github.com/docker-library/tomcat - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
OpenSSL issues with Tomcat 9.0 on Corretto
Hello. We've been using Tomcat 9 OpenJDK(8) images for a while, but are now trying to switch to Corretto. The problem we ran into is that tomcat-native is built with OpenSSL 1.0 libraries. That makes it impossible to use Ed25519 certificates. I don't think it's possible to rectify that at runtime. Are there any plans to switch to using OpenSSL 1.1 instead? Especially considering that the OpenJDK variant is built with 1.1? Thank you! -- With best of best regards Pawel S. Veselov - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Connection header override
Hello!
Tomcat 9.0.x
I'd like to force connection closure on some endpoints.
I'm trying this on a simple JSP page.
If I call response.setHeader("Connection","close"), I see that the
response has "Connection: close, keep-alive".
I assume Tomcat inserts the keep-alive part. It looks like the browsers
still close the connection based on this, but I was wondering if it's
possible to have Tomcat honor the header value set by the application.
I was also wondering what does it mean - when multiple connection tokens
are specified for a connection header. Breezing through the RFCs I can't
find a clear statement on that except that multiple connection tokens are
allowed in the header...
Thank you!
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Generating passwords digests for 9.0.27
Hello. I'm doing something where I need to generate a password for a tomcat user that is authenticated using org.apache.catalina.realm.UserDatabaseRealm with "sha" digest, the user database is produced by org.apache.catalina.users.MemoryUserDatabaseFactory from an xml file (standard conf/tomcat-users.xml) Reading https://tomcat.apache.org/tomcat-9.0-doc/realm-howto.html#Digested_Passwords I see that it says: If you are writing an application that needs to calculate digested passwords dynamically, call the static Digest() method of the org.apache.catalina.realm.RealmBase class, passing the cleartext password, the digest algorithm name and the encoding as arguments. This method will return the digested password. However, there is no static method Digest in org.apache.catalina.realm.RealmBase. What is the proper way to programmatically generate a proper password hash? Thank you, Pawel.
Re: Fwd: Tomcat-embed-core-9.0.12.jar bug about Content-Length Corrupting Parsing logic for Subsequent Request
Sorry for a rather rude intrusion. On Thu, Feb 7, 2019 at 4:18 PM Christopher Schultz wrote: > Chunked encoding is like sending a bunch of small HTTP message-pieces > (I have to be careful about my wording here, since "part" actually > means something in multipart messages May be just "chunks"? :) > Then, you send each chunk with a chunk size (in decimal bytes), a CRLF > pair, then your content, and another CRLF pair. The final chunk has a > length of 0 (zero): > > 32 > [32 bytes of data] > 128 > [128 bytes of data] > 0 Chunk sizes are expressed in hexadecimal notation, though, not decimal. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: database pool and minIdle support in 8
Apologies for reviving a zombie. On Sat, Feb 3, 2018 at 8:43 AM, Pawel Veselov <pawel.vese...@gmail.com> wrote: >>>> What is the problem with failures during pool initialization? >>> ConnectionPool.init() attempts to borrow initialSize worth of >>> connections. Imagine your database server is having problems at >>> this time. This will make init() fail, and the pool creation fail, >>> and your webapp is dead in the water after that >> Is it? Does init() in fact fail? Which pol are you talking about? > org.apache.tomcat.jdbc.pool.ConnectionPool. And actually, it doesn't. > I'm sure it did at some point before, that's why I set initialSize to > 0 (it's an old application), but I tested it on 8.0.49 and in 8.0.9 > just now, and the pool does come up and binds. So this is a non-issue. I thought I tested this, by shutting down the DB and restarting the application. But now I see that initialization does fail if a connection can not be created during pool creation. Running this on 8.0.39. Full exception is pasted into https://pastebin.com/SShyhdix The initialization errors can be controlled by isIgnoreExceptionOnPreLoad JMX property. I'll have to see how to set that before creating the pool. Thank you. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: database pool and minIdle support in 8
Chris, On Fri, Feb 2, 2018 at 2:20 PM, Christopher Schultz <ch...@christopherschultz.net> wrote: > On 2/2/18 1:47 PM, Pawel Veselov wrote: >>> On 2/1/18 6:08 PM, Pawel Veselov wrote: >>>> On Thu, Feb 1, 2018 at 1:02 PM, Mark Thomas <ma...@apache.org> >>>> wrote: >>>>> On 01/02/18 20:57, Pawel Veselov wrote: >>>>>> Hello. >>>>>> >>>>>> It looks like in tomcat 8 (looking at master's HEAD), the >>>>>> minIdle support is broken. According to docs, minIdle >>>>>> supposed to do : "The minimum number of established >>>>>> connections that should be kept in the pool at all times. >>>>>> The connection pool can shrink below this number if >>>>>> validation queries fail." >>>>>> >>>>>> I see that pool cleaner thread checks if the minIdle is >>>>>> *less* than pool size, and only then invokes checkIdle(). >>>>>> checkIdle() then will remove(!) connections from idle pool >>>>>> until the value drops down to minIdle. But I don't see any >>>>>> code that will add connections when minIdle is not met, and >>>>>> the documentation suggests that that's the intent. >>>>>> >>>>>> Am I misunderstanding something? Is there a way to keep a >>>>>> level of connections in the pool? I don't want to have >>>>>> initialSize control that, as I need to at 0, to prevent >>>>>> failures during pool initialization... >>>>> But that is what initial size is for. >>>> Indisputably so. However, that would be the only other way to >>>> achieve this, as long as the database connection don't die. I apologize, I actually misread what Mark has said here, reading the exact opposite of what he wrote. Have I read it correctly, I would have phrased my response differently, though I don't think that there is a difference at this point in this discussion. >>> What is the problem with failures during pool initialization? >> ConnectionPool.init() attempts to borrow initialSize worth of >> connections. Imagine your database server is having problems at >> this time. This will make init() fail, and the pool creation fail, >> and your webapp is dead in the water after that > Is it? Does init() in fact fail? Which pol are you talking about? org.apache.tomcat.jdbc.pool.ConnectionPool. And actually, it doesn't. I'm sure it did at some point before, that's why I set initialSize to 0 (it's an old application), but I tested it on 8.0.49 and in 8.0.9 just now, and the pool does come up and binds. So this is a non-issue. > If your webapp needs the pool to be working on startup, then delaying > the pool-initialization until "later" will only delay it a few ms. The point here was that I either have an existing pool with no connections, rather than no pool at all. >> it's unlikely that there are mechanisms to attempt to recreate the >> connection pool. > The connection pool is just fine. It just doesn't contain any connection > s. Right. Again, I had an assumption that initialization failed, otherwise my point had no sense. >> This complicates automatic recovery. An existing pool can recover >> easily, once the database is back online after a failure, the >> connections will be vended normally. > I would expect that any pool would recover when the database becomes > available again, regardless of when the first connection was attempted. Same here. >>> If you don't want the pool to make initialSize connections when >>> it starts, when DO you want the pool to create those connections? >>> During the first "cleaning"? >> I don't have strong expectations on exactly when the pool is to be >> inflated. minIdle suggests that it should happen at some point, may >> be that can be "cleaning", though this thread should probably then >> be renamed as "maintenance". If not inflated in time, then any >> requested connection will populate the pool, that's already what >> ends up going on in my case. >> >> In all cases, I think that either minIdle should be reimplemented, >> or at least the documentation should be changed from saying "The >> minimum number of established connections that should be kept in >> the pool at all times" to "Minimum idle connections that will >> remain in the pool after cleaning", with a note that idle >> connections are not created by the pool itself to fulfill that >> amount. > This may simply be an issue of semantics. When
Re: database pool and minIdle support in 8
> On 2/1/18 6:08 PM, Pawel Veselov wrote: >> On Thu, Feb 1, 2018 at 1:02 PM, Mark Thomas <ma...@apache.org> >> wrote: >>> On 01/02/18 20:57, Pawel Veselov wrote: >>>> Hello. >>>> >>>> It looks like in tomcat 8 (looking at master's HEAD), the >>>> minIdle support is broken. According to docs, minIdle supposed >>>> to do : "The minimum number of established connections that >>>> should be kept in the pool at all times. The connection pool >>>> can shrink below this number if validation queries fail." >>>> >>>> I see that pool cleaner thread checks if the minIdle is *less* >>>> than pool size, and only then invokes checkIdle(). checkIdle() >>>> then will remove(!) connections from idle pool until the value >>>> drops down to minIdle. But I don't see any code that will add >>>> connections when minIdle is not met, and the documentation >>>> suggests that that's the intent. >>>> >>>> Am I misunderstanding something? Is there a way to keep a level >>>> of connections in the pool? I don't want to have initialSize >>>> control that, as I need to at 0, to prevent failures during >>>> pool initialization... >>> But that is what initial size is for. >> >> Indisputably so. However, that would be the only other way to >> achieve this, as long as the database connection don't die. > > What is the problem with failures during pool initialization? ConnectionPool.init() attempts to borrow initialSize worth of connections. Imagine your database server is having problems at this time. This will make init() fail, and the pool creation fail, and your webapp is dead in the water after that, it's unlikely that there are mechanisms to attempt to recreate the connection pool. This complicates automatic recovery. An existing pool can recover easily, once the database is back online after a failure, the connections will be vended normally. > If you > don't want the pool to make initialSize connections when it starts, > when DO you want the pool to create those connections? During the > first "cleaning"? I don't have strong expectations on exactly when the pool is to be inflated. minIdle suggests that it should happen at some point, may be that can be "cleaning", though this thread should probably then be renamed as "maintenance". If not inflated in time, then any requested connection will populate the pool, that's already what ends up going on in my case. In all cases, I think that either minIdle should be reimplemented, or at least the documentation should be changed from saying "The minimum number of established connections that should be kept in the pool at all times" to "Minimum idle connections that will remain in the pool after cleaning", with a note that idle connections are not created by the pool itself to fulfill that amount. The reason prefer the pool to auto-saturate, is that so sudden demand for high amount of database connections doesn't require a lot of new connections to be established, as that's the slowest part of the process. Thank you, Pawel. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: database pool and minIdle support in 8
On Thu, Feb 1, 2018 at 1:02 PM, Mark Thomas <ma...@apache.org> wrote: > On 01/02/18 20:57, Pawel Veselov wrote: >> Hello. >> >> It looks like in tomcat 8 (looking at master's HEAD), the minIdle >> support is broken. According to docs, minIdle supposed to do : "The >> minimum number of established connections that should be kept in the >> pool at all times. The connection pool can shrink below this number if >> validation queries fail." >> >> I see that pool cleaner thread checks if the minIdle is *less* than >> pool size, and only then invokes checkIdle(). checkIdle() then will >> remove(!) connections from idle pool until the value drops down to >> minIdle. But I don't see any code that will add connections when >> minIdle is not met, and the documentation suggests that that's the >> intent. >> >> Am I misunderstanding something? Is there a way to keep a level of >> connections in the pool? I don't want to have initialSize control >> that, as I need to at 0, to prevent failures during pool >> initialization... > But that is what initial size is for. Indisputably so. However, that would be the only other way to achieve this, as long as the database connection don't die. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
database pool and minIdle support in 8
Hello. It looks like in tomcat 8 (looking at master's HEAD), the minIdle support is broken. According to docs, minIdle supposed to do : "The minimum number of established connections that should be kept in the pool at all times. The connection pool can shrink below this number if validation queries fail." I see that pool cleaner thread checks if the minIdle is *less* than pool size, and only then invokes checkIdle(). checkIdle() then will remove(!) connections from idle pool until the value drops down to minIdle. But I don't see any code that will add connections when minIdle is not met, and the documentation suggests that that's the intent. Am I misunderstanding something? Is there a way to keep a level of connections in the pool? I don't want to have initialSize control that, as I need to at 0, to prevent failures during pool initialization... Thank you, Pawel. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: How to make database connection pool show in JMX
Keiichi,
On Thu, Jan 25, 2018 at 1:48 AM, Keiichi Fujino <kfuj...@apache.org> wrote:
>
> Hi
>
> The Docs of jdbc-pool describe like this.
> ==
> If you're running outside of a container, you can register the DataSource
> yourself under any object name you specify, and it propagates the
> registration to the underlying pool. To do this you would call
> mBeanServer.registerMBean(dataSource.getPool().getJmxPool(),objectname).
> Prior to this call, ensure that the pool has been created by calling
> dataSource.createPool().
> ==
> http://tomcat.apache.org/tomcat-9.0-doc/jdbc-pool.html#JMX
>
> Thus you must register the DataSource as MBean.
Thank you. I dug around more after your pointers.
I am using 8.0.x (I should have said that). I found a few things
problematic with JMX around data sources.
I ended up calling MBeanRegistration.preRegister() on the data source.
This way I don't have to cast it to Tomcat classes. It does the job,
but not in a right way. I looked around and I noticed these facts:
- DataSource.createPool() is always called anyway when the data source
is created, by DataSourceFactory.createDataSource()
- Pool object is not supposed to be registered JMX object, there is a
special JMX object that is created is jmxEnabled is true.
- DataSource implementation looks like a JMX bean, it implements
MBeanRegistration and ConnectionPoolMBean, but trying to register it
directly throws a non-compliant JMX bean exception
- MBeanRegistration implementation of the DataSource violates API
contract. It doesn't return the ObjectName that is being used, and it
actually registers the bean with JMX (it shouldn't)
I had problems with that last fact, because if I register the bean, I
need to clean it up on shutdown, but I can't get the actual name that
the bean is registered as (except for copying the code that
preRegister ends up calling). I ended up fishing the bean out by
querying it before unregistering it.
I hope things are, in fact, better in 9.
Thank you,
Pawel.
> 2018-01-24 10:48 GMT+09:00 Pawel Veselov <pawel.vese...@gmail.com>:
>
> > Hello.
> >
> > I'd like to get some JMX stats out of the JDBC connection pools. But
> > they don't seem to register
> > in JMX, even though they are based on ConnectionPoolMBean.
> >
> > I do create the pools programmatically, by binding the factory into
> > the JNDI, the creation snippet is
> > copied below. When I search for JMX objects, I don't see anything that
> > looks like the pool info,
> > but I'm also not sure what the object name is supposed to be. I expect
> > it to be in "Catalina"
> > domain, though some of the code I saw suggests it may be in
> > "tomcat.jdbc" domain instead... I
> > don't have any "tomcat." domains at all.
> >
> > Any clues are appreciated.
> >
> > --
> > Properties p = new Properties();
> >
> > p.setProperty("type", "javax.sql.Datasource");
> > p.setProperty("defaultAutoCommit", "false");
> > // <...> set some other properties here
> > // enable JMX
> > p.setProperty("jmxEnabled", String.valueOf(Boolean.TRUE));
> > // JNDI of the actual data source
> > p.setProperty("dataSourceJNDI", pooledDSName);
> > Class dsfClass =
> > Class.forName("org.apache.tomcat.jdbc.pool.DataSourceFactory");
> > Object dsf = dsfClass.newInstance();
> > Method m = dsfClass.getDeclaredMethod("createDataSource",
> > Properties.class);
> > return m.invoke(dsf, p);
> > // the result is bound into JNDI
> > --
> >
> > Thank you!
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
How to make database connection pool show in JMX
Hello.
I'd like to get some JMX stats out of the JDBC connection pools. But
they don't seem to register
in JMX, even though they are based on ConnectionPoolMBean.
I do create the pools programmatically, by binding the factory into
the JNDI, the creation snippet is
copied below. When I search for JMX objects, I don't see anything that
looks like the pool info,
but I'm also not sure what the object name is supposed to be. I expect
it to be in "Catalina"
domain, though some of the code I saw suggests it may be in
"tomcat.jdbc" domain instead... I
don't have any "tomcat." domains at all.
Any clues are appreciated.
--
Properties p = new Properties();
p.setProperty("type", "javax.sql.Datasource");
p.setProperty("defaultAutoCommit", "false");
// <...> set some other properties here
// enable JMX
p.setProperty("jmxEnabled", String.valueOf(Boolean.TRUE));
// JNDI of the actual data source
p.setProperty("dataSourceJNDI", pooledDSName);
Class dsfClass =
Class.forName("org.apache.tomcat.jdbc.pool.DataSourceFactory");
Object dsf = dsfClass.newInstance();
Method m = dsfClass.getDeclaredMethod("createDataSource",
Properties.class);
return m.invoke(dsf, p);
// the result is bound into JNDI
--
Thank you!
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
