Robert - (Sorry for top-posting, but this is just about GlassFish 4.1, not Dan’s problem.)
GlassFish 4.1 is a bit hard to find these days, but it can be downloaded from here: https://javaee.github.io/glassfish/download Also, 4.1 refuses to run on a modern JVM, resulting in this message: GlassFish requires Java SE version 6. Your JDK is version 0 Mine is currently 23.0.2; not sure how far back one will have to go… - Chuck > On 2025 Aug 16, at 14:02, Robert Turner <rtur...@e-djuster.ca.INVALID> wrote: > > Dan, > > You are doing well to keep up -- we keep throwing lots at you. I think we > all find this issue so bizarre we really want to figure it out. We are, > even if it doesn't look like it, slowly eliminating some potential causes. > > I've put a few comments inline, and then a larger update on some testing I > was doing this morning to try to reproduce what you are observing in your > live system. > > Questions / suggestions for Dan marked with @Dan > > On Sat, Aug 16, 2025 at 2:31 PM Chuck Caldarale <n82...@gmail.com > <mailto:n82...@gmail.com>> wrote: > >> >>> On 2025 Aug 16, at 12:44, Daniel Schwartz <d...@danielgschwartz.com> >> wrote: >>> >>> From: Chuck Caldarale <n82...@gmail.com> >>> Sent: Saturday, August 16, 2025 1:33 PM >>> To: Tomcat Users List <users@tomcat.apache.org> >>> Subject: Re: [EXTERNAL EMAIL] How to access a REST service >>> >>>> Immediately after doing this, the message copied below started >> appearing repeatedly in the server.log and my website stopped working---the >> DB evidently became inaccessible. However, Glassfish was still running, >> and as soon as I set the maximum pool size back to 1000, these messages >> stopped coming and the website started working again. >>> >>> Indicative of an orphaned connection (resource leak). >>> >>> DGS: Why is this? What do those messages have to do with a resource >> leak? >> >> >> When the pool is at its maximum and all connections are in use, GlassFish >> will hold requests for a configurable amount of time (max-wait, defaults to >> 60 seconds), and only abort the request and log the situation after the >> caller has been suspended for that amount of time. Since your web service’s >> response time is very quick, this indicates the only allocated connection >> has stayed in-use for over a minute - something leaked it. >> >> When the maximum pool size is set to your usual value, GlassFish >> eventually cleans things up because it periodically discards all existing >> connections and acquires new ones just in case they have become unusable >> from the DB point of view. >> > > Just to add that there is another situation, but is unlikely based on the > default HTTP thread count maximums. It could be that you are getting a > large amount of connections faster than they can be processed -- i.e. let's > say 100 per 10 ms would overflow the connection pool if you had a limit of > 50 and your average response time was 10 ms. This was why I asked about > your settings for HTTP thread limits. > > I will check Glassfish 4.1 a bit later today, but with Glassfish 7.0.25 > ("out of the box") the limit on the HTTP processing threads was 5, and > without increasing that, it's not possible to have more than 5 typical > requests being processed at once. That means that if each request never > acquires more than 1 connection (which is the case in all the examples you > have provided), the connection pool usage should never grow beyond 5. > > So, unless you have increased the HTTP processing thread limits from > defaults to a value larger than the default thread pool maximum size, or > the deployment you are using does such, you shouldn't have seen an issue > originally if all the connections are being released to the pool. The "peak > value", as tracked in Glassfish, should never exceed this thread count > unless a connection is not being returned to the pool. > > Thus, the conclusion that there is a leak somewhere, but where... I'm not > sure at this point. > This is where I think we need to focus on -- possibly by dialing back your > settings, and also by looking at what's going on with the robot requests -- > are they causing faults that aren't visible"? > (more on this at the end) > > @Dan: It would be a really good idea though (as it's been said, and been > acknowledged) to wrap all the connection / statement / result-set > acquisitions with try-with-resources. Not only will it rule out leaks from > those going forward, it will also simplify your code and make it easier to > read. I would do this sooner so we can try to rule out leaks from those > activities. > > > >> >>> So this is very interesting, and points to a possible line of >> investigation. If the web crawlers are throwing garbage at your app, that >> could possibly be resulting in losing track of connections, statements, >> result sets, etc. You should carefully check your error paths and make sure >> that all DB-related resources are being properly disposed of when invalid >> input is received. >>> >>> DGS: I'm fairly sure that all connections and prepared statements are >> being closed, but will look again. This would be happening in the code >> fragment that tries to retrieve the list of holidays with calculated dates >> (not the countries). This is somewhat complex and could have problems. >> >> >> Your GetCountryList code from a previous message doesn’t dispose of the >> ResultSet object; depending on the implementation details of the JDBC >> driver and pool, closing the result set may not be required, but it’s >> always good practice to do so. Since the ResultSet object contains a >> reference to the Connection (and vice-versa), it may be interfering with >> reuse of the connection until a garbage collection happens. >> > > > Okay, so while I was in between things this morning I did a whole bunch of > testing. > > With Glassfish 7.0.25, with JDK 21, and the latest PostgreSQL JDBC driver, > I was unable to reproduce the "leak" problem using code very similar to > Dan's. > > I tried all of the following: > - enabling and disabling the JDBC object wrapping in Glassfish > - using the "singleton" data source or retrieving one each request > - not closing the Statement or the ResultSet / or closing them > - not closing the Connection or closing it > > The only situation where I could reproduce a leak (and very quickly) was to > fail to close the connection. (remove it from the try-with-resources > block). I could not get the lack of closing the ResultSet or the Statement > to trigger a "leak" in the connection pool -- but that doesn't mean one can > ignore that, only that it didn't definitely cause the issue in my test > setup. > > I was not however running any "real queries". I was just running "SELECT 1" > each time and returning the integer to the caller. So no data was coming > from the client that could be triggering other potential failures. > > My next plan is to switch to MySQL JDBC (as I noted this is what Dan is > using) and see if maybe there is something different with that JDBC driver > and do the same tests. > > @Dan: Could you share the exact version of the JDBC driver you are using? > > And then, I'll try rolling back to Glassfish 4.1.x and see if that makes a > difference. > > @Dan: Is there a minor version, or is it 4.1.latest ? > > > Dan: it would also be a good idea to "verify" the parameters coming in for > "valid values" without using a DB query (if that is possible) so as to > avoid the robots triggering DB connections. Ultimately this is a risk of a > DoS type of attack. You can also employ rate limiting as well to help avoid > such problems. > > @Dan: I think we also need to change your "printing to Stdout" to print to > the Glassfish logger. The problem with printing to Stdout is that only the > HTTP client making the request that triggers these logs will see them. This > could be "masking" some errors where you aren't doing the testing yourself. > (I could be wrong about where Stdout output goes though -- so if you do see > those in the logs, then ignore this item. I will validate that shortly > myself in Glassfish 7.0.25). > > @Dan: Have you been able to get a rough count of the request rate you are > seeing on your server, so we have some idea if it's even possible to be > seeing an overload situation or if we can rule that out. > > > Robert > (who is clearly spending way too much time on Dan's issue, but is very > puzzled and wants to get to the bottom of it, and as a result has now used > Glassfish for the first time... :) )
