On Sat, Aug 16, 2025, 15:44 Chuck Caldarale <n82...@gmail.com> wrote:
> Robert - > > (Sorry for top-posting, but this is just about GlassFish 4.1, not Dan’s > problem.) > > GlassFish 4.1 is a bit hard to find these days, but it can be downloaded > from here: https://javaee.github.io/glassfish/download > > Also, 4.1 refuses to run on a modern JVM, resulting in this message: > > GlassFish requires Java SE version 6. Your JDK is version 0 > > Mine is currently 23.0.2; not sure how far back one will have to go… > > - Chuck > Thanks for the heads up. That might be a bit tricky then... Maybe that won't happen then. > > > > On 2025 Aug 16, at 14:02, Robert Turner <rtur...@e-djuster.ca.INVALID> > wrote: > > > > Dan, > > > > You are doing well to keep up -- we keep throwing lots at you. I think we > > all find this issue so bizarre we really want to figure it out. We are, > > even if it doesn't look like it, slowly eliminating some potential > causes. > > > > I've put a few comments inline, and then a larger update on some testing > I > > was doing this morning to try to reproduce what you are observing in your > > live system. > > > > Questions / suggestions for Dan marked with @Dan > > > > On Sat, Aug 16, 2025 at 2:31 PM Chuck Caldarale <n82...@gmail.com > <mailto:n82...@gmail.com>> wrote: > > > >> > >>> On 2025 Aug 16, at 12:44, Daniel Schwartz <d...@danielgschwartz.com> > >> wrote: > >>> > >>> From: Chuck Caldarale <n82...@gmail.com> > >>> Sent: Saturday, August 16, 2025 1:33 PM > >>> To: Tomcat Users List <users@tomcat.apache.org> > >>> Subject: Re: [EXTERNAL EMAIL] How to access a REST service > >>> > >>>> Immediately after doing this, the message copied below started > >> appearing repeatedly in the server.log and my website stopped > working---the > >> DB evidently became inaccessible. However, Glassfish was still running, > >> and as soon as I set the maximum pool size back to 1000, these messages > >> stopped coming and the website started working again. > >>> > >>> Indicative of an orphaned connection (resource leak). > >>> > >>> DGS: Why is this? What do those messages have to do with a resource > >> leak? > >> > >> > >> When the pool is at its maximum and all connections are in use, > GlassFish > >> will hold requests for a configurable amount of time (max-wait, > defaults to > >> 60 seconds), and only abort the request and log the situation after the > >> caller has been suspended for that amount of time. Since your web > service’s > >> response time is very quick, this indicates the only allocated > connection > >> has stayed in-use for over a minute - something leaked it. > >> > >> When the maximum pool size is set to your usual value, GlassFish > >> eventually cleans things up because it periodically discards all > existing > >> connections and acquires new ones just in case they have become unusable > >> from the DB point of view. > >> > > > > Just to add that there is another situation, but is unlikely based on the > > default HTTP thread count maximums. It could be that you are getting a > > large amount of connections faster than they can be processed -- i.e. > let's > > say 100 per 10 ms would overflow the connection pool if you had a limit > of > > 50 and your average response time was 10 ms. This was why I asked about > > your settings for HTTP thread limits. > > > > I will check Glassfish 4.1 a bit later today, but with Glassfish 7.0.25 > > ("out of the box") the limit on the HTTP processing threads was 5, and > > without increasing that, it's not possible to have more than 5 typical > > requests being processed at once. That means that if each request never > > acquires more than 1 connection (which is the case in all the examples > you > > have provided), the connection pool usage should never grow beyond 5. > > > > So, unless you have increased the HTTP processing thread limits from > > defaults to a value larger than the default thread pool maximum size, or > > the deployment you are using does such, you shouldn't have seen an issue > > originally if all the connections are being released to the pool. The > "peak > > value", as tracked in Glassfish, should never exceed this thread count > > unless a connection is not being returned to the pool. > > > > Thus, the conclusion that there is a leak somewhere, but where... I'm not > > sure at this point. > > This is where I think we need to focus on -- possibly by dialing back > your > > settings, and also by looking at what's going on with the robot requests > -- > > are they causing faults that aren't visible"? > > (more on this at the end) > > > > @Dan: It would be a really good idea though (as it's been said, and been > > acknowledged) to wrap all the connection / statement / result-set > > acquisitions with try-with-resources. Not only will it rule out leaks > from > > those going forward, it will also simplify your code and make it easier > to > > read. I would do this sooner so we can try to rule out leaks from those > > activities. > > > > > > > >> > >>> So this is very interesting, and points to a possible line of > >> investigation. If the web crawlers are throwing garbage at your app, > that > >> could possibly be resulting in losing track of connections, statements, > >> result sets, etc. You should carefully check your error paths and make > sure > >> that all DB-related resources are being properly disposed of when > invalid > >> input is received. > >>> > >>> DGS: I'm fairly sure that all connections and prepared statements are > >> being closed, but will look again. This would be happening in the code > >> fragment that tries to retrieve the list of holidays with calculated > dates > >> (not the countries). This is somewhat complex and could have problems. > >> > >> > >> Your GetCountryList code from a previous message doesn’t dispose of the > >> ResultSet object; depending on the implementation details of the JDBC > >> driver and pool, closing the result set may not be required, but it’s > >> always good practice to do so. Since the ResultSet object contains a > >> reference to the Connection (and vice-versa), it may be interfering with > >> reuse of the connection until a garbage collection happens. > >> > > > > > > Okay, so while I was in between things this morning I did a whole bunch > of > > testing. > > > > With Glassfish 7.0.25, with JDK 21, and the latest PostgreSQL JDBC > driver, > > I was unable to reproduce the "leak" problem using code very similar to > > Dan's. > > > > I tried all of the following: > > - enabling and disabling the JDBC object wrapping in Glassfish > > - using the "singleton" data source or retrieving one each request > > - not closing the Statement or the ResultSet / or closing them > > - not closing the Connection or closing it > > > > The only situation where I could reproduce a leak (and very quickly) was > to > > fail to close the connection. (remove it from the try-with-resources > > block). I could not get the lack of closing the ResultSet or the > Statement > > to trigger a "leak" in the connection pool -- but that doesn't mean one > can > > ignore that, only that it didn't definitely cause the issue in my test > > setup. > > > > I was not however running any "real queries". I was just running "SELECT > 1" > > each time and returning the integer to the caller. So no data was coming > > from the client that could be triggering other potential failures. > > > > My next plan is to switch to MySQL JDBC (as I noted this is what Dan is > > using) and see if maybe there is something different with that JDBC > driver > > and do the same tests. > > > > @Dan: Could you share the exact version of the JDBC driver you are using? > > > > And then, I'll try rolling back to Glassfish 4.1.x and see if that makes > a > > difference. > > > > @Dan: Is there a minor version, or is it 4.1.latest ? > > > > > > Dan: it would also be a good idea to "verify" the parameters coming in > for > > "valid values" without using a DB query (if that is possible) so as to > > avoid the robots triggering DB connections. Ultimately this is a risk of > a > > DoS type of attack. You can also employ rate limiting as well to help > avoid > > such problems. > > > > @Dan: I think we also need to change your "printing to Stdout" to print > to > > the Glassfish logger. The problem with printing to Stdout is that only > the > > HTTP client making the request that triggers these logs will see them. > This > > could be "masking" some errors where you aren't doing the testing > yourself. > > (I could be wrong about where Stdout output goes though -- so if you do > see > > those in the logs, then ignore this item. I will validate that shortly > > myself in Glassfish 7.0.25). > > > > @Dan: Have you been able to get a rough count of the request rate you are > > seeing on your server, so we have some idea if it's even possible to be > > seeing an overload situation or if we can rule that out. > > > > > > Robert > > (who is clearly spending way too much time on Dan's issue, but is very > > puzzled and wants to get to the bottom of it, and as a result has now > used > > Glassfish for the first time... :) ) > >
