Re: Problem posting to Tomcat ssl connector ..
On Fri, May 28, 2021 at 01:32:47PM -0600, John Dale wrote: > Page loads fine. > > Other pages load fine. > > SSL handshakes are working until .. > > An Ajax post with a base64 encoded image in the data. > > I debugged up to the request and it's fine. > > I debugged the server and it's not reaching my component. > > Tomcat is killing the connection for some reason. > > Thought it might be maxSavePostSize .. bumped that up in the SSL connector. > > No dice. > > Request post is around 300K. > > What's going on!? I suppose there's nothing useful in the logs? Can anyone suggest adjustments that might log relevant observations? -- Mark H. Wood Lead Technology Analyst University Library Indiana University - Purdue University Indianapolis 755 W. Michigan Street Indianapolis, IN 46202 317-274-0749 www.ulib.iupui.edu signature.asc Description: PGP signature
Re: [OT] web app big memory usage?
Cris, On 6/1/21 09:17, Berneburg, Cris J. - US wrote: Hi Chris [lots of snippage] cb> One of our web apps is using a "lot" of memory, specifically a big cb> user query. We'd like to find out why. cb> 1. Is there a way to analyze uncollected garbage? cb> * AWS EC2 instance. cb> * There are other TC instances on the same server. cb> * Each TC instance has multiple apps. cs> What's the goal? Do you just Want To Know, or are you trying cs> to solve an actual problem. a. Barely enough memory to distribute among the multiple TC instances and the apps they support. A big enough user query (no throttling) causes OOME's. Attempting to determine if the code is being wasteful in some way and therefore could be made more efficient. What's the database? And the driver? MySQL Connector/J used to (still does?) read 100% of the results into the heap before Statement.executeQuery() returns unless you specifically tell it not to. So if your query returns 1M rows, you might bust your heap. It's entirely possible that other drivers do similar things. b. It's a dev app server (EC2) which hosts diff stages in the dev process - dev, test, and prototype streams. Multiple TC instances (3) because multiple copies of the apps don't play nice with each other. That is, we can't just rename the WAR files and expect the deployed apps to stay inside that context name (I think). You might want to look into that, eventually. If they aren't playing together nicely, they are not "good" servlet citizens. Solving those issues may improve other things. *shrug* c. I don't want to debug the code. I'm relatively new to the project, unfamiliar with some of the code, and anticipate getting lost in the weeds. See point #1 below. ;-) > cs> If you have a bunch of garbage that's not being cleaned up, cs> usually it's because there is simply no need to do so. The GC cs> is behaving according to the 3 laws of rob..., er, 3 virtues of cs> computing[1]: cs> cs>1. Laziness: nothing needs that memory so... meh cs>2. Impatience: gotta clean that Eden space quick cs>3. Hubris: if I ever need more memory, I know where to find it cs> [1] http://threevirtues.com/ Ha ha ha! :-) cs> How long does the query take to run? Dunno about the time on the DB query itself. From the user's point of view, a full minute plus. cs> What kind of query is it? Are we talking about something like SQL Yup. Classic RDMS back-end. cs> or some in-memory database or something which really does cs> take a lot of memory for the application to fulfill the request? Nah, nothing that fancy. The only fancy part is using node.js for the front-end. I followed Amit's and John's suggestion of using Eclipse Memory Analyzer Tool's "Keep unreachable options" when running a query from the app client. Digging deeper into the Leak Suspects Report, I saw a StringBuilder - 264MB for the supporting byte array and 264MB for the returned String, about 790MB total for that piece of the pie. Contents were simply the JSON query results returned to the client. No mystery there. Yep: runaway string concatenation. This is a devolution of the "Connector/J reads the whole result set into memory before returning" thing I mentioned above. Most JSON endpoints return arbitrarily large JSON responses and most client applications just go "duh, read the JSON, then process it". If your JSON is big, well, then you need a lot of memory to store it all if that' who you do things. If you want to deal with JSON at scale, you need to process it in a streaming fashion. The only library I know that can do streaming JSON is Noggit, which was developed for use with Solr (I think, maybe it came from elsewhere before that). Anyway, it's ... not for the faint of heart. But if you can figure out out, you can handle petabytes of JSON with a tiny heap. I suspect that repeating the process with multiple queries will reveal multiple StringBuilder's each containing big honking JSON results. Very likely. You might want to throttle/serialize queries you expect to have big responses so that only e.g. 2 of them can be running at a time. Maybe all is well when they come one-at-a-time, but if you try to handle 5 concurrent "big responses" you bust your heap. The short-term solution is to simply not allow that much concurrency. So the issue may not be a problem with efficiency so much as one of simple memory hogging. This can happen with almost any application. We serve ... many clients simultaneously and we run with a maximum of 10 connections in our db connection pool on each app server. It sounds "too small" but it handles the user-traffic without any user complaints. (At least StringBuilder is being used instead of plus-sign String concatenation.) In Java "..." + "..." uses a StringBuilder (unless the compiler can perform the concatenation at compile-time, in which case there is no concatenation at all). In some code, "..." + "..." is just fine and I hate i
Re: Pi Based Java Work
John, On 5/28/21 20:21, John Dale wrote: MariaDB has a sorting/limit problem that I haven't reported, yet (had to work around it). I'm interested in this. What is happening to you? -chris On 5/28/21, Christopher Schultz wrote: John, On 5/28/21 15:32, John Dale wrote: I debugged the server and it's not reaching my component. > > Request post is around 300K. Tomcat 9 on a raspberry pi 4 (w00t!). Maybe you are still just waiting around for that tiny CPU to run all that bytecode. /snark Seriously, though, I'd be interested to hear about your pi-based Java work in another thread. I have 2 Pi 4s and 2 Pi Zeros that I haven't managed to do anything with besides running EmulationStation (which I highly recommend for anyone who grew up with an NES. Pew-pew!). -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Problem posting to Tomcat ssl connector ..
John, On 5/28/21 20:17, John Dale wrote: ran apt-get install tomcat9 and it upgraded these packages: libtomcat9-java tomcat9 tomcat9-common Still did not resolve the issue, however. Looking for a guide to manually upgrade a package installed with apt-get. > > [repeated from elsewhere in the thread] Package: tomcat9 Version: 9.0.31-1~deb10u4 You may be misinterpreting what you are reading. Debian (and downstream) maintain stable versions of their packages. Basically, they (Debian, specifically) want a SUPER stable package that admins don't have to worry is going to blow-up on them with what should be minor upgrades. They usually pick a version to go with a major release and then add patches for things like security and ignore the new features, etc. and other things that might change. This is more difficult with a project like Tomcat which doesn't actually publish patches, but instead publishes new versions. I'm not sure what Raspbian does to upstream packages (probably nothing) or if they are Debian-based or Ubuntu-based. Ubuntu seems to me to be following the unstable branches of everything and less conservative than Debian, for better or worse. You may find that this bug isn't fixed in your version. You may also find that it won't be fixed until a new (Raspbian) release in the future. That future may take a while. You could look at the package availability in other branches of Raspbian (or Debian, whichever is appropriate) to see if maybe the unstable branch contains a better version (for you). You could also just install it yourself from official ASF packages any time. -chris On 5/28/21, Christopher Schultz wrote: John, On 5/28/21 15:32, John Dale wrote: I debugged the server and it's not reaching my component. > > Request post is around 300K. Tomcat 9 on a raspberry pi 4 (w00t!). Maybe you are still just waiting around for that tiny CPU to run all that bytecode. /snark Seriously, though, I'd be interested to hear about your pi-based Java work in another thread. I have 2 Pi 4s and 2 Pi Zeros that I haven't managed to do anything with besides running EmulationStation (which I highly recommend for anyone who grew up with an NES. Pew-pew!). -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: [OT] web app big memory usage?
Hi Chris [lots of snippage] cb> One of our web apps is using a "lot" of memory, specifically a big cb> user query. We'd like to find out why. cb> 1. Is there a way to analyze uncollected garbage? cb> * AWS EC2 instance. cb> * There are other TC instances on the same server. cb> * Each TC instance has multiple apps. cs> What's the goal? Do you just Want To Know, or are you trying cs> to solve an actual problem. a. Barely enough memory to distribute among the multiple TC instances and the apps they support. A big enough user query (no throttling) causes OOME's. Attempting to determine if the code is being wasteful in some way and therefore could be made more efficient. b. It's a dev app server (EC2) which hosts diff stages in the dev process - dev, test, and prototype streams. Multiple TC instances (3) because multiple copies of the apps don't play nice with each other. That is, we can't just rename the WAR files and expect the deployed apps to stay inside that context name (I think). c. I don't want to debug the code. I'm relatively new to the project, unfamiliar with some of the code, and anticipate getting lost in the weeds. See point #1 below. ;-) cs> If you have a bunch of garbage that's not being cleaned up, cs> usually it's because there is simply no need to do so. The GC cs> is behaving according to the 3 laws of rob..., er, 3 virtues of cs> computing[1]: cs> cs>1. Laziness: nothing needs that memory so... meh cs>2. Impatience: gotta clean that Eden space quick cs>3. Hubris: if I ever need more memory, I know where to find it cs> [1] http://threevirtues.com/ Ha ha ha! :-) cs> How long does the query take to run? Dunno about the time on the DB query itself. From the user's point of view, a full minute plus. cs> What kind of query is it? Are we talking about something like SQL Yup. Classic RDMS back-end. cs> or some in-memory database or something which really does cs> take a lot of memory for the application to fulfill the request? Nah, nothing that fancy. The only fancy part is using node.js for the front-end. I followed Amit's and John's suggestion of using Eclipse Memory Analyzer Tool's "Keep unreachable options" when running a query from the app client. Digging deeper into the Leak Suspects Report, I saw a StringBuilder - 264MB for the supporting byte array and 264MB for the returned String, about 790MB total for that piece of the pie. Contents were simply the JSON query results returned to the client. No mystery there. I suspect that repeating the process with multiple queries will reveal multiple StringBuilder's each containing big honking JSON results. So the issue may not be a problem with efficiency so much as one of simple memory hogging. (At least StringBuilder is being used instead of plus-sign String concatenation.) -- Cris Berneburg CACI Senior Software Engineer This electronic message contains information from CACI International Inc or subsidiary companies, which may be company sensitive, proprietary, privileged or otherwise protected from disclosure. The information is intended to be used solely by the recipient(s) named above. If you are not an intended recipient, be aware that any review, disclosure, copying, distribution or use of this transmission or its contents is prohibited. If you have received this transmission in error, please notify the sender immediately.
RE: [OT] web app big memory usage?
Hi Raghunath cb> One of our web apps is using a "lot" of memory, cb> specifically a big user query. We'd like to find out why. cb> 1. Is there a way to analyze uncollected garbage? rm> You could try using the Oracle utility - "jstat" - for analyzing rm> the GC in an active Java process (PID) rm> The "gcold" option helps us to peep into the Old Generation area rm> jstat -gcold PID rm> jstat -gcoldcapacity PID rm> rm> https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jstat.html That sounds interesting. :-) I'll look into it! -- Cris Berneburg CACI Senior Software Engineer This electronic message contains information from CACI International Inc or subsidiary companies, which may be company sensitive, proprietary, privileged or otherwise protected from disclosure. The information is intended to be used solely by the recipient(s) named above. If you are not an intended recipient, be aware that any review, disclosure, copying, distribution or use of this transmission or its contents is prohibited. If you have received this transmission in error, please notify the sender immediately. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Enhancement: Additional user attributes queried by (some) realms
On Tue, Jun 1, 2021 at 12:46 PM Carsten Klein wrote: > > > On 01/06/2021 10:18, Mark Thomas wrote: > > > I don't know if you can. I suspect not. By all means see if you can. I'm > > mildly curious to find out the answer. Whether you can or not, you don't > > need to. > > I found nothing to re-trigger the Travis CI build so far. However, now > the CI test is 'successful'. Did you do anything about it? > I've restarted it few hours ago. Only members of github.com/apache/tomcat team can restart builds via the TravisCI UI. And most probably Apache Infra team members. > > Carsten > > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: Enhancement: Additional user attributes queried by (some) realms
On 01/06/2021 10:18, Mark Thomas wrote: I don't know if you can. I suspect not. By all means see if you can. I'm mildly curious to find out the answer. Whether you can or not, you don't need to. I found nothing to re-trigger the Travis CI build so far. However, now the CI test is 'successful'. Did you do anything about it? Carsten - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Enhancement: Additional user attributes queried by (some) realms
On 01/06/2021 08:39, Carsten Klein wrote: Mark, On 01/06/2021 09:28, Mark Thomas wrote: We have been seeing that a lot lately. As far as I can tell, it is an issue with Travis CI. Can you use the PR anyway? Yes. We don't have a strict CI must pass rule. Whether or not a PR is applied a decision that is made by a committer. The tools are useful and inform that decision but the final say is made by a committer using their judgement. Can/must I re-trigger the Travis build? I don't know if you can. I suspect not. By all means see if you can. I'm mildly curious to find out the answer. Whether you can or not, you don't need to. Thanks, Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Enhancement: Additional user attributes queried by (some) realms
Mark, On 01/06/2021 09:28, Mark Thomas wrote: We have been seeing that a lot lately. As far as I can tell, it is an issue with Travis CI. Can you use the PR anyway? Can/must I re-trigger the Travis build? Carsten - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Encoding of LocalStrings_xy.properties files
Mark, On 01/06/2021 09:15, Mark Thomas wrote: Start Tomcat with: catalina jpda run (or start but I typically use run as I nearly always want to see what is logged to the console) In Eclipse go to Debug > Debug Configurations > Remote Java Application > New Configuration. Browse to the project and then click Debug. Tomcat's default jpda config matches Eclipse's so so should then have a remote debug session set up with your Tomcat instance. Trying that soon. Many thanks. Carsten - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Enhancement: Additional user attributes queried by (some) realms
On 29/05/2021 13:28, Carsten Klein wrote: Mark, On 27/05/2021 18:56, Carsten Klein wrote: Concerning removal of class UserDatabaseRealm.UserDatabasePrincipal: I will provide a PR and file a corresponding issue in Bugzilla soon. My PR and Bugzilla issue are present. However, Travis CI build failed on arm64 architecture for the PR with "An error occurred while generating the build script." All other tests/jobs (including my local builds) were fine. Currently, I have no idea how to deal with that. We have been seeing that a lot lately. As far as I can tell, it is an issue with Travis CI. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Encoding of LocalStrings_xy.properties files
On 28/05/2021 10:13, Carsten Klein wrote: Mark, On 28/05/2021 10:35, Mark Thomas wrote: No doubt that UTF-8 is the better encoding for messages and language files. And yes, my Eclipse actually does not use the version built by Ant. I use the start-tomcat.launch configuration file for starting Tomcat. Actually it only takes a startup-class name. So, it must obviously use the JARs built by Eclipse. The trick is, that in the build.xml file, you are actually converting message files: Simple. However, you do that after having them copied. While copying, you use filtering-copy and specify ISO-8859-1 as the file's encoding: [...] Should be UTF-8 now? Strictly, yes. Practically, it makes no difference because the filters that are applied do find and replacement with ASCII strings and are highly unlikely to ever be anything other than ASCII. I'll get that updated. Back to the Eclipse. I guess there is not much difference between calling Ant from the console and using Eclipse's Ant support (Run As -> Ant build). But, how to start that with support for debugging in Eclipse (may be a dumb questing, I know)? Start Tomcat with: catalina jpda run (or start but I typically use run as I nearly always want to see what is logged to the console) In Eclipse go to Debug > Debug Configurations > Remote Java Application > New Configuration. Browse to the project and then click Debug. Tomcat's default jpda config matches Eclipse's so so should then have a remote debug session set up with your Tomcat instance. Mark Carsten - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
