I'm not sure where the OSC locking and unlocking happens off the top of my head, but if it's waiting on a lock, it could still be the database. You can set up database monitoring on cpu/ram/io just to rule that out as a culprit. Otherwise, if it is your app, you might find the problem running your app through a profiler.
Also, you can check to see if your app is not returning stats on javamonitor. That usually indicates a deadlock. If you're running multiple instances, you could be failing over to a new instance after a timeout due to the instance being deadlocked. If you see your instances deadlocking, you can configure jmx and then use something like jstat to detect where your deadlock is happening. It was a bit of a pain setting that up on my prod apps, but it was definitely worth it. You can see all kinds of useful info on your running prod apps and find problems you didn't even know you had. Also, set up monitoring on your app server. It might just be you're being hammered by a botnet and that's bogging down all your network IO you use for your DB. Or you could have a rouge application eating all your app server cpu. Whenever you're stuck, just set up more and more monitoring. :) It's never a waste of time, and you can often find your problem that way. ________________________________ From: OCsite <o...@ocs.cz> Sent: Thursday, February 1, 2024 7:39 PM To: OCsite via Webobjects-dev <webobjects-dev@lists.apple.com> Cc: Jérémy DE ROYER <jeremy.dero...@ingencys.net>; Ramsey Gurley <ramsey.gur...@practicemojo.com> Subject: Re: loooong saveChanges prep-stage You don't often get email from o...@ocs.cz. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Thanks again! Though I do not really think the database would be the culprit. Does actually EOF do anything with the DB when saving changes before DatabaseContextDelegate.databaseContextWillPerformAdaptorOperations is called? I might be wrong as so often, but I don't think so; I believe it contacts the DB for the first time only after that. Thanks and all the best, OC On 1. 2. 2024, at 6:12, Ramsey Gurley <ramsey.gur...@practicemojo.com> wrote: Are you hosting your database? If so, don't rule out a problem outside your application and in your database. It could be low memory. If you can't fit all the tables/indexes into memory, you fall off a performance cliff by going to disk. It could also be an antivirus scanner that kicked off on your DB's data directory. Once it starts scanning and locking files, it can ruin performance. If you can catch it during these slow periods, ssh into the box and check with top and iostat. Look for something on the machine tying up resources. It could be a poorly tuned database too, your box has plenty of memory, but your database isn't configured correctly to use it. For example, if you're using Postgres: https://pgtune.leopard.in.ua/ It could even be below your database config and in your OS config. On linux, it's highly dependent on which file system you are using (ext4, btrfs, xfs, etc), but maybe you have insufficent read-ahead, or some other filesystem setting. Even if you're not using Postgres, you should probably get a copy of Postgresql High Performance and read the first few chapters which covers hardware tuning below the database. It could also be your database is busy reindexing or freeing up table space. On postgres, vacuuming and reindexing can lock tables until the job is done. This can take several minutes or more depending on the size of your table. It could be the autovacuum doing it. If you're not vacuuming/reindexing, that can hurt your performance too, since indexes will grow to exceed your available memory and drop you off the performance cliff. You can do the maintenance and get around the locking problems using something like pg_repack. ________________________________ From: OCsite <o...@ocs.cz> Sent: Wednesday, January 31, 2024 7:59 PM To: OCsite via Webobjects-dev <webobjects-dev@lists.apple.com> Cc: Jérémy DE ROYER <jeremy.dero...@ingencys.net>; Ramsey Gurley <ramsey.gur...@practicemojo.com> Subject: Re: loooong saveChanges prep-stage You don't often get email from o...@ocs.cz. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Thanks, guys! I am pretty sure though the problem can't be a background process either reading for a long time or saving for a long time, for I do use the ERXAdaptorChannelDelegate.trace logs and through DatabaseContextDelegate.databaseContextWillPerformAdaptorOperations I log each save — and there's nothing like that in the log in the vicinity of those long saveChanges, alas. Thus the culprit must be something else. Perhaps indeed something locks the OSC pretty often and for a long time, but that something is neither a long SELECT which would log through ERXAdaptorChannelDelegate.trace, nor another unrelated save, which would log through DatabaseContextDelegate.databaseContextWillPerformAdaptorOperations. Besides, it does not really feel like OSC locks caused by another thread. Meantime, I've rigged an awk script to compute how long each saveChanges takes, and it looks like this: - for a long time, all is OK - when the save times begin to grow, they keep consistently long (e.g., about 30 s each, or about 50 s each) for each save for awhile (a quarter or half an hour), before things get back to normal If another thread locked OSC, it would most probably mean some saveChanges would be long, but some quick; it does not seem probable a background thread would consistently keep OSC locked so that each saveChanges takes roughly the same (long) time. This rather feels by something at the beginning of saveChanges becomes slow. This would most probably happen under the OSC lock, and given the way it works, does not seem really plausible that it is simply waiting to acquire the lock itself. For the moment, I'm rather outta ideas :( Thanks again and all the best, OC ________________________________ Confidentiality Notice: This email, including all attachments and replies thereto, are covered by the Electronic Communications Privacy Act, 18 U.S.C. Sections 2510-2521 and are legally privileged. This information is confidential, and intended only for the use of the individuals or entities named above. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or the taking of any action in reliance on the contents of this transmitted information is strictly prohibited. Please notify us if you have received this transmission in error. Thank you. ________________________________ Confidentiality Notice: This email, including all attachments and replies thereto, are covered by the Electronic Communications Privacy Act, 18 U.S.C. Sections 2510-2521 and are legally privileged. This information is confidential, and intended only for the use of the individuals or entities named above. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or the taking of any action in reliance on the contents of this transmitted information is strictly prohibited. Please notify us if you have received this transmission in error. Thank you.
_______________________________________________ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
