Re: [OT] Best way to log requests from a servlet and to a database?
Caldarale, Charles R wrote: From: Brian Braun [mailto:brianbr...@gmail.com] Subject: Re: [OT] Best way to log requests from a servlet and to a database? However, if I get significantly more requests, this may not be enough because MySQL will get slower and the queue will get full. I think I could start using this queue, and if it gets full I could failover to Amazons Queue service. What do you think? I think it would be cheaper for you to just spool the excess logging objects to your own local disk if the in-memory queue reaches some predetermined limit. However, if your DB server really can't keep up, you'll need some strategy to just discard log entries or reject requests until it can, even if you overflow to local disk. Not being myself a Java specialist, and given the expressed requirements and transactions volume, my instinctive reaction would be to just write the log entries as some kind of text line into a local disk file, and then process that disk file with a totally separate program, to create the required log entries in the back-end database. The idea is to make the action of writing the log entries as fast, simple and lightweight as possible for the real-time process (the webapp), and leave the brunt of the work to a separate process which can run at a more leisurely pace, deal with access contention to the back-end db, only run at specified intervals and times etc. The local disk file can be rotated regularly, to prevent each file instance from becoming too large, and the process which reads the files could only process the rotated ones (to avoid contention with the webapp on the current logfile). Such a design would provide a kind of buffering automatically, with a predictable time and memory overhead for the webapp writing each log entry to the local logfile. Considering the disk capacities available nowadays, I believe that it would be quite unlikely that one would ever reach the point where the logfiles volume would get too large to fit. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Best way to log requests from a servlet and to a database?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Brian, On 1/26/13 11:29 PM, Brian Braun wrote: I finally found this, implemented it and works great! http://docs.oracle.com/javase/6/docs/api/java/util/concurrent/LinkedBlockingQueue.html However, if I get significantly more requests, this may not be enough because MySQL will get slower and the queue will get full. MySQL shouldn't get slower unless something is badly misconfigured. If you have lots of indexes on your logging table, you're doing it wrong. What table type, etc. are you using? MySQL might not be the best solution for logging. Do you really need ACID? I think I could start using this queue, and if it gets full I could failover to Amazons Queue service. What do you think? I think if you expect to failover, you should consider why that would happen. If you can't handle your normal load using just MySQL, then you may as well go for the complete Amazon solution. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlEGlnsACgkQ9CaO5/Lv0PBgaQCfa+e50e7hirdyD/o6ngLqCxyv EkUAoKyzk+L/L7YAdytKs+5aUNtqED+W =t6eB -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: [OT] Best way to log requests from a servlet and to a database?
From: Brian Braun [mailto:brianbr...@gmail.com] Subject: Re: Best way to log requests from a servlet and to a database? (Marking this off-topic, since it has nothing to do with Tomcat.) My current method can hold about 3000 threads until memory collapses. I'm looking to replace this method with something more efficient in terms of RAM usage. Something like a buffer, where each item needs far less RAM than the kind of threads I'm creating now. Then when it gets full I will use Amazon's queue as a failover mechanism.Any ideas? :-) Instead of using one additional thread per log entry, use just _one_ additional logging service thread. When each request processing thread has prepared the object representing the log entry to be made, that object should be placed on a synchronized FIFO queue. If the logging service thread is idle, the request processing thread should mark the logging service thread as active and wake it up before unlocking the queue; if the logging service thread is already active, the request processing thread just unlocks the queue after enqueuing its entry. When the logging service thread wakes up, it must lock the queue, remove _all_ the entries currently on the queue, unlock the queue, and send all the removed entries to the database in as few calls as possible. Once that's done, the logging service thread relocks the queue, checks for any new arrivals and repeats as needed. If no new arrivals, it marks itself as inactive, and goes to sleep on the queue. No polling required. You can use the standard synchronization methods (wait(), notify(), etc.) for all this (safest), or the newer but easier to abuse java.util.concurrent operations. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Best way to log requests from a servlet and to a database?
Hi chuck, I finally found this, implemented it and works great! http://docs.oracle.com/javase/6/docs/api/java/util/concurrent/LinkedBlockingQueue.html However, if I get significantly more requests, this may not be enough because MySQL will get slower and the queue will get full. I think I could start using this queue, and if it gets full I could failover to Amazons Queue service. What do you think? On Sat, Jan 26, 2013 at 9:06 AM, Caldarale, Charles R chuck.caldar...@unisys.com wrote: From: Brian Braun [mailto:brianbr...@gmail.com] Subject: Re: Best way to log requests from a servlet and to a database? (Marking this off-topic, since it has nothing to do with Tomcat.) My current method can hold about 3000 threads until memory collapses. I'm looking to replace this method with something more efficient in terms of RAM usage. Something like a buffer, where each item needs far less RAM than the kind of threads I'm creating now. Then when it gets full I will use Amazon's queue as a failover mechanism.Any ideas? :-) Instead of using one additional thread per log entry, use just _one_ additional logging service thread. When each request processing thread has prepared the object representing the log entry to be made, that object should be placed on a synchronized FIFO queue. If the logging service thread is idle, the request processing thread should mark the logging service thread as active and wake it up before unlocking the queue; if the logging service thread is already active, the request processing thread just unlocks the queue after enqueuing its entry. When the logging service thread wakes up, it must lock the queue, remove _all_ the entries currently on the queue, unlock the queue, and send all the removed entries to the database in as few calls as possible. Once that's done, the logging service thread relocks the queue, checks for any new arrivals and repeats as needed. If no new arrivals, it marks itself as inactive, and goes to sleep on the queue. No polling required. You can use the standard synchronization methods (wait(), notify(), etc.) for all this (safest), or the newer but easier to abuse java.util.concurrent operations. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: [OT] Best way to log requests from a servlet and to a database?
From: Brian Braun [mailto:brianbr...@gmail.com] Subject: Re: [OT] Best way to log requests from a servlet and to a database? However, if I get significantly more requests, this may not be enough because MySQL will get slower and the queue will get full. I think I could start using this queue, and if it gets full I could failover to Amazons Queue service. What do you think? I think it would be cheaper for you to just spool the excess logging objects to your own local disk if the in-memory queue reaches some predetermined limit. However, if your DB server really can't keep up, you'll need some strategy to just discard log entries or reject requests until it can, even if you overflow to local disk. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
