Re: [OT] Best way to log requests from a servlet and to a database?
Caldarale, Charles R wrote: From: Brian Braun [mailto:brianbr...@gmail.com] Subject: Re: [OT] Best way to log requests from a servlet and to a database? However, if I get significantly more requests, this may not be enough because MySQL will get slower and the queue will get full. I think I could start using this queue, and if it gets full I could failover to Amazons Queue service. What do you think? I think it would be cheaper for you to just spool the excess logging objects to your own local disk if the in-memory queue reaches some predetermined limit. However, if your DB server really can't keep up, you'll need some strategy to just discard log entries or reject requests until it can, even if you overflow to local disk. Not being myself a Java specialist, and given the expressed requirements and transactions volume, my instinctive reaction would be to just write the log entries as some kind of text line into a local disk file, and then process that disk file with a totally separate program, to create the required log entries in the back-end database. The idea is to make the action of writing the log entries as fast, simple and lightweight as possible for the real-time process (the webapp), and leave the brunt of the work to a separate process which can run at a more leisurely pace, deal with access contention to the back-end db, only run at specified intervals and times etc. The local disk file can be rotated regularly, to prevent each file instance from becoming too large, and the process which reads the files could only process the rotated ones (to avoid contention with the webapp on the current logfile). Such a design would provide a kind of buffering automatically, with a predictable time and memory overhead for the webapp writing each log entry to the local logfile. Considering the disk capacities available nowadays, I believe that it would be quite unlikely that one would ever reach the point where the logfiles volume would get too large to fit. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Best way to log requests from a servlet and to a database?
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Brian,
On 1/25/13 9:00 PM, Brian Braun wrote:
Until recently, all this happened in a syncronous way. I mean,
before the Tomcat thread delivered the response, it had to create
the records in the database table. The problem is that this log
used to take more than 90% of the total time, and even worse: when
the database got slower then the service took about 10-15 seconds
instead of just 20 milliseconds.
Ouch.
What kind of tables are you using? How many records count as a log?
In how many tables? How many indexes do you have on that/those table/s?
I recetly made an improvement: Each Tomcat thread creates an extra
thread doing a (new Thread(new certain Object)).start(); that
takes care of the SQL insertion in an asyncronous way, so the
response gets to the clients faster. But these threads take too
much RAM when MySQL runs slower and threads multiply, and with a
few thousands of them the JVM Tomcat runs of the memory.
Yup: you'll at least want to run an Executor to limit the number of
new threads you create. That's a bad suggestion, anyway, because there
are better techniques you can use.
What I need is to be able to accept as much HTTP requests as
possible, to log every one of them as fast as possible (not
syncronously), and to make everything fast and with a very low
usage of RAM when MySQL gets slow and inserts need to queue. I
think I need some kind of queue to buffer the entries when the
speed of http request is higher than the speed of insertions in the
database log.
So, if you don't need synchronous log-writing does that mean you can
also tolerate some logs being missed? They are kind of two sides of
the came coin.
I'm thinking about these ideas:
1- Creating some kind of FIFO queue myself, maybe using some of
those Apache commons collections, and the some kind of thread that
polls the collection and creates the database records. But what
collection should I use? And how should I program the thread that
polls it, so it won't monopolize the CPU? I think that a Do while
(true) would eat the CPU cycles. And that about making it
thread safe? How to do it?
You can use while(!stop) { Object.wait(); dequeue(); } which will be
somewhat reasonable. Then you set stop when you want to shut-down
your queue-processor (like from a ServletContextListener). Be sure to
make that member volatile.
2- log4J? I have never used it directly, but it seems that this
framework is algo designed to creat appenders that talk to the
database. Would that be the way to do it?
If you can deal with the restrictions of log4j (basically, you
generate a line of text and that can get inserted into the db). I
haven't checked how the JDBCAppender works, but I would hope they
write more than one message to the db in each transaction.
Honestly, I would try log4j: it's already written for you and you can
log to it very simply and configure it simply as well. So, it's cheap
to compare it to your current situation.
- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEAREIAAYFAlEGlIIACgkQ9CaO5/Lv0PAEsACfVvL/sAxua5yYF2UvARPzt4jS
M6EAoIJ71sbsYQmUwHmfqWKCoLDFYCwS
=LlXJ
-END PGP SIGNATURE-
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Best way to log requests from a servlet and to a database?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Brian, On 1/26/13 11:29 PM, Brian Braun wrote: I finally found this, implemented it and works great! http://docs.oracle.com/javase/6/docs/api/java/util/concurrent/LinkedBlockingQueue.html However, if I get significantly more requests, this may not be enough because MySQL will get slower and the queue will get full. MySQL shouldn't get slower unless something is badly misconfigured. If you have lots of indexes on your logging table, you're doing it wrong. What table type, etc. are you using? MySQL might not be the best solution for logging. Do you really need ACID? I think I could start using this queue, and if it gets full I could failover to Amazons Queue service. What do you think? I think if you expect to failover, you should consider why that would happen. If you can't handle your normal load using just MySQL, then you may as well go for the complete Amazon solution. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlEGlnsACgkQ9CaO5/Lv0PBgaQCfa+e50e7hirdyD/o6ngLqCxyv EkUAoKyzk+L/L7YAdytKs+5aUNtqED+W =t6eB -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Best way to log requests from a servlet and to a database?
My current method can hold about 3000 threads until memory collapses. I'm looking to replace this method with something more efficient in terms of RAM usage. Something like a buffer, where each item needs far less RAM than the kind of threads I'm creating now. Then when it gets full I will use Amazon's queue as a failover mechanism.Any ideas? :-) On Sat, Jan 26, 2013 at 12:25 AM, Hassan Schroeder hassan.schroe...@gmail.com wrote: On Fri, Jan 25, 2013 at 8:16 PM, Brian Braun brianbr...@gmail.com wrote: OK, Amazon's solution would be too expensive to use as a first option. I would like to use it but just after a first queue is full. This first queue would be something running in my own host, using RAM to host the buffer. Any ideas on how to create this? Since you already have an implicit interface -- ...doing a (new Thread(new certain Object)).start(); write some tests describing the behavior of that class as a resource (in this case memory) threshold approaches. The actual design will probably be pretty apparent. Or at least you'll have a good start. /* been there, done that, drunk the TDD koolaid :-) */ -- Hassan Schroeder hassan.schroe...@gmail.com http://about.me/hassanschroeder twitter: @hassan - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: [OT] Best way to log requests from a servlet and to a database?
From: Brian Braun [mailto:brianbr...@gmail.com] Subject: Re: Best way to log requests from a servlet and to a database? (Marking this off-topic, since it has nothing to do with Tomcat.) My current method can hold about 3000 threads until memory collapses. I'm looking to replace this method with something more efficient in terms of RAM usage. Something like a buffer, where each item needs far less RAM than the kind of threads I'm creating now. Then when it gets full I will use Amazon's queue as a failover mechanism.Any ideas? :-) Instead of using one additional thread per log entry, use just _one_ additional logging service thread. When each request processing thread has prepared the object representing the log entry to be made, that object should be placed on a synchronized FIFO queue. If the logging service thread is idle, the request processing thread should mark the logging service thread as active and wake it up before unlocking the queue; if the logging service thread is already active, the request processing thread just unlocks the queue after enqueuing its entry. When the logging service thread wakes up, it must lock the queue, remove _all_ the entries currently on the queue, unlock the queue, and send all the removed entries to the database in as few calls as possible. Once that's done, the logging service thread relocks the queue, checks for any new arrivals and repeats as needed. If no new arrivals, it marks itself as inactive, and goes to sleep on the queue. No polling required. You can use the standard synchronization methods (wait(), notify(), etc.) for all this (safest), or the newer but easier to abuse java.util.concurrent operations. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Best way to log requests from a servlet and to a database?
Hi chuck, I finally found this, implemented it and works great! http://docs.oracle.com/javase/6/docs/api/java/util/concurrent/LinkedBlockingQueue.html However, if I get significantly more requests, this may not be enough because MySQL will get slower and the queue will get full. I think I could start using this queue, and if it gets full I could failover to Amazons Queue service. What do you think? On Sat, Jan 26, 2013 at 9:06 AM, Caldarale, Charles R chuck.caldar...@unisys.com wrote: From: Brian Braun [mailto:brianbr...@gmail.com] Subject: Re: Best way to log requests from a servlet and to a database? (Marking this off-topic, since it has nothing to do with Tomcat.) My current method can hold about 3000 threads until memory collapses. I'm looking to replace this method with something more efficient in terms of RAM usage. Something like a buffer, where each item needs far less RAM than the kind of threads I'm creating now. Then when it gets full I will use Amazon's queue as a failover mechanism.Any ideas? :-) Instead of using one additional thread per log entry, use just _one_ additional logging service thread. When each request processing thread has prepared the object representing the log entry to be made, that object should be placed on a synchronized FIFO queue. If the logging service thread is idle, the request processing thread should mark the logging service thread as active and wake it up before unlocking the queue; if the logging service thread is already active, the request processing thread just unlocks the queue after enqueuing its entry. When the logging service thread wakes up, it must lock the queue, remove _all_ the entries currently on the queue, unlock the queue, and send all the removed entries to the database in as few calls as possible. Once that's done, the logging service thread relocks the queue, checks for any new arrivals and repeats as needed. If no new arrivals, it marks itself as inactive, and goes to sleep on the queue. No polling required. You can use the standard synchronization methods (wait(), notify(), etc.) for all this (safest), or the newer but easier to abuse java.util.concurrent operations. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: [OT] Best way to log requests from a servlet and to a database?
From: Brian Braun [mailto:brianbr...@gmail.com] Subject: Re: [OT] Best way to log requests from a servlet and to a database? However, if I get significantly more requests, this may not be enough because MySQL will get slower and the queue will get full. I think I could start using this queue, and if it gets full I could failover to Amazons Queue service. What do you think? I think it would be cheaper for you to just spool the excess logging objects to your own local disk if the in-memory queue reaches some predetermined limit. However, if your DB server really can't keep up, you'll need some strategy to just discard log entries or reject requests until it can, even if you overflow to local disk. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Best way to log requests from a servlet and to a database?
On Fri, Jan 25, 2013 at 6:00 PM, Brian Braun brianbr...@gmail.com wrote: What I need is to be able to accept as much HTTP requests as possible, to log every one of them as fast as possible (not syncronously), and to make everything fast and with a very low usage of RAM when MySQL gets slow and inserts need to queue. I think I need some kind of queue to buffer the entries when the speed of http request is higher than the speed of insertions in the database log. 3- Using some kind of any other framework that specializes in this? http://aws.amazon.com/sqs/ Let us know how that works out if you try it :-) -- Hassan Schroeder hassan.schroe...@gmail.com http://about.me/hassanschroeder twitter: @hassan - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Best way to log requests from a servlet and to a database?
Hi Hassan, I forgot to mention that I am already using Amazon's cloud (EC2+load balancer) so I love to see a suggestion that mentions it! I will definitely check your advice. The problem is that it costs money to use it, and I would love to just use some kind of framework that uses my own RAM to host a buffer/queue. However, creating my own queue would not escalate to hundreds or thousands of requests per second, so considering that I was already considering some kind of queue service and actually I thought about Amazon's. but I did it while sleeping so I forgot to mention it in my list :-) Thanks again for your suggestion! On Fri, Jan 25, 2013 at 9:32 PM, Hassan Schroeder hassan.schroe...@gmail.com wrote: On Fri, Jan 25, 2013 at 6:00 PM, Brian Braun brianbr...@gmail.com wrote: What I need is to be able to accept as much HTTP requests as possible, to log every one of them as fast as possible (not syncronously), and to make everything fast and with a very low usage of RAM when MySQL gets slow and inserts need to queue. I think I need some kind of queue to buffer the entries when the speed of http request is higher than the speed of insertions in the database log. 3- Using some kind of any other framework that specializes in this? http://aws.amazon.com/sqs/ Let us know how that works out if you try it :-) -- Hassan Schroeder hassan.schroe...@gmail.com http://about.me/hassanschroeder twitter: @hassan - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Best way to log requests from a servlet and to a database?
OK, Amazon's solution would be too expensive to use as a first option. I would like to use it but just after a first queue is full. This first queue would be something running in my own host, using RAM to host the buffer. Any ideas on how to create this? On Fri, Jan 25, 2013 at 9:44 PM, Brian Braun brianbr...@gmail.com wrote: Hi Hassan, I forgot to mention that I am already using Amazon's cloud (EC2+load balancer) so I love to see a suggestion that mentions it! I will definitely check your advice. The problem is that it costs money to use it, and I would love to just use some kind of framework that uses my own RAM to host a buffer/queue. However, creating my own queue would not escalate to hundreds or thousands of requests per second, so considering that I was already considering some kind of queue service and actually I thought about Amazon's. but I did it while sleeping so I forgot to mention it in my list :-) Thanks again for your suggestion! On Fri, Jan 25, 2013 at 9:32 PM, Hassan Schroeder hassan.schroe...@gmail.com wrote: On Fri, Jan 25, 2013 at 6:00 PM, Brian Braun brianbr...@gmail.com wrote: What I need is to be able to accept as much HTTP requests as possible, to log every one of them as fast as possible (not syncronously), and to make everything fast and with a very low usage of RAM when MySQL gets slow and inserts need to queue. I think I need some kind of queue to buffer the entries when the speed of http request is higher than the speed of insertions in the database log. 3- Using some kind of any other framework that specializes in this? http://aws.amazon.com/sqs/ Let us know how that works out if you try it :-) -- Hassan Schroeder hassan.schroe...@gmail.com http://about.me/hassanschroeder twitter: @hassan - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
