-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Brian,
On 1/25/13 9:00 PM, Brian Braun wrote: > Until recently, all this happened in a syncronous way. I mean, > before the Tomcat thread delivered the response, it had to create > the records in the database table. The problem is that this log > used to take more than 90% of the total time, and even worse: when > the database got slower then the service took about 10-15 seconds > instead of just 20 milliseconds. Ouch. What kind of tables are you using? How many records count as a "log"? In how many tables? How many indexes do you have on that/those table/s? > I recetly made an improvement: Each Tomcat thread creates an extra > thread doing a "(new Thread(new certain Object)).start();" that > takes care of the SQL insertion in an asyncronous way, so the > response gets to the clients faster. But these threads take too > much RAM when MySQL runs slower and threads multiply, and with a > few thousands of them the JVM Tomcat runs of the memory. Yup: you'll at least want to run an Executor to limit the number of new threads you create. That's a bad suggestion, anyway, because there are better techniques you can use. > What I need is to be able to accept as much HTTP requests as > possible, to log every one of them as fast as possible (not > syncronously), and to make everything fast and with a very low > usage of RAM when MySQL gets slow and inserts need to queue. I > think I need some kind of queue to buffer the entries when the > speed of http request is higher than the speed of insertions in the > database log. So, if you don't need synchronous log-writing does that mean you can also tolerate some logs being missed? They are kind of two sides of the came coin. > I'm thinking about these ideas: > > 1- Creating some kind of FIFO queue myself, maybe using some of > those Apache commons collections, and the some kind of thread that > polls the collection and creates the database records. But what > collection should I use? And how should I program the thread that > polls it, so it won't monopolize the CPU? I think that a "Do while > (true)...." would eat the CPU cycles. And that about making it > thread safe? How to do it? You can use while(!stop) { Object.wait(); dequeue(); } which will be somewhat reasonable. Then you set "stop" when you want to shut-down your queue-processor (like from a ServletContextListener). Be sure to make that member volatile. > 2- log4J? I have never used it directly, but it seems that this > framework is algo designed to creat "appenders" that talk to the > database. Would that be the way to do it? If you can deal with the restrictions of log4j (basically, you generate a line of text and that can get inserted into the db). I haven't checked how the JDBCAppender works, but I would hope they write more than one message to the db in each transaction. Honestly, I would try log4j: it's already written for you and you can log to it very simply and configure it simply as well. So, it's cheap to compare it to your current situation. - -chris -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlEGlIIACgkQ9CaO5/Lv0PAEsACfVvL/sAxua5yYF2UvARPzt4jS M6EAoIJ71sbsYQmUwHmfqWKCoLDFYCwS =LlXJ -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org