Cool sounds very useful. Now I have some reading for the weekend :) How hard would it be by default to seperate out the slow query logging from the rest. Would'nt that make any performance regressions stand out like a saw thumb. It would also represent a logical seperation.
Alan Alan Berg Group Education and Research Services Central Computer Services University of Amsterdam ________________________________ From: [email protected] [[email protected]] on behalf of Zach A. Thomas [[email protected]] Sent: 18 May 2012 19:24 To: OAE Development Subject: Re: [oae-dev] Log parsing and introductions. One of the very good features of Sling is the strangely named Logging Logger Configuration. You can use it to create individual logs as narrowly scoped as you like (at the package level or even to an individual class). And for each config, you also specify its own logging level. This is great for diagnosis when you don't want to pollute your main log with a torrent of DEBUG messages. There's a description of using this in the nakamura bootcamp document: https://confluence.sakaiproject.org/display/KERNDOC/5+Explore+the+server Good to have you, Alan! regards, Zach On May 18, 2012, at 6:20 AM, Berg, Alan wrote: Hi list, I am working closely with Kent Fitzgerald on testing Sakai OAE. Over the coming months I intend to reach out and ask for feedback through this list. I am not too old to learn :) I will be initially posting any work I do at: https://github.com/AlanBerg/SakaiOAE-Open My first practical comments and set of questions is about the out of the box logs from Sakai OAE. E.G: /sling/logs/error.log I find the log rich in useful information that system administrators need to understand to keep their systems stable. However, the volume and types of information makes for grep only review. Therefore, someone needs to write a non traditional log parser to separate out the different sorts of information and allow connection to automation such as a Continuous Integration Server. The parser should also look for new bug types as Sakai OAE is evolving rapidly. Here is a script that works by whitelisting each line of the log file. https://github.com/AlanBerg/SakaiOAE-Open/tree/master/SCRIPTS/logparser It looks for a set of known patterns which it then ignores or a set of patterns that it considers urgent. The patterns vary per log level. Any pattern that is not understood is added to an uncaught file. A summary is generated in HTML, slow queries are separated out into another file and the urgent to resolve bugs into the third. The log patterns are in text format in a configuration file. It is easy to update, but because of the richness of the log file patterns needs to be expanded against a wide range of realistic data. I will update slightly and hook into Jenkins the OAE's CI server. Now the questions. 0) Has anyone done anything similar. If so, lets compare notes :) 1. Can someone give me feedback on the tool. It is one thing to build and another to be useful. 2. If anyone wants to share some real logs other than the QA servers I can expand the patterns. 3. Feature requests (within reason) are welcome. 4. I will document how to hook up to a CI server, It can send notifications when certain events occur and draw custom plots over time. 5. The slow log is now separated out it mentions specifically slow queries. Does anyone work with these logs systematically. If so can you share. Looking forward to feedback. You are most welcome to contact me. I promise a positive response for a positive response :) Alan Berg _______________________________________________ oae-dev mailing list [email protected]<mailto:[email protected]> http://collab.sakaiproject.org/mailman/listinfo/oae-dev
_______________________________________________ oae-dev mailing list [email protected] http://collab.sakaiproject.org/mailman/listinfo/oae-dev
