is there a way to unsubscribe from this list? thank you. Sent from my iPad
On Jan 25, 2011, at 7:12 PM, "John P. Rouillard" <rou...@cs.umb.edu> wrote: > > In message <4d3f7f27.5080...@umn.edu>, > Tim Peiffer writes: > >> I am interested in following Radius Detail records as a log input >> source. They are vertically aligned with attributes as 'tag = value' >> pairs. Are there any best practices for parsing such types of input >> with SEC? I tried using RegexpN, but found a number of the records were >> variable - some had other attributes that were being tracked, causing >> the number of lines to be somewhat inconsistent. In short, the parsing >> looked pretty gross. What words of wisdom can anyone impart for this >> kind of 'log' data? >> >> Mon Jan 24 23:33:38 2011 >> User-Name = "JoeUser" >> NAS-Port = 70496256 >> Service-Type = Framed-User >> Framed-Protocol = PPP >> Framed-IP-Address = 10.10.3.125 >> Called-Station-Id = "10.21.217.82" >> Calling-Station-Id = "172.16.29.46" >> Acct-Status-Type = Start >> Acct-Delay-Time = 2 >> Acct-Session-Id = "B1A039A3" >> Acct-Authentic = RADIUS >> NAS-Port-Type = Virtual >> Tunnel-Client-Endpoint = 172.16.29.46 >> NAS-IP-Address = 192.168.249.28 >> Timestamp = 1295933616 >> >> Tue Jan 25 01:15:57 2011 >> User-Name = "JoeUser" >> NAS-Port = 70496256 >> Service-Type = Framed-User >> Framed-Protocol = PPP >> Framed-IP-Address = 10.10.3.125 >> Called-Station-Id = "10.21.217.82" >> Calling-Station-Id = "172.16.29.46" >> Acct-Status-Type = Stop >> Acct-Delay-Time = 2 >> Acct-Input-Octets = 13855207 >> Acct-Output-Octets = 56621822 >> Acct-Session-Id = "B1A039A3" >> Acct-Authentic = RADIUS >> Acct-Session-Time = 6139 >> Acct-Input-Packets = 66890 >> Acct-Output-Packets = 84696 >> Acct-Terminate-Cause = User-Request >> NAS-Port-Type = Virtual >> Tunnel-Client-Endpoint = 172.16.29.46 >> NAS-IP-Address = 192.168.249.28 >> Timestamp = 1295939755 > > > I would treat this using a classic line at a time parser. I answered a > similar problem for gathering java stack traces a few years back but > sadly I can't find the message in any of the online archives anymore. > > It looks like every one of your events is started with a non space > character in column 1 and ends with a timestamp line. > > I wrote this recipe for recognizing a multiline pattern when the > number of lines is unknown for a class I taught a couple of years ago. > It may give you an idea. Sadly the message referenced in this recipe > no longer exists. > > ===== > SEC supports the use of a counter on the end of the a number of the > pattern types such as SubStr3 or RegExp10. The number N makes the > pattern match against the last N lines in SEC's input buffer every > time a new line is read. > > But you can have error messages in a multiline log message where the > number of lines changes depending on how the error was triggered. > > For example you may want to find and report all > java.lang.OutOfMemoryError messages in a java stack dump and record > all the lines that led up to that. The problem is that the call stack > is dynamic and you may not know the number of elements in the stack > that is needed to set the N. > > For example: > > Exception in thread "main" java.lang.OutOfMemoryError: PermGen space > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:620) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124) > at > org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:1847) > at > org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:873) > at > org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1326) > at > org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1205) > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:620) > > or > > Exception in thread "main" java.lang.OutOfMemoryError > at java.lang.Thread.start(Native Method) > at com.ibm.ctg.server.ThreadManager.createObject(ThreadManager.java:345) > at com.ibm.ctg.server.ThreadManager.<init>(ThreadManager.java:131) > at com.ibm.ctg.server.ManagedResources.<init>(ManagedResources.java:106) > at com.ibm.ctg.server.JGate.main(JGate.java:895) > > or > > 2008-04-23 06:18:50,418 [ERROR] > axis:org.apache.axis2.transport.http.AxisServlet unable to create new native > thread > java.lang.OutOfMemoryError: unable to create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Thread.java:574) > at > org.apache.commons.net.telnet.TelnetInputStream._start(TelnetInputStream.java:97) > at > org.apache.commons.net.telnet.TelnetClient._connectAction_(TelnetClient.java:104) > at org.apache.commons.net.SocketClient.connect(SocketClient.java:163) > at edu.internet2.hopi.dragon.DragonCSA.login(DragonCSA.java:60) > at net.es.oscars.pss.dragon.VlsrPSS.createPath(VlsrPSS.java:244) > at net.es.oscars.pss.PathSetupManager.create(PathSetupManager.java:91) > at > net.es.oscars.oscars.PathSetupAdapter.create(PathSetupAdapter.java:84) > at > net.es.oscars.oscars.OSCARSSkeleton.createPath(OSCARSSkeleton.java:438) > at > net.es.oscars.oscars.OSCARSMessageReceiverInOut.invokeBusinessLogic(OSCARSMessageReceiverInOut.java:174) > at > org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40) > > > To parse these out, use an idea discussed in: > http://sourceforge.net/mailarchive/message.php?msg_id=200510101206.j9AC6Srn029641%40mx1.cs.umb.edu. > Basically > you create a line at a time parser that recognizes the first line of a > dump and also recognized following lines. Note that the first line of > java stack dumps are not indented while the continuation lines are > indented. > > The following rules collect all the java error messages that occur > within 10 seconds after an OutOfMemory error. First identify the > starter line by looking for a line that does not start with a space > when we are not already collecting a java stackdump using this rule: > > type=single > rem = rule 1 > desc=Identify first line of stack dump > ptype = regexp > pattern = ^[^\s].*java\.lang\.OutOfMemoryError > context = !collecting_java_stackdump > action = create collecting_java_stackdump 10 \ > report collecting_java_stackdump \ > /bin/mailx -s "java OOM error" adminuser; \ > add $0 collecting_java_stackdump > > > This rule identifies the first line of a stack dump as a line that > does not have a space in the first column and contains the > OutOfMemoryError identifier. When we trigger this rule, it creates the > collecting_java_stackdump context which exists for 10 seconds. The > collecting_java_stackdump context has a lifetime of 10 seconds and > reports the contents of the context when it expires. As the last step, > the starting line is added to the collecting_java_stackdump context. > > Next we collect all the additional lines in the log file using: > > type=single > desc = accumulate continuation lines > rem = rule 2 > ptype = regexp > pattern = .? > context = collecting_java_stackdump > action = add $0 collecting_java_stackdump > > Note that this rule fires anytime the collecting_java_stackdump > context is valid, up to 10 seconds after the starter line is > recognized. > > Now suppose we want to collect only the one error message and not all > the errors for a 10 second period. One way of doing this is to add > another rule before the first rule that clears (and reports) the > collecting_java_stackdump context. Given the format of the java error > (starter line has a non-whitespace character in column 1 and > continuation lines start with whitespace) you can add the rule: > > type=single > desc = close current stackdump, start new one > rem = rule 0 > continue=takenext > ptype = regexp > pattern = ^[^\s] > context = collecting_java_stackdump > action = obsolete collecting_java_stackdump > > before rule 1. Rule 0 matches the start of a new stackdump only if we > are already collecting a stackdump. If we aren't matching a stackdump, > this rule is ignored. When this rule fires, it obsoletes (and thus > removes the collecting_java_stackdump context) but obsolete causes the > context expiration action to trigger. The expiration action reports > the collected stackdump. This rule has the continue keyword set to > takenext so that the event that triggered this rule is passed to the > following rule(s) to determine if the new stackdump should be > collected and reported. > =========== > > This doesn't discuss how to match a pattern where the interesting bit > is in the middle of the multiline sequence. > > For example, suppose I only wanted to report an out of memory error > multi-line event that contains: > > > org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1326 > > I create a single rule that matches this line if we are in the middle > of a stackdump, and this rule indicates that the line of interest is > present by creating a new context using the alias command. E.G. > > type=single > desc = mark this as an interesting stackdump > rem = put this before rule 2 > ptype = regexp > pattern = WebappClassLoader.loadClass.WebappClassLoader.java:1326 > context = collecting_java_stackdump > action = alias collecting_java_stackdump it_is_interesting > > and change rule 0 to be two rules, one that reports and one that > doesn't: > > type=single > desc = close current stackdump, report it and start new one > rem = rule 0a > continue=takenext > ptype = regexp > pattern = ^[^\s] > context = collecting_java_stackdump && it_is_interesting > action = obsolete collecting_java_stackdump > > and follow this with rule 0b > > type=single > desc = only if rule 0a did not fire does collecting_java_stackdump exist > rem = so delete the context it since it's not interesting and start a new one > rem = rule 0b > continue=takenext > ptype = regexp > pattern = ^[^\s] > context = collecting_java_stackdump > action = delete collecting_java_stackdump > > You can add more than one copy of the "mark this as an interesting > stackdump" single rule. Have each one create a unique context. Then > modify rule 0a so that it requires all the contexts to exist before it > triggers the report. > > type=single > desc = close current stackdump, report it and start new one > rem = rule 0a > continue=takenext > ptype = regexp > pattern = ^[^\s] > context = collecting_java_stackdump && it_is_interesting && \ > it_has_a_second_interesting_line && it_has_a_third_line_too > action = obsolete collecting_java_stackdump > > So does this give you any ideas? > > -- > -- rouilj > John Rouillard > =========================================================================== > My employers don't acknowledge my existence much less my opinions. > > ------------------------------------------------------------------------------ > Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)! > Finally, a world-class log management solution at an even better price-free! > Download using promo code Free_Logger_4_Dev2Dev. Offer expires > February 28th, so secure your free ArcSight Logger TODAY! > http://p.sf.net/sfu/arcsight-sfd2d > _______________________________________________ > Simple-evcorr-users mailing list > Simple-evcorr-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users ------------------------------------------------------------------------------ Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)! Finally, a world-class log management solution at an even better price-free! Download using promo code Free_Logger_4_Dev2Dev. Offer expires February 28th, so secure your free ArcSight Logger TODAY! http://p.sf.net/sfu/arcsight-sfd2d _______________________________________________ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
