My quick and short response after this, I'm planning to use NXlog at resource side and imo NXlog is capable to use RegEX. I saw many examples. NXlog will send the structured log in GELF format to Graylog and graylog can parse it.
Regards. On Tue, Feb 2, 2016 at 5:32 PM, Joi Owen <[email protected]> wrote: > As far as I know, using the regular regexp rules in graylog, there isn't a > way to tell it to just split on white space, and there isn't a way to tell > it to capture multiple values in one pattern. Not with the interfaces we > have available, anyway. > > In perl, I could do something like: > ($field1,$field2,field3) = split(/\s+/, $logline); > > I don't think the graylog system is nearly that powerful. > > The first pattern I gave you extracts the contents of the first field. > The ^ specifies the pattern starts at the beginning of the line. The > (.+)\s says to capture all characters and that it will end with a space. > Normally, .+ means 'match any character forever', but the rest of the > pattern has to match as well. The rest of it, \d+/\d+/20\d+ will only > match on the date field. So, by matching the date field there, we prevent > .+ from matching anything past the last space before the date, and so it > captures the entire sourceserver field. > > sourceserver: ^(.+)\s\d+/\d+/20\d\d\s+ > > I don't know anything about creating content packs. > > I think if you really want to use regexp, you just need to find yourself > a good tutorial and do some reading. This sort of basic regular expression > tutoring is likely outside the scope of this group. > > On Tue, Feb 2, 2016 at 3:46 AM, Mehmet Ali Büyükkarakaş < > [email protected]> wrote: > >> Hello again, >> >> I thinked a little bit about your response. >> >> In my log example, my assumptions are below ; (Dont get me wrong, I'm >> just trying to understand a regex coders point of view) >> >> - The first string until the first space char is the "SourceServer". It >> can vary on hostname number of characters. So do I have to write a complex >> regex for this ? If no, how can I define this as a field ? >> - and so... >> >> All I want to ask is, can I tell to the parser something like this ? >> >> " All the strings or numbers between spaces are fields. Your delimiter is >> the "space" char." >> >> >> bl-db01 02/01/2016 21:16:53.000000 14762 140124060886784 52 2 0 Total >> number of pauses: 0 >> >> My second question is, lets say that I figured out how to create correct >> regex statements. How will I insert into a content pack ? Any guide about >> this to forward me ? >> >> Best regards. >> Mehmet >> >> On Mon, Feb 1, 2016 at 10:18 PM, Joi Owen <[email protected]> wrote: >> >>> This is the sort of thing that can take some trial and error to get >>> perfect, and I doubt anyone on this list has messages of exactly this >>> format hitting their own servers, so it will be hard for us to give you >>> proven tested-and-good answers, we can only point you in the right >>> direction. >>> >>> You need a different rule for each variable you wish to create. If >>> every message in your log is of this form, you could start with something >>> like: (in some places I use a \s instead of a space just in case your mail >>> client reformats the strings and makes things even more confusing.) >>> >>> sourceserver: ^(.+)\s\d+/\d+/20\d\d\s+ >>> date: ^.+\s(\d+/\d+/20\d+) >>> time: ^.+\s(\d\d:\d\d:\d\d\.\d+ ) >>> pid: ^.+\s\d\d:\d\d:\d\d.\d+\s(\d+) \d+ >>> threadid: ^.+\s\d\d:\d\d:\d\d.\d+\s\d+s(\d+)\s+\d+\s+ >>> seq: ^.+\s\d\d:\d\d:\d\d.\d+\s\d+\s\d+\s+\d+\s+(\d+)\s+ >>> seq: ^.+\s\d\d:\d\d:\d\d.\d+\s\d+\s\d+\s+\d+\s+\d+\s+(\d+) >>> messageid: ^.+\s\d\d:\d\d:\d\d.\d+\s\d+\s\d+\s+\d+\s+\d+\s+\d+\s+(\d)\s+ >>> Message: ^.+\s\d\d:\d\d:\d\d.\d+\s\d+\s\d+\s+\d+\s+\d+\s+\d+\s+\d\s+(.+) >>> >>> None of this is tested, this is just what my first attempts would be. >>> And the graylog help page really is the best place to start if you don't >>> have any working examples to study. >>> >>> It really does help build regexp if the input has something that will be >>> consistently recognizable. In your sample, the only fields that are easy >>> to key on are the source field (because it comes first) the date field >>> (because it contains /) and the time field (because it contains : and . >>> between the digits.) All the rest are just counting field so of digits and >>> capturing the correct one for each variable. >>> >>> >>> >>> >>> On Mon, Feb 1, 2016 at 1:46 PM, Mehmet Ali Büyükkarakaş < >>> [email protected]> wrote: >>> >>>> Hello everybody, >>>> >>>> I have a log like this from Doubletake for Linux. >>>> >>>> bl-db01 02/01/2016 21:16:53.000000 14762 140124060886784 52 2 0 Total >>>> number of pauses: 0 >>>> >>>> The fields should be >>>> sourceserver, date, time, PID, ThreadID, SequenceNumber, Severity, >>>> MessageID, Message >>>> >>>> I want to put this raw syslog msg to fields and index in Graylog. >>>> Could you help me please to solve it quickly ? >>>> >>>> And how can I learn to convert these raw msgs and put into fields of >>>> graylog ? I have some resources about RegEx but using regex into graylog is >>>> not clear for me. (Dont redirect me to graylog help page, please) >>>> Thank you in advance. >>>> >>>> Mehmet >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Graylog Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/graylog2/10e8f9c2-ed14-4583-bf99-977748f24b13%40googlegroups.com >>>> <https://groups.google.com/d/msgid/graylog2/10e8f9c2-ed14-4583-bf99-977748f24b13%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> >>> No matter what we think of Linux versus FreeBSD, etc., the one thing I >>> really like about Linux is that it has Microsoft worried. Anything >>> that kicks a monopoly in the pants has got to be good for something. >>> - Chris Johnson >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "Graylog Users" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/graylog2/B-Uv3787TmE/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/graylog2/CAL5rfGW198fvX0B%2BZ-Gy2D%3D_WyYqKWAwX9yaDeYses9WUiwayg%40mail.gmail.com >>> <https://groups.google.com/d/msgid/graylog2/CAL5rfGW198fvX0B%2BZ-Gy2D%3D_WyYqKWAwX9yaDeYses9WUiwayg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> >> >> ------------------------------------------------------------------------------------------------- >> Alice - "Bana hangi yoldan gitmem gerektigini söyler misin?" >> "Bu neyi istedigine ve neye ulasmaya çalistigina bagli" dedi kedi >> "Sey, bilmem ki ??? " dedi Alice >> "O zaman hangi yoldan gittigin farketmez" dedi kedi. >> >> Alice Harikalar Diyarinda >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Graylog Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/graylog2/CACv48hDdr8cQprsrMKMgED4wVBpXUTqP6Zzk1ibpnQr7N%3DoJbQ%40mail.gmail.com >> <https://groups.google.com/d/msgid/graylog2/CACv48hDdr8cQprsrMKMgED4wVBpXUTqP6Zzk1ibpnQr7N%3DoJbQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > > No matter what we think of Linux versus FreeBSD, etc., the one thing I > really like about Linux is that it has Microsoft worried. Anything > that kicks a monopoly in the pants has got to be good for something. > - Chris Johnson > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Graylog Users" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/graylog2/B-Uv3787TmE/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/graylog2/CAL5rfGVhF6ATcaPVPDRA2Dp3orJxCRpfdpOmnwNA6SqWukGfHg%40mail.gmail.com > <https://groups.google.com/d/msgid/graylog2/CAL5rfGVhF6ATcaPVPDRA2Dp3orJxCRpfdpOmnwNA6SqWukGfHg%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- ------------------------------------------------------------------------------------------------- Alice - "Bana hangi yoldan gitmem gerektigini söyler misin?" "Bu neyi istedigine ve neye ulasmaya çalistigina bagli" dedi kedi "Sey, bilmem ki ??? " dedi Alice "O zaman hangi yoldan gittigin farketmez" dedi kedi. Alice Harikalar Diyarinda -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CACv48hDydQ%3Dk%2B5jqn6oCQMS8g-iGo1LPjLEH1JLM2M_7pDhREA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
