I finally end up writing my own error log fluentd custom parser in ruby. It's working now.
Thanks for help anyways, much appreciated On 11 January 2018 at 14:26, Aziz Rozyev <[email protected]> wrote: > Hi, > > seems, that fluentd has an nginx_parser plugin already, another solution > that probably should work is to use the grep filters, > something as follows: > > <fitler foo.bar> > @type grep > <regexp> > key client > patter ^client.*\ $ > </regexp> > <regexp> > key server > pattern ^server.*\ $ > </regexp> > <regexp> > key host > pattern ^host.*$ > </regexp> > <regexp> > key zone > pattern ^zone.*\ $ > </regexp> > ….. > </filter> > > > then use record_trasformer type, to make further modifications. But, I > didn’t tried above, > probably it’s something that better to be asked from fluentd community.. > > > br, > Aziz. > > > > > > > On 10 Jan 2018, at 15:23, mohit Agrawal <[email protected]> wrote: > > > > Thanks Aziz for this, I get your point, but can we do awking in fluentd > cons file ? Basically we are looking for realtime awking a nginx error log > file, how heavy this would be according to you. > > > > On 10 January 2018 at 17:44, Aziz Rozyev <[email protected]> wrote: > > If you need parse exactly the same format, as you’ve shown in you > question, it’s fairly easy to create something e.g. perl/awk/sed script. > > > > for instance: > > > > ################# tst.awk ################# > > BEGIN {FS = "," } > > { > > split($1, m, "\ ") > > printf "%s", "{ " > > printf "%s",$2 > > printf "%s",$3 > > printf "%s",$5 > > printf "%s",$4 > > printf "reason: %s %s %s %s \"%s\"\n", m[6], m[7], m[8], m[9], m[10] > > print " }” > > > > } > > ############################################# > > > > > > result: > > > > echo 2018/01/10 06:26:31 [error] 13485#13485: *64285471 limiting > connections by zone "rl_conn", client: xx.xx.xx.xx, server: www.xyz.com, > request: "GET /api/xyz HTTP/1.1", host: "www.xyz.com" | awk -f > /tmp/test.awk > > { client: xx.xx.xx.xx server: www.xyz.com host: www.xyz.com request: > GET /api/xyz HTTP/1.1reason: limiting connections by zone "rl_conn" > > } > > > > > > br, > > Aziz. > > > > > > > > > > > > > On 10 Jan 2018, at 14:45, mohit Agrawal <[email protected]> > wrote: > > > > > > Yeah I have tried grok / regex pattern as well. But not extensive > success that I achieved. grok didn't work for me, I tried regex then it was > able to segregate time , pid, tid, log_level and message. I also need > message break up for above pattern > > > > > > On 10 January 2018 at 17:12, Aziz Rozyev <[email protected]> wrote: > > > Hi Mohit, > > > > > > check the second reply. I’m not sure that there is a conventional > pretty printing > > > tools for nginx error log. > > > > > > > > > br, > > > Aziz. > > > > > > > > > > > > > > > > > > > On 10 Jan 2018, at 14:37, mohit Agrawal <[email protected]> > wrote: > > > > > > > > Hi Aziz, > > > > > > > > log_format directive only provides formatting for access log, I am > looking to format error.log which doesn't take log_format directive. > > > > Above example that I gave is just for nginx error logs. > > > > > > > > Thanks > > > > > > > > On 10 January 2018 at 15:26, Aziz Rozyev <[email protected]> wrote: > > > > btw, after re-reading the your questing, it looks like you need > something like logstash grok filter. > > > > > > > > br, > > > > Aziz. > > > > > > > > > > > > > > > > > > > > > > > > > On 10 Jan 2018, at 11:45, mohit Agrawal <[email protected]> > wrote: > > > > > > > > > > Hi , > > > > > > > > > > I am looking to parse nginx error log so as to find out which > particular IP is throttled during specific amount of time on connection > throttling / request throttling. The format looks like : > > > > > > > > > > 2018/01/10 06:26:31 [error] 13485#13485: *64285471 limiting > connections by zone "rl_conn", client: xx.xx.xx.xx, server: www.xyz.com, > request: "GET /api/xyz HTTP/1.1", host: "www.xyz.com" > > > > > And the sample that I am looking for is : > > > > > > > > > > {client: "xx.xx.xx.xx", server: "www.xyz.com", host: "www.xyz.com", > "request": "GET /api/xyz HTTP/1.1", reason: "limiting connections by zone > "rl_conn""} > > > > > so that I can pass it through ELK stack and find out the root ip > which is causing issue. > > > > > > > > > > > > > > > -- > > > > > Mohit Agrawal > > > > > _______________________________________________ > > > > > nginx mailing list > > > > > [email protected] > > > > > http://mailman.nginx.org/mailman/listinfo/nginx > > > > > > > > _______________________________________________ > > > > nginx mailing list > > > > [email protected] > > > > http://mailman.nginx.org/mailman/listinfo/nginx > > > > > > > > > > > > > > > > -- > > > > Mohit Agrawal > > > > > > > > > > > > > > > -- > > > Mohit Agrawal > > > > > > > > > > -- > > Mohit Agrawal > > -- Mohit Agrawal
_______________________________________________ nginx mailing list [email protected] http://mailman.nginx.org/mailman/listinfo/nginx
