Hey, Sorry for the late reply but :
I created a small utility to forward JSON objects (from stdin or a file) to a GELF UDP server.. It's written in C and is fast enough to live as an agent next on your varnish server without adding too much load and without the need of a full scripting environment (python/ruby/js). https://github.com/gboddin/json2gelf There's an exemple on how to create a JSON packet from varnishncsa here : https://github.com/gboddin/json2gelf/blob/master/scripts/varnish2json Just pipe varnish2json in json2gelf to start sending gelf packets to your graylog/logstash/<your gelf product here>. We're using it in production, feel free to use it for any application able to output json logging. It avoids UDP fragmentation by using the chunk specification of the GELF protocol. Shall your MTU be exotic, you can always change the #define in json2gelf.c Hope it helps ! On Sunday, November 1, 2015 at 3:11:37 AM UTC+1, Dhyaneshwaran Muralidharan wrote: > > Hi Razvan, > > Can you please explain how exactly you converted the varnish logs to Json > ? I was trying to do the same but was worried if the log itself could have > strings which could make the Json invalid . > > Currently we have the format like below : > -F %h %u %t "%r" %s %B "%{Referer}i" "%{User-agent}i" > > --Dhyan > > On Wednesday, December 3, 2014 at 6:23:57 PM UTC-5, Razvan Grigore wrote: >> >> Hello and thank you for this great piece of software! >> >> I joined one of the author's live presentations in Hamburg, and since >> then I am using graylog2 as logging solution for the company I work for. >> >> Most of the basic stuff works great, it's just the varnish logging I >> can't seem to make it work. I found very little about this couple online - >> very strange, no extractors, no group topic, nothing. Varnish is great! >> >> My idea was to pipe the logs via simple UDP from varnishncsa (log daemon) >> instead of storing them in a file and have rsyslog parse them and send the >> to graylog - avoid unnecessary IO. >> >> Since varnishncsa has the -F option, I specified a JSON format according >> to GELF specification, and this works great, I can see all entries in stdin >> like a tail -f of the log file, all valid JSON. >> >> Now the part that is not working, is that I try to pipe this output to nc >> (netcat) via UDP. The strange this is, SOME logs appear, after long time, >> but most of them are lost. >> >> I tried the nc command with both -w 1 and without, seems the same, my >> guess is that netcat keeps the connection open and sends multiple messages >> until the max packet size is reached, making graylog receive invalid >> formatted JSON. >> >> I should mention that the input is working, I can receive messages in >> GELF format from PHP via UDP on that port without any problem. If needed, I >> can provide the exact command I am running. >> >> Thank you for your help! >> Razvan, HH >> > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/1a6175b6-cd81-4832-9460-6b7b07dd7d51%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
