Finally I reach to ship my log file in a json format! (That's one small
step for a mankind, one giant leap for me) :-)
I'm using the imfile protocol (log files are alredy preformatted in
json) and a teamplate (that I shameless found in internet)
$template
jsonLog,"{\"type\":\"%programname%\",\"host\":\"%HOSTNAME%\",\"role\":\"apache
custom log\",\"message\":\"%rawmsg:::json%\"}\n"
And this is what logstash receive
"type" => "apache-access",
"host" => "centos7_test2",
"role" => "apache custom log",
"message" => "{ \"vhost\": \"192.168.122.226\", \"host\":
\"192.168.122.1\", \"sllProtocol\": \"-\", \"chiper\" : \"-\",
\"client\" : \"-\", \"user\": \"-\", \"timestamp\":
\"06-04-201616:31:10CEST\", \"id\": \"-\", \"protocol\": \"HTTP/1.1\",
\"method\": \"GET\", \"alive\": \"6\", \"urlpath\":
\"/noindex/css/fonts/Bold/OpenSans-Bold.ttf\", \"urlquery\": \"\",
\"status\":\" 404\", \"bytes\": \"238\", \"header\":
\"http://192.168.122.226/noindex/css/open-sans.css\";, \"useragent\":
\"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101
Firefox/45.0\", \"duration\": \"330\", \"connection\": \"+\",
\"bReceived\": \"421\", \"bSent\": \"473\" }",
"@version" => "1",
"@timestamp" => "2016-04-06T14:31:13.609Z",
"port" => 34414
So, at the moment i have all the variable and i need to divide them one
by one.
I can do that changing the template?
I konw, maybe for most of you this is a really retarded question but for
me, that have just started to use linux, json and regular syntax it's
really fucking hard!
Thanks again and sorry if I disturb you but I have no idea where else I
can ask for help.
On 06/04/2016 10:28, Giulio Vaccari wrote:
> Thank you for your reply.
>
> By the way, I must use logstash because I must parse log from many OS,
> some of them are quite old so in many case I must grok the log using
> logstash.
>
> So, now I'm trying to ship the file piping the into rsyslog using this
> configuration :
>
> #LogFormat "{ \
> #\"vhost\": \"%V\", \
> #\"host\": \"%h\", \
> #\"protocol\": \"%{SSL_PROTOCOL}x\", \
> #\"chiper\" : \"%{SSL_CIPHER}x\", \
> #\"client\" : \"%{SSL_CLIENT_S_DN_CN}x\", \
> #\"user\": \"%u\", \
> #\"timestamp\": \"%{%d-%m-%Y%H:%M:%S%Z}t\", \
> #\"id\": \"%{JSESSIONID}C\", \
> #\"protocol\": \"%H\", \
> #\"method\": \"%m\", \
> #\"alive\": \"%k\", \
> #\"urlpath\": \"%U\", \
> #\"urlquery\": \"%q\", \
> #\"status\":\" %>s\", \
> #\"bytes\": \"%b\", \
> #\"header\": \"%{Referer}i\", \
> #\"useragent\": \"%{User-agent}i\", \
> #\"duration\": \"%D\", \
> #\"connection\": \"%X\", \
> #\"bReceived\": \"%I\", \
> #\"bSent\": \"%O\" \
> #}" test.log
>
> CustomLog "|/usr/bin/logger -t httpd -p local6.info" combined
>
> As test, when I write them into a file I receive this as output:
>
> { "vhost": "192.168.122.226", "host": "192.168.122.1", "protocol": "-",
> "chiper" : "-", "client" : "-", "user": "-", "timestamp":
> "05-04-201617:04:01CEST", "id": "-", "protocol": "HTTP/1.1", "method":
> "GET", "alive": "43", "urlpath":
> "/noindex/css/fonts/Bold/OpenSans-Bold.ttf", "urlquery": "", "status":"
> 404", "bytes": "238", "header":
> "http://192.168.122.226/noindex/css/open-sans.css";, "useragent":
> "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101
> Firefox/45.0", "duration": "316", "connection": "+", "bReceived": "421",
> "bSent": "473" }
>
> That is a correct json format.
>
> then I add the follow option into rsyslog
>
> local6.*
> @@192.168.122.32:5514
>
> When i refresh my apache test server, it send the log...But are different!
>
> "<182>Apr 6 09:39:57 centos7_test2 httpd: 192.168.122.1 - -
> [06/Apr/2016:09:39:57 +0200] \"GET
> /noindex/css/fonts/Bold/OpenSans-Bold.ttf HTTP/1.1\" 404 238
> \"http://192.168.122.226/noindex/css/open-sans.css\"; \"Mozilla/5.0 (X11;
> Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0\"",
>
> Not only the form but also the content, seems like that I'm piping the
> standard apache log format and not the custom one :\
>
> And again many many thanks... I now that maybe for you those are really
> stupid questions but I'm new of this job and it's really hard for me.
>
> On 06/04/2016 09:12, David Lang wrote:
>> well, I'm now guessing as to what you mean, but if the file is all
>> json, then
>> the $msg variable should be the json content you are interested in. So
>> if you
>> create a template like:
>>
>> $template json,"%msg%\n"
>>
>> and then have a filter that sends the logs from that imfile input to
>> logstash,
>> and configure logstash to parse the input as json, it will probably do
>> something
>> approximatly what you want.
>>
>> There are several dozen ways to send messages between rsyslog and
>> logstash, none
>> of them are "the one right" way to do so.
>>
>> Personally, I would deliver normal syslog messages to logstash and
>> configure it
>> to parse them, or I would eliminate logstash entirely and deliver to
>> whatever
>> destination you have logstash hooked to (especially since in current
>> rsyslog
>> versions there is a module to let you use the inefficient regex Grok
>> filters if
>> you think you need them). But there is no one right way to do things,
>> both
>> programs talk a lot of protocols and have extensive parsing/formatting
>> options.
>>
>> David Lang
>>
>> On Wed, 6 Apr 2016, Giulio Vaccari wrote:
>>
>>> Date: Wed, 6 Apr 2016 09:04:05 +0200
>>> From: Giulio Vaccari <[email protected]>
>>> Reply-To: rsyslog-users <[email protected]>
>>> To: rsyslog-users <[email protected]>
>>> Subject: Re: [rsyslog] json format
>>>
>>> Hi and sorry for the late answer, yesterday I simply give up.
>>>
>>> I don't want waste time of the community, I only really would like to
>>> know how I should proceed with a rsyslog v. 7.4.7
>>> Imfile work for send file alredy formatted in json? Or I must use a
>>> rsyslog pattern? Or I must use some other option that I still don't
>>> know?
>>>
>>> Again, many thanks
>>>
>>> On 05/04/2016 15:30, David Lang wrote:
>>>> On Tue, 5 Apr 2016, Giulio Vaccari wrote:
>>>>
>>>>> First of all, thank you for yuo reply
>>>>>
>>>>> Simply i have no more ideas about how to do this!
>>>>>
>>>>> I'm using rsyslog 7.4.7 (standard centos7)
>>>>> My first attempt was to create a a Json file using Apache and ship it
>>>>> using Rsyslog using "imfile"... Seems that it don't work... (json file
>>>>> is in a valid json format)
>>>>>
>>>>> Then I try to create a json pattern in rsyslog that mutate a log like
>>>>> this one:
>>>>>
>>>>> www.test.example.com 132.168.192.123 TLSv1.2
>>>>> EABDE-IIA-AES128-GMM-SHDF434 - - 31/03/2016 05:39:47 CEST
>>>>> VvycA8ErsYMAAClUQhwAAABH - HTTP/1.1 GET 0 /test.txt "" 404 295 "-"
>>>>> "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
>>>>> 415 +
>>>>> 661 3860
>>>>>
>>>>> Andnothing
>>>> what is the config that you are trying and failing?
>>>>
>>>>> Then I try to ship a json preformatted file using an rsylog pattern!
>>>>> (Yes, I know, this is stupid but I was really in shorts of ideas... In
>>>>> my tired mind this "procedure" should be able load the variable
>>>>> from an
>>>>> apache json log file, load them into the template and ship them to
>>>>> logstash)... You can immaginate by yourself the result...
>>>>>
>>>>> So...What is the correct way to proceed??? I find many sites but
>>>>> most of
>>>>> the are simply not clear or theyr solution don't work for me.
>>>> what does the file you are trying to ship look like?
>>>>
>>>> The first thing to do, anytime you have trouble with your output, is
>>>> to write a
>>>> local log in the format RSYSLOG_DebugFormat because that will let
>>>> you see
>>>> exactly what is in every variable. Once you have the variables defined
>>>> the way
>>>> you think they are, then creating a format to chip them to logstash is
>>>> pretty
>>>> easy.
>>>>
>>>> David Lang
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>>>> if you DON'T LIKE THAT.
>>>>
>>>> AVVISO DI RISERVATEZZA Informazioni riservate possono essere contenute
>>>> nel messaggio o nei suoi allegati. Se non siete i destinatari indicati
>>>> nel messaggio, o responsabili per la sua consegna alla persona, o se
>>>> avete ricevuto il messaggio per errore, siete pregati di non
>>>> trascriverlo, copiarlo o inviarlo ad alcuno. In tal caso vi invitiamo
>>>> a cancellare il messaggio ed i suoi allegati. Grazie.
>>>> CONFIDENTIALITY NOTICE Confidential information may be contained in
>>>> this message or in its attachments. If you are not the addressee
>>>> indicated in this message, or responsible for message delivering to
>>>> that person, or if you have received this message in error, you may
>>>> not transcribe, copy or deliver this message to anyone. In that case,
>>>> you should delete this message and its attachments. Thank you.
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>> POST if you DON'T LIKE THAT.
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>> if you DON'T LIKE THAT.
>>
>> AVVISO DI RISERVATEZZA Informazioni riservate possono essere contenute
>> nel messaggio o nei suoi allegati. Se non siete i destinatari indicati
>> nel messaggio, o responsabili per la sua consegna alla persona, o se
>> avete ricevuto il messaggio per errore, siete pregati di non
>> trascriverlo, copiarlo o inviarlo ad alcuno. In tal caso vi invitiamo
>> a cancellare il messaggio ed i suoi allegati. Grazie.
>> CONFIDENTIALITY NOTICE Confidential information may be contained in
>> this message or in its attachments. If you are not the addressee
>> indicated in this message, or responsible for message delivering to
>> that person, or if you have received this message in error, you may
>> not transcribe, copy or deliver this message to anyone. In that case,
>> you should delete this message and its attachments. Thank you.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
>
> AVVISO DI RISERVATEZZA Informazioni riservate possono essere contenute nel
> messaggio o nei suoi allegati. Se non siete i destinatari indicati nel
> messaggio, o responsabili per la sua consegna alla persona, o se avete
> ricevuto il messaggio per errore, siete pregati di non trascriverlo, copiarlo
> o inviarlo ad alcuno. In tal caso vi invitiamo a cancellare il messaggio ed i
> suoi allegati. Grazie.
> CONFIDENTIALITY NOTICE Confidential information may be contained in this
> message or in its attachments. If you are not the addressee indicated in this
> message, or responsible for message delivering to that person, or if you have
> received this message in error, you may not transcribe, copy or deliver this
> message to anyone. In that case, you should delete this message and its
> attachments. Thank you.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
