And I'm here again with another (probably really stupid for you) question.
As I wrote in the previous mail I reach to send the entire message as
json and now i need to divide it in many fields.
I have create a new custom log that give me only one variable (the host)
and I'm trying to send it to logstash alredy formatted.
I'have try many template (this is one of them)
$template jsonLog,"{%msg:::json:message%{%host-text:::jsonf:host-text%}}"
But what I receive back is more or less always the same
"message" => "{{ \\\"host\\\": \\\"192.168.122.1\\\", }{**INVALID
PROPERTY NAME**",
"tags" => [
[0] "_jsonparsefailure"
],
"@version" => "1",
"@timestamp" => "2016-04-08T10:26:31.288Z",
"host" => "192.168.122.226",
"port" => 38649,
"type" => "apache"
Logstash can read the correct variable (host nuber is correct) but I
receive always the same error :/
Can someone please tell me where I'm wrong?
On 06/04/2016 17:30, David Lang wrote:
> It's actually very easy in rsyslog, if you are using the right tools on a
> current version. I'll post later with info.
>
> In any language, processing JSON as if it was just string data is hard.
>
> David Lang
>
> On Wed, 6 Apr 2016, Giulio Vaccari wrote:
>
>> Date: Wed, 6 Apr 2016 16:55:07 +0200
>> From: Giulio Vaccari <[email protected]>
>> Reply-To: rsyslog-users <[email protected]>
>> To: rsyslog-users <[email protected]>
>> Subject: Re: [rsyslog] json format
>>
>> Finally I reach to ship my log file in a json format! (That's one small
>> step for a mankind, one giant leap for me) :-)
>> I'm using the imfile protocol (log files are alredy preformatted in
>> json) and a teamplate (that I shameless found in internet)
>>
>> $template
>> jsonLog,"{\"type\":\"%programname%\",\"host\":\"%HOSTNAME%\",\"role\":\"apache
>>
>> custom log\",\"message\":\"%rawmsg:::json%\"}\n"
>>
>> And this is what logstash receive
>>
>> "type" => "apache-access",
>> "host" => "centos7_test2",
>> "role" => "apache custom log",
>> "message" => "{ \"vhost\": \"192.168.122.226\", \"host\":
>> \"192.168.122.1\", \"sllProtocol\": \"-\", \"chiper\" : \"-\",
>> \"client\" : \"-\", \"user\": \"-\", \"timestamp\":
>> \"06-04-201616:31:10CEST\", \"id\": \"-\", \"protocol\": \"HTTP/1.1\",
>> \"method\": \"GET\", \"alive\": \"6\", \"urlpath\":
>> \"/noindex/css/fonts/Bold/OpenSans-Bold.ttf\", \"urlquery\": \"\",
>> \"status\":\" 404\", \"bytes\": \"238\", \"header\":
>> \"http://192.168.122.226/noindex/css/open-sans.css\";, \"useragent\":
>> \"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101
>> Firefox/45.0\", \"duration\": \"330\", \"connection\": \"+\",
>> \"bReceived\": \"421\", \"bSent\": \"473\" }",
>> "@version" => "1",
>> "@timestamp" => "2016-04-06T14:31:13.609Z",
>> "port" => 34414
>>
>> So, at the moment i have all the variable and i need to divide them one
>> by one.
>> I can do that changing the template?
>>
>> I konw, maybe for most of you this is a really retarded question but for
>> me, that have just started to use linux, json and regular syntax it's
>> really fucking hard!
>>
>> Thanks again and sorry if I disturb you but I have no idea where else I
>> can ask for help.
>>
>>
>>
>> On 06/04/2016 10:28, Giulio Vaccari wrote:
>>> Thank you for your reply.
>>>
>>> By the way, I must use logstash because I must parse log from many OS,
>>> some of them are quite old so in many case I must grok the log using
>>> logstash.
>>>
>>> So, now I'm trying to ship the file piping the into rsyslog using this
>>> configuration :
>>>
>>> #LogFormat "{ \
>>> #\"vhost\": \"%V\", \
>>> #\"host\": \"%h\", \
>>> #\"protocol\": \"%{SSL_PROTOCOL}x\", \
>>> #\"chiper\" : \"%{SSL_CIPHER}x\", \
>>> #\"client\" : \"%{SSL_CLIENT_S_DN_CN}x\", \
>>> #\"user\": \"%u\", \
>>> #\"timestamp\": \"%{%d-%m-%Y%H:%M:%S%Z}t\", \
>>> #\"id\": \"%{JSESSIONID}C\", \
>>> #\"protocol\": \"%H\", \
>>> #\"method\": \"%m\", \
>>> #\"alive\": \"%k\", \
>>> #\"urlpath\": \"%U\", \
>>> #\"urlquery\": \"%q\", \
>>> #\"status\":\" %>s\", \
>>> #\"bytes\": \"%b\", \
>>> #\"header\": \"%{Referer}i\", \
>>> #\"useragent\": \"%{User-agent}i\", \
>>> #\"duration\": \"%D\", \
>>> #\"connection\": \"%X\", \
>>> #\"bReceived\": \"%I\", \
>>> #\"bSent\": \"%O\" \
>>> #}" test.log
>>>
>>> CustomLog "|/usr/bin/logger -t httpd -p local6.info" combined
>>>
>>> As test, when I write them into a file I receive this as output:
>>>
>>> { "vhost": "192.168.122.226", "host": "192.168.122.1", "protocol": "-",
>>> "chiper" : "-", "client" : "-", "user": "-", "timestamp":
>>> "05-04-201617:04:01CEST", "id": "-", "protocol": "HTTP/1.1", "method":
>>> "GET", "alive": "43", "urlpath":
>>> "/noindex/css/fonts/Bold/OpenSans-Bold.ttf", "urlquery": "", "status":"
>>> 404", "bytes": "238", "header":
>>> "http://192.168.122.226/noindex/css/open-sans.css";, "useragent":
>>> "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101
>>> Firefox/45.0", "duration": "316", "connection": "+", "bReceived":
>>> "421",
>>> "bSent": "473" }
>>>
>>> That is a correct json format.
>>>
>>> then I add the follow option into rsyslog
>>>
>>> local6.*
>>> @@192.168.122.32:5514
>>>
>>> When i refresh my apache test server, it send the log...But are
>>> different!
>>>
>>> "<182>Apr 6 09:39:57 centos7_test2 httpd: 192.168.122.1 - -
>>> [06/Apr/2016:09:39:57 +0200] \"GET
>>> /noindex/css/fonts/Bold/OpenSans-Bold.ttf HTTP/1.1\" 404 238
>>> \"http://192.168.122.226/noindex/css/open-sans.css\"; \"Mozilla/5.0
>>> (X11;
>>> Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0\"",
>>>
>>> Not only the form but also the content, seems like that I'm piping the
>>> standard apache log format and not the custom one :\
>>>
>>> And again many many thanks... I now that maybe for you those are really
>>> stupid questions but I'm new of this job and it's really hard for me.
>>>
>>> On 06/04/2016 09:12, David Lang wrote:
>>>> well, I'm now guessing as to what you mean, but if the file is all
>>>> json, then
>>>> the $msg variable should be the json content you are interested in. So
>>>> if you
>>>> create a template like:
>>>>
>>>> $template json,"%msg%\n"
>>>>
>>>> and then have a filter that sends the logs from that imfile input to
>>>> logstash,
>>>> and configure logstash to parse the input as json, it will probably do
>>>> something
>>>> approximatly what you want.
>>>>
>>>> There are several dozen ways to send messages between rsyslog and
>>>> logstash, none
>>>> of them are "the one right" way to do so.
>>>>
>>>> Personally, I would deliver normal syslog messages to logstash and
>>>> configure it
>>>> to parse them, or I would eliminate logstash entirely and deliver to
>>>> whatever
>>>> destination you have logstash hooked to (especially since in current
>>>> rsyslog
>>>> versions there is a module to let you use the inefficient regex Grok
>>>> filters if
>>>> you think you need them). But there is no one right way to do things,
>>>> both
>>>> programs talk a lot of protocols and have extensive parsing/formatting
>>>> options.
>>>>
>>>> David Lang
>>>>
>>>> On Wed, 6 Apr 2016, Giulio Vaccari wrote:
>>>>
>>>>> Date: Wed, 6 Apr 2016 09:04:05 +0200
>>>>> From: Giulio Vaccari <[email protected]>
>>>>> Reply-To: rsyslog-users <[email protected]>
>>>>> To: rsyslog-users <[email protected]>
>>>>> Subject: Re: [rsyslog] json format
>>>>>
>>>>> Hi and sorry for the late answer, yesterday I simply give up.
>>>>>
>>>>> I don't want waste time of the community, I only really would like to
>>>>> know how I should proceed with a rsyslog v. 7.4.7
>>>>> Imfile work for send file alredy formatted in json? Or I must use a
>>>>> rsyslog pattern? Or I must use some other option that I still don't
>>>>> know?
>>>>>
>>>>> Again, many thanks
>>>>>
>>>>> On 05/04/2016 15:30, David Lang wrote:
>>>>>> On Tue, 5 Apr 2016, Giulio Vaccari wrote:
>>>>>>
>>>>>>> First of all, thank you for yuo reply
>>>>>>>
>>>>>>> Simply i have no more ideas about how to do this!
>>>>>>>
>>>>>>> I'm using rsyslog 7.4.7 (standard centos7)
>>>>>>> My first attempt was to create a a Json file using Apache and
>>>>>>> ship it
>>>>>>> using Rsyslog using "imfile"... Seems that it don't work...
>>>>>>> (json file
>>>>>>> is in a valid json format)
>>>>>>>
>>>>>>> Then I try to create a json pattern in rsyslog that mutate a log
>>>>>>> like
>>>>>>> this one:
>>>>>>>
>>>>>>> www.test.example.com 132.168.192.123 TLSv1.2
>>>>>>> EABDE-IIA-AES128-GMM-SHDF434 - - 31/03/2016 05:39:47 CEST
>>>>>>> VvycA8ErsYMAAClUQhwAAABH - HTTP/1.1 GET 0 /test.txt "" 404 295 "-"
>>>>>>> "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
>>>>>>> 415 +
>>>>>>> 661 3860
>>>>>>>
>>>>>>> Andnothing
>>>>>> what is the config that you are trying and failing?
>>>>>>
>>>>>>> Then I try to ship a json preformatted file using an rsylog
>>>>>>> pattern!
>>>>>>> (Yes, I know, this is stupid but I was really in shorts of
>>>>>>> ideas... In
>>>>>>> my tired mind this "procedure" should be able load the variable
>>>>>>> from an
>>>>>>> apache json log file, load them into the template and ship them to
>>>>>>> logstash)... You can immaginate by yourself the result...
>>>>>>>
>>>>>>> So...What is the correct way to proceed??? I find many sites but
>>>>>>> most of
>>>>>>> the are simply not clear or theyr solution don't work for me.
>>>>>> what does the file you are trying to ship look like?
>>>>>>
>>>>>> The first thing to do, anytime you have trouble with your output, is
>>>>>> to write a
>>>>>> local log in the format RSYSLOG_DebugFormat because that will let
>>>>>> you see
>>>>>> exactly what is in every variable. Once you have the variables
>>>>>> defined
>>>>>> the way
>>>>>> you think they are, then creating a format to chip them to
>>>>>> logstash is
>>>>>> pretty
>>>>>> easy.
>>>>>>
>>>>>> David Lang
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> http://www.rsyslog.com/professional-services/
>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>>>>> POST
>>>>>> if you DON'T LIKE THAT.
>>>>>>
>>>>>> AVVISO DI RISERVATEZZA Informazioni riservate possono essere
>>>>>> contenute
>>>>>> nel messaggio o nei suoi allegati. Se non siete i destinatari
>>>>>> indicati
>>>>>> nel messaggio, o responsabili per la sua consegna alla persona, o se
>>>>>> avete ricevuto il messaggio per errore, siete pregati di non
>>>>>> trascriverlo, copiarlo o inviarlo ad alcuno. In tal caso vi
>>>>>> invitiamo
>>>>>> a cancellare il messaggio ed i suoi allegati. Grazie.
>>>>>> CONFIDENTIALITY NOTICE Confidential information may be contained in
>>>>>> this message or in its attachments. If you are not the addressee
>>>>>> indicated in this message, or responsible for message delivering to
>>>>>> that person, or if you have received this message in error, you may
>>>>>> not transcribe, copy or deliver this message to anyone. In that
>>>>>> case,
>>>>>> you should delete this message and its attachments. Thank you.
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>>>> POST if you DON'T LIKE THAT.
>>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>>>> if you DON'T LIKE THAT.
>>>>
>>>> AVVISO DI RISERVATEZZA Informazioni riservate possono essere contenute
>>>> nel messaggio o nei suoi allegati. Se non siete i destinatari indicati
>>>> nel messaggio, o responsabili per la sua consegna alla persona, o se
>>>> avete ricevuto il messaggio per errore, siete pregati di non
>>>> trascriverlo, copiarlo o inviarlo ad alcuno. In tal caso vi invitiamo
>>>> a cancellare il messaggio ed i suoi allegati. Grazie.
>>>> CONFIDENTIALITY NOTICE Confidential information may be contained in
>>>> this message or in its attachments. If you are not the addressee
>>>> indicated in this message, or responsible for message delivering to
>>>> that person, or if you have received this message in error, you may
>>>> not transcribe, copy or deliver this message to anyone. In that case,
>>>> you should delete this message and its attachments. Thank you.
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>> POST if you DON'T LIKE THAT.
>>>
>>> AVVISO DI RISERVATEZZA Informazioni riservate possono essere
>>> contenute nel messaggio o nei suoi allegati. Se non siete i
>>> destinatari indicati nel messaggio, o responsabili per la sua
>>> consegna alla persona, o se avete ricevuto il messaggio per errore,
>>> siete pregati di non trascriverlo, copiarlo o inviarlo ad alcuno. In
>>> tal caso vi invitiamo a cancellare il messaggio ed i suoi allegati.
>>> Grazie.
>>> CONFIDENTIALITY NOTICE Confidential information may be contained in
>>> this message or in its attachments. If you are not the addressee
>>> indicated in this message, or responsible for message delivering to
>>> that person, or if you have received this message in error, you may
>>> not transcribe, copy or deliver this message to anyone. In that
>>> case, you should delete this message and its attachments. Thank you.
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> POST if you DON'T LIKE THAT.
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if you DON'T LIKE THAT.
>
> AVVISO DI RISERVATEZZA Informazioni riservate possono essere contenute
> nel messaggio o nei suoi allegati. Se non siete i destinatari indicati
> nel messaggio, o responsabili per la sua consegna alla persona, o se
> avete ricevuto il messaggio per errore, siete pregati di non
> trascriverlo, copiarlo o inviarlo ad alcuno. In tal caso vi invitiamo
> a cancellare il messaggio ed i suoi allegati. Grazie.
> CONFIDENTIALITY NOTICE Confidential information may be contained in
> this message or in its attachments. If you are not the addressee
> indicated in this message, or responsible for message delivering to
> that person, or if you have received this message in error, you may
> not transcribe, copy or deliver this message to anyone. In that case,
> you should delete this message and its attachments. Thank you.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
