Thank you for your reply.
By the way, I must use logstash because I must parse log from many OS,
some of them are quite old so in many case I must grok the log using
logstash.
So, now I'm trying to ship the file piping the into rsyslog using this
configuration :
#LogFormat "{ \
#\"vhost\": \"%V\", \
#\"host\": \"%h\", \
#\"protocol\": \"%{SSL_PROTOCOL}x\", \
#\"chiper\" : \"%{SSL_CIPHER}x\", \
#\"client\" : \"%{SSL_CLIENT_S_DN_CN}x\", \
#\"user\": \"%u\", \
#\"timestamp\": \"%{%d-%m-%Y%H:%M:%S%Z}t\", \
#\"id\": \"%{JSESSIONID}C\", \
#\"protocol\": \"%H\", \
#\"method\": \"%m\", \
#\"alive\": \"%k\", \
#\"urlpath\": \"%U\", \
#\"urlquery\": \"%q\", \
#\"status\":\" %>s\", \
#\"bytes\": \"%b\", \
#\"header\": \"%{Referer}i\", \
#\"useragent\": \"%{User-agent}i\", \
#\"duration\": \"%D\", \
#\"connection\": \"%X\", \
#\"bReceived\": \"%I\", \
#\"bSent\": \"%O\" \
#}" test.log
CustomLog "|/usr/bin/logger -t httpd -p local6.info" combined
As test, when I write them into a file I receive this as output:
{ "vhost": "192.168.122.226", "host": "192.168.122.1", "protocol": "-",
"chiper" : "-", "client" : "-", "user": "-", "timestamp":
"05-04-201617:04:01CEST", "id": "-", "protocol": "HTTP/1.1", "method":
"GET", "alive": "43", "urlpath":
"/noindex/css/fonts/Bold/OpenSans-Bold.ttf", "urlquery": "", "status":"
404", "bytes": "238", "header":
"http://192.168.122.226/noindex/css/open-sans.css";, "useragent":
"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101
Firefox/45.0", "duration": "316", "connection": "+", "bReceived": "421",
"bSent": "473" }
That is a correct json format.
then I add the follow option into rsyslog
local6.*
@@192.168.122.32:5514
When i refresh my apache test server, it send the log...But are different!
"<182>Apr 6 09:39:57 centos7_test2 httpd: 192.168.122.1 - -
[06/Apr/2016:09:39:57 +0200] \"GET
/noindex/css/fonts/Bold/OpenSans-Bold.ttf HTTP/1.1\" 404 238
\"http://192.168.122.226/noindex/css/open-sans.css\"; \"Mozilla/5.0 (X11;
Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0\"",
Not only the form but also the content, seems like that I'm piping the
standard apache log format and not the custom one :\
And again many many thanks... I now that maybe for you those are really
stupid questions but I'm new of this job and it's really hard for me.
On 06/04/2016 09:12, David Lang wrote:
> well, I'm now guessing as to what you mean, but if the file is all
> json, then
> the $msg variable should be the json content you are interested in. So
> if you
> create a template like:
>
> $template json,"%msg%\n"
>
> and then have a filter that sends the logs from that imfile input to
> logstash,
> and configure logstash to parse the input as json, it will probably do
> something
> approximatly what you want.
>
> There are several dozen ways to send messages between rsyslog and
> logstash, none
> of them are "the one right" way to do so.
>
> Personally, I would deliver normal syslog messages to logstash and
> configure it
> to parse them, or I would eliminate logstash entirely and deliver to
> whatever
> destination you have logstash hooked to (especially since in current
> rsyslog
> versions there is a module to let you use the inefficient regex Grok
> filters if
> you think you need them). But there is no one right way to do things,
> both
> programs talk a lot of protocols and have extensive parsing/formatting
> options.
>
> David Lang
>
> On Wed, 6 Apr 2016, Giulio Vaccari wrote:
>
>> Date: Wed, 6 Apr 2016 09:04:05 +0200
>> From: Giulio Vaccari <[email protected]>
>> Reply-To: rsyslog-users <[email protected]>
>> To: rsyslog-users <[email protected]>
>> Subject: Re: [rsyslog] json format
>>
>> Hi and sorry for the late answer, yesterday I simply give up.
>>
>> I don't want waste time of the community, I only really would like to
>> know how I should proceed with a rsyslog v. 7.4.7
>> Imfile work for send file alredy formatted in json? Or I must use a
>> rsyslog pattern? Or I must use some other option that I still don't
>> know?
>>
>> Again, many thanks
>>
>> On 05/04/2016 15:30, David Lang wrote:
>>> On Tue, 5 Apr 2016, Giulio Vaccari wrote:
>>>
>>>> First of all, thank you for yuo reply
>>>>
>>>> Simply i have no more ideas about how to do this!
>>>>
>>>> I'm using rsyslog 7.4.7 (standard centos7)
>>>> My first attempt was to create a a Json file using Apache and ship it
>>>> using Rsyslog using "imfile"... Seems that it don't work... (json file
>>>> is in a valid json format)
>>>>
>>>> Then I try to create a json pattern in rsyslog that mutate a log like
>>>> this one:
>>>>
>>>> www.test.example.com 132.168.192.123 TLSv1.2
>>>> EABDE-IIA-AES128-GMM-SHDF434 - - 31/03/2016 05:39:47 CEST
>>>> VvycA8ErsYMAAClUQhwAAABH - HTTP/1.1 GET 0 /test.txt "" 404 295 "-"
>>>> "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
>>>> 415 +
>>>> 661 3860
>>>>
>>>> Andnothing
>>>
>>> what is the config that you are trying and failing?
>>>
>>>> Then I try to ship a json preformatted file using an rsylog pattern!
>>>> (Yes, I know, this is stupid but I was really in shorts of ideas... In
>>>> my tired mind this "procedure" should be able load the variable
>>>> from an
>>>> apache json log file, load them into the template and ship them to
>>>> logstash)... You can immaginate by yourself the result...
>>>>
>>>> So...What is the correct way to proceed??? I find many sites but
>>>> most of
>>>> the are simply not clear or theyr solution don't work for me.
>>>
>>> what does the file you are trying to ship look like?
>>>
>>> The first thing to do, anytime you have trouble with your output, is
>>> to write a
>>> local log in the format RSYSLOG_DebugFormat because that will let
>>> you see
>>> exactly what is in every variable. Once you have the variables defined
>>> the way
>>> you think they are, then creating a format to chip them to logstash is
>>> pretty
>>> easy.
>>>
>>> David Lang
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>>> if you DON'T LIKE THAT.
>>>
>>> AVVISO DI RISERVATEZZA Informazioni riservate possono essere contenute
>>> nel messaggio o nei suoi allegati. Se non siete i destinatari indicati
>>> nel messaggio, o responsabili per la sua consegna alla persona, o se
>>> avete ricevuto il messaggio per errore, siete pregati di non
>>> trascriverlo, copiarlo o inviarlo ad alcuno. In tal caso vi invitiamo
>>> a cancellare il messaggio ed i suoi allegati. Grazie.
>>> CONFIDENTIALITY NOTICE Confidential information may be contained in
>>> this message or in its attachments. If you are not the addressee
>>> indicated in this message, or responsible for message delivering to
>>> that person, or if you have received this message in error, you may
>>> not transcribe, copy or deliver this message to anyone. In that case,
>>> you should delete this message and its attachments. Thank you.
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> POST if you DON'T LIKE THAT.
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if you DON'T LIKE THAT.
>
> AVVISO DI RISERVATEZZA Informazioni riservate possono essere contenute
> nel messaggio o nei suoi allegati. Se non siete i destinatari indicati
> nel messaggio, o responsabili per la sua consegna alla persona, o se
> avete ricevuto il messaggio per errore, siete pregati di non
> trascriverlo, copiarlo o inviarlo ad alcuno. In tal caso vi invitiamo
> a cancellare il messaggio ed i suoi allegati. Grazie.
> CONFIDENTIALITY NOTICE Confidential information may be contained in
> this message or in its attachments. If you are not the addressee
> indicated in this message, or responsible for message delivering to
> that person, or if you have received this message in error, you may
> not transcribe, copy or deliver this message to anyone. In that case,
> you should delete this message and its attachments. Thank you.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
