Thank you for your reply.
By the way, I must use logstash because I must parse log from many OS,
some of them are quite old so in many case I must grok the log using
logstash.
So, now I'm trying to ship the file piping the into rsyslog using this
configuration :
#LogFormat "{ \
#\"vhost\": \"%V\", \
#\"host\": \"%h\", \
#\"protocol\": \"%{SSL_PROTOCOL}x\", \
#\"chiper\" : \"%{SSL_CIPHER}x\", \
#\"client\" : \"%{SSL_CLIENT_S_DN_CN}x\", \
#\"user\": \"%u\", \
#\"timestamp\": \"%{%d-%m-%Y%H:%M:%S%Z}t\", \
#\"id\": \"%{JSESSIONID}C\", \
#\"protocol\": \"%H\", \
#\"method\": \"%m\", \
#\"alive\": \"%k\", \
#\"urlpath\": \"%U\", \
#\"urlquery\": \"%q\", \
#\"status\":\" %>s\", \
#\"bytes\": \"%b\", \
#\"header\": \"%{Referer}i\", \
#\"useragent\": \"%{User-agent}i\", \
#\"duration\": \"%D\", \
#\"connection\": \"%X\", \
#\"bReceived\": \"%I\", \
#\"bSent\": \"%O\" \
#}" test.log
CustomLog "|/usr/bin/logger -t httpd -p local6.info" combined
As test, when I write them into a file I receive this as output:
{ "vhost": "192.168.122.226", "host": "192.168.122.1", "protocol": "-",
"chiper" : "-", "client" : "-", "user": "-", "timestamp":
"05-04-201617:04:01CEST", "id": "-", "protocol": "HTTP/1.1", "method":
"GET", "alive": "43", "urlpath":
"/noindex/css/fonts/Bold/OpenSans-Bold.ttf", "urlquery": "", "status":"
404", "bytes": "238", "header":
"http://192.168.122.226/noindex/css/open-sans.css";, "useragent":
"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101
Firefox/45.0", "duration": "316", "connection": "+", "bReceived": "421",
"bSent": "473" }
That is a correct json format.
then I add the follow option into rsyslog
local6.*
@@192.168.122.32:5514
When i refresh my apache test server, it send the log...But are different!
"<182>Apr 6 09:39:57 centos7_test2 httpd: 192.168.122.1 - -
[06/Apr/2016:09:39:57 +0200] \"GET
/noindex/css/fonts/Bold/OpenSans-Bold.ttf HTTP/1.1\" 404 238
\"http://192.168.122.226/noindex/css/open-sans.css\"; \"Mozilla/5.0 (X11;
Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0\"",
Not only the form but also the content, seems like that I'm piping the
standard apache log format and not the custom one :\
And again many many thanks... I now that maybe for you those are really
stupid questions but I'm new of this job and it's really hard for me.
On 06/04/2016 09:12, David Lang wrote:
well, I'm now guessing as to what you mean, but if the file is all
json, then
the $msg variable should be the json content you are interested in. So
if you
create a template like:
$template json,"%msg%\n"
and then have a filter that sends the logs from that imfile input to
logstash,
and configure logstash to parse the input as json, it will probably do
something
approximatly what you want.
There are several dozen ways to send messages between rsyslog and
logstash, none
of them are "the one right" way to do so.
Personally, I would deliver normal syslog messages to logstash and
configure it
to parse them, or I would eliminate logstash entirely and deliver to
whatever
destination you have logstash hooked to (especially since in current
rsyslog
versions there is a module to let you use the inefficient regex Grok
filters if
you think you need them). But there is no one right way to do things,
both
programs talk a lot of protocols and have extensive parsing/formatting
options.
David Lang
On Wed, 6 Apr 2016, Giulio Vaccari wrote:
Date: Wed, 6 Apr 2016 09:04:05 +0200
From: Giulio Vaccari <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] json format
Hi and sorry for the late answer, yesterday I simply give up.
I don't want waste time of the community, I only really would like to
know how I should proceed with a rsyslog v. 7.4.7
Imfile work for send file alredy formatted in json? Or I must use a
rsyslog pattern? Or I must use some other option that I still don't
know?
Again, many thanks
On 05/04/2016 15:30, David Lang wrote:
On Tue, 5 Apr 2016, Giulio Vaccari wrote:
First of all, thank you for yuo reply
Simply i have no more ideas about how to do this!
I'm using rsyslog 7.4.7 (standard centos7)
My first attempt was to create a a Json file using Apache and ship it
using Rsyslog using "imfile"... Seems that it don't work... (json file
is in a valid json format)
Then I try to create a json pattern in rsyslog that mutate a log like
this one:
www.test.example.com 132.168.192.123 TLSv1.2
EABDE-IIA-AES128-GMM-SHDF434 - - 31/03/2016 05:39:47 CEST
VvycA8ErsYMAAClUQhwAAABH - HTTP/1.1 GET 0 /test.txt "" 404 295 "-"
"Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
415 +
661 3860
Andnothing
what is the config that you are trying and failing?
Then I try to ship a json preformatted file using an rsylog pattern!
(Yes, I know, this is stupid but I was really in shorts of ideas... In
my tired mind this "procedure" should be able load the variable
from an
apache json log file, load them into the template and ship them to
logstash)... You can immaginate by yourself the result...
So...What is the correct way to proceed??? I find many sites but
most of
the are simply not clear or theyr solution don't work for me.
what does the file you are trying to ship look like?
The first thing to do, anytime you have trouble with your output, is
to write a
local log in the format RSYSLOG_DebugFormat because that will let
you see
exactly what is in every variable. Once you have the variables defined
the way
you think they are, then creating a format to chip them to logstash is
pretty
easy.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if you DON'T LIKE THAT.
AVVISO DI RISERVATEZZA Informazioni riservate possono essere contenute
nel messaggio o nei suoi allegati. Se non siete i destinatari indicati
nel messaggio, o responsabili per la sua consegna alla persona, o se
avete ricevuto il messaggio per errore, siete pregati di non
trascriverlo, copiarlo o inviarlo ad alcuno. In tal caso vi invitiamo
a cancellare il messaggio ed i suoi allegati. Grazie.
CONFIDENTIALITY NOTICE Confidential information may be contained in
this message or in its attachments. If you are not the addressee
indicated in this message, or responsible for message delivering to
that person, or if you have received this message in error, you may
not transcribe, copy or deliver this message to anyone. In that case,
you should delete this message and its attachments. Thank you.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if you DON'T LIKE THAT.
AVVISO DI RISERVATEZZA Informazioni riservate possono essere contenute
nel messaggio o nei suoi allegati. Se non siete i destinatari indicati
nel messaggio, o responsabili per la sua consegna alla persona, o se
avete ricevuto il messaggio per errore, siete pregati di non
trascriverlo, copiarlo o inviarlo ad alcuno. In tal caso vi invitiamo
a cancellare il messaggio ed i suoi allegati. Grazie.
CONFIDENTIALITY NOTICE Confidential information may be contained in
this message or in its attachments. If you are not the addressee
indicated in this message, or responsible for message delivering to
that person, or if you have received this message in error, you may
not transcribe, copy or deliver this message to anyone. In that case,
you should delete this message and its attachments. Thank you.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
AVVISO DI RISERVATEZZA Informazioni riservate possono essere contenute nel
messaggio o nei suoi allegati. Se non siete i destinatari indicati nel
messaggio, o responsabili per la sua consegna alla persona, o se avete ricevuto
il messaggio per errore, siete pregati di non trascriverlo, copiarlo o inviarlo
ad alcuno. In tal caso vi invitiamo a cancellare il messaggio ed i suoi
allegati. Grazie.
CONFIDENTIALITY NOTICE Confidential information may be contained in this
message or in its attachments. If you are not the addressee indicated in this
message, or responsible for message delivering to that person, or if you have
received this message in error, you may not transcribe, copy or deliver this
message to anyone. In that case, you should delete this message and its
attachments. Thank you.
