Understood. But we have a case where we have to support dots in field names.
On 11/18/2016 08:00 AM, Rainer Gerhards wrote:
And, as I wrote, rsyslog already removes dots...
Rainer
Sent from phone, thus brief.
Am 18.11.2016 15:55 schrieb "Rich Megginson" <[email protected]>:
On 11/18/2016 05:27 AM, Ryan Ward wrote:
Supporting dots are back in the latest elasticsearch 5.0
And, if you don't want to use 5.0 just yet, 2.4.1 supports dots in field
names if you use the -Dmapper.allow_dots_in_name=true flag.
On Fri, Nov 18, 2016 at 6:16 AM, chenlin rao <[email protected]>
wrote:
There is a logstash plugin named logstash-filter-de_dot.
And because logstash has a force queue to send a larger bulk_size to ES
which is better in ES recommandition, I suggest to use
rsyslog-omelasticsearch only for not so high load -- but need to watch it
as quickly-- use case.
2016-11-18 18:53 GMT+08:00 Rainer Gerhards <[email protected]>:
2016-11-18 11:51 GMT+01:00 David Lang <[email protected]>:
On Fri, 18 Nov 2016, Simon Tiong wrote:
On 18/11/2016 5:39 PM, David Lang wrote:
On Fri, 18 Nov 2016, Peter Viskup wrote:
Dear all,
at the moment our message flow is setup like this:
rsyslog remote client->rsyslog server->local
file->Logstash->ElasticSearch
Logstash does any special message processing, just forwards messages
to
ES.
We are considering the use of omelasticsearch module for direct
forward to ES (skip local file store and LogStash).
What could be the consequences or drawbacks? What are your
experiences
you can share?
going directly to ES will be MUCH faster :-)
one problem with recent versions of ES is that it won't accept any
json
that has a '.' in the object name. I don't think Logstash fixes this,
but I
don't know what it does in those cases.
David Lang
Hi Peter,
ES not accepting '.', but here is a workaround way. Under
logstash.conf
you might do some converting, for example. Hope it helps.
the question was how to not use logstash and what the
problems/advantages of
not using logstash are.
mutate {
rename => {
"id.orig_h" => "id_orig_h"
"id.orig_p" => "id_orig_p"
"id.resp_h" => "id_resp_h"
"id.resp_p" => "id_resp_p"
}
}
this requires that you know all the values that have '.' in their name.
If
you know that you can also manually do the same thing in rsyslog.
recent omelasticsearch handles that itself:
https://github.com/rsyslog/rsyslog/commit/0f5595c0742fce2f5f
cfe86fda22bc
091330f368
Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
