Hi Edmundo,
thanks for hints. It helped me better understand GL but not solved my issue.
I accept suggested workaround with dummy value and fact that some fields
are not removable (probably GL internal design).
Current "Message Processors Configuration":
1. GeoIP Resolver
2. Message Filter Chain
3. Pipeline Processor
I tested some scenarios:
1. rewrite on non "message" field
rule "function changer"
when
has_field("fruit")
then
set_field("fruit", "x");
end
Result: working as expected. Field fruit rewriten with dummy value.
2. rewrite on "message" field
rule "function changer"
when
has_field("message")
then
set_field("message", "x");
end
Result: not working as expected. All content of message (in sense of
original message form source) was rewriten with dummy value (x), not only
rewrite of "message" field.
I tried also combination of new custom stream and link the rewrite rule
with it but with same result.
I feel that it could help to combine custom stream and rules and stages but
no idea which way...
Any idea from you?
Thanks in advance.
Regards,
Nimmie
On Friday, June 10, 2016 at 5:53:00 PM UTC+2, Edmundo Alvarez wrote:
> Looking at the documentation, you cannot remove the "message" field, as it
> is a required field. As far as I know, the only thing you could do with the
> pipeline processor, is to set the "message" field content to some dummy
> value, e.g. "discarded", by using the `set_field` function.
>
> If that is good enough for you, the rule you shared looks otherwise fine,
> with two remarks:
>
> - If you use a JSON extractor with data from the "message" field, you need
> to ensure that the processors are in the right order. Check the "Message
> Processors Configuration" section under System -> Configurations
> - Connecting it to "Incoming messages stream" will drop the "message"
> field on all messages that are not routed into any stream
>
> I hope that helps.
>
> Regards,
> Edmundo
>
> > On 10 Jun 2016, at 12:43, nimmie <nim...@tutanota.com <javascript:>>
> wrote:
> >
> > Hi,
> > to be honest I tried the pipelines first, but without positive result.
> > May be I am missing something.
> >
> > My pipeline setup:
> > 1. rule definition
> > rule "remove"
> > when
> > has_field("message")
> > then
> > remove_field("message");
> > end
> >
> > 2. rule linked with new pipeline (stage 0) and the pipeline linked with
> "Incoming messages stream".
> > Is that OK?
> >
> > I suppose no graylog restart is needed. How can I make troubleshooting
> (check correct work) of this setup?
> >
> > Thanks.
> >
> > Nimmie
> >
> >
> > On Friday, June 10, 2016 at 11:21:18 AM UTC+2, Edmundo Alvarez wrote:
> > Hi,
> >
> > I know it's not what you asked for, but I think using the new pipeline
> processor can help you with that. You can find it in Graylog 2.0, if you
> want to try it. Here's some documentation:
> http://docs.graylog.org/en/2.0/pages/pipelines.html
> >
> > Regards,
> > Edmundo
> >
> > > On 10 Jun 2016, at 10:41, nimmie <nim...@tutanota.com> wrote:
> > >
> > > Hi,
> > > I would like to ask you how to deal with $SUBJ?
> > >
> > > Additional question marks:
> > > a. I need something like logstash mutate -> remove_field
> functionality: feasible in Graylog?
> > > b. after some examination: is the Graylog correct/only tool "drools"?
> (pipeline procesing still marked as experimental)
> > > c. what is exact processing order in Graylog (from my issue point of
> view): 1. input filter with extractor (created via GUI) 2. drools file
> *.drl?
> > > d. where is the best starting point for learning drools? not the link
> in graylog docs, but where can I find e.g. list of "methods" (e.g.
> "add.Field")?
> > >
> > > My current set-up:
> > > 1. local file with logs
> > > 2. logstah for file processing (LS sending json to RabbitMQ)
> > > 3. RabbiMQ
> > > 4. Graylog (input with RawAMPQ): used JSON extractor - working, but I
> need to remove "message" field (remove data duplication)
> > >
> > > Thanks in advance.
> > > nimmie
> > >
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups "Graylog Users" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> an email to graylog2+u...@googlegroups.com.
> > > To view this discussion on the web visit
> https://groups.google.com/d/msgid/graylog2/b49e3a46-3e46-457d-bd04-5cc34942c2dd%40googlegroups.com.
>
>
> > > For more options, visit https://groups.google.com/d/optout.
> >
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "Graylog Users" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to graylog2+u...@googlegroups.com <javascript:>.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/graylog2/e8ad54fa-1ebe-449a-86a5-a95f0b88680b%40googlegroups.com.
>
>
> > For more options, visit https://groups.google.com/d/optout.
>
>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/ee8122c0-7988-4240-8676-cfe9fc88aaf5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
