poorejc commented on issue #243: URL: https://github.com/apache/incubator-flagon-useralejs/issues/243#issuecomment-1109852891
@EandrewJones thanks! This is great context. After I read this, I'm still leaning on those nulls being tags... in fact, they could actually be `grokparsefailure` tags. Anytime a given message wasn't parsed correctly by a given filter, etc., you get a `grokparsefailure` added to tags... That's to say that I'm not sure UserALE.js is injecting the `nulls` at the end of the message. Nonetheless, I'll do a little more digging into this... One thing that I should mention is that LogStash actually creates something like a NiFi "flowfile". Meaning what gets sent to ELK isn't the original message, but new JSON. In fact, if you're using our Flagon example 6.8 ELK build, if you look in the Pipeline dir under Logstash, you'll find that we actually `mutate` our the original message payload and headers from the "flowfile", so that the original message payload isn't posted to Elastic (would be added to index under "message" key), in addition to the newly created, parsed JSON. See below from [logstash.conf](https://github.com/UMD-ARLIS/docker-elk/blob/flagon-configs/logstash/pipeline/logstash.conf): ``` filter { mutate { remove_field => [ "message", "@version", "headers"] } ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@flagon.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
