I'm not sure that's true in the general case. Of the errors I've had with our elk stack, upward of 95% have been caused by type errors (json field should be an int but is an object); some small handful have failed because a message was truncated somewhere asking the line; a smaller number have failed because somebody hand-crafted json and forgot about a trailing comma or quote. Overwhelmingly, the data aren't corrupted: they were invalid at source in a way that would still allow them to be read as plain Unicode strings.
Obviously I accept that given enough data, I'll see more interesting failure modes that need more thought, but reading from the errorfile and pushing to a separate error index would work very well in our environment. On Fri, 2 Dec 2016, 08:43 David Lang, <[email protected]> wrote: On Fri, 2 Dec 2016, Bob Gregory wrote: > You may well be able to insert the rejected log into a different index. > Most of our failed logs are down to a mismatch between the mapping config > and the fields in json logs. > > An error index that treats the whole message as a single blob should work > fine. what bytes would need to be escaped? what if it's invalid unicode junk, etc. almost by definition we are talking about corrupt data. David Lang _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
