John W Higgins wrote: > On Sun, Apr 15, 2018 at 11:08 AM, David Arnold <dar@xoe.solutions> wrote: > > > >This would appear to solve multiline issues within Fluent..... > > >https://docs.fluentd.org/v0.12/articles/parser_multiline > > > > I definitely looked at that, but what guarantees do I have that the > > sequence is always ERROR/STATEMENT/DETAIL? And not the other way round? > > Have you asked that question? You seem to at least have opened the source > code - did you try to figure out what the logging format is?
I looked at this a couple of days ago. I think parsing with this library is possible to a certain extent, and the problems stem from limitations of the library. So, it turns out that the firstline can be set to a single regex that searches for PANIC, FATAL, ERROR, WARNING, LOG, NOTICE, DEBUG. That's always the first line in any postgres log event. A log event contains some subsequent lines. Those start either with a tab (which is a continuation of the previous line) or with one of DETAIL, CONTEXT, HINT, STATEMENT, QUERY. This seems very simple to parse (just add lineN patterns for those), *except* that the messages can be multiline too; and where would you assign the continuation lines for each of those? parser_multiline does not support that. Another thing worth keeping in mind is that you need to change the regex depending on log_line_prefix, which sounds very painful. All in all, the best approach might be to create a specific parser_postgresql.rb plugin. Seems much easier to write two dozen lines of Ruby than change all of PostgreSQL's logging infrastructure. -- Álvaro Herrera https://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
