>Why? The newlines aren't meaningfully different from other characters you need to parse? The data isn't actually stored in a newline separated fashion, that's just one byte with that meaning
I miss the details, but I believe that stdout is usually parsed and streamed simply line by line. Like in: "Important: do not attempt to add multiline support in your regular expressions if you are using Tail <https://fluentbit.io/documentation/0.13/input/tail.html> input plugin since each line is handled as a separated entity. Instead use Tail Multiline <https://fluentbit.io/documentation/0.13/input/tail.html#multiline> support configuration feature." https://fluentbit.io/documentation/0.13/parser/regular_expression.html El dom., 15 abr. 2018 a las 13:08, David Arnold (<dar@xoe.solutions>) escribió: > >This would appear to solve multiline issues within Fluent..... > >https://docs.fluentd.org/v0.12/articles/parser_multiline > > I definitely looked at that, but what guarantees do I have that the > sequence is always ERROR/STATEMENT/DETAIL? And not the other way round? > And it only works with tail logging from log file so I cannot use a native > docker logging driver which streams event by event. > This again prohibits me the usage of host-global docker logging driver > configuration as my standard option for host provisioning. > > >Is the issue that there are line breaks in things like lines 7-9? > No way to parse that with a line by line regex cleanly. Fluent-multi line > is no real multi line regex, it's just some logic to emulate multi line > regex. I believe the reason might be that true multi line regex would be > way to resource demanding to run intercalatingly on a moving set of lines > of unknown cardinality. > > El dom., 15 abr. 2018 a las 13:00, David Arnold (<dar@xoe.solutions>) > escribió: > >> >It looks like the thread skipped over the problem space for the >> solution space pretty fast >> >> OK, I apologize, it seemed to me from the feedback that the problem was >> already uncontested. To verify/falsify that was the objective of my >> previous mail :) >> >> >Can you elaborate? >> >> Sure. >> CSV Logs: https://pastebin.com/uwfmRdU7 >> stdout: >> https://gist.github.com/blaggacao/d389940ef7b223eebc707096b536967d >> >> CSV shows line breaks, STDOUT shows ERROR/FATAL and detail on different >> lines, not an easy problem to stream-parse reliably (without some kind of a >> buffer, etc)... >> >> El dom., 15 abr. 2018 a las 12:46, Christophe Pettus (<x...@thebuild.com>) >> escribió: >> >>> >>> > On Apr 15, 2018, at 10:39, David Arnold <dar@xoe.solutions> wrote: >>> > >>> > In the light of the specific use case / problem for this thread to be >>> born, what exactly would you suggest? >>> >>> It looks like the thread skipped over the problem space for the solution >>> space pretty fast; I see your note: >>> >>> > I have reviewed some log samples and all DO contain some kind of multi >>> line logs which are very uncomfortable to parse reliably in a log streamer. >>> >>> ... but I don't see any actual examples of those. Can you elaborate? >>> >>> -- >>> -- Christophe Pettus >>> x...@thebuild.com >>> >>> -- >> [image: XOE Solutions] <http://xoe.solutions/> DAVID ARNOLD >> Gerente General >> xoe.solutions >> dar@xoe.solutions >> +57 (315) 304 13 68 >> *Confidentiality Note: * This email may contain confidential and/or >> private information. If you received this email in error please delete and >> notify sender. >> *Environmental Consideration: * Please avoid printing this email on >> paper, unless really necessary. >> > -- > [image: XOE Solutions] <http://xoe.solutions/> DAVID ARNOLD > Gerente General > xoe.solutions > dar@xoe.solutions > +57 (315) 304 13 68 > *Confidentiality Note: * This email may contain confidential and/or > private information. If you received this email in error please delete and > notify sender. > *Environmental Consideration: * Please avoid printing this email on > paper, unless really necessary. > -- [image: XOE Solutions] <http://xoe.solutions/> DAVID ARNOLD Gerente General xoe.solutions dar@xoe.solutions +57 (315) 304 13 68 *Confidentiality Note: * This email may contain confidential and/or private information. If you received this email in error please delete and notify sender. *Environmental Consideration: * Please avoid printing this email on paper, unless really necessary.