Awesome. I personally think that the only practical solution for multiline logging is mandatory sentencing guidelines at the federal level.
On Wed, Feb 7, 2018 at 4:08 PM, Charles Givre <[email protected]> wrote: > Hi Kunal, > As implemented it doesn’t do multiline logfiles. I wrote this for a > specific client a while ago and it’s proven VERY useful so I thought I’d > contribute it. > I would like to get this in there and then add multiline capability. > — C > > > > > On Feb 7, 2018, at 21:28, Kunal Khatua <[email protected]> wrote: > > > > I think I'm jumping the gun, because I haven’t yet tried out your PR. > > > > But to explain why I mentioned LogStash is because the primary challenge > (IMO) of creating a log file reader is that the format can be wildly > different and there is no standard format. So, what is needed is a good > mechanism to consume the logs with the right Regex feature. LogStash comes > with a Grok parser that does (IMHO) a fantastic job of parsing & tokenizing > the logs. > > > > The logback XML that I have for drill defines this format: > > <appender name="FILE" class="ch.qos.logback.core. > rolling.RollingFileAppender"> > > <encoder> > > <pattern>%date{ISO8601} %property{HOSTNAME} [%thread] %-5level > %logger{36} - %msg%n</pattern> > > </encoder> > > </appender> > > > > The one that comes default with Drill is > > <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender"> > > <encoder> > > <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - > %msg%n</pattern> > > </encoder> > > </appender> > > And > > <appender name="FILE" class="ch.qos.logback.core. > rolling.RollingFileAppender"> > > <encoder> > > <pattern>%date{ISO8601} [%thread] %-5level %logger{36} - > %msg%n</pattern> > > </encoder> > > </appender> > > > > Notice how all three patterns are different. > > > > A quick glance of the PR hints towards a fairly limited scope of log > files that can be processed (though I could be wrong). > > > > A good way to test the log reader should be to simply look at the web > UI's http://<hostname>:8047/logs link and pick out those logs for > processing/parsing. > > > > I did stitch up something using ELK (ElasticSearch+LogStash+Kibana) to > process Drill logs, but that was back in 2015. If we can get something like > that into a storage plugin for Drill, that would probably go much farther. > I could share what I did back then and figure out a way to use that > approach and libraries to leverage this. > > > > -----Original Message----- > > From: Charles Givre [mailto:[email protected]] > > Sent: Wednesday, February 07, 2018 1:08 PM > > To: [email protected] > > Subject: Re: Test cases for Drill-6104: Added Logfile Reader > > > > Hi Kunal, > > I just don’t know how to craft one with all the Drill internals. Is > there an example that I you can point me to? > > > >> On Feb 7, 2018, at 18:38, Kunal Khatua <[email protected]> wrote: > >> > >> How about using the Drill logs as a use case? > >> > >> You have drillbit.out and drillbit_hostname.log to consume. It would be > interesting to see how multiline log entries are handled. > >> > >> Logstash does an excellent job IMO, but that's more for parsing. > >> > >> -----Original Message----- > >> From: Charles Givre [mailto:[email protected]] > >> Sent: Wednesday, February 07, 2018 2:32 AM > >> To: [email protected] > >> Subject: Test cases for Drill-6104: Added Logfile Reader > >> > >> Hello all, > >> I submitted this PR for a logfile parser for Drill (https://urldefense. > proofpoint.com/v2/url?u=https-3A__github.com_apache_drill_ > pull_1114&d=DwIFAg&c=cskdkSMqhcnjZxdQVpwTXg&r=-cT6otg6lpT_XkmYy7yg3A&m= > oyYUEV4U-85UnHzphkWP57ikKiUPhdBpBw7F9HZGZZ4&s=rmM0FHOFV2_cyScnz1qtDz_ > zJpJjkPEB_2jT1WsujT0&e= <https://urldefense.proofpoint.com/v2/url?u=https- > 3A__github.com_apache_drill_pull_1114&d=DwIFAg&c= > cskdkSMqhcnjZxdQVpwTXg&r=-cT6otg6lpT_XkmYy7yg3A&m=oyYUEV4U- > 85UnHzphkWP57ikKiUPhdBpBw7F9HZGZZ4&s=rmM0FHOFV2_cyScnz1qtDz_ > zJpJjkPEB_2jT1WsujT0&e=>) . I need to write unit tests for it, however I > really have no idea how to do so. Could someone point me to an example or > something so that the PR will pass the CI tests? > >> TIA, > >> - C > >> > >> > >> > > > >
