> On Feb. 28, 2017, 2:24 p.m., Balázs Donát Bessenyei wrote: > > flume-ng-doc/sphinx/FlumeUserGuide.rst > > Lines 1191 (patched) > > <https://reviews.apache.org/r/50438/diff/2/?file=1638337#file1638337line1191> > > > > Can you please clarify this setting? > > qiao wen wrote: > It's explained in the function readMultilineEventPre() and > readMultilineEventNext(). And you can see the effect in TestTaildirSource. > > Previous: If not matched, this line is not part of previous event when > the buffer event is not null. > Then create a new event with buffer event's message and > put the current line into the > cleared buffer event. > Next: If not matched, this line is not part of next event. Then > merge the current line into the > buffer event and create a new event with the merged > message. > > private Event readMultilineEventPre(LineResult line, boolean match, > Pattern pattern) > throws IOException { > Event event = null; > Matcher m = pattern.matcher(new String(line.line)); > boolean find = m.find(); > match = (find && match) || (!find && !match); > byte[] lineBytes = toOriginBytes(line); > if (match) { > /** If matched, merge it to the buffer event. */ > mergeEvent(line); > } else { > /** > * If not matched, this line is not part of previous event when the > buffer event is not null. > * Then create a new event with buffer event's message and put the > current line into the > * cleared buffer event. > */ > if (bufferEvent != null) { > event = EventBuilder.withBody(bufferEvent.getBody()); > } > bufferEvent = null; > bufferEvent = EventBuilder.withBody(lineBytes); > if (line.lineSepInclude) { > bufferEvent.getHeaders().put("lineCount", "1"); > } > long now = System.currentTimeMillis(); > > bufferEvent.getHeaders().put(TimestampInterceptor.Constants.TIMESTAMP, > Long.toString(now)); > } > return event; > } > > private Event readMultilineEventNext(LineResult line, boolean match, > Pattern pattern) > throws IOException { > Event event = null; > Matcher m = pattern.matcher(new String(line.line)); > boolean find = m.find(); > match = (find && match) || (!find && !match); > if (match) { > /** If matched, merge it to the buffer event. */ > mergeEvent(line); > } else { > /** > * If not matched, this line is not part of next event. Then merge > the current line into the > * buffer event and create a new event with the merged message. > */ > mergeEvent(line); > event = EventBuilder.withBody(bufferEvent.getBody()); > bufferEvent = null; > } > return event; > }
When I read the code, it's perfectly clear. However, it isn't necessarily so for users. I think including some description in the user guide would be very useful. - Balázs Donát ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50438/#review167074 ----------------------------------------------------------- On Feb. 17, 2017, 11:13 a.m., qiao wen wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/50438/ > ----------------------------------------------------------- > > (Updated Feb. 17, 2017, 11:13 a.m.) > > > Review request for Flume. > > > Repository: flume-git > > > Description > ------- > > TaidirSource defaults to LINE, this has issue when multiline log events like > stack traces and have request/responses. Following part is Java traceback > logs. We expect to have log line start regex Key to aggregate all the log > lines till the next regex key is found. > 2016-07-16 14:59:43,956 ERROR lifecycleSupervisor-1-7 LifecycleSupervisor.run > - Unable to start EventDrivenSourceRunner: { > source:cn.yottabyte.flume.source.http.HTTPSource{name:sourceHttp,state:IDLE} > } - Exception follows. > java.lang.IllegalStateException: Running HTTP Server found in source: > sourceHttp before I started one. Will not attempt to start. > at com.google.common.base.Preconditions.checkState(Preconditions.java:145) > at > cn.yottabyte.flume.source.http.HTTPSource.startHttpSourceServer(HTTPSource.java:170) > at cn.yottabyte.flume.source.http.HTTPSource.start(HTTPSource.java:166) > at > org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44) > at > org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > > Diffs > ----- > > flume-ng-doc/sphinx/FlumeUserGuide.rst afa6625 > > flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/ReliableTaildirEventReader.java > 8838320 > > flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/TailFile.java > 42474c4 > > flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/TaildirSource.java > a107a01 > > flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/TaildirSourceConfigurationConstants.java > f2347f3 > > flume-ng-sources/flume-taildir-source/src/test/java/org/apache/flume/source/taildir/TestTaildirSource.java > 097ee0b > > > Diff: https://reviews.apache.org/r/50438/diff/2/ > > > Testing > ------- > > All tests in TestTaildirSource passed. > > > Thanks, > > qiao wen > >
