[jira] [Commented] (FLUME-2584) Need graceful shutdown of HDFSEventSink
[ https://issues.apache.org/jira/browse/FLUME-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14638101#comment-14638101 ] Format Fang commented on FLUME-2584: but the file with ".tmp" extention won't be removed, how should i do? > Need graceful shutdown of HDFSEventSink > --- > > Key: FLUME-2584 > URL: https://issues.apache.org/jira/browse/FLUME-2584 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: v1.5.1 > Environment: Tested with Flume 1.5.0 and 1.5.2 on Redhat 6. >Reporter: Sverre Bakke >Priority: Minor > > When pressing ctrl+c (or otherwise signal to kill the flume process), the > HDFS sink seems to have issues dealing with interruption. 100% of the times I > do this I get the following errors back: > 2015-01-05 11:16:52,448 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN > - > org.apache.flume.sink.hdfs.BucketWriter.callWithTimeout(BucketWriter.java:752)] >Unexpected Exception null > java.lang.InterruptedException > at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:400) > at java.util.concurrent.FutureTask.get(FutureTask.java:199) > at > org.apache.flume.sink.hdfs.BucketWriter.callWithTimeout(BucketWriter.java:725) > at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:262) > at > org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:554) > at > org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:426) > at > org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) > at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) > at java.lang.Thread.run(Thread.java:745) > 2015-01-05 11:16:52,451 (SinkRunner-PollingRunner-DefaultSinkProcessor) > [ERROR - > org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:467)] > proc ess failed > java.lang.RuntimeException: java.lang.InterruptedException > at com.google.common.base.Throwables.propagate(Throwables.java:156) > at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:291) > at > org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:554) > at > org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:426) > at > org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) > at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.InterruptedException > at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:400) > at java.util.concurrent.FutureTask.get(FutureTask.java:199) > at > org.apache.flume.sink.hdfs.BucketWriter.callWithTimeout(BucketWriter.java:725) > at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:262) > ... 5 more > 2015-01-05 11:16:52,452 (SinkRunner-PollingRunner-DefaultSinkProcessor) > [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] > Unable to deliver event. Exception follows. > org.apache.flume.EventDeliveryException: java.lang.RuntimeException: > java.lang.InterruptedException > at > org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:471) > at > org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) > at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException: java.lang.InterruptedException > at com.google.common.base.Throwables.propagate(Throwables.java:156) > at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:291) > at > org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:554) > at > org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:426) > ... 3 more > Caused by: java.lang.InterruptedException > at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:400) > at java.util.concurrent.FutureTask.get(FutureTask.java:199) > at > org.apache.flume.sink.hdfs.BucketWriter.callWithTimeout(BucketWriter.java:725) > at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:262) > ... 5 more > or > 2015-01-05 11:20:34,706 (agent-shutdown-hook) [DEBUG - > org.apache.flume.SinkRunner.stop(SinkRunner.java:104)] Waiting for runner > thread to exit > 2015-01-05 11:20:34,709 (SinkRunner-PollingRunner-DefaultSinkProcessor) > [ERROR - > org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:467)] > process failed > java.lang.InterruptedException: Timed out before HDFS call was made. Your > hdfs.ca
[jira] [Commented] (FLUME-2498) Implement Taildir Source
[ https://issues.apache.org/jira/browse/FLUME-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14637926#comment-14637926 ] Hari Shreedharan commented on FLUME-2498: - [~jrufus]/[~roshan_naik] - Do you think one of you would be able take a look at this one? > Implement Taildir Source > > > Key: FLUME-2498 > URL: https://issues.apache.org/jira/browse/FLUME-2498 > Project: Flume > Issue Type: New Feature > Components: Sinks+Sources >Reporter: Satoshi Iijima > Fix For: v1.7.0 > > Attachments: FLUME-2498-2.patch, FLUME-2498.patch > > > This is the proposal of implementing a new tailing source. > This source watches the specified files, and tails them in nearly real-time > once appends are detected to these files. > * This source is reliable and will not miss data even when the tailing files > rotate. > * It periodically writes the last read position of each file in a position > file using the JSON format. > * If Flume is stopped or down for some reason, it can restart tailing from > the position written on the existing position file. > * It can add event headers to each tailing file group. > A attached patch includes a config documentation of this. > This source requires Unix-style file system and Java 1.7 or later. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2498) Implement Taildir Source
[ https://issues.apache.org/jira/browse/FLUME-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14637397#comment-14637397 ] sutanu das commented on FLUME-2498: --- Can this patch be backported for Flume 1.5 please? We run Hortonworks Flume 1.5.2.2 and they will not backport this patch for us. Reason we need this patch: 1. We want to restart log-files ingestion as events at a point which flume stopped/recovered, yet, the loglines keep growing/appending 2. We want to get logfile even if logs rotate with new_names eg tail.log.x get rotated to tail.log.y -- where -F of exec source doesnt work and spoolDir doesnt work either (b/c of timestamp limitations) > Implement Taildir Source > > > Key: FLUME-2498 > URL: https://issues.apache.org/jira/browse/FLUME-2498 > Project: Flume > Issue Type: New Feature > Components: Sinks+Sources >Reporter: Satoshi Iijima > Fix For: v1.7.0 > > Attachments: FLUME-2498-2.patch, FLUME-2498.patch > > > This is the proposal of implementing a new tailing source. > This source watches the specified files, and tails them in nearly real-time > once appends are detected to these files. > * This source is reliable and will not miss data even when the tailing files > rotate. > * It periodically writes the last read position of each file in a position > file using the JSON format. > * If Flume is stopped or down for some reason, it can restart tailing from > the position written on the existing position file. > * It can add event headers to each tailing file group. > A attached patch includes a config documentation of this. > This source requires Unix-style file system and Java 1.7 or later. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2746) How to include this Flume Patch in Flume 1.5.2 ?
[ https://issues.apache.org/jira/browse/FLUME-2746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14637371#comment-14637371 ] sutanu das commented on FLUME-2746: --- 1. How can we include this patch in Flume 1.5.2 ? (note we run Hortonworks Flume 1.5.2) we have an immediate use case for this patch/Tarildir source Patch : https://issues.apache.org/jira/secure/attachment/12692432/FLUME-2498-2.patch +Taildir Source 2. When will be Flume 1.7 available for install which includes this patch/source? > How to include this Flume Patch in Flume 1.5.2 ? > > > Key: FLUME-2746 > URL: https://issues.apache.org/jira/browse/FLUME-2746 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: v1.5.1, v1.7.0 > Environment: 1. We run Hortontowns flume 1.5.2 but how can we include > this patch (sourceType) -- > https://issues.apache.org/jira/secure/attachment/12692432/FLUME-2498-2.patch > in Flume 1.5.2 ? > FLUME-2498-2.patch > 2. When is FLume 1.7 which has this sourceType (+Taildir Source) available > for Install ? >Reporter: sutanu das > Fix For: v1.7.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLUME-2746) How to include this Flume Patch in Flume 1.5.2 ?
sutanu das created FLUME-2746: - Summary: How to include this Flume Patch in Flume 1.5.2 ? Key: FLUME-2746 URL: https://issues.apache.org/jira/browse/FLUME-2746 Project: Flume Issue Type: Bug Components: Sinks+Sources Affects Versions: v1.5.1, v1.7.0 Environment: 1. We run Hortontowns flume 1.5.2 but how can we include this patch (sourceType) -- https://issues.apache.org/jira/secure/attachment/12692432/FLUME-2498-2.patch in Flume 1.5.2 ? FLUME-2498-2.patch 2. When is FLume 1.7 which has this sourceType (+Taildir Source) available for Install ? Reporter: sutanu das Fix For: v1.7.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Reg MultiLine Deserializer for Tail Source and Spool Dir source
Is there any new about this? 2015-05-09 20:51 GMT+02:00 Hari Shreedharan : > The deserializer interface does not restrict reads to lines. You can always > write one that reads stack traces. If you end up writing one, we'd gladly > review it to include it in flume itself. > > On Saturday, May 9, 2015, Siva kotipalli > wrote: > > > Hi, > > > >The Spool directory source by default consider the carriage return as > > event delimiter, when we have stack traces spread across more than one > > line, they get stored as differnt events. > >Do we have any plans for including MultilineDeserizer for Spool data > > source, i created my own multiline deserializer used it via > configuration, > > but it would nice to have one included in the release. > > > >The same case as Tail datasource., where there is no option to include > > multiple deserializer and i created my own custom source to support > single > > line and multiline log statements. > > > > The above queries are with reference to Flume 1.5.2 > > > > Thanks & Regards > > Siva > > > > > -- > > Thanks, > Hari >
[jira] [Commented] (FLUME-2745) HbaseEventSerializer do not close
[ https://issues.apache.org/jira/browse/FLUME-2745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14636434#comment-14636434 ] Format Fang commented on FLUME-2745: I created a pull request in github. https://github.com/apache/flume/pull/20 > HbaseEventSerializer do not close > - > > Key: FLUME-2745 > URL: https://issues.apache.org/jira/browse/FLUME-2745 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: v1.5.0 >Reporter: Format Fang > > HbaseEventSerializer in HbaseSink do not close in stop method. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLUME-2745) HbaseEventSerializer do not close
Format Fang created FLUME-2745: -- Summary: HbaseEventSerializer do not close Key: FLUME-2745 URL: https://issues.apache.org/jira/browse/FLUME-2745 Project: Flume Issue Type: Bug Components: Sinks+Sources Affects Versions: v1.5.0 Reporter: Format Fang HbaseEventSerializer in HbaseSink do not close in stop method. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
