Hi, There is nothing CDH specific in this discussion so let's move to the upstream user list which can be subscribed to here:
http://flume.apache.org/mailinglists.html When appending just a single like you are likely hitting the default batch size, which is 20. Try appending more than 20 lines or changing the batch size to 1: http://flume.apache.org/FlumeUserGuide.html#exec-source Brock On Mon, Nov 26, 2012 at 9:39 AM, Benc <[email protected]> wrote: > Well nothing. If I start it up it all looks good. If I do > > echo test >> /tmp/ben.log changing the config to read that. I don't see > anything in the log. > > I just see this in the flume.log > > 26 Nov 2012 15:37:39,263 INFO [lifecycleSupervisor-1-1] > (org.apache.flume.source.ExecSource.start:155) - Exec source starting with > command:tail -f /tmp/ben.log > > if for instance I change the command to "cat /etc/hosts" > > Then I see data in the log > > 26 Nov 2012 15:38:47,628 INFO [lifecycleSupervisor-1-2] > (org.apache.flume.source.ExecSource.start:155) - Exec source starting with > command:cat /etc/hosts > 26 Nov 2012 15:38:47,645 INFO [pool-4-thread-1] > (org.apache.flume.source.ExecSource$ExecRunnable.run:307) - Command [cat > /etc/hosts] exited with 0 > 26 Nov 2012 15:38:47,648 INFO > [SinkRunner-PollingRunner-DefaultSinkProcessor] > (org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{} body: > 23 23 ## } > 26 Nov 2012 15:38:47,648 INFO > [SinkRunner-PollingRunner-DefaultSinkProcessor] > (org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{} body: > 23 20 48 6F 73 74 20 44 61 74 61 62 61 73 65 # Host Database } > 26 Nov 2012 15:38:47,648 INFO > [SinkRunner-PollingRunner-DefaultSinkProcessor] > (org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{} body: > 23 # } > > > > > > > On Monday, November 26, 2012 3:30:18 PM UTC, Brock Noland wrote: > >> Hi, >> >> What error or behavior are you seeing that you do not expect? >> >> Also, as someone else said, I would replace -f with -F. >> >> Brock >> >> >> On Mon, Nov 26, 2012 at 9:27 AM, Benc <[email protected]> wrote: >> >>> Here is my config >>> >>> agent.sources = tail >>> >>> agent.channels = memoryChannel >>> >>> agent.sinks = loggerSink >>> >>> >>> agent.sources.tail.type = exec >>> >>> agent.sources.tail.command = tail -f /Users/pj/work/flume_test.log >>> >>> agent.sources.tail.channels = memoryChannel >>> >>> >>> agent.sinks.loggerSink.channel = memoryChannel >>> >>> agent.sinks.loggerSink.type = logger >>> >>> >>> agent.channels.memoryChannel.**type = memory >>> >>> agent.channels.memoryChannel.**capacity = 100 >>> >>> On Monday, November 26, 2012 3:26:55 PM UTC, Brock Noland wrote: >>> >>>> Hi, >>>> >>>> Can you share your configuration? Perhaps post it to pastbin? >>>> >>>> Brock >>>> >>>> >>>> On Tue, Nov 20, 2012 at 11:10 AM, Benc <[email protected]>wrote: >>>> >>>>> Hi James >>>>> >>>>> I would like to get all of our application log files into HDFS is the >>>>> high level overview, so I cam monitor for any errors. I did see that you >>>>> can use the exec command, but when migrating up from flume to flumeNG tail >>>>> does not work, or my configuration does not seem to work using exec = tail >>>>> -f log.name >>>>> >>>>> I know that tail is not great as in the event of a failure it does not >>>>> pick up from where it last read. >>>>> >>>>> >>>>> On Tuesday, November 20, 2012 4:50:38 PM UTC, James Hogarth wrote: >>>>>> >>>>>> >>>>>> >>>>>> On Tuesday, 20 November 2012 16:38:53 UTC, Benc wrote: >>>>>>> >>>>>>> All with the new FlumeNG has anyone built a log reader that would >>>>>>> replace tail? >>>>>> >>>>>> >>>>>> Can you describe your use case a little more? >>>>>> >>>>>> The documentation for FlumeNG talks a bit about this and how the exec >>>>>> source can be used with tail -F >>>>>> >>>>>> James >>>>>> >>>>>> -- >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/** >>>> mr**unit/ <http://incubator.apache.org/mrunit/> >>>> >>> -- >>> >>> >>> >>> >> >> >> >> -- >> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/** >> mrunit/ <http://incubator.apache.org/mrunit/> >> > -- > > > > -- Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
