I've found the solution, I need to put "\r" in my custom executable code.
And the exception is gone in the beginning of logging period.
However, after I execute it for forty minutes++ same exception appeared
again. I guess I need to increase the memory size.
Refer to this snapshot for the exception:
2012-05-27 12:25:03,290 ERROR source.ExecSource: Failed while running
command: /home/ubuntu/testExec/intervalPrinting
org.apache.flume.ChannelException: Space for commit to queue couldn't be
acquired Sinks are likely not keeping up with sources, or the buffer size
is too tight
at
org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:84)
at
org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
at
org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:178)
at org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:258)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
2012-05-27 12:25:03,542 INFO sink.LoggerSink: Event: { headers:{} body: 6C
69 6E 65 20 23 32 31 34 31 line #2141 }
arinto
On Sun, May 27, 2012 at 11:39 AM, Arinto Murdopo <[email protected]> wrote:
> Hi there,
>
> I encountered this exception when I'm using exec source and logger sink:
>
> ERROR source.ExecSource: Failed while running command:
> /home/ubuntu/testExec/intervalPrinting
> org.apache.flume.ChannelException: Space for commit to queue couldn't be
> acquired Sinks are likely not keeping up with sources, or the buffer size
> is too tight
> at
> org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:84)
> at
> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
> at
> org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:178)
> at org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:258)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> My flume.conf:
> exec-agent.sources = intervalPrinting
> exec-agent.channels = memoryChannel
> exec-agent.sinks = loggerSink
>
> # For each one of the sources, the type is defined
> exec-agent.sources.intervalPrinting.type = exec
> exec-agent.sources.intervalPrinting.command =
> /home/ubuntu/testExec/intervalPrinting
>
> # The channel can be defined as follows.
> exec-agent.sources.intervalPrinting.channels = memoryChannel
>
> # Each sink's type must be defined
> exec-agent.sinks.loggerSink.type = logger
>
> #Specify the channel the sink should use
> exec-agent.sinks.loggerSink.channel = memoryChannel
>
> # Each channel's type is defined.
> exec-agent.channels.memoryChannel.type = memory
>
> # Other config values specific to each type of channel(sink or source)
> # can be defined as well
> # In this case, it specifies the capacity of the memory channel
> exec-agent.channels.memoryChannel.capacity = 100
>
> *intervalPrinting *is a simple C program that print a running number from
> 1 every one second. (1, 2, 3....)
> Something like SequenceGenerator but less frequent.
>
> Another strange thing is the logger doesn't automatically update the log
> file using the value printed by *intervalPrinting* in stdout. Only after
> the aforementioned exception occurs, logger updates the log file using
> value from *intervalPrinting *as shown below. Not that the logging
> timestamps are always 09:17:42. I except it to be updated every second
> (like sequence generator, but in slower interval), synchronized with the
> behavior of *intervalPrinting. *So, any idea how to solve these problems
> (exception and strange behavior on updating the log file)?
>
> 2012-05-27 09:17:42,180 INFO sink.LoggerSink: Event: { headers:{} body: 31
> 1 }
> 2012-05-27 09:17:42,181 INFO sink.LoggerSink: Event: { headers:{} body: 32
> 2 }
> 2012-05-27 09:17:42,181 INFO sink.LoggerSink: Event: { headers:{} body: 33
> 3 }
> 2012-05-27 09:17:42,181 INFO sink.LoggerSink: Event: { headers:{} body: 34
> 4 }
> 2012-05-27 09:17:42,181 INFO sink.LoggerSink: Event: { headers:{} body: 35
> 5 }
> 2012-05-27 09:17:42,182 INFO sink.LoggerSink: Event: { headers:{} body: 36
> 6 }
> 2012-05-27 09:17:42,182 INFO sink.LoggerSink: Event: { headers:{} body: 37
> 7 }
> 2012-05-27 09:17:42,182 INFO sink.LoggerSink: Event: { headers:{} body: 38
> 8 }
> 2012-05-27 09:17:42,182 INFO sink.LoggerSink: Event: { headers:{} body: 39
> 9 }
> ..... [10 to 89 are omitted to save space]
> 2012-05-27 09:17:42,216 INFO sink.LoggerSink: Event: { headers:{} body: 39
> 30 90 }
> 2012-05-27 09:17:42,216 INFO sink.LoggerSink: Event: { headers:{} body: 39
> 31 91 }
> 2012-05-27 09:17:42,217 INFO sink.LoggerSink: Event: { headers:{} body: 39
> 32 92 }
> 2012-05-27 09:17:42,217 INFO sink.LoggerSink: Event: { headers:{} body: 39
> 33 93 }
> 2012-05-27 09:17:42,217 INFO sink.LoggerSink: Event: { headers:{} body: 39
> 34 94 }
> 2012-05-27 09:17:42,217 INFO sink.LoggerSink: Event: { headers:{} body: 39
> 35 95 }
> 2012-05-27 09:17:42,217 INFO sink.LoggerSink: Event: { headers:{} body: 39
> 36 96 }
> 2012-05-27 09:17:42,217 INFO sink.LoggerSink: Event: { headers:{} body: 39
> 37 97 }
> 2012-05-27 09:17:42,218 INFO sink.LoggerSink: Event: { headers:{} body: 39
> 38 98 }
> 2012-05-27 09:17:42,218 INFO sink.LoggerSink: Event: { headers:{} body: 39
> 39 99 }
> 2012-05-27 09:17:42,218 INFO sink.LoggerSink: Event: { headers:{} body: 31
> 30 30 100 }
>
> arinto
>
