Flume source cannot put data into channel unless there is free space. You can try increasing the capacity of the channel.
2015-08-12 2:05 GMT+08:00 George Blazer <[email protected]>: > Hellok > > I'm getting "Channel is full" error because ChannelFillPercentage is > nearly at 100%. > > What can I look at? > > 2015-08-11 17:55:51,173 (pool-5-thread-1) [INFO - > org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:238)] > Last read was never committed - resetting mark position. > 2015-08-11 17:55:54,195 (pool-5-thread-1) [WARN - > org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:239)] > *The channel is full, and cannot write data now. The source will try again > after 4000 milliseconds* > > curl localhost:5653/metrics > > {"SINK.s3-sink":{"BatchCompleteCount":"0","ConnectionFailedCount":"1","EventDrainAttemptCount":"0","ConnectionCreatedCount":"0","Type":"SINK","BatchEmptyCount":"0","ConnectionClosedCount":"0","EventDrainSuccessCount":"0","StopTime":"0","StartTime":"1439315713336","BatchUnderflowCount":"0"},"SOURCE.spooling-directory":{"OpenConnectionCount":"0","Type":"SOURCE","AppendBatchAcceptedCount":"0","AppendBatchReceivedCount":"10","EventAcceptedCount":"0","AppendReceivedCount":"0","StopTime":"0","StartTime":"1439315713378","EventReceivedCount":"1000","AppendAcceptedCount":"0"},"CHANNEL.fileChannel":{"EventPutSuccessCount":"0","ChannelFillPercentage":"99.9956","Type":"CHANNEL","StopTime":"0","EventPutAttemptCount":"450","ChannelSize":"999956","StartTime":"1439315713327","EventTakeSuccessCount":"0","ChannelCapacity":"1000000","EventTakeAttemptCount":"1"}} > > My current config is: > > agent_messaging.sinks = s3-sink > agent_messaging.sources = spooling-directory > agent_messaging.channels = fileChannel > > agent_messaging.channels.fileChannel.type = file > agent_messaging.channels.fileChannel.checkpointDir = /mnt/flume/checkpoint > agent_messaging.channels.fileChannel.dataDirs = /mnt/flume/data > > agent_messaging.sources.spooling-directory.channels = fileChannel > agent_messaging.sources.spooling-directory.type = spooldir > agent_messaging.sources.spooling-directory.deletePolicy = immediate > agent_messaging.sources.spooling-directory.spoolDir = /mnt/flume/spool > > agent_messaging.sources.spooling-directory.interceptors = sr i1 ts > agent_messaging.sources.spooling-directory.interceptors.sr.type = > org.apache.flume.interceptor.SearchAndReplaceInterceptor$Builder > agent_messaging.sources.spooling-directory.interceptors.sr.searchPattern = > (\\d{4})/(\\d{2})/(\\d{2}) (\\d{2}):(\\d{2}):(\\d{2}) > agent_messaging.sources.spooling-directory.interceptors.sr.replaceString = > $1-$2-$3 $4:$5:$6 > > agent_messaging.sources.spooling-directory.interceptors.i1.type = > regex_extractor > agent_messaging.sources.spooling-directory.interceptors.i1.regex = > ^(?:\\n)?(\\d\\d\\d\\d-\\d\\d-\\d\\d\\s\\d\\d:\\d\\d:\\d\\d) > agent_messaging.sources.spooling-directory.interceptors.i1.serializers = s1 > agent_messaging.sources.spooling-directory.interceptors.i1.serializers.s1.type > = org.apache.flume.interceptor.RegexExtractorInterceptorMillisSerializer > > agent_messaging.sources.spooling-directory.interceptors.i1.serializers.s1.name > = timestamp > agent_messaging.sources.spooling-directory.interceptors.i1.serializers.s1.pattern > = yyyy-MM-dd HH:mm:ss > > agent_messaging.sources.spooling-directory.interceptors.ts.type = > org.apache.flume.interceptor.TimestampInterceptor$Builder > agent_messaging.sources.spooling-directory.interceptors.ts.preserveExisting > = true > > agent_messaging.sources.spooling-directory.deserializer.maxLineLength = > 10000 > > agent_messaging.sinks.s3-sink.channel = fileChannel > agent_messaging.sinks.s3-sink.type = hdfs > agent_messaging.sinks.s3-sink.hdfs.path = s3n://SECRETS@logs > /events/%Y-%m-%d > agent_messaging.sinks.s3-sink.hdfs.filePrefix = prod-role-8599226c > agent_messaging.sinks.s3-sink.hdfs.rollInterval = 0 > agent_messaging.sinks.s3-sink.hdfs.rollSize = 67108864 > agent_messaging.sinks.s3-sink.hdfs.rollCount = 0 > agent_messaging.sinks.s3-sink.hdfs.serializer = TEXT > agent_messaging.sinks.s3-sink.hdfs.fileType = DataStream > agent_messaging.sinks.s3-sink.hdfs.closeTries = 3 > agent_messaging.sinks.s3-sink.hdfs.idleTimeout = 0 > agent_messaging.sinks.s3-sink.hdfs.callTimeout = 180000 >
