Snehal Waghmare created FLUME-3137:
--------------------------------------
Summary: GC overhead limit exceeded in KafkaSink
Key: FLUME-3137
URL: https://issues.apache.org/jira/browse/FLUME-3137
Project: Flume
Issue Type: Bug
Reporter: Snehal Waghmare
My configuration file:
tier2.sources = source1
tier2.channels = channel
tier2.sinks = sink1
tier2.sources.source1.type = tailDir
tier2.sources.source1.channels = channel
tier2.sources.source1.filegroups = f1
tier2.sources.source1.filegroups.f1 = /path-to-logs/processed_logs/.*.log*
tier2.sources.source1.positionFile =
/path-to-file/apache-flume-1.7.0bin/.flume/taildir_position.json
tier2.channels.channel.type = memory
tier2.channels.channel.capacity = 10000
tier2.channels.channel.transactionCapacity = 10000
tier2.channels.channel.use-fast-replay = true
tier2.sinks.sink1.channel = channel
tier2.sinks.sink1.type = org.apache.flume.sink.kafka.KafkaSink
tier2.sinks.sink1.kafka.topic = topicname
tier2.sinks.sink1.kafka.bootstrap.servers=kafkaserverIP:9092
tier2.sinks.sink1.kafka.flumeBatchSize = 1000
Number of log files are located at folder "/path-to-logs/processed_logs/". Now
when flume agent is started it tries to send a few number of files but after
that gives the following error:
[ERROR -
org.apache.flume.source.taildir.TaildirSource.writePosition(TaildirSource.java:334)]
Failed writing positionFile
java.lang.OutOfMemoryError: GC overhead limit exceeded
at com.google.common.collect.ImmutableMap.of(ImmutableMap.java:95)
at
org.apache.flume.source.taildir.TaildirSource.toPosInfoJson(TaildirSource.java:349)
at
org.apache.flume.source.taildir.TaildirSource.writePosition(TaildirSource.java:330)
at
org.apache.flume.source.taildir.TaildirSource.access$600(TaildirSource.java:59)
at
org.apache.flume.source.taildir.TaildirSource$PositionWriterRunnable.run(TaildirSource.java:320)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
