Hello Everyone,
Facing a problem with flume spool.
Below is my configuration,
# Please paste flume.conf here. Example:
# Sources, channels, and sinks are defined per
# agent name, in this case 'pnm'.
pnm.sources = SPOOL
pnm.channels = MemChannel
pnm.sinks = AVRO
# For each source, channel, and sink, set
# standard properties.
pnm.sources.SPOOL.type = spooldir
pnm.sources.SPOOL.spoolDir = /home/s_sdldalplhdxxxedh/pnm-poll-results
pnm.sources.SPOOL.channels = MemChannel MemChannel2
pnm.sources.SPOOL.fileHeader = true
pnm.sources.SPOOL.deletePolicy = immediate
pnm.sources.SPOOL.consumeOrder = oldest
pnm.sources.SPOOL.batchSize = 1
pnm.sources.SPOOL.interceptors = time
pnm.sources.SPOOL.interceptors.time.type =
org.apache.flume.interceptor.TimestampInterceptor$Builder
pnm.sources.SPOOL.deserializer =
com.suddenlink.flume.WholeFileDeserializer$Builder
pnm.sinks.AVRO.type = avro
pnm.sinks.AVRO.channel = MemChannel
pnm.sinks.AVRO.hostname = sdldalplhdw01.suddenlink.cequel3.com
pnm.sinks.AVRO.port = 40001
pnm.sinks.AVRO.batchSize = 1
pnm.sinks.AVRO.connect-timeout = 40000
# pnm.sinks.HDFS.type = hdfs
# pnm.sinks.HDFS.channel = MemChannel2
# pnm.sinks.HDFS.hdfs.path = /user/flume/poll/%Y/%m/%d/%H/
# pnm.sinks.HDFS.hdfs.fileType = DataStream
# pnm.sinks.HDFS.hdfs.writeFormat = Text
# pnm.sinks.HDFS.hdfs.batchSize = 100
# pnm.sinks.HDFS.hdfs.rollSize = 0
# pnm.sinks.HDFS.hdfs.rollCount = 1000
# pnm.sinks.HDFS.hdfs.rollInterval = 600
# Other properties are specific to each type of
# source, channel, or sink. In this case, we
# specify the capacity of the memory channel.
pnm.channels.MemChannel.capacity = 1000000
pnm.channels.MemChannel.type = memory
# pnm.channels.MemChannel2.capacity = 10000
# pnm.channels.MemChannel2.type = memory
*Error Log file*
org.apache.flume.source.SpoolDirectorySource
FATAL: Spool Directory source SPOOL: { spoolDir:
/home/s_sdldalplhdxxxedh/pnm-poll-results }: Uncaught exception in
SpoolDirectorySource thread. Restart or reconfigure Flume to continue
processing.
java.lang.IllegalStateException: File should not roll when commit is
outstanding.
at
org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:235)
at
org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:228)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
I have tried to restart the agent, but the same error is appearing.
Regards,
Nikhil
