---------- Forwarded message ----------
From: NGuyen thi Kim Tuyen <[email protected]>
Date: Mon, Jan 28, 2013 at 2:14 PM
Subject: Flume-NG 1.3.1 : Spooling dir source : java.io.IOException: Stream
closed
To: [email protected]
Hi ,
Please help me .
I want to use Flume in the following case :
Spooling directory source --> FileChannel --> HBase sink . But I have some
problems with Spooling directory source :
Here is my test flume.conf :
t-game-db194.sources = test-hbase
t-game-db194.sinks = sink-hbase
t-game-db194.channels = hbase-channel
#source spoolDir
t-game-db194.sources.test-hbase.type = spooldir
t-game-db194.sources.test-hbase.spoolDir =/var/log/testhbase
t-game-db194.sources.test-hbase.fileHeader = true
t-game-db194.sources.test-hbase.channels = hbase-channel
#file Channel
t-game-db194.channels.hbase-channel.type = file
t-game-db194.channels.hbase-channel.checkpointDir =
/var/log/flume-ng/checkpoint
t-game-db194.channels.hbase-channel.dataDir = /var/log/flume-ng/filedata
#sink
t-game-db194.sinks.sink-hbase.type = logger
t-game-db194.sinks.sink-hbase.channel = hbase-channel
And I tested : echo "tuyen ssssssssss " >> "/var/log/testhbase/hbase_1.log"
. The first event is OK , but the next events are not work . Here is
flume.log
28 Jan 2013 13:16:47,424 INFO [lifecycleSupervisor-1-0]
(org.apache.flume.source.SpoolDirectorySource.start:64) -
SpoolDirectorySource source starting with directory:/var/log/testhbase
28 Jan 2013 13:16:47,732 INFO [pool-7-thread-1]
(org.apache.flume.client.avro.SpoolingFileLineReader.retireCurrentFile:229)
- Preparing to move file /var/log/testhbase/hbase_1.log to
/var/log/testhbase/hbase_1.log.COMPLETED
28 Jan 2013 13:16:48,436 INFO
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.flume.sink.LoggerSink.process:70) - Event: {
headers:{file=/var/log/testhbase/hbase_1.log} body: 74 75 79 65 6E 20 73 73
73 73 73 73 73 73 73 73 tuyen ssssssssss }
28 Jan 2013 13:17:08,836 INFO [pool-7-thread-1]
(org.apache.flume.client.avro.SpoolingFileLineReader.retireCurrentFile:229)
- Preparing to move file /var/log/testhbase/hbase_1.log to
/var/log/testhbase/hbase_1.log.COMPLETED
28 Jan 2013 13:17:08,837 ERROR [pool-7-thread-1]
(org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run:148)
- Uncaught exception in Runnable
java.lang.IllegalStateException: File name has been re-used with different
files. Spooling assumption violated for
/var/log/testhbase/hbase_1.log.COMPLETED
at
org.apache.flume.client.avro.SpoolingFileLineReader.retireCurrentFile(SpoolingFileLineReader.java:272)
at
org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:185)
at
org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:135)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
28 Jan 2013 13:17:09,340 ERROR [pool-7-thread-1]
(org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run:148)
- Uncaught exception in Runnable
java.io.IOException: Stream closed
at java.io.BufferedReader.ensureOpen(BufferedReader.java:97)
at java.io.BufferedReader.readLine(BufferedReader.java:292)
at java.io.BufferedReader.readLine(BufferedReader.java:362)
at
org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:180)
at
org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:135)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
Are there more documents about Flume-ng spooling source , beside
http://flume.apache.org/FlumeUserGuide.html#spooling-directory-source ?
Could you please give me some advice ?
--
Nguyễn Thị Kim Tuyên
Computer Science Engineering
HCMC University Of Technology.
--
Nguyễn Thị Kim Tuyên
Computer Science Engineering
HCMC University Of Technology.
