The spooling directory source assumes that the files in the directory your are spooling are immutable.
java.lang.**IllegalStateException: File name has been re-used with different files. Spooling assumption violated for /var/log/testhbase/hbase_1.**log.COMPLETED This message is indicative that a filename has been changed. Have you considered using some unique identifier (e.g. timestamp) when copying log files into this directory? On Sun, Jan 27, 2013 at 9:06 PM, Mike Percy <[email protected]> wrote: > bcc: [email protected] > > No version of CDH currently ships with Flume 1.3.1, so redirecting this > question to the [email protected] user list. > > Regards, > Mike > > On Sun, Jan 27, 2013 at 8:56 PM, NGuyen thi Kim Tuyen < > [email protected]> wrote: > >> I'm using Flume-Ng 1.3.1 . >> >> Vào 11:33:49 UTC+7 Thứ hai, ngày 28 tháng một năm 2013, NGuyen thi Kim >> Tuyen đã viết: >> >>> Hi , >>> >>> Please help me . >>> >>> I want to use Flume in the following case : >>> Spooling directory source --> FileChannel --> HBase sink . But I have >>> some problems with Spooling directory source : >>> >>> Here is my test flume.conf : >>> t-game-db194.sources = test-hbase >>> >>> t-game-db194.sinks = sink-hbase >>> >>> t-game-db194.channels = hbase-channel >>> >>> #source spoolDir >>> t-game-db194.sources.test-**hbase.type = spooldir >>> >>> t-game-db194.sources.test-**hbase.spoolDir =/var/log/testhbase >>> >>> t-game-db194.sources.test-**hbase.fileHeader = true >>> >>> t-game-db194.sources.test-**hbase.channels = hbase-channel >>> >>> #file Channel >>> t-game-db194.channels.hbase-**channel.type = file >>> >>> t-game-db194.channels.hbase-**channel.checkpointDir = >>> /var/log/flume-ng/checkpoint >>> >>> t-game-db194.channels.hbase-**channel.dataDir = >>> /var/log/flume-ng/filedata >>> >>> >>> #sink >>> t-game-db194.sinks.sink-hbase.**type = logger >>> >>> t-game-db194.sinks.sink-hbase.**channel = hbase-channel >>> >>> And I tested : echo "tuyen ssssssssss " >> "/var/log/testhbase/hbase_1.* >>> *log" . The first event is OK , but the next events are not work . Here >>> is flume.log >>> >>> 28 Jan 2013 13:16:47,424 INFO [lifecycleSupervisor-1-0] >>> (org.apache.flume.source.**SpoolDirectorySource.start:64) - >>> SpoolDirectorySource source starting with directory:/var/log/testhbase >>> 28 Jan 2013 13:16:47,732 INFO [pool-7-thread-1] >>> (org.apache.flume.client.avro.**SpoolingFileLineReader.**retireCurrentFile:229) >>> - Preparing to move file /var/log/testhbase/hbase_1.log to >>> /var/log/testhbase/hbase_1.**log.COMPLETED >>> 28 Jan 2013 13:16:48,436 INFO >>> [SinkRunner-PollingRunner-**DefaultSinkProcessor] >>> (org.apache.flume.sink.**LoggerSink.process:70) - Event: { >>> headers:{file=/var/log/**testhbase/hbase_1.log} body: 74 75 79 65 6E 20 >>> 73 73 73 73 73 73 73 73 73 73 tuyen ssssssssss } >>> >>> 28 Jan 2013 13:17:08,836 INFO [pool-7-thread-1] >>> (org.apache.flume.client.avro.**SpoolingFileLineReader.**retireCurrentFile:229) >>> - Preparing to move file /var/log/testhbase/hbase_1.log to >>> /var/log/testhbase/hbase_1.**log.COMPLETED >>> 28 Jan 2013 13:17:08,837 ERROR [pool-7-thread-1] >>> (org.apache.flume.source.**SpoolDirectorySource$** >>> SpoolDirectoryRunnable.run:**148) - Uncaught exception in Runnable >>> java.lang.**IllegalStateException: File name has been re-used with >>> different files. Spooling assumption violated for >>> /var/log/testhbase/hbase_1.**log.COMPLETED >>> at org.apache.flume.client.avro.**SpoolingFileLineReader.** >>> retireCurrentFile(**SpoolingFileLineReader.java:**272) >>> at org.apache.flume.client.avro.**SpoolingFileLineReader.**readLines(** >>> SpoolingFileLineReader.java:**185) >>> at org.apache.flume.source.**SpoolDirectorySource$** >>> SpoolDirectoryRunnable.run(**SpoolDirectorySource.java:135) >>> at java.util.concurrent.**Executors$RunnableAdapter.** >>> call(Executors.java:441) >>> at java.util.concurrent.**FutureTask$Sync.** >>> innerRunAndReset(FutureTask.**java:317) >>> at java.util.concurrent.**FutureTask.runAndReset(**FutureTask.java:150) >>> at java.util.concurrent.**ScheduledThreadPoolExecutor$** >>> ScheduledFutureTask.access$**101(**ScheduledThreadPoolExecutor.** >>> java:98) >>> at java.util.concurrent.**ScheduledThreadPoolExecutor$** >>> ScheduledFutureTask.**runPeriodic(**ScheduledThreadPoolExecutor.** >>> java:180) >>> at java.util.concurrent.**ScheduledThreadPoolExecutor$** >>> ScheduledFutureTask.run(**ScheduledThreadPoolExecutor.**java:204) >>> at java.util.concurrent.**ThreadPoolExecutor$Worker.** >>> runTask(ThreadPoolExecutor.**java:886) >>> at java.util.concurrent.**ThreadPoolExecutor$Worker.run(** >>> ThreadPoolExecutor.java:908) >>> at java.lang.Thread.run(Thread.**java:662) >>> 28 Jan 2013 13:17:09,340 ERROR [pool-7-thread-1] >>> (org.apache.flume.source.**SpoolDirectorySource$** >>> SpoolDirectoryRunnable.run:**148) - Uncaught exception in Runnable >>> java.io.IOException: Stream closed >>> at java.io.BufferedReader.**ensureOpen(BufferedReader.**java:97) >>> at java.io.BufferedReader.**readLine(BufferedReader.java:**292) >>> at java.io.BufferedReader.**readLine(BufferedReader.java:**362) >>> at org.apache.flume.client.avro.**SpoolingFileLineReader.**readLines(** >>> SpoolingFileLineReader.java:**180) >>> at org.apache.flume.source.**SpoolDirectorySource$** >>> SpoolDirectoryRunnable.run(**SpoolDirectorySource.java:135) >>> at java.util.concurrent.**Executors$RunnableAdapter.** >>> call(Executors.java:441) >>> at java.util.concurrent.**FutureTask$Sync.** >>> innerRunAndReset(FutureTask.**java:317) >>> at java.util.concurrent.**FutureTask.runAndReset(**FutureTask.java:150) >>> at java.util.concurrent.**ScheduledThreadPoolExecutor$** >>> ScheduledFutureTask.access$**101(**ScheduledThreadPoolExecutor.** >>> java:98) >>> at java.util.concurrent.**ScheduledThreadPoolExecutor$** >>> ScheduledFutureTask.**runPeriodic(**ScheduledThreadPoolExecutor.** >>> java:180) >>> at java.util.concurrent.**ScheduledThreadPoolExecutor$** >>> ScheduledFutureTask.run(**ScheduledThreadPoolExecutor.**java:204) >>> at java.util.concurrent.**ThreadPoolExecutor$Worker.** >>> runTask(ThreadPoolExecutor.**java:886) >>> >>> >>> Are there more documents about Flume-ng spooling source , beside >>> http://flume.apache.**org/FlumeUserGuide.html#** >>> spooling-directory-source<http://flume.apache.org/FlumeUserGuide.html#spooling-directory-source> >>> ? >>> >>> Could you please give me some advice ? >>> >>> >>> -- >> >> >> >> > >
