bcc: [email protected] No version of CDH currently ships with Flume 1.3.1, so redirecting this question to the [email protected] user list.
Regards, Mike On Sun, Jan 27, 2013 at 8:56 PM, NGuyen thi Kim Tuyen <[email protected] > wrote: > I'm using Flume-Ng 1.3.1 . > > Vào 11:33:49 UTC+7 Thứ hai, ngày 28 tháng một năm 2013, NGuyen thi Kim > Tuyen đã viết: > >> Hi , >> >> Please help me . >> >> I want to use Flume in the following case : >> Spooling directory source --> FileChannel --> HBase sink . But I have >> some problems with Spooling directory source : >> >> Here is my test flume.conf : >> t-game-db194.sources = test-hbase >> >> t-game-db194.sinks = sink-hbase >> >> t-game-db194.channels = hbase-channel >> >> #source spoolDir >> t-game-db194.sources.test-**hbase.type = spooldir >> >> t-game-db194.sources.test-**hbase.spoolDir =/var/log/testhbase >> >> t-game-db194.sources.test-**hbase.fileHeader = true >> >> t-game-db194.sources.test-**hbase.channels = hbase-channel >> >> #file Channel >> t-game-db194.channels.hbase-**channel.type = file >> >> t-game-db194.channels.hbase-**channel.checkpointDir = >> /var/log/flume-ng/checkpoint >> >> t-game-db194.channels.hbase-**channel.dataDir = >> /var/log/flume-ng/filedata >> >> >> #sink >> t-game-db194.sinks.sink-hbase.**type = logger >> >> t-game-db194.sinks.sink-hbase.**channel = hbase-channel >> >> And I tested : echo "tuyen ssssssssss " >> "/var/log/testhbase/hbase_1.**log" >> . The first event is OK , but the next events are not work . Here is >> flume.log >> >> 28 Jan 2013 13:16:47,424 INFO [lifecycleSupervisor-1-0] >> (org.apache.flume.source.**SpoolDirectorySource.start:64) - >> SpoolDirectorySource source starting with directory:/var/log/testhbase >> 28 Jan 2013 13:16:47,732 INFO [pool-7-thread-1] >> (org.apache.flume.client.avro.**SpoolingFileLineReader.**retireCurrentFile:229) >> - Preparing to move file /var/log/testhbase/hbase_1.log to >> /var/log/testhbase/hbase_1.**log.COMPLETED >> 28 Jan 2013 13:16:48,436 INFO >> [SinkRunner-PollingRunner-**DefaultSinkProcessor] >> (org.apache.flume.sink.**LoggerSink.process:70) - Event: { >> headers:{file=/var/log/**testhbase/hbase_1.log} body: 74 75 79 65 6E 20 >> 73 73 73 73 73 73 73 73 73 73 tuyen ssssssssss } >> >> 28 Jan 2013 13:17:08,836 INFO [pool-7-thread-1] >> (org.apache.flume.client.avro.**SpoolingFileLineReader.**retireCurrentFile:229) >> - Preparing to move file /var/log/testhbase/hbase_1.log to >> /var/log/testhbase/hbase_1.**log.COMPLETED >> 28 Jan 2013 13:17:08,837 ERROR [pool-7-thread-1] (org.apache.flume.source. >> **SpoolDirectorySource$**SpoolDirectoryRunnable.run:**148) - Uncaught >> exception in Runnable >> java.lang.**IllegalStateException: File name has been re-used with >> different files. Spooling assumption violated for >> /var/log/testhbase/hbase_1.**log.COMPLETED >> at org.apache.flume.client.avro.**SpoolingFileLineReader.** >> retireCurrentFile(**SpoolingFileLineReader.java:**272) >> at org.apache.flume.client.avro.**SpoolingFileLineReader.**readLines(** >> SpoolingFileLineReader.java:**185) >> at org.apache.flume.source.**SpoolDirectorySource$** >> SpoolDirectoryRunnable.run(**SpoolDirectorySource.java:135) >> at java.util.concurrent.**Executors$RunnableAdapter.** >> call(Executors.java:441) >> at java.util.concurrent.**FutureTask$Sync.**innerRunAndReset(FutureTask. >> **java:317) >> at java.util.concurrent.**FutureTask.runAndReset(**FutureTask.java:150) >> at java.util.concurrent.**ScheduledThreadPoolExecutor$** >> ScheduledFutureTask.access$**101(**ScheduledThreadPoolExecutor.**java:98) >> at java.util.concurrent.**ScheduledThreadPoolExecutor$** >> ScheduledFutureTask.**runPeriodic(**ScheduledThreadPoolExecutor.** >> java:180) >> at java.util.concurrent.**ScheduledThreadPoolExecutor$** >> ScheduledFutureTask.run(**ScheduledThreadPoolExecutor.**java:204) >> at java.util.concurrent.**ThreadPoolExecutor$Worker.** >> runTask(ThreadPoolExecutor.**java:886) >> at java.util.concurrent.**ThreadPoolExecutor$Worker.run(** >> ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.**java:662) >> 28 Jan 2013 13:17:09,340 ERROR [pool-7-thread-1] (org.apache.flume.source. >> **SpoolDirectorySource$**SpoolDirectoryRunnable.run:**148) - Uncaught >> exception in Runnable >> java.io.IOException: Stream closed >> at java.io.BufferedReader.**ensureOpen(BufferedReader.**java:97) >> at java.io.BufferedReader.**readLine(BufferedReader.java:**292) >> at java.io.BufferedReader.**readLine(BufferedReader.java:**362) >> at org.apache.flume.client.avro.**SpoolingFileLineReader.**readLines(** >> SpoolingFileLineReader.java:**180) >> at org.apache.flume.source.**SpoolDirectorySource$** >> SpoolDirectoryRunnable.run(**SpoolDirectorySource.java:135) >> at java.util.concurrent.**Executors$RunnableAdapter.** >> call(Executors.java:441) >> at java.util.concurrent.**FutureTask$Sync.**innerRunAndReset(FutureTask. >> **java:317) >> at java.util.concurrent.**FutureTask.runAndReset(**FutureTask.java:150) >> at java.util.concurrent.**ScheduledThreadPoolExecutor$** >> ScheduledFutureTask.access$**101(**ScheduledThreadPoolExecutor.**java:98) >> at java.util.concurrent.**ScheduledThreadPoolExecutor$** >> ScheduledFutureTask.**runPeriodic(**ScheduledThreadPoolExecutor.** >> java:180) >> at java.util.concurrent.**ScheduledThreadPoolExecutor$** >> ScheduledFutureTask.run(**ScheduledThreadPoolExecutor.**java:204) >> at java.util.concurrent.**ThreadPoolExecutor$Worker.** >> runTask(ThreadPoolExecutor.**java:886) >> >> >> Are there more documents about Flume-ng spooling source , beside >> http://flume.apache.**org/FlumeUserGuide.html#**spooling-directory-source<http://flume.apache.org/FlumeUserGuide.html#spooling-directory-source> >> ? >> >> Could you please give me some advice ? >> >> >> -- > > > >
