Thank for your reply . If spooling source only works on "done" , immutable files , it's not suitable my problem . I think I 'll use exec tail command instead . But warning from http://flume.apache.org/FlumeUserGuide.html#exec-source : The problem with ExecSource and other asynchronous sources is that the source can not guarantee that if there is a failure to put the event into the Channel the client knows about it. ..... For stronger reliability guarantees, consider the Spooling Directory Source or direct integration with Flume via the SDK.
I'm still considering between ExecSource and Log4jAppender . http://www.slideshare.net/sematext/search-analytics-with-flume-and-hbase Could you share me your opinion ? On Mon, Jan 28, 2013 at 2:29 PM, Mike Percy <[email protected]> wrote: > Hi Nguyễn, > The spooling source only works on "done", immutable files. So they have to > be atomically moved and they cannot be modified after being placed into the > spooling directory. > > Regards, > Mike > > > On Sun, Jan 27, 2013 at 11:14 PM, NGuyen thi Kim Tuyen < > [email protected]> wrote: > >> Hi , >> >> Please help me . >> >> I want to use Flume in the following case : >> Spooling directory source --> FileChannel --> HBase sink . But I have >> some problems with Spooling directory source : >> >> Here is my test flume.conf : >> t-game-db194.sources = test-hbase >> >> t-game-db194.sinks = sink-hbase >> >> t-game-db194.channels = hbase-channel >> >> #source spoolDir >> t-game-db194.sources.test-hbase.type = spooldir >> >> t-game-db194.sources.test-hbase.spoolDir =/var/log/testhbase >> >> t-game-db194.sources.test-hbase.fileHeader = true >> >> t-game-db194.sources.test-hbase.channels = hbase-channel >> >> #file Channel >> t-game-db194.channels.hbase-channel.type = file >> >> t-game-db194.channels.hbase-channel.checkpointDir = >> /var/log/flume-ng/checkpoint >> >> t-game-db194.channels.hbase-channel.dataDir = /var/log/flume-ng/filedata >> >> >> #sink >> t-game-db194.sinks.sink-hbase.type = logger >> >> t-game-db194.sinks.sink-hbase.channel = hbase-channel >> >> And I tested : echo "tuyen ssssssssss " >> >> "/var/log/testhbase/hbase_1.log" . The first event is OK , but the next >> events are not work . Here is flume.log >> >> 28 Jan 2013 13:16:47,424 INFO [lifecycleSupervisor-1-0] >> (org.apache.flume.source.SpoolDirectorySource.start:64) - >> SpoolDirectorySource source starting with directory:/var/log/testhbase >> 28 Jan 2013 13:16:47,732 INFO [pool-7-thread-1] >> (org.apache.flume.client.avro.SpoolingFileLineReader.retireCurrentFile:229) >> - Preparing to move file /var/log/testhbase/hbase_1.log to >> /var/log/testhbase/hbase_1.log.COMPLETED >> 28 Jan 2013 13:16:48,436 INFO >> [SinkRunner-PollingRunner-DefaultSinkProcessor] >> (org.apache.flume.sink.LoggerSink.process:70) - Event: { >> headers:{file=/var/log/testhbase/hbase_1.log} body: 74 75 79 65 6E 20 73 73 >> 73 73 73 73 73 73 73 73 tuyen ssssssssss } >> >> 28 Jan 2013 13:17:08,836 INFO [pool-7-thread-1] >> (org.apache.flume.client.avro.SpoolingFileLineReader.retireCurrentFile:229) >> - Preparing to move file /var/log/testhbase/hbase_1.log to >> /var/log/testhbase/hbase_1.log.COMPLETED >> 28 Jan 2013 13:17:08,837 ERROR [pool-7-thread-1] >> (org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run:148) >> - Uncaught exception in Runnable >> java.lang.IllegalStateException: File name has been re-used with >> different files. Spooling assumption violated for >> /var/log/testhbase/hbase_1.log.COMPLETED >> at >> org.apache.flume.client.avro.SpoolingFileLineReader.retireCurrentFile(SpoolingFileLineReader.java:272) >> at >> org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:185) >> at >> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:135) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) >> at >> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) >> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:662) >> 28 Jan 2013 13:17:09,340 ERROR [pool-7-thread-1] >> (org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run:148) >> - Uncaught exception in Runnable >> java.io.IOException: Stream closed >> at java.io.BufferedReader.ensureOpen(BufferedReader.java:97) >> at java.io.BufferedReader.readLine(BufferedReader.java:292) >> at java.io.BufferedReader.readLine(BufferedReader.java:362) >> at >> org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:180) >> at >> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:135) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) >> at >> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) >> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> >> >> Are there more documents about Flume-ng spooling source , beside >> http://flume.apache.org/FlumeUserGuide.html#spooling-directory-source ? >> >> Could you please give me some advice ? >> >> -- >> Nguyễn Thị Kim Tuyên >> Computer Science Engineering >> HCMC University Of Technology. > > > -- Nguyễn Thị Kim Tuyên Computer Science Engineering HCMC University Of Technology.
