Yes Flume-1.3.0 supports a source called Spool Directory Source which does this.
On Thu, Nov 29, 2012 at 10:59 AM, kashif khan <[email protected]>wrote: > Thanks Alexander > > Yeah you right, I was not read the document properly, sorry for that. once > read the documents, I have done and now uploading the data. > > I have just question about Flume 1.3. (which to be release very soon) Is > it support to automatically detect the file in folder and upload into HDFS? > > Many thanks > > Best regards > > > On Thu, Nov 29, 2012 at 3:39 PM, Alexander Alten-Lorenz < > [email protected]> wrote: > >> Hi, >> >> agent.channels.memoryChannel.transactionCapacity=1000 >> >> Is wrong. You cant have equal or more transaction capacity in a channel >> as configured capacity. Use the opposite, when you want to use it. >> from our Guide: >> >> capacity NUM The max number of events stored in the >> channel >> transactionCapacity NUM The max number of events stored in the >> channel per transaction >> >> Try this: >> >> agent.channels.memoryChannel.capacity = 1000 >> agent.channels.memoryChannel.transactionCapacity=10 >> >> cheers >> - Alex >> >> >> On Nov 29, 2012, at 1:03 PM, kashif khan <[email protected]> wrote: >> >> > Hi, >> > >> > I am just struggling to learn the flume and doing some testing. I am >> > running two agents (agent, agent1). The agent used to upload the log >> data >> > into hdfs and agent1 used as logger. The configuration of two agents as: >> > >> > agent.sources = tail >> > agent.channels = memoryChannel >> > agent.sinks = hdfs-clusterSink >> > >> > agent.sources.tail.type = exec >> > agent.sources.tail.command = tail -f /var/log/flume-ng/flume.log >> > agent.sources.tail.channels = memoryChannel >> > >> > agent.sinks.hdfs-clusterSink. >> > channel = memoryChannel >> > agent.sinks.hdfs-clusterSink.type = hdfs >> > agent.sinks.hdfs-clusterSink.hdfs.path = hdfs:// >> > hadoop1.example.com/user/root/Test/ >> > >> > >> > agent.channels.memoryChannel.type = memory >> > agent.channels.memoryChannel.transactionCapacity=1000 >> > agent.channels.memoryChannel.capacity = 100 >> > >> > >> > >> > >> > agent1.sources = source1 >> > agent1.sinks = sink1 >> > agent1.channels = channel1 >> > >> > # Describe/configure source1 >> > agent1.sources.source1.type = netcat >> > agent1.sources.source1.bind = localhost >> > agent1.sources.source1.port = 44444 >> > >> > # Describe sink1 >> > agent1.sinks.sink1.type = logger >> > >> > # Use a channel which buffers events in memory >> > agent1.channels.channel1.type = memory >> > agent1.channels.channel1.capacity = 1000 >> > agent1.channels.channel1.transactionCapactiy = 100 >> > >> > # Bind the source and sink to the channel >> > agent1.sources.source1.channels = channel1 >> > agent1.sinks.sink1.channel = channel1 >> > >> > >> > I dont know why it does not upload the log file into hdfs. where I am >> doing >> > mistake . If anyone who have solution please let me know. >> > >> > >> > The log file as: >> > >> > >> > 29 Nov 2012 11:49:13,046 INFO [main] >> > (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting >> > lifecycle supervisor 1 >> > 29 Nov 2012 11:49:13,050 INFO [main] >> > (org.apache.flume.node.FlumeNode.start:54) - Flume node starting - >> agent >> > 29 Nov 2012 11:49:13,051 INFO [lifecycleSupervisor-1-0] >> > (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203) >> - >> > Node manager starting >> > 29 Nov 2012 11:49:13,053 INFO [lifecycleSupervisor-1-0] >> > (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting >> > lifecycle supervisor 10 >> > 29 Nov 2012 11:49:13,052 INFO [lifecycleSupervisor-1-2] >> > (org.apache.flume.conf.file.AbstractFileConfigurationProvider.start:67) >> - >> > Configuration provider starting >> > 29 Nov 2012 11:49:13,054 INFO [conf-file-poller-0] >> > >> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:195) >> > - Reloading configuration file:/etc/flume-ng/conf/flume.conf >> > 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0] >> > >> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912) >> > - Added sinks: hdfs-clusterSink Agent: agent >> > 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0] >> > >> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998) >> > - Processing:hdfs-clusterSink >> > 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0] >> > >> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998) >> > - Processing:sink1 >> > 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0] >> > >> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998) >> > - Processing:hdfs-clusterSink >> > 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0] >> > >> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998) >> > - Processing:sink1 >> > 29 Nov 2012 11:49:13,058 INFO [conf-file-poller-0] >> > >> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912) >> > - Added sinks: sink1 Agent: agent1 >> > 29 Nov 2012 11:49:13,058 INFO [conf-file-poller-0] >> > >> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998) >> > - Processing:hdfs-clusterSink >> > 29 Nov 2012 11:49:13,087 INFO [conf-file-poller-0] >> > (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:122) - >> > Post-validation flume configuration contains configuration for agents: >> > [agent, agent1] >> > 29 Nov 2012 11:49:13,088 INFO [conf-file-poller-0] >> > >> (org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:249) >> > - Creating channels >> > 29 Nov 2012 11:49:13,090 ERROR [conf-file-poller-0] >> > >> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:204) >> > - Failed to load configuration data. Exception follows. >> > java.lang.IllegalStateException >> > at >> > com.google.common.base.Preconditions.checkState(Preconditions.java:129) >> > at >> > org.apache.flume.channel.MemoryChannel.configure(MemoryChannel.java:189) >> > at >> org.apache.flume.conf.Configurables.configure(Configurables.java:41) >> > at >> > >> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:271) >> > at >> > >> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:221) >> > at >> > >> org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123) >> > at >> > >> org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38) >> > at >> > >> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202) >> > at >> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) >> > at >> > >> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) >> > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) >> > at >> > >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) >> > at >> > >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) >> > at >> > >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) >> > at >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> > at >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> > at java.lang.Thread.run(Thread.java:662) >> > 29 Nov 2012 11:49:22,157 INFO [main] >> > (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting >> > lifecycle supervisor 1 >> > 29 Nov 2012 11:49:22,161 INFO [main] >> > (org.apache.flume.node.FlumeNode.start:54) - Flume node starting - >> agent1 >> > 29 Nov 2012 11:49:22,162 INFO [lifecycleSupervisor-1-0] >> > (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203) >> - >> > Node manager starting >> > 29 Nov 2012 11:49:22,164 INFO [lifecycleSupervisor-1-0] >> > (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting >> > lifecycle supervisor 10 >> > 29 Nov 2012 11:49:22,162 INFO [lifecycleSupervisor-1-1] >> > (org.apache.flume.conf.file.AbstractFileConfigurationProvider.start:67) >> - >> > Configuration provider starting >> > 29 Nov 2012 11:49:22,165 INFO [conf-file-poller-0] >> > >> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:195) >> > - Reloading configuration file:/etc/flume-ng/conf/flume.conf >> > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0] >> > >> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912) >> > - Added sinks: hdfs-clusterSink Agent: agent >> > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0] >> > >> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998) >> > - Processing:hdfs-clusterSink >> > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0] >> > >> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998) >> > - Processing:sink1 >> > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0] >> > >> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998) >> > - Processing:hdfs-clusterSink >> > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0] >> > >> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998) >> > - Processing:sink1 >> > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0] >> > >> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912) >> > - Added sinks: sink1 Agent: agent1 >> > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0] >> > >> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998) >> > - Processing:hdfs-clusterSink >> > 29 Nov 2012 11:49:22,195 INFO [conf-file-poller-0] >> > (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:122) - >> > Post-validation flume configuration contains configuration for agents: >> > [agent, agent1] >> > 29 Nov 2012 11:49:22,195 INFO [conf-file-poller-0] >> > >> (org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:249) >> > - Creating channels >> > 29 Nov 2012 11:49:22,221 INFO [conf-file-poller-0] >> > (org.apache.flume.instrumentation.MonitoredCounterGroup.<init>:68) - >> > Monitoried counter group for type: CHANNEL, name: channel1, registered >> > successfully. >> > 29 Nov 2012 11:49:22,222 INFO [conf-file-poller-0] >> > >> (org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:273) >> > - created channel channel1 >> > 29 Nov 2012 11:49:22,233 INFO [conf-file-poller-0] >> > (org.apache.flume.sink.DefaultSinkFactory.create:70) - Creating >> instance >> > of sink: sink1, type: logger >> > 29 Nov 2012 11:49:22,237 INFO [conf-file-poller-0] >> > >> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:106) >> > - Starting new configuration:{ >> > sourceRunners:{source1=EventDrivenSourceRunner: { >> > source:org.apache.flume.source.NetcatSource{name:source1,state:IDLE} }} >> > sinkRunners:{sink1=SinkRunner: { >> > policy:org.apache.flume.sink.DefaultSinkProcessor@4430d82dcounterGroup:{ >> > name:null counters:{} } }} >> > channels:{channel1=org.apache.flume.channel.MemoryChannel{name: >> channel1}} } >> > 29 Nov 2012 11:49:22,238 INFO [conf-file-poller-0] >> > >> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:113) >> > - Starting Channel channel1 >> > 29 Nov 2012 11:49:22,239 INFO [lifecycleSupervisor-1-1] >> > (org.apache.flume.instrumentation.MonitoredCounterGroup.start:82) - >> > Component type: CHANNEL, name: channel1 started >> > 29 Nov 2012 11:49:22,239 INFO [conf-file-poller-0] >> > >> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:128) >> > - Waiting for channel: channel1 to start. Sleeping for 500 ms >> > 29 Nov 2012 11:49:22,739 INFO [conf-file-poller-0] >> > >> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:141) >> > - Starting Sink sink1 >> > 29 Nov 2012 11:49:22,740 INFO [conf-file-poller-0] >> > >> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:152) >> > - Starting Source source1 >> > 29 Nov 2012 11:49:22,741 INFO [lifecycleSupervisor-1-3] >> > (org.apache.flume.source.NetcatSource.start:147) - Source starting >> > 29 Nov 2012 11:49:22,752 INFO [lifecycleSupervisor-1-3] >> > (org.apache.flume.source.NetcatSource.start:161) - Created >> > serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444] >> > 29 Nov 2012 11:49:33,327 INFO >> > [SinkRunner-PollingRunner-DefaultSinkProcessor] >> > (org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{} >> body: >> > 68 65 6C 6C 6F 0D hello. } >> > 29 Nov 2012 11:49:42,328 INFO >> > [SinkRunner-PollingRunner-DefaultSinkProcessor] >> > (org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{} >> body: >> > 68 6F 77 20 69 73 20 67 6F 69 6E 67 0D how is going. } >> > 29 Nov 2012 11:49:46,359 INFO >> > [SinkRunner-PollingRunner-DefaultSinkProcessor] >> > (org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{} >> body: >> > 65 76 65 72 79 20 74 68 69 6E 67 73 20 61 72 65 every things are } >> > >> > >> > Many thanks >> > >> > Best regards, >> >> -- >> Alexander Alten-Lorenz >> http://mapredit.blogspot.com >> German Hadoop LinkedIn Group: http://goo.gl/N8pCF >> >> > -- Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
