Alex is right and our error message there needs much improvement. I have created a JIRA here https://issues.apache.org/jira/browse/FLUME-1744
On Thu, Nov 29, 2012 at 9:39 AM, Alexander Alten-Lorenz <[email protected] > wrote: > Hi, > > agent.channels.memoryChannel.transactionCapacity=1000 > > Is wrong. You cant have equal or more transaction capacity in a channel as > configured capacity. Use the opposite, when you want to use it. > from our Guide: > > capacity NUM The max number of events stored in the > channel > transactionCapacity NUM The max number of events stored in the > channel per transaction > > Try this: > > agent.channels.memoryChannel.capacity = 1000 > agent.channels.memoryChannel.transactionCapacity=10 > > cheers > - Alex > > > On Nov 29, 2012, at 1:03 PM, kashif khan <[email protected]> wrote: > > > Hi, > > > > I am just struggling to learn the flume and doing some testing. I am > > running two agents (agent, agent1). The agent used to upload the log data > > into hdfs and agent1 used as logger. The configuration of two agents as: > > > > agent.sources = tail > > agent.channels = memoryChannel > > agent.sinks = hdfs-clusterSink > > > > agent.sources.tail.type = exec > > agent.sources.tail.command = tail -f /var/log/flume-ng/flume.log > > agent.sources.tail.channels = memoryChannel > > > > agent.sinks.hdfs-clusterSink. > > channel = memoryChannel > > agent.sinks.hdfs-clusterSink.type = hdfs > > agent.sinks.hdfs-clusterSink.hdfs.path = hdfs:// > > hadoop1.example.com/user/root/Test/ > > > > > > agent.channels.memoryChannel.type = memory > > agent.channels.memoryChannel.transactionCapacity=1000 > > agent.channels.memoryChannel.capacity = 100 > > > > > > > > > > agent1.sources = source1 > > agent1.sinks = sink1 > > agent1.channels = channel1 > > > > # Describe/configure source1 > > agent1.sources.source1.type = netcat > > agent1.sources.source1.bind = localhost > > agent1.sources.source1.port = 44444 > > > > # Describe sink1 > > agent1.sinks.sink1.type = logger > > > > # Use a channel which buffers events in memory > > agent1.channels.channel1.type = memory > > agent1.channels.channel1.capacity = 1000 > > agent1.channels.channel1.transactionCapactiy = 100 > > > > # Bind the source and sink to the channel > > agent1.sources.source1.channels = channel1 > > agent1.sinks.sink1.channel = channel1 > > > > > > I dont know why it does not upload the log file into hdfs. where I am > doing > > mistake . If anyone who have solution please let me know. > > > > > > The log file as: > > > > > > 29 Nov 2012 11:49:13,046 INFO [main] > > (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting > > lifecycle supervisor 1 > > 29 Nov 2012 11:49:13,050 INFO [main] > > (org.apache.flume.node.FlumeNode.start:54) - Flume node starting - agent > > 29 Nov 2012 11:49:13,051 INFO [lifecycleSupervisor-1-0] > > (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203) > - > > Node manager starting > > 29 Nov 2012 11:49:13,053 INFO [lifecycleSupervisor-1-0] > > (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting > > lifecycle supervisor 10 > > 29 Nov 2012 11:49:13,052 INFO [lifecycleSupervisor-1-2] > > (org.apache.flume.conf.file.AbstractFileConfigurationProvider.start:67) > - > > Configuration provider starting > > 29 Nov 2012 11:49:13,054 INFO [conf-file-poller-0] > > > (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:195) > > - Reloading configuration file:/etc/flume-ng/conf/flume.conf > > 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0] > > > (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912) > > - Added sinks: hdfs-clusterSink Agent: agent > > 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0] > > > (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998) > > - Processing:hdfs-clusterSink > > 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0] > > > (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998) > > - Processing:sink1 > > 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0] > > > (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998) > > - Processing:hdfs-clusterSink > > 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0] > > > (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998) > > - Processing:sink1 > > 29 Nov 2012 11:49:13,058 INFO [conf-file-poller-0] > > > (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912) > > - Added sinks: sink1 Agent: agent1 > > 29 Nov 2012 11:49:13,058 INFO [conf-file-poller-0] > > > (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998) > > - Processing:hdfs-clusterSink > > 29 Nov 2012 11:49:13,087 INFO [conf-file-poller-0] > > (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:122) - > > Post-validation flume configuration contains configuration for agents: > > [agent, agent1] > > 29 Nov 2012 11:49:13,088 INFO [conf-file-poller-0] > > > (org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:249) > > - Creating channels > > 29 Nov 2012 11:49:13,090 ERROR [conf-file-poller-0] > > > (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:204) > > - Failed to load configuration data. Exception follows. > > java.lang.IllegalStateException > > at > > com.google.common.base.Preconditions.checkState(Preconditions.java:129) > > at > > org.apache.flume.channel.MemoryChannel.configure(MemoryChannel.java:189) > > at > org.apache.flume.conf.Configurables.configure(Configurables.java:41) > > at > > > org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:271) > > at > > > org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:221) > > at > > > org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123) > > at > > > org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38) > > at > > > org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202) > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > > at > > > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > 29 Nov 2012 11:49:22,157 INFO [main] > > (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting > > lifecycle supervisor 1 > > 29 Nov 2012 11:49:22,161 INFO [main] > > (org.apache.flume.node.FlumeNode.start:54) - Flume node starting - > agent1 > > 29 Nov 2012 11:49:22,162 INFO [lifecycleSupervisor-1-0] > > (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203) > - > > Node manager starting > > 29 Nov 2012 11:49:22,164 INFO [lifecycleSupervisor-1-0] > > (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting > > lifecycle supervisor 10 > > 29 Nov 2012 11:49:22,162 INFO [lifecycleSupervisor-1-1] > > (org.apache.flume.conf.file.AbstractFileConfigurationProvider.start:67) > - > > Configuration provider starting > > 29 Nov 2012 11:49:22,165 INFO [conf-file-poller-0] > > > (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:195) > > - Reloading configuration file:/etc/flume-ng/conf/flume.conf > > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0] > > > (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912) > > - Added sinks: hdfs-clusterSink Agent: agent > > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0] > > > (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998) > > - Processing:hdfs-clusterSink > > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0] > > > (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998) > > - Processing:sink1 > > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0] > > > (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998) > > - Processing:hdfs-clusterSink > > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0] > > > (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998) > > - Processing:sink1 > > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0] > > > (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912) > > - Added sinks: sink1 Agent: agent1 > > 29 Nov 2012 11:49:22,168 INFO [conf-file-poller-0] > > > (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998) > > - Processing:hdfs-clusterSink > > 29 Nov 2012 11:49:22,195 INFO [conf-file-poller-0] > > (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:122) - > > Post-validation flume configuration contains configuration for agents: > > [agent, agent1] > > 29 Nov 2012 11:49:22,195 INFO [conf-file-poller-0] > > > (org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:249) > > - Creating channels > > 29 Nov 2012 11:49:22,221 INFO [conf-file-poller-0] > > (org.apache.flume.instrumentation.MonitoredCounterGroup.<init>:68) - > > Monitoried counter group for type: CHANNEL, name: channel1, registered > > successfully. > > 29 Nov 2012 11:49:22,222 INFO [conf-file-poller-0] > > > (org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:273) > > - created channel channel1 > > 29 Nov 2012 11:49:22,233 INFO [conf-file-poller-0] > > (org.apache.flume.sink.DefaultSinkFactory.create:70) - Creating instance > > of sink: sink1, type: logger > > 29 Nov 2012 11:49:22,237 INFO [conf-file-poller-0] > > > (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:106) > > - Starting new configuration:{ > > sourceRunners:{source1=EventDrivenSourceRunner: { > > source:org.apache.flume.source.NetcatSource{name:source1,state:IDLE} }} > > sinkRunners:{sink1=SinkRunner: { > > policy:org.apache.flume.sink.DefaultSinkProcessor@4430d82dcounterGroup:{ > > name:null counters:{} } }} > > channels:{channel1=org.apache.flume.channel.MemoryChannel{name: > channel1}} } > > 29 Nov 2012 11:49:22,238 INFO [conf-file-poller-0] > > > (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:113) > > - Starting Channel channel1 > > 29 Nov 2012 11:49:22,239 INFO [lifecycleSupervisor-1-1] > > (org.apache.flume.instrumentation.MonitoredCounterGroup.start:82) - > > Component type: CHANNEL, name: channel1 started > > 29 Nov 2012 11:49:22,239 INFO [conf-file-poller-0] > > > (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:128) > > - Waiting for channel: channel1 to start. Sleeping for 500 ms > > 29 Nov 2012 11:49:22,739 INFO [conf-file-poller-0] > > > (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:141) > > - Starting Sink sink1 > > 29 Nov 2012 11:49:22,740 INFO [conf-file-poller-0] > > > (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:152) > > - Starting Source source1 > > 29 Nov 2012 11:49:22,741 INFO [lifecycleSupervisor-1-3] > > (org.apache.flume.source.NetcatSource.start:147) - Source starting > > 29 Nov 2012 11:49:22,752 INFO [lifecycleSupervisor-1-3] > > (org.apache.flume.source.NetcatSource.start:161) - Created > > serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444] > > 29 Nov 2012 11:49:33,327 INFO > > [SinkRunner-PollingRunner-DefaultSinkProcessor] > > (org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{} > body: > > 68 65 6C 6C 6F 0D hello. } > > 29 Nov 2012 11:49:42,328 INFO > > [SinkRunner-PollingRunner-DefaultSinkProcessor] > > (org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{} > body: > > 68 6F 77 20 69 73 20 67 6F 69 6E 67 0D how is going. } > > 29 Nov 2012 11:49:46,359 INFO > > [SinkRunner-PollingRunner-DefaultSinkProcessor] > > (org.apache.flume.sink.LoggerSink.process:70) - Event: { headers:{} > body: > > 65 76 65 72 79 20 74 68 69 6E 67 73 20 61 72 65 every things are } > > > > > > Many thanks > > > > Best regards, > > -- > Alexander Alten-Lorenz > http://mapredit.blogspot.com > German Hadoop LinkedIn Group: http://goo.gl/N8pCF > > -- Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
