Re: flume-ng agent startup problem

[Jagadish Bihani]

Hi Thanks all for the inputs. After the initial problem I was able to start flume except in one scenario in which I use HDFS as sink. I have a production machine with hadoop-0.20.1 installed. I have installed latest flume 1.2.0. It works fine for all the configurations (at least which I tried) except when HDFS sink is used. Test: ---------  I used both netcat listener as the source of the agent and HDFS is sink. Then I start the agent using the command "bin/flume-ng agent -n agent1 -c conf -f conf/flume_hdfs.conf --classpath /MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop-0.20.1-core.jar -Dflume.root.logger=DEBUG,console"  with DEBUG logging mode enabled. I don't get any error/exception. I use "/usr/sbin/lsof -i:<port_no>" command to check whether the source is actually bound to that port and it doesn't return any port. But when I use file sink instead of HDFS sink and run lsof it correctly shows me the port on which it is listening. Thus when HDFS sink is used even source part of agent doesn't work and it doesn't give any exception. And nothing is written to HDFS sink. P.S. I have checked the user,permission details of HDFS. They are fine. I have run flume on my other machines with different version of hadoop (0.23 & 1.0). It has run HDFS sink properly there. Does flume support hadoop-0.20.1 or there is something I am missing??? This is my Configuration: ----------------------------------------- agent1.sources = sequencer agent1.sinks  =hdfsSink fileSink agent1.sinks  =fileSink agent1.channels =memoryChannel fileChannel agent1.sources.sequencer.type=seq agent1.sinks.hdfsSink.type=hdfs agent1.sinks.hdfsSink.hdfs.path=hdfs://MLNameNode2001:54310/flume agent1.sinks.fileSink.type=file_roll agent1.sinks.fileSink.sink.directory=/home/hadoop/flume/output agent1.channels.memoryChannel.type=memory agent1.channels.memoryChannel.capacity=10000 agent1.channels.memoryChannel.transactionCapacity=100 agent1.channels.fileChannel.type=file agent1.channels.fileChannel.checkpointDir=/home/hadoop/flume/channel/checkpointDir agent1.channels.fileChannel.dataDirs=/home/hadoop/flume/channel/dataDir agent1.sources.sequencer.channels = fileChannel agent1.sinks.hdfsSink.channel = fileChannel This is the log which I get: ---------------------------------------------------------- bin/flume-ng agent -n agent1 -c conf -f conf/flume_hdfs.conf --classpath /MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop -0.20.1-core.jar -Dflume.root.logger=DEBUG,console + exec /usr/java/jdk1.6.0_12/bin/java -Xmx20m -Dflume.root.logger=DEBUG,console -cp '/home/hadoop/flume/apache-flume-1.2.0/conf:/home/hadoop/flume/apache-flume-1.2.0/li b/*:/MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop-0.20.1-core.jar' -Djava.library.path= org.apache.flume.node.Application -n agent1 -f conf/flume_hdfs.conf 2012-08-10 10:56:50,604 (main) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 1 2012-08-10 10:56:50,607 (main) [INFO - org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node starting - agent1 2012-08-10 10:56:50,611 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)]  Configuration provider starting 2012-08-10 10:56:50,611 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)] Node ma nager starting 2012-08-10 10:56:50,613 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervis or 9 2012-08-10 10:56:50,614 (lifecycleSupervisor-1-0) [DEBUG - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:191)] Node manager started 2012-08-10 10:56:50,614 (lifecycleSupervisor-1-2) [DEBUG - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:86) ] Configuration provider started 2012-08-10 10:56:50,614 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProv ider.java:188)] Checking file:conf/flume_hdfs.conf for changes 2012-08-10 10:56:50,614 (conf-file-poller-0) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvi der.java:195)] Reloading configuration file:conf/flume_hdfs.conf 2012-08-10 10:56:50,618 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:fi leSink 2012-08-10 10:56:50,619 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:992)] Created cont ext for fileSink: sink.directory 2012-08-10 10:56:50,619 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:hd fsSink 2012-08-10 10:56:50,620 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:992)] Created cont ext for hdfsSink: type 2012-08-10 10:56:50,620 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:hd fsSink 2012-08-10 10:56:50,620 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:hd fsSink 2012-08-10 10:56:50,614 (lifecycleSupervisor-1-0) [DEBUG - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:191)] Node m anager started 2012-08-10 10:56:50,614 (lifecycleSupervisor-1-2) [DEBUG - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:86) ] Configuration provider started 2012-08-10 10:56:50,614 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProv ider.java:188)] Checking file:conf/flume_hdfs.conf for changes 2012-08-10 10:56:50,614 (conf-file-poller-0) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvi der.java:195)] Reloading configuration file:conf/flume_hdfs.conf 2012-08-10 10:56:50,618 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:fi leSink 2012-08-10 10:56:50,619 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:992)] Created cont ext for fileSink: sink.directory 2012-08-10 10:56:50,619 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:hd fsSink 2012-08-10 10:56:50,620 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:992)] Created cont ext for hdfsSink: type 2012-08-10 10:56:50,620 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:hd fsSink 2012-08-10 10:56:50,620 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:hd fsSink 2012-08-10 10:56:50,620 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:fileSink 2012-08-10 10:56:50,622 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:902)] Added sinks: fileSink Agent: agent1 2012-08-10 10:56:50,622 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:295)] Starting validation of configuration for agent: agent1, initial-configuration: AgentConfiguration[agent1] SOURCES: {sequencer={ parameters:{channels=fileChannel, type=seq} }} CHANNELS: {memoryChannel={ parameters:{transactionCapacity=100, capacity=10000, type=memory} }, fileChannel={ parameters:{checkpointDir=/home/hadoop/flume/channel/checkpointDir, dataDirs=/home/hadoop/flume/channel/dataDir, type=file} }} SINKS: {hdfsSink={ parameters:{hdfs.path=hdfs://MLNameNode2001:54310/flume, type=hdfs, channel=fileChannel} }, fileSink={ parameters:{type=file_roll, sink.directory=/home/hadoop/flume/output} }} 2012-08-10 10:56:50,627 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateChannels(FlumeConfiguration.java:450)] Created channel memoryChannel 2012-08-10 10:56:50,627 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateChannels(FlumeConfiguration.java:450)] Created channel fileChannel 2012-08-10 10:56:50,639 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:649)] Creating s On 08/08/2012 10:27 PM, Hari Shreedharan wrote: Could you share your configuration with us? Your class path is quite bare, and remember that Hadoop does not ship with Flume. Can you take a look at the logs and see if you notice any errors?  Note that Flume "not printing" stuff is quite normal - flume is not exactly interactive. You should look for logs - it does not print stuff on the console. It will print to the console, only the exec command it uses to start up flume - which is what you are seeing. The logs should give you more details. Did you check the logs to make sure Flume is not working at all? Hari --  Hari Shreedharan On Wednesday, August 8, 2012 at 9:49 AM, Patrick Wendell wrote: Jagadish, If you use the new page and it still doesn't work, can you send me the output of jstack against the flume pid? I've noticed some similar issues in the past (silent startup failures) and I'm also trying to debug. - Patrick On Wed, Aug 8, 2012 at 12:49 AM, alo alt <wget.n...@gmail.com> wrote: HI, This page is outdated. Please follow: http://flume.apache.org/FlumeUserGuide.html#setup Since you haven't attached logs or error messages it's hard to say what happen. best - Alex Jagadish Bihani wrote: Hi I have downloaded the tarball of latest flume-ng 1.2.0. I have JAVA_HOME properly set. To begin with I have followed the instructions in " https://cwiki.apache.org/FLUME/getting-started.html" as it is. And even for that basic example: My flume agent stucks printing the following output and then nothing happens: *bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -n agent1 + exec /usr/java/jdk1.6.0_12/bin/java -Xmx20m -cp '/root/flume/apache-flume-1.2.0/conf:/root/flume/apache-flume-1.2.0/lib/*' -Djava.library.path= org.apache.flume.node.Application -f conf/flume.conf -n agent1* Please guide me what might be the cause. Regards, Jagadish Jagadish Bihani August 8, 2012 9:45 AM Hi I have downloaded the tarball of latest flume-ng 1.2.0. I have JAVA_HOME properly set. To begin with I have followed the instructions in " https://cwiki.apache.org/FLUME/getting-started.html" as it is. And even for that basic example: My flume agent stucks printing the following output and then nothing happens:  bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -n agent1 + exec /usr/java/jdk1.6.0_12/bin/java -Xmx20m -cp '/root/flume/apache-flume-1.2.0/conf:/root/flume/apache-flume-1.2.0/lib/*' -Djava.library.path= org.apache.flume.node.Application -f conf/flume.conf -n agent1 Please guide me what might be the cause. Regards, Jagadish -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF