Hi
Thanks all for the inputs. After the initial problem I was able
to start flume except in one scenario in
which I use HDFS as sink.
I have a production machine with hadoop-0.20.1 installed. I have
installed latest flume 1.2.0.
It works fine for all the configurations (at least which I
tried) except when HDFS sink is used.
Test:
---------
I used both netcat listener as the source of the agent and HDFS
is sink. Then I start the agent using
the command
"bin/flume-ng agent -n agent1 -c conf -f
conf/flume_hdfs.conf --classpath
/MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop-0.20.1-core.jar
-Dflume.root.logger=DEBUG,console"
with DEBUG logging mode enabled. I don't get any
error/exception. I use
"/usr/sbin/lsof -i:<port_no>"
command to check whether the source
is actually bound to that port and it doesn't return any port.
But when I use
file sink instead of HDFS sink and run
lsof it correctly shows me the port on which
it is listening.
Thus when HDFS sink is used even source part of agent doesn't
work and it doesn't give any exception. And nothing is written
to
HDFS sink.
P.S. I have checked the user,permission details of HDFS. They
are fine.
I have run flume on my other machines with different version of
hadoop (0.23 & 1.0). It has run HDFS sink properly there.
Does flume support hadoop-0.20.1 or there is something I am
missing???
This is my Configuration:
-----------------------------------------
agent1.sources = sequencer
agent1.sinks =hdfsSink fileSink
agent1.sinks =fileSink
agent1.channels =memoryChannel fileChannel
agent1.sources.sequencer.type=seq
agent1.sinks.hdfsSink.type=hdfs
agent1.sinks.hdfsSink.hdfs.path=hdfs://MLNameNode2001:54310/flume
agent1.sinks.fileSink.type=file_roll
agent1.sinks.fileSink.sink.directory=/home/hadoop/flume/output
agent1.channels.memoryChannel.type=memory
agent1.channels.memoryChannel.capacity=10000
agent1.channels.memoryChannel.transactionCapacity=100
agent1.channels.fileChannel.type=file
agent1.channels.fileChannel.checkpointDir=/home/hadoop/flume/channel/checkpointDir
agent1.channels.fileChannel.dataDirs=/home/hadoop/flume/channel/dataDir
agent1.sources.sequencer.channels = fileChannel
agent1.sinks.hdfsSink.channel = fileChannel
This is the log which I get:
----------------------------------------------------------
bin/flume-ng agent -n agent1 -c conf -f conf/flume_hdfs.conf
--classpath /MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop
-0.20.1-core.jar -Dflume.root.logger=DEBUG,console
+ exec /usr/java/jdk1.6.0_12/bin/java -Xmx20m
-Dflume.root.logger=DEBUG,console -cp
'/home/hadoop/flume/apache-flume-1.2.0/conf:/home/hadoop/flume/apache-flume-1.2.0/li
b/*:/MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop-0.20.1-core.jar'
-Djava.library.path= org.apache.flume.node.Application -n agent1
-f conf/flume_hdfs.conf
2012-08-10 10:56:50,604 (main) [INFO -
org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
Starting lifecycle supervisor 1
2012-08-10 10:56:50,607 (main) [INFO -
org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume
node starting - agent1
2012-08-10 10:56:50,611 (lifecycleSupervisor-1-2) [INFO -
org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)]
Configuration provider starting
2012-08-10 10:56:50,611 (lifecycleSupervisor-1-0) [INFO -
org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)]
Node ma
nager starting
2012-08-10 10:56:50,613 (lifecycleSupervisor-1-0) [INFO -
org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
Starting lifecycle supervis
or 9
2012-08-10 10:56:50,614 (lifecycleSupervisor-1-0) [DEBUG -
org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:191)]
Node manager started
2012-08-10 10:56:50,614 (lifecycleSupervisor-1-2) [DEBUG -
org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:86)
] Configuration provider started
2012-08-10 10:56:50,614 (conf-file-poller-0) [DEBUG -
org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProv
ider.java:188)] Checking
file:conf/flume_hdfs.conf
for changes
2012-08-10 10:56:50,614 (conf-file-poller-0) [INFO -
org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvi
der.java:195)] Reloading configuration
file:conf/flume_hdfs.conf
2012-08-10 10:56:50,618 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
Processing:fi
leSink
2012-08-10 10:56:50,619 (conf-file-poller-0) [DEBUG -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:992)]
Created cont
ext for fileSink: sink.directory
2012-08-10 10:56:50,619 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
Processing:hd
fsSink
2012-08-10 10:56:50,620 (conf-file-poller-0) [DEBUG -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:992)]
Created cont
ext for hdfsSink: type
2012-08-10 10:56:50,620 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
Processing:hd
fsSink
2012-08-10 10:56:50,620 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
Processing:hd
fsSink
2012-08-10 10:56:50,614 (lifecycleSupervisor-1-0) [DEBUG -
org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:191)]
Node m
anager started
2012-08-10 10:56:50,614 (lifecycleSupervisor-1-2) [DEBUG -
org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:86)
] Configuration provider started
2012-08-10 10:56:50,614 (conf-file-poller-0) [DEBUG -
org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProv
ider.java:188)] Checking
file:conf/flume_hdfs.conf
for changes
2012-08-10 10:56:50,614 (conf-file-poller-0) [INFO -
org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvi
der.java:195)] Reloading configuration
file:conf/flume_hdfs.conf
2012-08-10 10:56:50,618 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
Processing:fi
leSink
2012-08-10 10:56:50,619 (conf-file-poller-0) [DEBUG -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:992)]
Created cont
ext for fileSink: sink.directory
2012-08-10 10:56:50,619 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
Processing:hd
fsSink
2012-08-10 10:56:50,620 (conf-file-poller-0) [DEBUG -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:992)]
Created cont
ext for hdfsSink: type
2012-08-10 10:56:50,620 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
Processing:hd
fsSink
2012-08-10 10:56:50,620 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
Processing:hd
fsSink
2012-08-10 10:56:50,620 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
Processing:fileSink
2012-08-10 10:56:50,622 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:902)]
Added sinks: fileSink Agent: agent1
2012-08-10 10:56:50,622 (conf-file-poller-0) [DEBUG -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:295)]
Starting validation of configuration for agent: agent1,
initial-configuration: AgentConfiguration[agent1]
SOURCES: {sequencer={ parameters:{channels=fileChannel,
type=seq} }}
CHANNELS: {memoryChannel={ parameters:{transactionCapacity=100,
capacity=10000, type=memory} }, fileChannel={
parameters:{checkpointDir=/home/hadoop/flume/channel/checkpointDir,
dataDirs=/home/hadoop/flume/channel/dataDir, type=file} }}
SINKS: {hdfsSink={
parameters:{hdfs.path=hdfs://MLNameNode2001:54310/flume,
type=hdfs, channel=fileChannel} }, fileSink={
parameters:{type=file_roll,
sink.directory=/home/hadoop/flume/output} }}
2012-08-10 10:56:50,627 (conf-file-poller-0) [DEBUG -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateChannels(FlumeConfiguration.java:450)]
Created channel memoryChannel
2012-08-10 10:56:50,627 (conf-file-poller-0) [DEBUG -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateChannels(FlumeConfiguration.java:450)]
Created channel fileChannel
2012-08-10 10:56:50,639 (conf-file-poller-0) [DEBUG -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:649)]
Creating s
On 08/08/2012 10:27 PM, Hari Shreedharan wrote:
