Hi
Thanks all for the inputs. After the initial problem
I was able to start flume except in one scenario in
which I use HDFS as sink.
I have a production machine with hadoop-0.20.1
installed. I have installed latest flume 1.2.0.
It works fine for all the configurations (at least
which I tried) except when HDFS sink is used.
Test:
---------
I used both netcat listener as the source of the
agent and HDFS is sink. Then I start the agent using
the command
"bin/flume-ng agent -n agent1 -c
conf -f conf/flume_hdfs.conf --classpath
/MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop-0.20.1-core.jar
-Dflume.root.logger=DEBUG,console"
with DEBUG logging mode enabled. I don't get any
error/exception. I use
"/usr/sbin/lsof
-i:<port_no>" command to check whether
the source
is actually bound to that port and it doesn't return
any port. But when I use
file sink instead of
HDFS sink and run lsof it correctly shows me
the port on which
it is listening.
Thus when HDFS sink is used even source part of
agent doesn't work and it doesn't give any
exception. And nothing is written to
HDFS sink.
P.S. I have checked the user,permission details of
HDFS. They are fine.
I have run flume on my other machines with different
version of hadoop (0.23 & 1.0). It has run HDFS
sink properly there.
Does flume support hadoop-0.20.1 or there is
something I am missing???
This is my Configuration:
-----------------------------------------
agent1.sources = sequencer
agent1.sinks =hdfsSink fileSink
agent1.sinks =fileSink
agent1.channels =memoryChannel fileChannel
agent1.sources.sequencer.type=seq
agent1.sinks.hdfsSink.type=hdfs
agent1.sinks.hdfsSink.hdfs.path=hdfs://MLNameNode2001:54310/flume
agent1.sinks.fileSink.type=file_roll
agent1.sinks.fileSink.sink.directory=/home/hadoop/flume/output
agent1.channels.memoryChannel.type=memory
agent1.channels.memoryChannel.capacity=10000
agent1.channels.memoryChannel.transactionCapacity=100
agent1.channels.fileChannel.type=file
agent1.channels.fileChannel.checkpointDir=/home/hadoop/flume/channel/checkpointDir
agent1.channels.fileChannel.dataDirs=/home/hadoop/flume/channel/dataDir
agent1.sources.sequencer.channels = fileChannel
agent1.sinks.hdfsSink.channel = fileChannel
This is the log which I get:
----------------------------------------------------------
bin/flume-ng agent -n agent1 -c conf -f
conf/flume_hdfs.conf --classpath
/MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop
-0.20.1-core.jar -Dflume.root.logger=DEBUG,console
+ exec /usr/java/jdk1.6.0_12/bin/java -Xmx20m
-Dflume.root.logger=DEBUG,console -cp
'/home/hadoop/flume/apache-flume-1.2.0/conf:/home/hadoop/flume/apache-flume-1.2.0/li
b/*:/MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop-0.20.1-core.jar'
-Djava.library.path=
org.apache.flume.node.Application -n agent1 -f
conf/flume_hdfs.conf
2012-08-10 10:56:50,604 (main) [INFO -
org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
Starting lifecycle supervisor 1
2012-08-10 10:56:50,607 (main) [INFO -
org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)]
Flume node starting - agent1
2012-08-10 10:56:50,611 (lifecycleSupervisor-1-2)
[INFO -
org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)]
Configuration provider starting
2012-08-10 10:56:50,611 (lifecycleSupervisor-1-0)
[INFO -
org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)]
Node ma
nager starting
2012-08-10 10:56:50,613 (lifecycleSupervisor-1-0)
[INFO -
org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
Starting lifecycle supervis
or 9
2012-08-10 10:56:50,614 (lifecycleSupervisor-1-0)
[DEBUG -
org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:191)]
Node manager started
2012-08-10 10:56:50,614 (lifecycleSupervisor-1-2)
[DEBUG -
org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:86)
] Configuration provider started
2012-08-10 10:56:50,614 (conf-file-poller-0) [DEBUG
-
org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProv
ider.java:188)] Checking
file:conf/flume_hdfs.conf
for changes
2012-08-10 10:56:50,614 (conf-file-poller-0) [INFO -
org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvi
der.java:195)] Reloading configuration
file:conf/flume_hdfs.conf
2012-08-10 10:56:50,618 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
Processing:fi
leSink
2012-08-10 10:56:50,619 (conf-file-poller-0) [DEBUG
-
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:992)]
Created cont
ext for fileSink: sink.directory
2012-08-10 10:56:50,619 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
Processing:hd
fsSink
2012-08-10 10:56:50,620 (conf-file-poller-0) [DEBUG
-
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:992)]
Created cont
ext for hdfsSink: type
2012-08-10 10:56:50,620 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
Processing:hd
fsSink
2012-08-10 10:56:50,620 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
Processing:hd
fsSink
2012-08-10 10:56:50,614 (lifecycleSupervisor-1-0)
[DEBUG -
org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:191)]
Node m
anager started
2012-08-10 10:56:50,614 (lifecycleSupervisor-1-2)
[DEBUG -
org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:86)
] Configuration provider started
2012-08-10 10:56:50,614 (conf-file-poller-0) [DEBUG
-
org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProv
ider.java:188)] Checking
file:conf/flume_hdfs.conf
for changes
2012-08-10 10:56:50,614 (conf-file-poller-0) [INFO -
org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvi
der.java:195)] Reloading configuration
file:conf/flume_hdfs.conf
2012-08-10 10:56:50,618 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
Processing:fi
leSink
2012-08-10 10:56:50,619 (conf-file-poller-0) [DEBUG
-
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:992)]
Created cont
ext for fileSink: sink.directory
2012-08-10 10:56:50,619 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
Processing:hd
fsSink
2012-08-10 10:56:50,620 (conf-file-poller-0) [DEBUG
-
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:992)]
Created cont
ext for hdfsSink: type
2012-08-10 10:56:50,620 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
Processing:hd
fsSink
2012-08-10 10:56:50,620 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
Processing:hd
fsSink
2012-08-10 10:56:50,620 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
Processing:fileSink
2012-08-10 10:56:50,622 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:902)]
Added sinks: fileSink Agent: agent1
2012-08-10 10:56:50,622 (conf-file-poller-0) [DEBUG
-
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:295)]
Starting validation of configuration for agent:
agent1, initial-configuration:
AgentConfiguration[agent1]
SOURCES: {sequencer={
parameters:{channels=fileChannel, type=seq} }}
CHANNELS: {memoryChannel={
parameters:{transactionCapacity=100, capacity=10000,
type=memory} }, fileChannel={
parameters:{checkpointDir=/home/hadoop/flume/channel/checkpointDir,
dataDirs=/home/hadoop/flume/channel/dataDir,
type=file} }}
SINKS: {hdfsSink={
parameters:{hdfs.path=hdfs://MLNameNode2001:54310/flume,
type=hdfs, channel=fileChannel} }, fileSink={
parameters:{type=file_roll,
sink.directory=/home/hadoop/flume/output} }}
2012-08-10 10:56:50,627 (conf-file-poller-0) [DEBUG
-
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateChannels(FlumeConfiguration.java:450)]
Created channel memoryChannel
2012-08-10 10:56:50,627 (conf-file-poller-0) [DEBUG
-
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateChannels(FlumeConfiguration.java:450)]
Created channel fileChannel
2012-08-10 10:56:50,639 (conf-file-poller-0) [DEBUG
-
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:649)]
Creating s
On 08/08/2012 10:27 PM, Hari Shreedharan wrote:
