Hello Prabhu,
You should have a look at your logs. Unless you've changed the settings,
they'll be in flume.log in whatever directory you executed from.
It looks to me like you are trying to use a header(%{host}) in the path,
but you are not providing that header. If you want to use it, you should
try using the host interceptor(which is better documented in the most
recent revisions). There may be other issues too, but it's hard to tell
without seeing the logs. Check them out and see if there's any
exceptions popping up
On 07/12/2012 03:19 AM, prabhu k wrote:
Hi,
While i am trying to load sample text file to HDFS sink, after
executing bin/flume-ng node --conf conf/ -f conf/flume.conf -n agent2
command, Am unable to view the file in hdfs.
i have executed like below.
root@md-trngpoc1:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT#
bin/flume-ng node --conf conf/ -f conf/flume.conf -n agent2
Warning: The "node" command is deprecated. Please use "agent" instead.
Info: Sourcing environment configuration script
/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf/flume-env.sh
Info: Including Hadoop libraries found via
(/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access
Info: Excluding
/usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-api-1.4.3.jar from
classpath
Info: Excluding
/usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar
from classpath
+ exec /usr/lib/jvm/java-6-sun-1.6.0.26/jre/bin/java -Xms100m -Xmx200m
-cp
'/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib/*:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/hadoop_dir/hadoop/libexec/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/local/hadoop_dir/hadoop/libexec/..:/usr/local/hadoop_dir/hadoop/libexec/../hadoop-core-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-thriftfs-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar'
-Djava.library.path=/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib:/usr/local/hadoop_dir/hadoop/libexec/../lib/native/Linux-i386-32
org.apache.flume.node.Application -f conf/flume.conf -n agent2
flume.conf
================
agent2.sources = tail
agent2.channels = memoryChannel
agent2.sinks = HDFS
agent2.sources.tail.type = exec
agent2.sources.tail.command = tail -F
/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/flume_test.txt
agent2.sources.tail.channels = memoryChannel
agent2.sinks.HDFS.channel = memoryChannel
agent2.sinks.HDFS.type = hdfs
agent2.sinks.HDFS.hdfs.path =
hdfs://10.5.114.110:54310/user/flume/'%{host}
<http://10.5.114.110:54310/user/flume/%27%%7Bhost%7D>'
agent2.sinks.HDFS.hdfs.file.Type = DataStream
agent2.channels.memoryChannel.type = memory
Please help me, if i have missed anything?
Thanks,
Prabhu
