can anyone help me on the below issue.early response i would appreciate. On Wed, Aug 22, 2012 at 3:13 PM, prabhu k <prabhu.fl...@gmail.com> wrote:
> Hi Users, > > I have followed the below link * > http://cloudfront.blogspot.in/2012/06/how-to-use-host-escape-sequence-in.html > * for sample text file to HDFS sink using tail source. > > and i have executed flume-ng like using below command. it seems got stuck. > attached flume.conf file and running script log. > > *bin/flume-ng agent -n agent1 -c /conf -f conf/flume.conf* > > Please suggest and help me on this issue. > > flume.conf > =========== > > agent1.sources = tail > agent1.channels = MemoryChannel-2 > agent1.sinks = HDFS > agent1.sources.tail.type = exec > agent1.sources.tail.command = tail -F > /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/flume_test.txt > agent1.sources.tail.channels = MemoryChannel-2 > agent1.sources.tail.interceptors = hostint > agent1.sources.tail.interceptors.hostint.type = > org.apache.flume.interceptor.HostInterceptor$Builder > agent1.sources.tail.interceptors.hostint.preserveExisting = true > agent1.sources.tail.interceptors.hostint.useIP = false > agent1.sinks.HDFS.channel = MemoryChannel-2 > agent1.sinks.HDFS.type = hdfs > agent1.sinks.HDFS.hdfs.path = hdfs://10.10.12.100:54310/user/%{host} > agent1.sinks.HDFS.hdfs.file.Type = dataStream > agent1.sinks.HDFS.hdfs.writeFormat = text > agent1.channels.MemoryChannel-2.type = memory > > > > script running output: > > /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT# bin/flume-ng > agent -n agent1 -c /conf -f conf/flume.conf > Info: Including Hadoop libraries found via > (/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access > Info: Excluding > /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-api-1.4.3.jar from > classpath > Info: Excluding > /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from > classpath > + exec /usr/lib/jvm/java-6-sun/bin/java -Xmx20m -cp > '/conf:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib/*:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/hadoop_dir/hadoop/libexec/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/local/hadoop_dir/hadoop/libexec/..:/usr/local/hadoop_dir/hadoop/libexec/../hadoop-core-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-thriftfs-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar' > -Djava.library.path=/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib:/usr/local/hadoop_dir/hadoop/libexec/../lib/native/Linux-i386-32 > org.apache.flume.node.Application -n agent1 -f conf/flume.conf > 12/08/22 15:01:08 INFO lifecycle.LifecycleSupervisor: Starting lifecycle > supervisor 1 > 12/08/22 15:01:08 INFO node.FlumeNode: Flume node starting - agent1 > 12/08/22 15:01:08 INFO nodemanager.DefaultLogicalNodeManager: Node manager > starting > 12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider: > Configuration provider starting > 12/08/22 15:01:08 INFO lifecycle.LifecycleSupervisor: Starting lifecycle > supervisor 10 > 12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider: > Reloading configuration file:conf/flume.conf > 12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS > 12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS > 12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS > 12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS > 12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS > 12/08/22 15:01:08 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent: > agent1 > 12/08/22 15:01:08 INFO conf.FlumeConfiguration: Post-validation flume > configuration contains configuration for agents: [agent1] > 12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider: > Creating channels > 12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider: > created channel MemoryChannel-2 > 12/08/22 15:01:08 INFO sink.DefaultSinkFactory: Creating instance of sink > HDFS typehdfs > > Thanks, > Prabhu. >
