Hi, means, flume is working. As I expected. On flume's side (the window from where you've started the agent), you have should seen a log entry who reports the successful received event.
Do you know what Flume does? - Alex On Jul 11, 2012, at 2:00 PM, prabhu k wrote: > Hi Alex, > > I have executed like below with the agent ip, connection got succeeded, but > where can i include this command? > > root@md-trngpoc1:/tmp# nc -v 10.5.114.110 54310 < /tmp/foo > Connection to 10.5.114.110 54310 port [tcp/*] succeeded! > > > On Wed, Jul 11, 2012 at 3:14 PM, alo alt <[email protected]> wrote: > >> >> example: >> >>>>>> echo "<13>Jun 20 12:12:12 host foo[345]: a syslog message" > /tmp/foo >>>>>> nc -v FLUME SYSLOG AGENT IP 5140 < /tmp/foo >> >> echo "<13>Jun 20 12:12:12 host foo[345]: a syslog message" > /tmp/foo >> nc -v 10.10.10.10 5140 < /tmp/foo >> >> 10.10.10.10 is the IP who the flume agent is waiting for a connection. >> >> man nc will help too. >> >> >> On Jul 11, 2012, at 11:36 AM, prabhu k wrote: >> >>> I did not get you, could you please elaborate bit more? >>> >>> you should replace FLUME SYSLOG AGENT IP with the ip of the agent! >>> >>> where i can add the FLUME SYSLOG AGENT IP ? >>> >>> On Wed, Jul 11, 2012 at 2:33 PM, alo alt <[email protected]> wrote: >>> >>>> you should replace FLUME SYSLOG AGENT IP with the ip of the agent! >>>> >>>> >>>> On Jul 11, 2012, at 11:01 AM, prabhu k wrote: >>>> >>>>> Hi Alex, >>>>> >>>>> I have executed the below commands, i am getting following output: >>>>> >>>>> root@md-trngpoc1 >>>> :/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT# >>>>> nc -v FLUME SYSLOG AGENT IP 5140 < /tmp/foo >>>>> nc: port range not valid >>>>> root@md-trngpoc1:/tmp# more foo >>>>> <13>Jun 20 12:12:12 host foo[345]: a syslog message >>>>> >>>>> Please suggest any idea on this issue. >>>>> >>>>> On Wed, Jul 11, 2012 at 2:15 PM, alo alt <[email protected]> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> there stuck nothing, flume is waiting for a event. Point a syslog >> server >>>>>> to the configured port or use the netcat example to see if the event >> can >>>>>> delivered: >>>>>> >>>>>> echo "<13>Jun 20 12:12:12 host foo[345]: a syslog message" > /tmp/foo >>>>>> nc -v FLUME SYSLOG AGENT IP 5140 < /tmp/foo >>>>>> >>>>>> - Alex >>>>>> >>>>>> >>>>>> On Jul 11, 2012, at 10:30 AM, prabhu k wrote: >>>>>> >>>>>>> Hi Alex, >>>>>>> >>>>>>> Thanks for the response. >>>>>>> >>>>>>> I have followed this link >>>>>>> http://mapredit.blogspot.de/2012/03/flumeng-evolution.html >>>>>>> and configured the syslog-agent.cnf and executed by using >>>>>>> bin/flume-ng agent -n syslog-agent -f conf/syslog-agent.cnf >>>>>>> >>>>>>> but, script running from past 45 mints, it seems struck up flume-ng >>>>>> script. >>>>>>> >>>>>>> Here is the syslog-agent.cnf file. >>>>>>> >>>>>>> syslog-agent.cnf >>>>>>> ===================== >>>>>>> syslog-agent.sources = Syslog >>>>>>> syslog-agent.channels = MemoryChannel-1 >>>>>>> syslog-agent.sinks = Console >>>>>>> >>>>>>> syslog-agent.sources.Syslog.type = syslogTcp >>>>>>> syslog-agent.sources.Syslog.port = 5140 >>>>>>> >>>>>>> syslog-agent.sources.Syslog.channels = MemoryChannel-1 >>>>>>> syslog-agent.sinks.Console.channel = MemoryChannel-1 >>>>>>> >>>>>>> syslog-agent.sinks.Console.type = logger >>>>>>> syslog-agent.channels.MemoryChannel-1.type = memory >>>>>>> >>>>>>> Execution log: >>>>>>> =================== >>>>>>> root@md-trngpoc1 >>>>>> :/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT# >>>>>>> bin/flume-ng agent -n syslog-agent -f conf/syslog-agent.cnf >>>>>>> Warning: No configuration directory set! Use --conf <dir> to >> override. >>>>>>> Info: Including Hadoop libraries found via >>>>>>> (/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access >>>>>>> Info: Excluding >>>>>>> /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-api-1.4.3.jar from >>>>>>> classpath >>>>>>> Info: Excluding >>>>>>> /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar >>>> from >>>>>>> classpath >>>>>>> + exec /usr/lib/jvm/java-6-sun/bin/java -Xmx20m -cp >>>>>>> >>>>>> >>>> >> '/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib/*:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/hadoop_dir/hadoop/libexec/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/local/hadoop_dir/hadoop/libexec/..:/usr/local/hadoop_dir/hadoop/libexec/../hadoop-core-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-thriftfs-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar' >>>>>>> >>>>>> >>>> >> -Djava.library.path=/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib:/usr/local/hadoop_dir/hadoop/libexec/../lib/native/Linux-i386-32 >>>>>>> org.apache.flume.node.Application -n syslog-agent -f >>>>>> conf/syslog-agent.cnf >>>>>>> 12/07/11 12:40:27 INFO lifecycle.LifecycleSupervisor: Starting >>>> lifecycle >>>>>>> supervisor 1 >>>>>>> 12/07/11 12:40:27 INFO node.FlumeNode: Flume node starting - >>>> syslog-agent >>>>>>> 12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager: Node >>>>>> manager >>>>>>> starting >>>>>>> 12/07/11 12:40:27 INFO lifecycle.LifecycleSupervisor: Starting >>>> lifecycle >>>>>>> supervisor 11 >>>>>>> 12/07/11 12:40:27 INFO >> properties.PropertiesFileConfigurationProvider: >>>>>>> Configuration provider starting >>>>>>> 12/07/11 12:40:27 INFO >> properties.PropertiesFileConfigurationProvider: >>>>>>> Reloading configuration file:conf/syslog-agent.cnf >>>>>>> 12/07/11 12:40:27 INFO conf.FlumeConfiguration: Processing:Console >>>>>>> 12/07/11 12:40:27 INFO conf.FlumeConfiguration: Processing:Console >>>>>>> 12/07/11 12:40:27 INFO conf.FlumeConfiguration: Added sinks: Console >>>>>> Agent: >>>>>>> syslog-agent >>>>>>> 12/07/11 12:40:27 INFO conf.FlumeConfiguration: Post-validation flume >>>>>>> configuration contains configuration for agents: [syslog-agent] >>>>>>> 12/07/11 12:40:27 INFO >> properties.PropertiesFileConfigurationProvider: >>>>>>> Creating channels >>>>>>> 12/07/11 12:40:27 INFO >> properties.PropertiesFileConfigurationProvider: >>>>>>> created channel MemoryChannel-1 >>>>>>> 12/07/11 12:40:27 INFO sink.DefaultSinkFactory: Creating instance of >>>> sink >>>>>>> Console typelogger >>>>>>> 12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager: >> Starting >>>>>> new >>>>>>> configuration:{ sourceRunners:{Syslog=EventDrivenSourceRunner: { >>>>>>> source:org.apache.flume.source.SyslogTcpSource@110c424 }} >>>>>>> sinkRunners:{Console=SinkRunner: { >>>>>>> policy:org.apache.flume.sink.DefaultSinkProcessor@1bd2664counterGroup >> :{ >>>>>>> name:null counters:{} } }} >>>>>>> >>>> channels:{MemoryChannel-1=org.apache.flume.channel.MemoryChannel@1238bd2 >> } >>>>>> } >>>>>>> 12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager: >> Starting >>>>>>> Channel MemoryChannel-1 >>>>>>> 12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager: >> Starting >>>>>> Sink >>>>>>> Console >>>>>>> 12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager: >> Starting >>>>>>> Source Syslog >>>>>>> 12/07/11 12:40:28 INFO source.SyslogTcpSource: Syslog TCP Source >>>>>> starting... >>>>>>> >>>>>>> Please let me know, if i am doing anything wrong? >>>>>>> >>>>>>> Thanks, >>>>>>> Prabhu >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Jul 11, 2012 at 11:48 AM, alo alt <[email protected]> >> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> what exactly is the issue? Error messages, things like that. >>>>>>>> >>>>>>>> You posted only debugging and exec logs, thats pretty normal. For a >>>>>> first >>>>>>>> short overview about flume take a look into: >>>>>>>> >>>>>>>> https://cwiki.apache.org/FLUME/getting-started.html >>>>>>>> >>>>>>>> >>>>>> >>>> >> http://archive.cloudera.com/cdh4/cdh/4/flume-ng-1.1.0-cdh4.0.0b2/FlumeUserGuide.html >>>>>>>> http://mapredit.blogspot.de/2012/03/flumeng-evolution.html >>>>>>>> >>>>>>>> regards, >>>>>>>> Alex >>>>>>>> >>>>>>>> On Jul 10, 2012, at 3:09 PM, prabhu k wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> While i am executing flume-ng, i am getting following out put on my >>>>>>>>> terminal, it seems got struck. >>>>>>>>> >>>>>>>>> root@md-trngpoc1 >>>>>>>> :/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT# >>>>>>>>> bin/flume-ng agent -n agent1 -c conf -f conf/agent1.conf >>>>>>>>> Info: Sourcing environment configuration script >>>>>>>>> >>>>>>>> >>>>>> >>>> >> /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf/flume-env.sh >>>>>>>>> Info: Including Hadoop libraries found via >>>>>>>>> (/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access >>>>>>>>> Info: Excluding >>>>>>>>> /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-api-1.4.3.jar >> from >>>>>>>>> classpath >>>>>>>>> Info: Excluding >>>>>>>>> /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar >>>>>> from >>>>>>>>> classpath >>>>>>>>> + exec /usr/lib/jvm/java-6-sun-1.6.0.26/jre/bin/java -Xms100m >>>> -Xmx200m >>>>>>>> -cp >>>>>>>>> >>>>>>>> >>>>>> >>>> >> '/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib/*:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/hadoop_dir/hadoop/libexec/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/local/hadoop_dir/hadoop/libexec/..:/usr/local/hadoop_dir/hadoop/libexec/../hadoop-core-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-thriftfs-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar' >>>>>>>>> >>>>>>>> >>>>>> >>>> >> -Djava.library.path=/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib:/usr/local/hadoop_dir/hadoop/libexec/../lib/native/Linux-i386-32 >>>>>>>>> org.apache.flume.node.Application -n agent1 -f conf/agent1.conf >>>>>>>>> >>>>>>>>> in execution summary: >>>>>>>>> ---------------------------------------- >>>>>>>>> Info: Including Hadoop libraries found via >>>>>>>>> (/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access >>>>>>>>> >>>>>>>>> hadoop lib path pointing to /usr/local/hadoop_dir/hadoop/bin/hadoop >>>>>>>>> directory insted of /usr/local/hadoop_dir/hadoop/lib path. >>>>>>>>> >>>>>>>>> Here my questions are; >>>>>>>>> >>>>>>>>> 1. in flume-ng script anything changes need in add_hadoop_paths >>>>>> function. >>>>>>>>> 2. anything wrong in flume-env.sh script >>>>>>>>> 3. anything wrong in agent1.conf file. >>>>>>>>> >>>>>>>>> Please find the attached configuaration details and let me know any >>>>>> more >>>>>>>>> details. >>>>>>>>> >>>>>>>>> Could you pleae suggest me on this issue. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Prabhu >>>>>>>>> <flume_config.txt> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Alexander Alten-Lorenz >>>>>>>> http://mapredit.blogspot.com >>>>>>>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF >>>>>>>> >>>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Alexander Alten-Lorenz >>>>>> http://mapredit.blogspot.com >>>>>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF >>>>>> >>>>>> >>>> >>>> >>>> -- >>>> Alexander Alten-Lorenz >>>> http://mapredit.blogspot.com >>>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF >>>> >>>> >> >> >> -- >> Alexander Alten-Lorenz >> http://mapredit.blogspot.com >> German Hadoop LinkedIn Group: http://goo.gl/N8pCF >> >> -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF
