I did not get you, could you please elaborate bit more? you should replace FLUME SYSLOG AGENT IP with the ip of the agent!
where i can add the FLUME SYSLOG AGENT IP ? On Wed, Jul 11, 2012 at 2:33 PM, alo alt <[email protected]> wrote: > you should replace FLUME SYSLOG AGENT IP with the ip of the agent! > > > On Jul 11, 2012, at 11:01 AM, prabhu k wrote: > > > Hi Alex, > > > > I have executed the below commands, i am getting following output: > > > > root@md-trngpoc1 > :/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT# > > nc -v FLUME SYSLOG AGENT IP 5140 < /tmp/foo > > nc: port range not valid > > root@md-trngpoc1:/tmp# more foo > > <13>Jun 20 12:12:12 host foo[345]: a syslog message > > > > Please suggest any idea on this issue. > > > > On Wed, Jul 11, 2012 at 2:15 PM, alo alt <[email protected]> wrote: > > > >> Hi, > >> > >> there stuck nothing, flume is waiting for a event. Point a syslog server > >> to the configured port or use the netcat example to see if the event can > >> delivered: > >> > >> echo "<13>Jun 20 12:12:12 host foo[345]: a syslog message" > /tmp/foo > >> nc -v FLUME SYSLOG AGENT IP 5140 < /tmp/foo > >> > >> - Alex > >> > >> > >> On Jul 11, 2012, at 10:30 AM, prabhu k wrote: > >> > >>> Hi Alex, > >>> > >>> Thanks for the response. > >>> > >>> I have followed this link > >>> http://mapredit.blogspot.de/2012/03/flumeng-evolution.html > >>> and configured the syslog-agent.cnf and executed by using > >>> bin/flume-ng agent -n syslog-agent -f conf/syslog-agent.cnf > >>> > >>> but, script running from past 45 mints, it seems struck up flume-ng > >> script. > >>> > >>> Here is the syslog-agent.cnf file. > >>> > >>> syslog-agent.cnf > >>> ===================== > >>> syslog-agent.sources = Syslog > >>> syslog-agent.channels = MemoryChannel-1 > >>> syslog-agent.sinks = Console > >>> > >>> syslog-agent.sources.Syslog.type = syslogTcp > >>> syslog-agent.sources.Syslog.port = 5140 > >>> > >>> syslog-agent.sources.Syslog.channels = MemoryChannel-1 > >>> syslog-agent.sinks.Console.channel = MemoryChannel-1 > >>> > >>> syslog-agent.sinks.Console.type = logger > >>> syslog-agent.channels.MemoryChannel-1.type = memory > >>> > >>> Execution log: > >>> =================== > >>> root@md-trngpoc1 > >> :/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT# > >>> bin/flume-ng agent -n syslog-agent -f conf/syslog-agent.cnf > >>> Warning: No configuration directory set! Use --conf <dir> to override. > >>> Info: Including Hadoop libraries found via > >>> (/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access > >>> Info: Excluding > >>> /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-api-1.4.3.jar from > >>> classpath > >>> Info: Excluding > >>> /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar > from > >>> classpath > >>> + exec /usr/lib/jvm/java-6-sun/bin/java -Xmx20m -cp > >>> > >> > '/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib/*:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/hadoop_dir/hadoop/libexec/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/local/hadoop_dir/hadoop/libexec/..:/usr/local/hadoop_dir/hadoop/libexec/../hadoop-core-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-thriftfs-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar' > >>> > >> > -Djava.library.path=/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib:/usr/local/hadoop_dir/hadoop/libexec/../lib/native/Linux-i386-32 > >>> org.apache.flume.node.Application -n syslog-agent -f > >> conf/syslog-agent.cnf > >>> 12/07/11 12:40:27 INFO lifecycle.LifecycleSupervisor: Starting > lifecycle > >>> supervisor 1 > >>> 12/07/11 12:40:27 INFO node.FlumeNode: Flume node starting - > syslog-agent > >>> 12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager: Node > >> manager > >>> starting > >>> 12/07/11 12:40:27 INFO lifecycle.LifecycleSupervisor: Starting > lifecycle > >>> supervisor 11 > >>> 12/07/11 12:40:27 INFO properties.PropertiesFileConfigurationProvider: > >>> Configuration provider starting > >>> 12/07/11 12:40:27 INFO properties.PropertiesFileConfigurationProvider: > >>> Reloading configuration file:conf/syslog-agent.cnf > >>> 12/07/11 12:40:27 INFO conf.FlumeConfiguration: Processing:Console > >>> 12/07/11 12:40:27 INFO conf.FlumeConfiguration: Processing:Console > >>> 12/07/11 12:40:27 INFO conf.FlumeConfiguration: Added sinks: Console > >> Agent: > >>> syslog-agent > >>> 12/07/11 12:40:27 INFO conf.FlumeConfiguration: Post-validation flume > >>> configuration contains configuration for agents: [syslog-agent] > >>> 12/07/11 12:40:27 INFO properties.PropertiesFileConfigurationProvider: > >>> Creating channels > >>> 12/07/11 12:40:27 INFO properties.PropertiesFileConfigurationProvider: > >>> created channel MemoryChannel-1 > >>> 12/07/11 12:40:27 INFO sink.DefaultSinkFactory: Creating instance of > sink > >>> Console typelogger > >>> 12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager: Starting > >> new > >>> configuration:{ sourceRunners:{Syslog=EventDrivenSourceRunner: { > >>> source:org.apache.flume.source.SyslogTcpSource@110c424 }} > >>> sinkRunners:{Console=SinkRunner: { > >>> policy:org.apache.flume.sink.DefaultSinkProcessor@1bd2664counterGroup:{ > >>> name:null counters:{} } }} > >>> > channels:{MemoryChannel-1=org.apache.flume.channel.MemoryChannel@1238bd2} > >> } > >>> 12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager: Starting > >>> Channel MemoryChannel-1 > >>> 12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager: Starting > >> Sink > >>> Console > >>> 12/07/11 12:40:27 INFO nodemanager.DefaultLogicalNodeManager: Starting > >>> Source Syslog > >>> 12/07/11 12:40:28 INFO source.SyslogTcpSource: Syslog TCP Source > >> starting... > >>> > >>> Please let me know, if i am doing anything wrong? > >>> > >>> Thanks, > >>> Prabhu > >>> > >>> > >>> > >>> On Wed, Jul 11, 2012 at 11:48 AM, alo alt <[email protected]> wrote: > >>> > >>>> Hi, > >>>> > >>>> what exactly is the issue? Error messages, things like that. > >>>> > >>>> You posted only debugging and exec logs, thats pretty normal. For a > >> first > >>>> short overview about flume take a look into: > >>>> > >>>> https://cwiki.apache.org/FLUME/getting-started.html > >>>> > >>>> > >> > http://archive.cloudera.com/cdh4/cdh/4/flume-ng-1.1.0-cdh4.0.0b2/FlumeUserGuide.html > >>>> http://mapredit.blogspot.de/2012/03/flumeng-evolution.html > >>>> > >>>> regards, > >>>> Alex > >>>> > >>>> On Jul 10, 2012, at 3:09 PM, prabhu k wrote: > >>>> > >>>>> Hi, > >>>>> > >>>>> While i am executing flume-ng, i am getting following out put on my > >>>>> terminal, it seems got struck. > >>>>> > >>>>> root@md-trngpoc1 > >>>> :/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT# > >>>>> bin/flume-ng agent -n agent1 -c conf -f conf/agent1.conf > >>>>> Info: Sourcing environment configuration script > >>>>> > >>>> > >> > /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf/flume-env.sh > >>>>> Info: Including Hadoop libraries found via > >>>>> (/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access > >>>>> Info: Excluding > >>>>> /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-api-1.4.3.jar from > >>>>> classpath > >>>>> Info: Excluding > >>>>> /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar > >> from > >>>>> classpath > >>>>> + exec /usr/lib/jvm/java-6-sun-1.6.0.26/jre/bin/java -Xms100m > -Xmx200m > >>>> -cp > >>>>> > >>>> > >> > '/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib/*:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/hadoop_dir/hadoop/libexec/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/local/hadoop_dir/hadoop/libexec/..:/usr/local/hadoop_dir/hadoop/libexec/../hadoop-core-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-thriftfs-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar' > >>>>> > >>>> > >> > -Djava.library.path=/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib:/usr/local/hadoop_dir/hadoop/libexec/../lib/native/Linux-i386-32 > >>>>> org.apache.flume.node.Application -n agent1 -f conf/agent1.conf > >>>>> > >>>>> in execution summary: > >>>>> ---------------------------------------- > >>>>> Info: Including Hadoop libraries found via > >>>>> (/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access > >>>>> > >>>>> hadoop lib path pointing to /usr/local/hadoop_dir/hadoop/bin/hadoop > >>>>> directory insted of /usr/local/hadoop_dir/hadoop/lib path. > >>>>> > >>>>> Here my questions are; > >>>>> > >>>>> 1. in flume-ng script anything changes need in add_hadoop_paths > >> function. > >>>>> 2. anything wrong in flume-env.sh script > >>>>> 3. anything wrong in agent1.conf file. > >>>>> > >>>>> Please find the attached configuaration details and let me know any > >> more > >>>>> details. > >>>>> > >>>>> Could you pleae suggest me on this issue. > >>>>> > >>>>> Thanks, > >>>>> Prabhu > >>>>> <flume_config.txt> > >>>> > >>>> > >>>> -- > >>>> Alexander Alten-Lorenz > >>>> http://mapredit.blogspot.com > >>>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF > >>>> > >>>> > >> > >> > >> -- > >> Alexander Alten-Lorenz > >> http://mapredit.blogspot.com > >> German Hadoop LinkedIn Group: http://goo.gl/N8pCF > >> > >> > > > -- > Alexander Alten-Lorenz > http://mapredit.blogspot.com > German Hadoop LinkedIn Group: http://goo.gl/N8pCF > >
