I pulled your source configuration. Here is the full configuration (single-node-custom-source.properties), removed my custom source from here
# Exampple for single node config # Base Config a1.sources=src1 a1.sinks=sink1 a1.channels=ch1 # Configure the source a1.sources.src1.type = org.apache.flume.source.http.HTTPSource a1.sources.src1.port = 9001 a1.sources.src1.bind = localhost a1.sources.src1.handler = org.apache.flume.source.http.JSONHandler # Sink Configuration a1.sinks.sink1.type=logger # Channel configuration a1.channels.ch1.type=memory a1.channels.ch1.capacity=1000 a1.channels.ch1.transactionCapacity=100 # Link stuff together a1.sources.src1.channels=ch1 a1.sinks.sink1.channel=ch1 and this is how I start my agent bin/flume-ng agent --conf-file conf/single-node-custom-source.properties --name a1 --conf ./conf/ -Dflume.root.logger=INFO,console On Fri, Jun 21, 2013 at 1:50 PM, Nickolay Kolev <[email protected]> wrote: > Hi Asish, > > I think I found the problem - there must be something wrong with the > configuration and the source is not created. Unfortunately I can not figure > what. > Did you test it with my configuration? > > Here is what I found in server log: > 2013-06-21 11:01:24,433 (conf-file-poller-0) [DEBUG - > org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:295)] > Starting validation of configuration for agent: agent1, > initial-configuration: AgentConfiguration[agent1] > SOURCES: {r1={ parameters:{port=9001, handler.nickname=json handler, > channels=ch1, type=org.apache.flume.source.http.HTTPSource, > handler=org.apache.flume.source.http.JSONHandler} } > } > CHANNELS: {ch1={ parameters:{transactionCapacity=1000, capacity=10000, > type=memory} }} > SINKS: {k1={ parameters:{type=file_roll, channel=ch1, > sink.directory=C:\Projects\nickyk\apache-flume-1.3.1-bin\log} }} > > 2013-06-21 11:01:24,451 (conf-file-poller-0) [DEBUG - > org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateChannels(FlumeConfiguration.java:450)] > Created channel ch1 > > 2013-06-21 11:01:24,451 (conf-file-poller-0) [WARN - > org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSources(FlumeConfiguration.java:489)] > Agent configuration for 'agent1' has no sources. > 2013-06-21 11:01:24,455 (conf-file-poller-0) [DEBUG - > org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:655)] > Creating sink: k1 using FILE_ROLL > 2013-06-21 11:01:24,457 (conf-file-poller-0) [DEBUG - > org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:353)] > Post validation configuration for agent1 > AgentConfiguration created without Configuration stubs for which only > basic syntactical validation was performed[agent1] > SOURCES: {r1={ parameters:{port=9001, handler.nickname=json handler, > channels=ch1, type=org.apache.flume.source.http.HTTPSource, > handler=org.apache.flume.source.http.JSONHandler} } > } > CHANNELS: {ch1={ parameters:{transactionCapacity=1000, capacity=10000, > type=memory} }} > SINKS: {k1={ parameters:{type=file_roll, channel=ch1, > sink.directory=C:\Projects\nickyk\apache-flume-1.3.1-bin\log} }} > > 2013-06-21 11:01:24,470 (conf-file-poller-0) [DEBUG - > org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:117)] > Channels:ch1 > > 2013-06-21 11:01:24,471 (conf-file-poller-0) [DEBUG - > org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:118)] > Sinks k1 > > 2013-06-21 11:01:24,471 (conf-file-poller-0) [DEBUG - > org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:119)] > Sources null > > best regards, > nickolay kolev > > > > > On Fri, Jun 21, 2013 at 10:40 AM, Ashish <[email protected]> wrote: > >> I am using Flume 1.3.1 and ran the example before sending. >> >> I don't have access to a windows system at the moment so can't test. >> >> This is output at my end on Flume Agent console >> >> 2013-06-21 13:03:54,913 (conf-file-poller-0) [INFO - >> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:152)] >> Starting Source src1 >> 2013-06-21 13:03:54,927 (lifecycleSupervisor-1-2) [INFO - >> org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] Logging to >> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via >> org.mortbay.log.Slf4jLog >> 2013-06-21 13:03:54,958 (lifecycleSupervisor-1-2) [INFO - >> org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] jetty-6.1.26 >> 2013-06-21 13:03:55,022 (lifecycleSupervisor-1-2) [INFO - >> org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] Started >> [email protected]:9001 >> 2013-06-21 13:09:36,992 (SinkRunner-PollingRunner-DefaultSinkProcessor) >> [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] >> Event: { headers:{timestamp=434324343, host=random_host.example.co >> m} body: 72 61 6E 64 6F 6D 5F 62 6F 64 79 random_body } >> 2013-06-21 13:09:36,992 (SinkRunner-PollingRunner-DefaultSinkProcessor) >> [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] >> Event: { headers:{namenode=namenode.example.com, datanode= >> random_datanode.example.com} body: 72 65 61 6C 6C 79 5F 72 61 6E 64 6F >> 6D 5F 62 6F really_random_bo } >> >> Flume dev's can comment further on this. >> >> >> On Fri, Jun 21, 2013 at 12:33 PM, Nickolay Kolev <[email protected]>wrote: >> >>> Hi Ashish, >>> >>> Thanks for pointing me that error. I am trying to read the code and this >>> is the correct full class name. (last time I wrote java code was in 1998 >>> and my knowledge are a lot out of date) >>> >>> Unfortunately the result is the same. >>> >curl -X POST -H "Content-Type: application/json" -d "[{ \"headers\" : { >>> \"timestamp\" : \"434324343\", \"host\" : \"random_host.example.com\" >>> }, \"body\" : \"random_body\" }, { \"headers\" : { \"namenode\" : \" >>> namenode.example.com\", \"datanode\" : \"random_datanode.example.com\" >>> }, \"body\" : \"really_random_body\"}]" http://localhost:9001 >>> >>> curl: (7) Failed connect to localhost:9001; No error >>> >>> Could be the reason if this errors the fact that I use Windows? I see in >>> wiki that Windows is not officially supported. >>> >>> Yesterday my first test was with netcat source and it worked as expected. >>> >>> >>> best regards, >>> nickolay kolev >>> >>> >>> On Fri, Jun 21, 2013 at 9:48 AM, Ashish <[email protected]> wrote: >>> >>>> update the handler config as >>>> >>>> agent1.sources.r1.handler = org.apache.flume.source.http.JSONHandler >>>> >>>> and try. I used your source config and it worked for me. Let us know if >>>> it work >>>> >>>> >>>> On Fri, Jun 21, 2013 at 12:04 PM, Nickolay Kolev <[email protected]>wrote: >>>> >>>>> Hi Hari and the all members of this list, >>>>> >>>>> Thanks for pointing me that error. This is actually my second attempt >>>>> to use Apache Flume - with avro client. The symptoms are the same - the is >>>>> no listener on the specified port and I am not able to connect. >>>>> >>>>> I started to evaluate Flume just yesterday and many things are new to >>>>> me and probably I do many mistakes. I want to point that I use Flume 1.3.1 >>>>> compiled by me if this is important to be mentioned. I have tested too >>>>> with >>>>> the official binary from Flume web site but the results are the same. >>>>> >>>>> Here is what I do to test HTTP source: >>>>> >>>>> my config file (httppost.conf): >>>>> agent1.sources = r1 >>>>> agent1.channels = ch1 >>>>> agent1.sinks = k1 >>>>> >>>>> agent1.sources.r1.type = org.apache.flume.source.http.HTTPSource >>>>> agent1.sources.r1.port = 9001 >>>>> agent1.sources.r1.channels = ch1 >>>>> #agent1.sources.r1.handler = org.example.rest.RestHandler >>>>> agent1.sources.r1.handler = org.apache.flume.http.JSONHandler >>>>> agent1.sources.r1.handler.nickname = json handler >>>>> >>>>> agent1.sinks.k1.type = file_roll >>>>> agent1.sinks.k1.channel = ch1 >>>>> >>>>> agent1.sinks.k1.sink.directory = >>>>> C:\\Projects\\nickyk\\apache-flume-1.3.1-bin\\log >>>>> >>>>> agent1.channels.ch1.type = memory >>>>> agent1.channels.ch1.capacity = 10000 >>>>> agent1.channels.ch1.transactionCapacity = 1000 >>>>> >>>>> my server bat file: >>>>> >>>>> set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin\ >>>>> set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe" >>>>> set JAVA_OPTS=-Xmx60m >>>>> set CONF=%FLUME_HOME%\conf\httppost.conf >>>>> set AGENT=agent1 >>>>> >>>>> %JAVA% %JAVA_OPTS% >>>>> -Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties -cp >>>>> "%FLUME_HOME%\lib\*" org.apache.flume.node.Application -f %CONF% -n >>>>> %AGENT% >>>>> >>>>> How I test: >>>>> >>>>> server console: >>>>> >"C:\Java\jdk1.6.0_45\bin\java.exe" -Xmx60m >>>>> -Dlog4j.configuration=file:///C:\Projects\nickyk\apache-flume-1.3.1-bin\\conf\log4j.properties >>>>> -cp "C:\Projects\nickyk\apache-flume-1.3.1-bin\\lib\*" >>>>> org.apache.flume.node.Application -f >>>>> C:\Projects\nickyk\apache-flume-1.3.1-bin\\conf\httppost.conf -n agent1 >>>>> >>>>> client console: >>>>> >C:\Projects\nickyk\apache-flume-1.3.1-bin>curl -X POST -H >>>>> "Content-Type: application/json" -d "[{ \"headers\" : { \"timestamp\" : >>>>> \"434324343\", \"host\" : \"random_host.example.co >>>>> m\" }, \"body\" : \"random_body\" }, { \"headers\" : { \"namenode\" : >>>>> \"namenode.example.com\", \"datanode\" : \"random_datanode.example.com\" >>>>> }, \"body\" : \"really_random_body\" >>>>> }]" http://localhost:9001 >>>>> curl: (7) Failed connect to localhost:9001; No error >>>>> >>>>> If I understand correct the documentation I should expect that on the >>>>> specified port (9001) it must be listener that will accept POST requests? >>>>> Is this correct? >>>>> Do I miss some important aspects that are critical to run Flume? >>>>> >>>>> Thanks in advance for your help. >>>>> >>>>> >>>>> best regards, >>>>> nickolay kolev >>>>> >>>>> >>>>> >>>>> On Thu, Jun 20, 2013 at 9:07 PM, Hari Shreedharan < >>>>> [email protected]> wrote: >>>>> >>>>>> Does not look like you are using the http source at all. Your source >>>>>> type needs to be HTTP >>>>>> >>>>>> >>>>>> Cheers, >>>>>> Hari >>>>>> >>>>>> On Thursday, June 20, 2013 at 8:57 AM, Nickolay Kolev wrote: >>>>>> >>>>>> Hi all, >>>>>> I am new to flume and all that logging stuff and probably many things >>>>>> are unclear to me despite I read the docs. >>>>>> I want to run Flume on windows server to create proof of concept >>>>>> solution. I was able to compile it successfully. I was able to make >>>>>> successful test with netcat. >>>>>> Next test that i wanted to do is with http source but unfortunately >>>>>> it did not pass. >>>>>> Here is what I do: >>>>>> >>>>>> my server config (httppost.conf): >>>>>> agent1.sources = r1 >>>>>> agent1.channels = memoryChannel >>>>>> agent1.sinks = k1 >>>>>> >>>>>> agent1.sources.r1.type = avro >>>>>> agent1.sources.r1.bind = localhost >>>>>> agent1.sources.r1.port = 9003 >>>>>> >>>>>> agent1.sinks.k1.type = file_roll >>>>>> agent1.sinks.k1.sink.directory = >>>>>> C:\\Projects\\nickyk\\apache-flume-1.3.1-bin\\log >>>>>> >>>>>> agent1.channels.memoryChannel.type = memory >>>>>> agent1.channels.memoryChannel.capacity = 10000 >>>>>> agent1.channels.memoryChannel.transactionCapacity = 1000 >>>>>> >>>>>> >>>>>> agent1.sources.r1.channels = memoryChannel >>>>>> agent1.sinks.k1.channel = memoryChannel >>>>>> >>>>>> I run the server with this bat file: >>>>>> set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin\ >>>>>> set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe" >>>>>> set JAVA_OPTS=-Xmx60m >>>>>> set CONF=%FLUME_HOME%\conf\httppost.conf >>>>>> set AGENT=agent1 >>>>>> >>>>>> %JAVA% %JAVA_OPTS% >>>>>> -Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties -cp >>>>>> "%FLUME_HOME%\lib\*" org.apache.flume.node.Application -f %CONF% -n >>>>>> %AGENT% >>>>>> >>>>>> netstat does not show any listener on port 9003? Is this normal or >>>>>> there must be a listener? >>>>>> >>>>>> I run avro client with this bat file: >>>>>> set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin >>>>>> set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe" >>>>>> set JAVA_OPTS=-Xmx60m >>>>>> set AVRO_HOST=localhost >>>>>> set AVRO_PORT=9003 >>>>>> set SENDFILE=%FLUME_HOME%\logs\data.log >>>>>> set HEADFILE=%FLUME_HOME%\conf\header.txt >>>>>> >>>>>> %JAVA% %JAVA_OPTS% >>>>>> -Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties -cp >>>>>> "%FLUME_HOME%\lib\*" org.apache.flume.client.avro.AvroCLIClient -H >>>>>> %AVRO_HOST% -p %AVRO_PORT% -F %SENDFILE% --headerFile %HEADFILE% >>>>>> >>>>>> >>>>>> Then I get this output: >>>>>> C:\Projects\nickyk\apache-flume-1.3.1-bin>"C:\Java\jdk1.6.0_45\bin\java.exe" >>>>>> -Xmx60m >>>>>> -Dlog4j.configuration=file:///C:\Projects\nickyk\apache-flume-1.3.1-bin\conf\log4j.properties >>>>>> -cp "C:\Projects\nickyk\apache-flume-1.3.1-bin\lib\*" >>>>>> org.apache.flume.client.avro.AvroCLIClient -H localhost -p 9003 -F >>>>>> C:\Projects\nickyk\apache-flume-1.3.1-bin\logs\data.log --headerFile >>>>>> C:\Projects\nickyk\apache-flume-1.3.1-bin\conf\header.txt >>>>>> 2013-06-20 17:55:58,285 (main) [DEBUG - >>>>>> org.apache.flume.client.avro.AvroCLIClient.parseHeaders(AvroCLIClient.java:101)] >>>>>> Inserting Header Key [filename] header value [datalog] >>>>>> 2013-06-20 17:55:58,287 (main) [DEBUG - >>>>>> org.apache.flume.client.avro.AvroCLIClient.parseHeaders(AvroCLIClient.java:101)] >>>>>> Inserting Header Key [host] header value [zhouhh] >>>>>> 2013-06-20 17:55:58,295 (main) [DEBUG - >>>>>> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:420)] >>>>>> Batch size string = 5 >>>>>> 2013-06-20 17:55:59,358 (main) [ERROR - >>>>>> org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:79)] >>>>>> Unable to open connection to Flume. Exception follows. >>>>>> org.apache.flume.FlumeException: NettyAvroRpcClient { host: >>>>>> localhost, port: 9003 }: RPC connection error >>>>>> at >>>>>> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117) >>>>>> at >>>>>> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93) >>>>>> at >>>>>> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507) >>>>>> at >>>>>> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169) >>>>>> at >>>>>> org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180) >>>>>> at >>>>>> org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71) >>>>>> Caused by: java.io.IOException: Error connecting to localhost/ >>>>>> 127.0.0.1:9003 >>>>>> at >>>>>> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261) >>>>>> at >>>>>> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203) >>>>>> at >>>>>> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152) >>>>>> at >>>>>> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106) >>>>>> ... 5 more >>>>>> Caused by: java.net.ConnectException: Connection refused: no further >>>>>> information >>>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >>>>>> at >>>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599) >>>>>> at >>>>>> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396) >>>>>> at >>>>>> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358) >>>>>> at >>>>>> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274) >>>>>> at >>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) >>>>>> at java.lang.Thread.run(Thread.java:662) >>>>>> 2013-06-20 17:55:59,369 (main) [DEBUG - >>>>>> org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:84)] >>>>>> Exiting >>>>>> >>>>>> >>>>>> My question is what I am doing wrong and what I need to test in order >>>>>> to fix this situatioin. >>>>>> >>>>>> Thanks in advance. >>>>>> >>>>>> best regards, >>>>>> >>>>>> >>>>>> >>>>> >>>> >>>> >>>> -- >>>> thanks >>>> ashish >>>> >>>> Blog: http://www.ashishpaliwal.com/blog >>>> My Photo Galleries: http://www.pbase.com/ashishpaliwal >>>> >>> >>> >> >> >> -- >> thanks >> ashish >> >> Blog: http://www.ashishpaliwal.com/blog >> My Photo Galleries: http://www.pbase.com/ashishpaliwal >> > > -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
