update the handler config as agent1.sources.r1.handler = org.apache.flume.source.http.JSONHandler
and try. I used your source config and it worked for me. Let us know if it work On Fri, Jun 21, 2013 at 12:04 PM, Nickolay Kolev <[email protected]> wrote: > Hi Hari and the all members of this list, > > Thanks for pointing me that error. This is actually my second attempt to > use Apache Flume - with avro client. The symptoms are the same - the is no > listener on the specified port and I am not able to connect. > > I started to evaluate Flume just yesterday and many things are new to me > and probably I do many mistakes. I want to point that I use Flume 1.3.1 > compiled by me if this is important to be mentioned. I have tested too with > the official binary from Flume web site but the results are the same. > > Here is what I do to test HTTP source: > > my config file (httppost.conf): > agent1.sources = r1 > agent1.channels = ch1 > agent1.sinks = k1 > > agent1.sources.r1.type = org.apache.flume.source.http.HTTPSource > agent1.sources.r1.port = 9001 > agent1.sources.r1.channels = ch1 > #agent1.sources.r1.handler = org.example.rest.RestHandler > agent1.sources.r1.handler = org.apache.flume.http.JSONHandler > agent1.sources.r1.handler.nickname = json handler > > agent1.sinks.k1.type = file_roll > agent1.sinks.k1.channel = ch1 > > agent1.sinks.k1.sink.directory = > C:\\Projects\\nickyk\\apache-flume-1.3.1-bin\\log > > agent1.channels.ch1.type = memory > agent1.channels.ch1.capacity = 10000 > agent1.channels.ch1.transactionCapacity = 1000 > > my server bat file: > > set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin\ > set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe" > set JAVA_OPTS=-Xmx60m > set CONF=%FLUME_HOME%\conf\httppost.conf > set AGENT=agent1 > > %JAVA% %JAVA_OPTS% > -Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties -cp > "%FLUME_HOME%\lib\*" org.apache.flume.node.Application -f %CONF% -n %AGENT% > > How I test: > > server console: > >"C:\Java\jdk1.6.0_45\bin\java.exe" -Xmx60m > -Dlog4j.configuration=file:///C:\Projects\nickyk\apache-flume-1.3.1-bin\\conf\log4j.properties > -cp "C:\Projects\nickyk\apache-flume-1.3.1-bin\\lib\*" > org.apache.flume.node.Application -f > C:\Projects\nickyk\apache-flume-1.3.1-bin\\conf\httppost.conf -n agent1 > > client console: > >C:\Projects\nickyk\apache-flume-1.3.1-bin>curl -X POST -H "Content-Type: > application/json" -d "[{ \"headers\" : { \"timestamp\" : \"434324343\", > \"host\" : \"random_host.example.co > m\" }, \"body\" : \"random_body\" }, { \"headers\" : { \"namenode\" : \" > namenode.example.com\", \"datanode\" : \"random_datanode.example.com\" }, > \"body\" : \"really_random_body\" > }]" http://localhost:9001 > curl: (7) Failed connect to localhost:9001; No error > > If I understand correct the documentation I should expect that on the > specified port (9001) it must be listener that will accept POST requests? > Is this correct? > Do I miss some important aspects that are critical to run Flume? > > Thanks in advance for your help. > > > best regards, > nickolay kolev > > > > On Thu, Jun 20, 2013 at 9:07 PM, Hari Shreedharan < > [email protected]> wrote: > >> Does not look like you are using the http source at all. Your source >> type needs to be HTTP >> >> >> Cheers, >> Hari >> >> On Thursday, June 20, 2013 at 8:57 AM, Nickolay Kolev wrote: >> >> Hi all, >> I am new to flume and all that logging stuff and probably many things are >> unclear to me despite I read the docs. >> I want to run Flume on windows server to create proof of concept >> solution. I was able to compile it successfully. I was able to make >> successful test with netcat. >> Next test that i wanted to do is with http source but unfortunately it >> did not pass. >> Here is what I do: >> >> my server config (httppost.conf): >> agent1.sources = r1 >> agent1.channels = memoryChannel >> agent1.sinks = k1 >> >> agent1.sources.r1.type = avro >> agent1.sources.r1.bind = localhost >> agent1.sources.r1.port = 9003 >> >> agent1.sinks.k1.type = file_roll >> agent1.sinks.k1.sink.directory = >> C:\\Projects\\nickyk\\apache-flume-1.3.1-bin\\log >> >> agent1.channels.memoryChannel.type = memory >> agent1.channels.memoryChannel.capacity = 10000 >> agent1.channels.memoryChannel.transactionCapacity = 1000 >> >> >> agent1.sources.r1.channels = memoryChannel >> agent1.sinks.k1.channel = memoryChannel >> >> I run the server with this bat file: >> set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin\ >> set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe" >> set JAVA_OPTS=-Xmx60m >> set CONF=%FLUME_HOME%\conf\httppost.conf >> set AGENT=agent1 >> >> %JAVA% %JAVA_OPTS% >> -Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties -cp >> "%FLUME_HOME%\lib\*" org.apache.flume.node.Application -f %CONF% -n %AGENT% >> >> netstat does not show any listener on port 9003? Is this normal or there >> must be a listener? >> >> I run avro client with this bat file: >> set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin >> set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe" >> set JAVA_OPTS=-Xmx60m >> set AVRO_HOST=localhost >> set AVRO_PORT=9003 >> set SENDFILE=%FLUME_HOME%\logs\data.log >> set HEADFILE=%FLUME_HOME%\conf\header.txt >> >> %JAVA% %JAVA_OPTS% >> -Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties -cp >> "%FLUME_HOME%\lib\*" org.apache.flume.client.avro.AvroCLIClient -H >> %AVRO_HOST% -p %AVRO_PORT% -F %SENDFILE% --headerFile %HEADFILE% >> >> >> Then I get this output: >> C:\Projects\nickyk\apache-flume-1.3.1-bin>"C:\Java\jdk1.6.0_45\bin\java.exe" >> -Xmx60m >> -Dlog4j.configuration=file:///C:\Projects\nickyk\apache-flume-1.3.1-bin\conf\log4j.properties >> -cp "C:\Projects\nickyk\apache-flume-1.3.1-bin\lib\*" >> org.apache.flume.client.avro.AvroCLIClient -H localhost -p 9003 -F >> C:\Projects\nickyk\apache-flume-1.3.1-bin\logs\data.log --headerFile >> C:\Projects\nickyk\apache-flume-1.3.1-bin\conf\header.txt >> 2013-06-20 17:55:58,285 (main) [DEBUG - >> org.apache.flume.client.avro.AvroCLIClient.parseHeaders(AvroCLIClient.java:101)] >> Inserting Header Key [filename] header value [datalog] >> 2013-06-20 17:55:58,287 (main) [DEBUG - >> org.apache.flume.client.avro.AvroCLIClient.parseHeaders(AvroCLIClient.java:101)] >> Inserting Header Key [host] header value [zhouhh] >> 2013-06-20 17:55:58,295 (main) [DEBUG - >> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:420)] >> Batch size string = 5 >> 2013-06-20 17:55:59,358 (main) [ERROR - >> org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:79)] >> Unable to open connection to Flume. Exception follows. >> org.apache.flume.FlumeException: NettyAvroRpcClient { host: localhost, >> port: 9003 }: RPC connection error >> at >> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117) >> at >> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93) >> at >> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507) >> at >> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169) >> at >> org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180) >> at >> org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71) >> Caused by: java.io.IOException: Error connecting to localhost/ >> 127.0.0.1:9003 >> at >> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261) >> at >> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203) >> at >> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152) >> at >> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106) >> ... 5 more >> Caused by: java.net.ConnectException: Connection refused: no further >> information >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599) >> at >> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396) >> at >> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358) >> at >> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274) >> at >> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) >> at java.lang.Thread.run(Thread.java:662) >> 2013-06-20 17:55:59,369 (main) [DEBUG - >> org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:84)] >> Exiting >> >> >> My question is what I am doing wrong and what I need to test in order to >> fix this situatioin. >> >> Thanks in advance. >> >> best regards, >> >> >> > -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
