Hi Ashish, Thanks for the support. I will continue with this issue. Hope I will find solution or at least confirmation that this is windows specific problem. Have a nice day
best regards, nickolay kolev On Fri, Jun 21, 2013 at 10:40 AM, Ashish <[email protected]> wrote: > I am using Flume 1.3.1 and ran the example before sending. > > I don't have access to a windows system at the moment so can't test. > > This is output at my end on Flume Agent console > > 2013-06-21 13:03:54,913 (conf-file-poller-0) [INFO - > org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:152)] > Starting Source src1 > 2013-06-21 13:03:54,927 (lifecycleSupervisor-1-2) [INFO - > org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] Logging to > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via > org.mortbay.log.Slf4jLog > 2013-06-21 13:03:54,958 (lifecycleSupervisor-1-2) [INFO - > org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] jetty-6.1.26 > 2013-06-21 13:03:55,022 (lifecycleSupervisor-1-2) [INFO - > org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] Started > [email protected]:9001 > 2013-06-21 13:09:36,992 (SinkRunner-PollingRunner-DefaultSinkProcessor) > [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] > Event: { headers:{timestamp=434324343, host=random_host.example.co > m} body: 72 61 6E 64 6F 6D 5F 62 6F 64 79 random_body } > 2013-06-21 13:09:36,992 (SinkRunner-PollingRunner-DefaultSinkProcessor) > [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] > Event: { headers:{namenode=namenode.example.com, datanode= > random_datanode.example.com} body: 72 65 61 6C 6C 79 5F 72 61 6E 64 6F 6D > 5F 62 6F really_random_bo } > > Flume dev's can comment further on this. > > > On Fri, Jun 21, 2013 at 12:33 PM, Nickolay Kolev <[email protected]>wrote: > >> Hi Ashish, >> >> Thanks for pointing me that error. I am trying to read the code and this >> is the correct full class name. (last time I wrote java code was in 1998 >> and my knowledge are a lot out of date) >> >> Unfortunately the result is the same. >> >curl -X POST -H "Content-Type: application/json" -d "[{ \"headers\" : { >> \"timestamp\" : \"434324343\", \"host\" : \"random_host.example.com\" }, >> \"body\" : \"random_body\" }, { \"headers\" : { \"namenode\" : \" >> namenode.example.com\", \"datanode\" : \"random_datanode.example.com\" >> }, \"body\" : \"really_random_body\"}]" http://localhost:9001 >> >> curl: (7) Failed connect to localhost:9001; No error >> >> Could be the reason if this errors the fact that I use Windows? I see in >> wiki that Windows is not officially supported. >> >> Yesterday my first test was with netcat source and it worked as expected. >> >> >> best regards, >> nickolay kolev >> >> >> On Fri, Jun 21, 2013 at 9:48 AM, Ashish <[email protected]> wrote: >> >>> update the handler config as >>> >>> agent1.sources.r1.handler = org.apache.flume.source.http.JSONHandler >>> >>> and try. I used your source config and it worked for me. Let us know if >>> it work >>> >>> >>> On Fri, Jun 21, 2013 at 12:04 PM, Nickolay Kolev <[email protected]>wrote: >>> >>>> Hi Hari and the all members of this list, >>>> >>>> Thanks for pointing me that error. This is actually my second attempt >>>> to use Apache Flume - with avro client. The symptoms are the same - the is >>>> no listener on the specified port and I am not able to connect. >>>> >>>> I started to evaluate Flume just yesterday and many things are new to >>>> me and probably I do many mistakes. I want to point that I use Flume 1.3.1 >>>> compiled by me if this is important to be mentioned. I have tested too with >>>> the official binary from Flume web site but the results are the same. >>>> >>>> Here is what I do to test HTTP source: >>>> >>>> my config file (httppost.conf): >>>> agent1.sources = r1 >>>> agent1.channels = ch1 >>>> agent1.sinks = k1 >>>> >>>> agent1.sources.r1.type = org.apache.flume.source.http.HTTPSource >>>> agent1.sources.r1.port = 9001 >>>> agent1.sources.r1.channels = ch1 >>>> #agent1.sources.r1.handler = org.example.rest.RestHandler >>>> agent1.sources.r1.handler = org.apache.flume.http.JSONHandler >>>> agent1.sources.r1.handler.nickname = json handler >>>> >>>> agent1.sinks.k1.type = file_roll >>>> agent1.sinks.k1.channel = ch1 >>>> >>>> agent1.sinks.k1.sink.directory = >>>> C:\\Projects\\nickyk\\apache-flume-1.3.1-bin\\log >>>> >>>> agent1.channels.ch1.type = memory >>>> agent1.channels.ch1.capacity = 10000 >>>> agent1.channels.ch1.transactionCapacity = 1000 >>>> >>>> my server bat file: >>>> >>>> set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin\ >>>> set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe" >>>> set JAVA_OPTS=-Xmx60m >>>> set CONF=%FLUME_HOME%\conf\httppost.conf >>>> set AGENT=agent1 >>>> >>>> %JAVA% %JAVA_OPTS% >>>> -Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties -cp >>>> "%FLUME_HOME%\lib\*" org.apache.flume.node.Application -f %CONF% -n %AGENT% >>>> >>>> How I test: >>>> >>>> server console: >>>> >"C:\Java\jdk1.6.0_45\bin\java.exe" -Xmx60m >>>> -Dlog4j.configuration=file:///C:\Projects\nickyk\apache-flume-1.3.1-bin\\conf\log4j.properties >>>> -cp "C:\Projects\nickyk\apache-flume-1.3.1-bin\\lib\*" >>>> org.apache.flume.node.Application -f >>>> C:\Projects\nickyk\apache-flume-1.3.1-bin\\conf\httppost.conf -n agent1 >>>> >>>> client console: >>>> >C:\Projects\nickyk\apache-flume-1.3.1-bin>curl -X POST -H >>>> "Content-Type: application/json" -d "[{ \"headers\" : { \"timestamp\" : >>>> \"434324343\", \"host\" : \"random_host.example.co >>>> m\" }, \"body\" : \"random_body\" }, { \"headers\" : { \"namenode\" : \" >>>> namenode.example.com\", \"datanode\" : \"random_datanode.example.com\" >>>> }, \"body\" : \"really_random_body\" >>>> }]" http://localhost:9001 >>>> curl: (7) Failed connect to localhost:9001; No error >>>> >>>> If I understand correct the documentation I should expect that on the >>>> specified port (9001) it must be listener that will accept POST requests? >>>> Is this correct? >>>> Do I miss some important aspects that are critical to run Flume? >>>> >>>> Thanks in advance for your help. >>>> >>>> >>>> best regards, >>>> nickolay kolev >>>> >>>> >>>> >>>> On Thu, Jun 20, 2013 at 9:07 PM, Hari Shreedharan < >>>> [email protected]> wrote: >>>> >>>>> Does not look like you are using the http source at all. Your source >>>>> type needs to be HTTP >>>>> >>>>> >>>>> Cheers, >>>>> Hari >>>>> >>>>> On Thursday, June 20, 2013 at 8:57 AM, Nickolay Kolev wrote: >>>>> >>>>> Hi all, >>>>> I am new to flume and all that logging stuff and probably many things >>>>> are unclear to me despite I read the docs. >>>>> I want to run Flume on windows server to create proof of concept >>>>> solution. I was able to compile it successfully. I was able to make >>>>> successful test with netcat. >>>>> Next test that i wanted to do is with http source but unfortunately it >>>>> did not pass. >>>>> Here is what I do: >>>>> >>>>> my server config (httppost.conf): >>>>> agent1.sources = r1 >>>>> agent1.channels = memoryChannel >>>>> agent1.sinks = k1 >>>>> >>>>> agent1.sources.r1.type = avro >>>>> agent1.sources.r1.bind = localhost >>>>> agent1.sources.r1.port = 9003 >>>>> >>>>> agent1.sinks.k1.type = file_roll >>>>> agent1.sinks.k1.sink.directory = >>>>> C:\\Projects\\nickyk\\apache-flume-1.3.1-bin\\log >>>>> >>>>> agent1.channels.memoryChannel.type = memory >>>>> agent1.channels.memoryChannel.capacity = 10000 >>>>> agent1.channels.memoryChannel.transactionCapacity = 1000 >>>>> >>>>> >>>>> agent1.sources.r1.channels = memoryChannel >>>>> agent1.sinks.k1.channel = memoryChannel >>>>> >>>>> I run the server with this bat file: >>>>> set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin\ >>>>> set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe" >>>>> set JAVA_OPTS=-Xmx60m >>>>> set CONF=%FLUME_HOME%\conf\httppost.conf >>>>> set AGENT=agent1 >>>>> >>>>> %JAVA% %JAVA_OPTS% >>>>> -Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties -cp >>>>> "%FLUME_HOME%\lib\*" org.apache.flume.node.Application -f %CONF% -n >>>>> %AGENT% >>>>> >>>>> netstat does not show any listener on port 9003? Is this normal or >>>>> there must be a listener? >>>>> >>>>> I run avro client with this bat file: >>>>> set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin >>>>> set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe" >>>>> set JAVA_OPTS=-Xmx60m >>>>> set AVRO_HOST=localhost >>>>> set AVRO_PORT=9003 >>>>> set SENDFILE=%FLUME_HOME%\logs\data.log >>>>> set HEADFILE=%FLUME_HOME%\conf\header.txt >>>>> >>>>> %JAVA% %JAVA_OPTS% >>>>> -Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties -cp >>>>> "%FLUME_HOME%\lib\*" org.apache.flume.client.avro.AvroCLIClient -H >>>>> %AVRO_HOST% -p %AVRO_PORT% -F %SENDFILE% --headerFile %HEADFILE% >>>>> >>>>> >>>>> Then I get this output: >>>>> C:\Projects\nickyk\apache-flume-1.3.1-bin>"C:\Java\jdk1.6.0_45\bin\java.exe" >>>>> -Xmx60m >>>>> -Dlog4j.configuration=file:///C:\Projects\nickyk\apache-flume-1.3.1-bin\conf\log4j.properties >>>>> -cp "C:\Projects\nickyk\apache-flume-1.3.1-bin\lib\*" >>>>> org.apache.flume.client.avro.AvroCLIClient -H localhost -p 9003 -F >>>>> C:\Projects\nickyk\apache-flume-1.3.1-bin\logs\data.log --headerFile >>>>> C:\Projects\nickyk\apache-flume-1.3.1-bin\conf\header.txt >>>>> 2013-06-20 17:55:58,285 (main) [DEBUG - >>>>> org.apache.flume.client.avro.AvroCLIClient.parseHeaders(AvroCLIClient.java:101)] >>>>> Inserting Header Key [filename] header value [datalog] >>>>> 2013-06-20 17:55:58,287 (main) [DEBUG - >>>>> org.apache.flume.client.avro.AvroCLIClient.parseHeaders(AvroCLIClient.java:101)] >>>>> Inserting Header Key [host] header value [zhouhh] >>>>> 2013-06-20 17:55:58,295 (main) [DEBUG - >>>>> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:420)] >>>>> Batch size string = 5 >>>>> 2013-06-20 17:55:59,358 (main) [ERROR - >>>>> org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:79)] >>>>> Unable to open connection to Flume. Exception follows. >>>>> org.apache.flume.FlumeException: NettyAvroRpcClient { host: localhost, >>>>> port: 9003 }: RPC connection error >>>>> at >>>>> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117) >>>>> at >>>>> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93) >>>>> at >>>>> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507) >>>>> at >>>>> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169) >>>>> at >>>>> org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180) >>>>> at >>>>> org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71) >>>>> Caused by: java.io.IOException: Error connecting to localhost/ >>>>> 127.0.0.1:9003 >>>>> at >>>>> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261) >>>>> at >>>>> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203) >>>>> at >>>>> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152) >>>>> at >>>>> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106) >>>>> ... 5 more >>>>> Caused by: java.net.ConnectException: Connection refused: no further >>>>> information >>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >>>>> at >>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274) >>>>> at >>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> 2013-06-20 17:55:59,369 (main) [DEBUG - >>>>> org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:84)] >>>>> Exiting >>>>> >>>>> >>>>> My question is what I am doing wrong and what I need to test in order >>>>> to fix this situatioin. >>>>> >>>>> Thanks in advance. >>>>> >>>>> best regards, >>>>> >>>>> >>>>> >>>> >>> >>> >>> -- >>> thanks >>> ashish >>> >>> Blog: http://www.ashishpaliwal.com/blog >>> My Photo Galleries: http://www.pbase.com/ashishpaliwal >>> >> >> > > > -- > thanks > ashish > > Blog: http://www.ashishpaliwal.com/blog > My Photo Galleries: http://www.pbase.com/ashishpaliwal >
