Re: Flume-NG Python avro Client

[Brock Noland]

The root cause here appears to be that the Python library has only an
HTTPTransciever and the FlumeSource is expecting data in
"NettyTransciever" format.

On Mon, Jul 23, 2012 at 7:00 AM,  <dinesh.simkh...@gridcore.se> wrote:
> Hello Mubarak,
> Thanks for your quick reply, is there any workaround for sending data to Avro 
> source using python ?
>
> Thanks
> simk
>
> -----Mubarak Seyed <se...@apple.com> wrote: -----
> To: user@flume.apache.org
> From: Mubarak Seyed <se...@apple.com>
> Date: 07/23/2012 01:19PM
> Subject: Re: Flume-NG Python avro Client
>
> Hi,
>
> This is a known issue in Avro. Please refer 
> https://issues.apache.org/jira/browse/AVRO-1111
>
>
>
> Thanks.
> -Mubarak
>
>
>
>
> On Jul 23, 2012, at 4:15 AM, dinesh.simkh...@gridcore.se wrote:
> Hi,
> I am trying to build a flume client in python that send message to Avro 
> source but always got 'java.lang.OutOfMemoryError: Java heap space'  error. I 
> tried same thing with java it works properly
> I have attached my source and error log, any help/suggestions exports out 
> there ?
>
> ---------- Source
> import sys
> import httplib
> import avro.ipc as ipc
> import avro.protocol as protocol
>
> server_addr = ('192.168.104.2', 3452)
> PROTOCOL = protocol.parse(open("flume.avpr").read())
>
> def sendData():
>     client = ipc.HTTPTransceiver(server_addr[0], server_addr[1])
>     requestor = ipc.Requestor(PROTOCOL, client)
>
>     event = dict()
>     event['headers'] = {'name': 'abc', 'address': 'zyx'}
>     event['body'] = bytes('hello')
>
>     params = dict()
>     params['event'] = event
>     print("Result : " + requestor.request('append', params))
>
>     client.close()
>
>
> if __name__ == '__main__':
>     sendData()
>
>
> ----------- error msg
> 2012-07-23 10:57:54,903 WARN ipc.NettyServer: Unexpected exception from 
> downstream.
> java.lang.OutOfMemoryError: Java heap space
>     at java.util.ArrayList.<init>(ArrayList.java:112)
>     at 
> org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decodePackHeader(NettyTransportCodec.java:154)
>     at 
> org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decode(NettyTransportCodec.java:131)
>     at 
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:282)
>     at 
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)
>     at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
>     at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
>     at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
>     at 
> org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
>     at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
>     at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     at java.lang.Thread.run(Thread.java:662)
>
>
>



-- 
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/