Hi Stefan, Thanks for the response (and apologies for the delay). The Thrift workaround (as you say, introduced in 1.4) is an interesting workaround. We are primarily a Java shop, but some of our tests are written in Python, which I was using for some quick exploration. If I get ambitious enough I may try Jython<->Python communication (since I don't want to use Flume for everything), and post back the results.
Steve On Mon, Jun 24, 2013 at 8:30 PM, Stefan Krawczyk <[email protected]>wrote: > Hi Steve, > > I'm new to the Avro community as well, but it just seems that nobody > really cares about Avro support outside of Java/C++... > > The issue you're seeing is because the python & ruby implementations are > using the HTTPTransceiver, whereas the Java server is using the > NettyTransceiver. They dont' talk the same language. I'm as surprised as > you are that such a change would be introduced without the other languages > having support (or providing documentation mentioning that they don't work). > > I asked this list earlier what it would take to get python to talk netty > but nobody responded... > > We're also using Flume, and the only reasonable option is to use Thrift > RPC & the Thrift Source (look at 1.4 branch of code) as the ingestion point > from our app. From there the rest of the flume pipeline can use Avro RPC. > The other option is to use the legacy avro source, but that source is > effectively deprecated... > > Hope that helps. > > Cheers, > > Stefan > > > On Thu, Jun 20, 2013 at 10:01 AM, Stephen Abrams <[email protected] > > wrote: > >> Hello! >> >> New to the Avro world. I have been evaluating Flume and testing various >> capabilities. I have run into an issue with the Avro RPC piece. So far, I >> have found that the Flume Avro RPC java client works with a Flume Avro >> Source as expected (I believe these implementations use Netty). However, >> when I tried using a Ruby RPC client to communicate with the Flume Avro >> Source, there was a failure and the source reported: >> >> org.apache.avro.AvroRuntimeException: Excessively large list allocation >> request detected: 539959368 items! Connection closed. >> >> I then tried this with a Python RPC client and had a similar issue. I >> suspected client/server Avro interop issues, so then verified that a Python >> Avro RPC client and Python Avro RPC server communicate correctly (using >> https://github.com/phunt/avro-rpc-quickstart). Also verified that Ruby >> Avro RPC client and Ruby Avro RPC server communicate correctly. However, >> using the Python Avro RPC client with Ruby Avro RPC server fails (client >> hangs), and Ruby client with Python server also fails (again, client hangs). >> >> I know I'm not providing exhaustive info here, but I'm surprised to find >> out-of-the-box interop problems. Do folks regularly use these >> client/servers in various mixes (Java/Ruby/Python)? Are there particular >> gotchas or configurations that are not obvious? >> >> Thank you, >> Steve >> > >
