Agreed - I'm all for Thrift. Though, I actually, contrary to Ryan, think that the existing HBaseRPC handler/client code is pretty good -- better than the equivalents from Thrift Java.
We could start by using Thrift serialization on our existing transport -- then maybe work towards contributing it upstream to the Thrift project. HDFS folks are potentially interested in doing that as well. -Todd On Fri, May 27, 2011 at 1:10 PM, Ryan Rawson <[email protected]> wrote: > I'm -1 on avro as a RPC format. Thrift is the way to go, any of the > advantages of smaller serialization of avro is lost by the sheer > complexity of avro and therefore the potential bugs. > > I understand the desire to have a pluggable RPC engine, but it feels > like the better approach would be to adopt a unified RPC and just be > done with it. I had a look at the HsHa mechanism in thrift and it is > very good, it in fact matches our 'handler' approach - async > recieving/sending of data, but single threaded for processing a > message. > > -ryan > > On Fri, May 27, 2011 at 1:00 PM, Andrew Purtell <[email protected]> wrote: >> Also needing, perhaps later, consideration: >> >> - HDFS-347 or not >> >> - Lucene embedding for hbase-search, though as a coprocessor this is >> already pretty much handled if we have platform support (therefore a >> platform module) for a HDFS that can do local read shortcutting and block >> placement requests >> >> - HFile v1 versus v2 >> >> Making decoupled development at several downstream sites manageable, with a >> home upstream for all the work, while simultaneously providing clean >> migration paths for users, basically. >> >> --- On Fri, 5/27/11, Andrew Purtell <[email protected]> wrote: >> >>> From: Andrew Purtell <[email protected]> >>> Subject: modular build and pluggable rpc >>> To: [email protected] >>> Date: Friday, May 27, 2011, 12:49 PM >>> From IRC: >>> >>> apurtell i propose we take the build modular as early as possible to >>> deal with multiple platform targets >>> apurtell secure vs nonsecure >>> apurtell 0.20 vs 0.22 vs trunk >>> apurtell i understand the maintenence issues with multiple rpc engines, >>> for example, but a lot of reflection twistiness is going to be worse >>> apurtell i propose we take up esammer on his offer >>> apurtell so branch 0.92 asap, get trunk modular and working against >>> multiple platform targets >>> apurtell especially if we're going to see rpc changes coming from >>> downstream projects... >>> apurtell also what about supporting secure and nonsecure clients with >>> the same deployment? >>> apurtell zookeeper does this >>> apurtell so that is selectable rpc engine per connection, with a >>> negotiation >>> apurtell we don't have or want to be crazy about it but a rolling >>> upgrade should be possible if for example we are taking in a new rpc from >>> fb (?) or cloudera (avro based?) >>> apurtell also looks like hlog modules for 0.20 vs 0.22 and successors >>> apurtell i think over time we can roadmap the rpc engines, if we have >>> multiple, by deprecation >>> apurtell now that we're on the edge of supporting both 0.20 and 0.22, >>> and secure vs nonsecure, let's get it as manageable as possible right away >>> >>> St^Ack_ apurtell: +1 >>> >>> apurtell also i think there is some interest in async rpc engine >>> >>> St^Ack_ we should stick this up on dev i'd say >>> >>> Best regards, >>> >>> - Andy >>> >>> Problems worthy of attack prove their worth by hitting >>> back. - Piet Hein (via Tom White) >>> >> > -- Todd Lipcon Software Engineer, Cloudera
