I don't disagree with any of this but the fact is we have compile time differences if going against secure Hadoop 0.20 or non-secure Hadoop 0.20.
So either we decide to punt on integration with secure Hadoop 0.20 or we deal with the compile time differences. If dealing with them, we can do it by reflection, which is brittle and can be difficult to understand and debug, and someone would have to do this work; or we can wholesale replace RPC with something based on Thrift, and someone would have to do the work; or we take the pluggable RPC changes that Gary has already developed and modularize the build, which Eric has already volunteered to do. - Andy --- On Fri, 5/27/11, Todd Lipcon <[email protected]> wrote: > From: Todd Lipcon <[email protected]> > Subject: Re: modular build and pluggable rpc > To: [email protected] > Cc: [email protected] > Date: Friday, May 27, 2011, 1:30 PM > Agreed - I'm all for Thrift. > > Though, I actually, contrary to Ryan, think that the > existing HBaseRPC > handler/client code is pretty good -- better than the > equivalents from > Thrift Java. > > We could start by using Thrift serialization on our > existing transport > -- then maybe work towards contributing it upstream to the > Thrift > project. HDFS folks are potentially interested in doing > that as well. > > -Todd > > On Fri, May 27, 2011 at 1:10 PM, Ryan Rawson <[email protected]> > wrote: > > I'm -1 on avro as a RPC format. Thrift is the way to > go, any of the > > advantages of smaller serialization of avro is lost by > the sheer > > complexity of avro and therefore the potential bugs. > > > > I understand the desire to have a pluggable RPC > engine, but it feels > > like the better approach would be to adopt a unified > RPC and just be > > done with it. I had a look at the HsHa mechanism in > thrift and it is > > very good, it in fact matches our 'handler' approach - > async > > recieving/sending of data, but single threaded for > processing a > > message. > > > > -ryan > > > > On Fri, May 27, 2011 at 1:00 PM, Andrew Purtell <[email protected]> > wrote: > >> Also needing, perhaps later, consideration: > >> > >> - HDFS-347 or not > >> > >> - Lucene embedding for hbase-search, though as a > coprocessor this is already pretty much handled if we have > platform support (therefore a platform module) for a HDFS > that can do local read shortcutting and block placement > requests > >> > >> - HFile v1 versus v2 > >> > >> Making decoupled development at several downstream > sites manageable, with a home upstream for all the work, > while simultaneously providing clean migration paths for > users, basically. > >> > >> --- On Fri, 5/27/11, Andrew Purtell <[email protected]> > wrote: > >> > >>> From: Andrew Purtell <[email protected]> > >>> Subject: modular build and pluggable rpc > >>> To: [email protected] > >>> Date: Friday, May 27, 2011, 12:49 PM > >>> From IRC: > >>> > >>> apurtell i propose we take the build > modular as early as possible to deal with multiple platform > targets > >>> apurtell secure vs nonsecure > >>> apurtell 0.20 vs 0.22 vs trunk > >>> apurtell i understand the maintenence > issues with multiple rpc engines, for example, but a lot of > reflection twistiness is going to be worse > >>> apurtell i propose we take up esammer on > his offer > >>> apurtell so branch 0.92 asap, get trunk > modular and working against multiple platform targets > >>> apurtell especially if we're going to > see rpc changes coming from downstream projects... > >>> apurtell also what about supporting > secure and nonsecure clients with the same deployment? > >>> apurtell zookeeper does this > >>> apurtell so that is selectable rpc > engine per connection, with a negotiation > >>> apurtell we don't have or want to be > crazy about it but a rolling upgrade should be possible if > for example we are taking in a new rpc from fb (?) or > cloudera (avro based?) > >>> apurtell also looks like hlog modules > for 0.20 vs 0.22 and successors > >>> apurtell i think over time we can > roadmap the rpc engines, if we have multiple, by > deprecation > >>> apurtell now that we're on the edge of > supporting both 0.20 and 0.22, and secure vs nonsecure, > let's get it as manageable as possible right away > >>> > >>> St^Ack_ apurtell: +1 > >>> > >>> apurtell also i think there is some > interest in async rpc engine > >>> > >>> St^Ack_ we should stick this up > on dev i'd say > >>> > >>> Best regards, > >>> > >>> - Andy > >>> > >>> Problems worthy of attack prove their worth by > hitting > >>> back. - Piet Hein (via Tom White) > >>> > >> > > > > > > -- > Todd Lipcon > Software Engineer, Cloudera >
