On Fri, May 27, 2011 at 2:38 PM, Andrew Purtell <[email protected]> wrote: > I don't disagree with any of this but the fact is we have compile time > differences if going against secure Hadoop 0.20 or non-secure Hadoop 0.20. > > So either we decide to punt on integration with secure Hadoop 0.20 or we deal > with the compile time differences. If dealing with them, we can do it by > reflection, which is brittle and can be difficult to understand and debug, > and someone would have to do this work; or we can wholesale replace RPC with > something based on Thrift, and someone would have to do the work; or we take > the pluggable RPC changes that Gary has already developed and modularize the > build, which Eric has already volunteered to do.
Yes, sorry for taking this discussion off-track. I think modularizing this part of the build and fixing things like the recent async response patch to work with that modularization is the right short-term solution. -Todd > > - Andy > > --- On Fri, 5/27/11, Todd Lipcon <[email protected]> wrote: > >> From: Todd Lipcon <[email protected]> >> Subject: Re: modular build and pluggable rpc >> To: [email protected] >> Cc: [email protected] >> Date: Friday, May 27, 2011, 1:30 PM >> Agreed - I'm all for Thrift. >> >> Though, I actually, contrary to Ryan, think that the >> existing HBaseRPC >> handler/client code is pretty good -- better than the >> equivalents from >> Thrift Java. >> >> We could start by using Thrift serialization on our >> existing transport >> -- then maybe work towards contributing it upstream to the >> Thrift >> project. HDFS folks are potentially interested in doing >> that as well. >> >> -Todd >> >> On Fri, May 27, 2011 at 1:10 PM, Ryan Rawson <[email protected]> >> wrote: >> > I'm -1 on avro as a RPC format. Thrift is the way to >> go, any of the >> > advantages of smaller serialization of avro is lost by >> the sheer >> > complexity of avro and therefore the potential bugs. >> > >> > I understand the desire to have a pluggable RPC >> engine, but it feels >> > like the better approach would be to adopt a unified >> RPC and just be >> > done with it. I had a look at the HsHa mechanism in >> thrift and it is >> > very good, it in fact matches our 'handler' approach - >> async >> > recieving/sending of data, but single threaded for >> processing a >> > message. >> > >> > -ryan >> > >> > On Fri, May 27, 2011 at 1:00 PM, Andrew Purtell <[email protected]> >> wrote: >> >> Also needing, perhaps later, consideration: >> >> >> >> - HDFS-347 or not >> >> >> >> - Lucene embedding for hbase-search, though as a >> coprocessor this is already pretty much handled if we have >> platform support (therefore a platform module) for a HDFS >> that can do local read shortcutting and block placement >> requests >> >> >> >> - HFile v1 versus v2 >> >> >> >> Making decoupled development at several downstream >> sites manageable, with a home upstream for all the work, >> while simultaneously providing clean migration paths for >> users, basically. >> >> >> >> --- On Fri, 5/27/11, Andrew Purtell <[email protected]> >> wrote: >> >> >> >>> From: Andrew Purtell <[email protected]> >> >>> Subject: modular build and pluggable rpc >> >>> To: [email protected] >> >>> Date: Friday, May 27, 2011, 12:49 PM >> >>> From IRC: >> >>> >> >>> apurtell i propose we take the build >> modular as early as possible to deal with multiple platform >> targets >> >>> apurtell secure vs nonsecure >> >>> apurtell 0.20 vs 0.22 vs trunk >> >>> apurtell i understand the maintenence >> issues with multiple rpc engines, for example, but a lot of >> reflection twistiness is going to be worse >> >>> apurtell i propose we take up esammer on >> his offer >> >>> apurtell so branch 0.92 asap, get trunk >> modular and working against multiple platform targets >> >>> apurtell especially if we're going to >> see rpc changes coming from downstream projects... >> >>> apurtell also what about supporting >> secure and nonsecure clients with the same deployment? >> >>> apurtell zookeeper does this >> >>> apurtell so that is selectable rpc >> engine per connection, with a negotiation >> >>> apurtell we don't have or want to be >> crazy about it but a rolling upgrade should be possible if >> for example we are taking in a new rpc from fb (?) or >> cloudera (avro based?) >> >>> apurtell also looks like hlog modules >> for 0.20 vs 0.22 and successors >> >>> apurtell i think over time we can >> roadmap the rpc engines, if we have multiple, by >> deprecation >> >>> apurtell now that we're on the edge of >> supporting both 0.20 and 0.22, and secure vs nonsecure, >> let's get it as manageable as possible right away >> >>> >> >>> St^Ack_ apurtell: +1 >> >>> >> >>> apurtell also i think there is some >> interest in async rpc engine >> >>> >> >>> St^Ack_ we should stick this up >> on dev i'd say >> >>> >> >>> Best regards, >> >>> >> >>> - Andy >> >>> >> >>> Problems worthy of attack prove their worth by >> hitting >> >>> back. - Piet Hein (via Tom White) >> >>> >> >> >> > >> >> >> >> -- >> Todd Lipcon >> Software Engineer, Cloudera >> > -- Todd Lipcon Software Engineer, Cloudera
