On Mon, Mar 19, 2012 at 9:17 AM, Suresh Srinivas <sur...@hortonworks.com> wrote: > I am sure Jitendra understands what Todd meant, given he was quite involved > in the work. As Jitendra said, I would like to keep the wire type from the > implementation type. Even for internal protocols. Rolling upgrades is > important.
How does the choice of whether we separate the types matter with regard to rolling upgrades? The translators are not required for rolling upgrade, so long as the protobufs are evolved correctly (ie not removing or adding required fields between adjacent versions). > > I understand where Todd is coming from. We did this work for 10 protocols. > But it is not like every day we introduce protocols and make huge protocol > changes. Given that I prefer to retain the structure we have in place. I > also volunteer to do this work, for protocols that are introduced. OK, fair enough. I may take you up on that at some point, Suresh ;-) Thanks for hearing me out. -Todd > > On Mon, Mar 19, 2012 at 8:11 AM, Robert Evans <ev...@yahoo-inc.com> wrote: > >> I think what we are talking about here is removing some of the extra >> layers of abstraction in java. The wire protocol used will be identical in >> either case. It is just that we would have to use the Protocol Buffer >> Builder APIs instead of wrapping them with our own custom getters/setters. >> >> I am +1 for reducing the layers needed to add/modify the RPC. >> >> --Bobby Evans >> >> On 3/19/12 2:17 AM, "Jitendra Pandey" <jiten...@hortonworks.com> wrote: >> >> Wire compatibility in hdfs private protocols between different >> components is also important for rolling upgrades. We do want to support >> upgrading different components of a cluster independent of each other and >> wire compatibility is one of the essential prerequisites. Therefore, even >> if some protocols are not exposed to the users, and are just used within >> different internal components, we still cannot afford to compromise on wire >> compatibility on those interfaces. >> >> On Sat, Mar 17, 2012 at 3:31 PM, Todd Lipcon <t...@cloudera.com> wrote: >> >> > Hi all, >> > >> > I've been working on some patches recently that required adding a new >> > protocol and some RPC calls that will be used entirely internally to >> > HDFS -- i.e the types and functions are never exposed to clients. The >> > process to do this involved: >> > 1) Add a new .proto file MyProtocol.proto with the types and the >> > service definition >> > 2) Add a new empty Java interface MyProtocolPB.java which adds the >> > ProtocolInfo and KerberosInfo annotations >> > 3) Add a new Java interface MyProtocol.java which duplicates the same >> > methods I defined in the protobuf service >> > 4) For each new type, create a new Java class which duplicates the >> > fields, getters, and setters from the protobuf messages >> > 5) Create a Client-Side Translator and Server-Side Translator class, >> > each containing a wrapper method for each of the calls >> > 6) Create a PBHelper class which contains two convert() methods for >> > each of the new types >> > >> > Given that we have many protocols that we never intend to expose, I >> > see little benefit to adding the indirection layer here. It only makes >> > the task of modifying the protocols quite onerous and full of >> > duplicate boilerplate code. >> > I'd like to propose that, when adding protocols that are meant to be >> > HDFS-private, we drop steps 3-6 and use the protobuf RPC engine >> > directly. Doing this doesn't force our hand or limit our options in >> > the future -- should we want to add an alternate mechanism one can >> > always add the indirection layer down the road. >> > >> > Thoughts? >> > >> > -Todd >> > -- >> > Todd Lipcon >> > Software Engineer, Cloudera >> > >> >> >> >> -- >> >> -- Todd Lipcon Software Engineer, Cloudera
