On Sun, Mar 6, 2011 at 9:25 PM, Suraj Varma <svarma...@gmail.com> wrote: > Thanks all for your insights into this. > > I would agree that providing mechanisms to support no-outage upgrades going > forward would really be widely beneficial. I was looking forward to Avro for > this reason. > > Some follow up questions: > 1) If asynchbase client to do this (i.e. talk wire protocol and adjust based > on server versions), why not the native hbase client? Is there something in > the native client design that would make this too hard / not worth > emulating?
Typically this has not been an issue. The particular design of the way that hadoop rpc (the rpc we use) makes it difficult to offer multiple protocol/version support. To "fix it" would more or less require rewriting the entire protocol stack. I'm glad we spent serious time making the base storage layer and query paths fast, since without those fundamentals a "better" RPC would be moot. From my measurements I dont think we are losing a lot of performance in our current RPC system, and unless we are very careful we'll lose a lot in a thrift/avro transition. > 2) Does asynchbase have any limitations (functionally or otherwise) compared > to the native HBase client? > > 3) If Avro were the "native" protocol that HBase & client talks through, > that is one thing (and that's what I'm hoping we end up with) - however, > isn't spinning up Avro gateways on each node (like what is currently > available) require folks to scale up two layers (Avro gateway layer + HBase > layer)? i.e. now we need to be worried about whether the Avro gateways can > handle the traffic, etc. The hbase client is fairly 'thick', it must intelligently route between different regionservers, handle errors, relook up meta data, use zookeeper to bootstrap, etc. This is part of making a scalable client though. Having the RPC serialization in thrift or avro would make it easier to write those kinds of clients for non-Java languages. The gateway approach will probably be necessary for a while alas. At SU I am not sure that the gateway is adding a lot of of latency to small queries, since average/median latency is around 1ms. One strategy is to deploy gateways on all client nodes and use localhost as much as possible. > In our application, we have Java clients talking directly to HBase. We > debated using Thrift or Stargate layer (even though we have a Java client) > just because of this easier upgrade-ability. But we finally decided to use > the native HBase client because we didn't want to have to scale two layers > rather than just HBase ... and Avro was on the road map. An HBase client > talking native Avro directly to RS (i.e. without intermediate "gateways" > would have worked - but that was a ways ... So again avro isn't going to be a magic bullet. Neither thrift. You can't just have a dumb client with little logic open up a socket and start talking to HBase. That isn't congruent with a scalable system unfortunately. You need your clients to be smart and do a bunch of work that otherwise would have to be done by a centralized type node or another middleman. Only if the client is smart can we send the minimal RPCs to the shortest network length. Other systems have servers bounce the requests to other servers but that can promote extra traffic at the cost of a simpler client. > I think now that we are in the .90s, an option to do no-outage upgrades > (from client's perspective) would be really beneficial. We'd all like this, it's formost in pretty much every committer's mind all the time. It's just a HUGE body of work. One that is fraught with perils and danger zones. For example it seemed avro would reign supreme, but the RPC landscape is shifting back towards thrift. > > Thanks, > --Suraj > > > On Sat, Mar 5, 2011 at 2:21 PM, Todd Lipcon <t...@cloudera.com> wrote: > >> On Sat, Mar 5, 2011 at 2:10 PM, Ryan Rawson <ryano...@gmail.com> wrote: >> > As for the past RPC, it's all well to complain that we didn't spend >> > more time making it more compatible, but in a world where evolving >> > features in an early platform is more important than keeping backwards >> > compatibility (how many hbase 18 jars want to talk to a modern >> > cluster? Like none.), I am confident we did the right choice. Moving >> > forward I think the goal should NOT be to maintain the current system >> > compatible at all costs, but to look at things like avro and thrift, >> > make a calculated engineering tradeoff and get ourselves on to a >> > extendable platform, even if there is a flag day. We aren't out of >> > the woods yet, but eventually we will be. >> >> Hear hear! +1! >> >> -Todd >> -- >> Todd Lipcon >> Software Engineer, Cloudera >> >
