Here are some thoughts: - Performance is important. - Unified protocol would help the use of tracer tags, which could give us top to bottom profiling. - Would like to do rolling restart of HBase, even under relatively major upgrades. - PHP (even pure php) bindings.
I'll poke at the code, what is the current state? On Tue, Jun 23, 2009 at 11:55 PM, Jeff Hammerbacher<[email protected]> wrote: > Ryan, > > The working plan is to use Avro to replace the current DataNode streaming > protocol, in addition to the HTTP-based protocol for fetching data during > the shuffle and the standard Hadoop IPC. Having a unified serialization and > RPC strategy should make enforcing things like security constraints and > versions far more straightforward. It would be great to see the HBase > community express their specific needs to the Avro folks so that we could > have unified serialization and RPC throughout the stack (not you, Michael). > > Later, > Jeff > > On Mon, Jun 22, 2009 at 9:36 PM, Ryan Rawson <[email protected]> wrote: > >> HBase has performance requirements on par with the datanode streaming >> protocol - we want to provide the data in our systems as fast as we can >> read >> and stream them. >> >> -ryan >> >> On Mon, Jun 22, 2009 at 9:19 PM, Amr Awadallah <[email protected]> wrote: >> >> > CCing avro-dev@ to comment, but my understanding is that it is currently >> > functioning and way superior to the alternatives you mention below :) >> > >> > -- amr >> > >> > >> > Andrew Purtell wrote: >> > >> >> I my opinion, we should not bother to wait for Avro. I've been hearing >> >> about it on and off for three months now. If it is ready the day we >> >> start work on 1015 for 0.21, and it fits the bill, fine, but otherwise >> >> we should just use Thrift or pbufs. Thrift may be preferable as its >> >> compact binary protocol is competitive with pbufs plus it has a fully >> >> implemented async rpc stack. I think this applies for both 1015 and >> >> 1295. Also I'm skeptical that something to supplant RMI won't have >> >> overheads related to that we don't need, e.g. transmitting class and >> >> method names as strings, etc. >> >> - Andy >> >> >> >> >> >> >> >> >> >> ________________________________ >> >> From: Jim Kellerman (POWERSET) <[email protected]> >> >> To: "[email protected]" <[email protected]> >> >> Sent: Saturday, June 20, 2009 3:26:58 PM >> >> Subject: RE: who's doing what for 0.21? >> >> >> >> I am also interested in 1295 (I have quite a bit of experience >> >> with cross data center replication), but more interested in >> >> getting more of the master into zookeeper. >> >> >> >> As for 1556, I might wait a bit. At the Cloudera off-site, one >> >> of the things talked about was doing something similar for >> >> Hadoop which we might leverage. >> >> >> >> What really needs to get done around builds is when you mark >> >> a Jira as patch available, we should do a patch build and test >> >> like Hadoop does. Noone has had time to do it to date, but if >> >> you are taking on the build, that would be a "nice to have". >> >> >> >> For 1015, should you wait for Avro? >> >> >> >> And if you missed it, here are the notes from the Cloudera off-site: >> >> http://wiki.apache.org/hadoop/DeveloperOffsite20090612 >> >> >> >> --- >> >> Jim Kellerman, Powerset (Live Search, Microsoft Corporation) >> >> >> >> >> >> >> >> >> >>> -----Original Message----- >> >>> From: Andrew Purtell [mailto:[email protected]] >> >>> Sent: Saturday, June 20, 2009 12:11 PM >> >>> To: [email protected] >> >>> Subject: who's doing what for 0.21? >> >>> >> >>> http://tinyurl.com/m7nt72 >> >>> >> >>> I have an interest in these: >> >>> >> >>> https://issues.apache.org/jira/browse/HBASE-1015 >> >>> https://issues.apache.org/jira/browse/HBASE-1295 >> >>> https://issues.apache.org/jira/browse/HBASE-1556 >> >>> >> >>> I think for 1015 and 1295, there is interest on the part of at least >> >>> myself, >> >>> dj_ryan, and jgray. dj_ryan was saying something about su executives >> >>> making >> >>> 1295 a priority for him. We should figure out how to divide up and >> >>> assign >> >>> out the work. >> >>> >> >>> Also, probably I'll end up taking on the grunt work of 1556, because >> >>> it >> >>> needs to be done. >> >>> >> >>> Have we set a time and place for the next dev meeting? >> >>> >> >>> - Andy >> >>> >> >>> >> >>> >> >> >> >> >> >> >> >> >> > >> >
