Quick question, if you have say 3000 records, on 13 servers, with each region being 1GB, how long do we expect those regions to load? (Master is dual core, with 8 GB RAM)
Also, what does this line mean: ZKUnassignedWatcher: ZK-EVENT-PROCESS: Got zkEvent NodeDeleted state:SyncConnected path:/hbase/UNASSIGNED/1735906890 -Jack On Mon, Sep 20, 2010 at 10:51 PM, Jack Levin <[email protected]> wrote: > Awesome, thanks!... I will give it a whirl on our test cluster. > > -Jack > > On Mon, Sep 20, 2010 at 10:15 PM, Ryan Rawson <[email protected]> wrote: >> So we are running this code in production: >> >> http://github.com/stumbleupon/hbase >> >> The branch off point is 8dc5a1a353ffc9fa57ac59618f76928b5eb31f6c, and >> everything past that is our rebase and cherry-picked changes. >> >> We use git to manage this internally, and don't use svn. Included is >> the LZO libraries we use checked directly into the code, and the >> assembly changes to publish those. >> >> So when we are ready to do a deploy, we do this: >> mvn install assembly:assembly >> (or include the -DskipTests to make it go faster) >> >> and then we have a new tarball to deploy. >> >> Note there is absolutely NO warranty here, not even that it will run >> for a microsecond... futhermore this is NOT an ASF release, just a >> courtesy. If there ever was to be a release it would look >> differently, because ASF releases cant include GPL code (this does) >> and depend on commercial releases of haoopp. >> >> Enjoy, >> -ryan >> >> On Mon, Sep 20, 2010 at 9:57 PM, Ryan Rawson <[email protected]> wrote: >>> no no, 20 GB heap per node. each node with 24-32gb ram, etc. >>> >>> we cant rely on the linux buffer cache to save us, so we have to cache >>> in hbase ram. >>> >>> :-) >>> >>> -ryan >>> >>> On Mon, Sep 20, 2010 at 9:44 PM, Jack Levin <[email protected]> wrote: >>>> 20GB+?, hmmm..... I do plan to run 50 regionserver nodes though, with >>>> 3 GB Heap likely, this should be plenty to rip through say, 350TB of >>>> data. >>>> >>>> -Jack >>>> >>>> On Mon, Sep 20, 2010 at 9:39 PM, Ryan Rawson <[email protected]> wrote: >>>>> yes that is the new ZK based coordination. when i publish the SU code >>>>> we have a patch which limits that and is faster. 2GB is a little >>>>> small for a regionserver memory... in my ideal world we'll be putting >>>>> 20GB+ of ram to regionserver. >>>>> >>>>> I just figured you were using the DEB/RPMs because your files were in >>>>> /usr/local... I usually run everything out of /home/hadoop b/c it >>>>> allows me to easily rsync as user hadoop. >>>>> >>>>> but you are on the right track yes :-) >>>>> >>>>> On Mon, Sep 20, 2010 at 9:32 PM, Jack Levin <[email protected]> wrote: >>>>>> Who said anything about deb :). I do use tarballs.... Yes, so what did >>>>>> it is the copy of that jar to under hbase/lib, and then full restart. >>>>>> Now here is a funny thing, the master shuddered for about 10 minutes, >>>>>> spewing those messages: >>>>>> >>>>>> 2010-09-20 21:23:45,826 DEBUG org.apache.hadoop.hbase.master.HMaster: >>>>>> Event NodeCreated with state SyncConnected with path >>>>>> /hbase/UNASSIGNED/97999366 >>>>>> 2010-09-20 21:23:45,827 DEBUG >>>>>> org.apache.hadoop.hbase.master.ZKMasterAddressWatcher: Got event >>>>>> NodeCreated with path /hbase/UNASSIGNED/97999366 >>>>>> 2010-09-20 21:23:45,827 DEBUG >>>>>> org.apache.hadoop.hbase.master.ZKUnassignedWatcher: ZK-EVENT-PROCESS: >>>>>> Got zkEvent NodeCreated state:SyncConnected >>>>>> path:/hbase/UNASSIGNED/97999366 >>>>>> 2010-09-20 21:23:45,827 DEBUG >>>>>> org.apache.hadoop.hbase.master.RegionManager: Created/updated >>>>>> UNASSIGNED zNode img15,normal052q.jpg,1285001686282.97999366 in state >>>>>> M2ZK_REGION_OFFLINE >>>>>> 2010-09-20 21:23:45,828 INFO >>>>>> org.apache.hadoop.hbase.master.RegionServerOperation: >>>>>> img13,p1000319tq.jpg,1284952655960.812544765 open on >>>>>> 10.103.2.3,60020,1285042333293 >>>>>> 2010-09-20 21:23:45,828 DEBUG >>>>>> org.apache.hadoop.hbase.master.ZKUnassignedWatcher: Got event type [ >>>>>> M2ZK_REGION_OFFLINE ] for region 97999366 >>>>>> 2010-09-20 21:23:45,828 DEBUG org.apache.hadoop.hbase.master.HMaster: >>>>>> Event NodeChildrenChanged with state SyncConnected with path >>>>>> /hbase/UNASSIGNED >>>>>> 2010-09-20 21:23:45,828 DEBUG >>>>>> org.apache.hadoop.hbase.master.ZKMasterAddressWatcher: Got event >>>>>> NodeChildrenChanged with path /hbase/UNASSIGNED >>>>>> 2010-09-20 21:23:45,828 DEBUG >>>>>> org.apache.hadoop.hbase.master.ZKUnassignedWatcher: ZK-EVENT-PROCESS: >>>>>> Got zkEvent NodeChildrenChanged state:SyncConnected >>>>>> path:/hbase/UNASSIGNED >>>>>> 2010-09-20 21:23:45,830 DEBUG >>>>>> org.apache.hadoop.hbase.master.BaseScanner: Current assignment of >>>>>> img150,,1284859678248.3116007 is not valid; >>>>>> serverAddress=10.103.2.1:60020, startCode=1285038205920 unknown. >>>>>> >>>>>> >>>>>> Does anyone know what they mean? At first it would kill one of my >>>>>> datanodes. But what helped is when I changed to heap size to 4GB for >>>>>> master and 2GB for datanode that was dying, and after 10 minutes I got >>>>>> into a clean state. >>>>>> >>>>>> -Jack >>>>>> >>>>>> >>>>>> On Mon, Sep 20, 2010 at 9:28 PM, Ryan Rawson <[email protected]> wrote: >>>>>>> yes, on every single machine as well, and restart. >>>>>>> >>>>>>> again, not sure how how you'd do this in a scalable manner with your >>>>>>> deb packages... on the source tarball you can just replace it, rsync >>>>>>> it out and done. >>>>>>> >>>>>>> :-) >>>>>>> >>>>>>> On Mon, Sep 20, 2010 at 8:56 PM, Jack Levin <[email protected]> wrote: >>>>>>>> ok, I found that file, do I replace hadoop-core.*.jar under >>>>>>>> /usr/lib/hbase/lib? >>>>>>>> Then restart, etc? All regionservers too? >>>>>>>> >>>>>>>> -Jack >>>>>>>> >>>>>>>> On Mon, Sep 20, 2010 at 8:40 PM, Ryan Rawson <[email protected]> >>>>>>>> wrote: >>>>>>>>> Well I don't really run CDH, I disagree with their rpm/deb packaging >>>>>>>>> policies and I have to highly recommend not using DEBs to install >>>>>>>>> software... >>>>>>>>> >>>>>>>>> So normally installing from tarball, the jar is in >>>>>>>>> <installpath>/hadoop-0.20.0-320/hadoop-core-0.20.2+320.jar >>>>>>>>> >>>>>>>>> On CDH/DEB edition, it's somewhere silly ... locate and find will be >>>>>>>>> your friend. It should be called hadoop-core-0.20.2+320.jar though! >>>>>>>>> >>>>>>>>> I'm working on a github publish of SU's production system, which uses >>>>>>>>> the cloudera maven repo to install the correct JAR in hbase so when >>>>>>>>> you type 'mvn assembly:assembly' to build your own hbase-*-bin.tar.gz >>>>>>>>> (the * being whatever version you specified in pom.xml) the cdh3b2 jar >>>>>>>>> comes pre-packaged. >>>>>>>>> >>>>>>>>> Stay tuned :-) >>>>>>>>> >>>>>>>>> -ryan >>>>>>>>> >>>>>>>>> On Mon, Sep 20, 2010 at 8:36 PM, Jack Levin <[email protected]> wrote: >>>>>>>>>> Ryan, hadoop jar, what is the usual path to the file? I just to to be >>>>>>>>>> sure, and where do I put it? >>>>>>>>>> >>>>>>>>>> -Jack >>>>>>>>>> >>>>>>>>>> On Mon, Sep 20, 2010 at 8:30 PM, Ryan Rawson <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>>> you need 2 more things: >>>>>>>>>>> >>>>>>>>>>> - restart hdfs >>>>>>>>>>> - make sure the hadoop jar from your install replaces the one we >>>>>>>>>>> ship with >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Mon, Sep 20, 2010 at 8:22 PM, Jack Levin <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>>> So, I switched to 0.89, and we already had CDH3 >>>>>>>>>>>> (hadoop-0.20-datanode-0.20.2+320-3.noarch), even though I added >>>>>>>>>>>> <name>dfs.support.append</name> as true to both hdfs-site.xml and >>>>>>>>>>>> hbase-site.xml, the master still reports this: >>>>>>>>>>>> >>>>>>>>>>>> You are currently running the HMaster without HDFS append support >>>>>>>>>>>> enabled. This may result in data loss. Please see the HBase wiki >>>>>>>>>>>> for >>>>>>>>>>>> details. >>>>>>>>>>>> Master Attributes >>>>>>>>>>>> Attribute Name Value Description >>>>>>>>>>>> HBase Version 0.89.20100726, r979826 HBase version and svn >>>>>>>>>>>> revision >>>>>>>>>>>> HBase Compiled Sat Jul 31 02:01:58 PDT 2010, stack When HBase >>>>>>>>>>>> version >>>>>>>>>>>> was compiled and by whom >>>>>>>>>>>> Hadoop Version 0.20.2, r911707 Hadoop version and svn revision >>>>>>>>>>>> Hadoop Compiled Fri Feb 19 08:07:34 UTC 2010, chrisdo When Hadoop >>>>>>>>>>>> version was compiled and by whom >>>>>>>>>>>> HBase Root Directory >>>>>>>>>>>> hdfs://namenode-rd.imageshack.us:9000/hbase Location >>>>>>>>>>>> of HBase home directory >>>>>>>>>>>> >>>>>>>>>>>> Any ideas whats wrong? >>>>>>>>>>>> >>>>>>>>>>>> -Jack >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Sep 20, 2010 at 5:47 PM, Ryan Rawson <[email protected]> >>>>>>>>>>>> wrote: >>>>>>>>>>>>> Hey, >>>>>>>>>>>>> >>>>>>>>>>>>> There is actually only 1 active branch of hbase, that being the >>>>>>>>>>>>> 0.89 >>>>>>>>>>>>> release, which is based on 'trunk'. We have snapshotted a series >>>>>>>>>>>>> of >>>>>>>>>>>>> 0.89 "developer releases" in hopes that people would try them our >>>>>>>>>>>>> and >>>>>>>>>>>>> start thinking about the next major version. One of these is >>>>>>>>>>>>> what SU >>>>>>>>>>>>> is running prod on. >>>>>>>>>>>>> >>>>>>>>>>>>> At this point tracking 0.89 and which ones are the 'best' peach >>>>>>>>>>>>> sets >>>>>>>>>>>>> to run is a bit of a contact sport, but if you are serious about >>>>>>>>>>>>> not >>>>>>>>>>>>> losing data it is worthwhile. SU is based on the most recent DR >>>>>>>>>>>>> with >>>>>>>>>>>>> a few minor patches of our own concoction brought in. If current >>>>>>>>>>>>> works, but some Master ops are slow, and there are a few patches >>>>>>>>>>>>> on >>>>>>>>>>>>> top of that. I'll poke about and see if its possible to publish >>>>>>>>>>>>> to a >>>>>>>>>>>>> github branch or something. >>>>>>>>>>>>> >>>>>>>>>>>>> -ryan >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Sep 20, 2010 at 5:16 PM, Jack Levin <[email protected]> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> Sounds, good, only reason I ask is because of this: >>>>>>>>>>>>>> >>>>>>>>>>>>>> There are currently two active branches of HBase: >>>>>>>>>>>>>> >>>>>>>>>>>>>> * 0.20 - the current stable release series, being maintained >>>>>>>>>>>>>> with >>>>>>>>>>>>>> patches for bug fixes only. This release series does not support >>>>>>>>>>>>>> HDFS >>>>>>>>>>>>>> durability - edits may be lost in the case of node failure. >>>>>>>>>>>>>> * 0.89 - a development release series with active feature and >>>>>>>>>>>>>> stability development, not currently recommended for production >>>>>>>>>>>>>> use. >>>>>>>>>>>>>> This release does support HDFS durability - cases in which edits >>>>>>>>>>>>>> are >>>>>>>>>>>>>> lost are considered serious bugs. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Are we talking about data loss in case of datanode going down >>>>>>>>>>>>>> while >>>>>>>>>>>>>> being written to, or RegionServer going down? >>>>>>>>>>>>>> >>>>>>>>>>>>>> -jack >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, Sep 20, 2010 at 4:09 PM, Ryan Rawson >>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>> We run 0.89 in production @ Stumbleupon. We also employ 3 >>>>>>>>>>>>>>> committers... >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> As for safety, you have no choice but to run 0.89. If you run >>>>>>>>>>>>>>> a 0.20 >>>>>>>>>>>>>>> release you will lose data. you must be on 0.89 and >>>>>>>>>>>>>>> CDH3/append-branch to achieve data durability, and there really >>>>>>>>>>>>>>> is no >>>>>>>>>>>>>>> argument around it. If you are doing your tests with 0.20.6 >>>>>>>>>>>>>>> now, I'd >>>>>>>>>>>>>>> stop and rebase those tests onto the latest DR announced on the >>>>>>>>>>>>>>> list. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -ryan >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mon, Sep 20, 2010 at 3:17 PM, Jack Levin <[email protected]> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> Hi Stack, see inline: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Mon, Sep 20, 2010 at 2:42 PM, Stack <[email protected]> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> Hey Jack: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks for writing. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> See below for some comments. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Mon, Sep 20, 2010 at 11:00 AM, Jack Levin >>>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Image-Shack gets close to two million image uploads per day, >>>>>>>>>>>>>>>>>> which are >>>>>>>>>>>>>>>>>> usually stored on regular servers (we have about 700), as >>>>>>>>>>>>>>>>>> regular >>>>>>>>>>>>>>>>>> files, and each server has its own host name, such as >>>>>>>>>>>>>>>>>> (img55). I've >>>>>>>>>>>>>>>>>> been researching on how to improve our backend design in >>>>>>>>>>>>>>>>>> terms of data >>>>>>>>>>>>>>>>>> safety and stumped onto the Hbase project. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Any other requirements other than data safety? (latency, etc). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Latency is the second requirement. We have some services that >>>>>>>>>>>>>>>> are >>>>>>>>>>>>>>>> very short tail, and can produce 95% cache hit rate, so I >>>>>>>>>>>>>>>> assume this >>>>>>>>>>>>>>>> would really put cache into good use. Some other services >>>>>>>>>>>>>>>> however, >>>>>>>>>>>>>>>> have about 25% cache hit ratio, in which case the latency >>>>>>>>>>>>>>>> should be >>>>>>>>>>>>>>>> 'adequate', e.g. if its slightly worse than getting data off >>>>>>>>>>>>>>>> raw disk, >>>>>>>>>>>>>>>> then its good enough. Safely is supremely important, then its >>>>>>>>>>>>>>>> availability, then speed. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Now, I think hbase is he most beautiful thing that happen to >>>>>>>>>>>>>>>>>> distributed DB world :). The idea is to store image files >>>>>>>>>>>>>>>>>> (about >>>>>>>>>>>>>>>>>> 400Kb on average into HBASE). >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I'd guess some images are much bigger than this. Do you ever >>>>>>>>>>>>>>>>> limit >>>>>>>>>>>>>>>>> the size of images folks can upload to your service? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The setup will include the following >>>>>>>>>>>>>>>>>> configuration: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 50 servers total (2 datacenters), with 8 GB RAM, dual core >>>>>>>>>>>>>>>>>> cpu, 6 x >>>>>>>>>>>>>>>>>> 2TB disks each. >>>>>>>>>>>>>>>>>> 3 to 5 Zookeepers >>>>>>>>>>>>>>>>>> 2 Masters (in a datacenter each) >>>>>>>>>>>>>>>>>> 10 to 20 Stargate REST instances (one per server, hash >>>>>>>>>>>>>>>>>> loadbalanced) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Whats your frontend? Why REST? It might be more efficient >>>>>>>>>>>>>>>>> if you >>>>>>>>>>>>>>>>> could run with thrift given REST base64s its payload IIRC >>>>>>>>>>>>>>>>> (check the >>>>>>>>>>>>>>>>> src yourself). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> For insertion we use Haproxy, and balance curl PUTs across >>>>>>>>>>>>>>>> multiple REST APIs. >>>>>>>>>>>>>>>> For reading, its a nginx proxy that does Content-type >>>>>>>>>>>>>>>> modification >>>>>>>>>>>>>>>> from image/jpeg to octet-stream, and vice versa, >>>>>>>>>>>>>>>> it then hits Haproxy again, which hits balanced REST. >>>>>>>>>>>>>>>> Why REST, it was the simplest thing to run, given that its >>>>>>>>>>>>>>>> supports >>>>>>>>>>>>>>>> HTTP, potentially we could rewrite something for thrift, as >>>>>>>>>>>>>>>> long as we >>>>>>>>>>>>>>>> can use http still to send and receive data (anyone wrote >>>>>>>>>>>>>>>> anything >>>>>>>>>>>>>>>> like that say in python, C or java?) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 40 to 50 RegionServers (will probably keep masters separate >>>>>>>>>>>>>>>>>> on dedicated boxes). >>>>>>>>>>>>>>>>>> 2 Namenode servers (one backup, highly available, will do >>>>>>>>>>>>>>>>>> fsimage and >>>>>>>>>>>>>>>>>> edits snapshots also) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> So far I got about 13 servers running, and doing about 20 >>>>>>>>>>>>>>>>>> insertions / >>>>>>>>>>>>>>>>>> second (file size ranging from few KB to 2-3MB, ave. 400KB). >>>>>>>>>>>>>>>>>> via >>>>>>>>>>>>>>>>>> Stargate API. Our frontend servers receive files, and I just >>>>>>>>>>>>>>>>>> fork-insert them into stargate via http (curl). >>>>>>>>>>>>>>>>>> The inserts are humming along nicely, without any noticeable >>>>>>>>>>>>>>>>>> load on >>>>>>>>>>>>>>>>>> regionservers, so far inserted about 2 TB worth of images. >>>>>>>>>>>>>>>>>> I have adjusted the region file size to be 512MB, and table >>>>>>>>>>>>>>>>>> block size >>>>>>>>>>>>>>>>>> to about 400KB , trying to match average access block to >>>>>>>>>>>>>>>>>> limit HDFS >>>>>>>>>>>>>>>>>> trips. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> As Todd suggests, I'd go up from 512MB... 1G at least. You'll >>>>>>>>>>>>>>>>> probably want to up your flush size from 64MB to 128MB or >>>>>>>>>>>>>>>>> maybe 192MB. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Yep, i will adjust to 1G. I thought flush was controlled by a >>>>>>>>>>>>>>>> function of memstore HEAP, something like 40%? Or are you >>>>>>>>>>>>>>>> talking >>>>>>>>>>>>>>>> about HDFS block size? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> So far the read performance was more than adequate, and of >>>>>>>>>>>>>>>>>> course write performance is nowhere near capacity. >>>>>>>>>>>>>>>>>> So right now, all newly uploaded images go to HBASE. But we >>>>>>>>>>>>>>>>>> do plan >>>>>>>>>>>>>>>>>> to insert about 170 Million images (about 100 days worth), >>>>>>>>>>>>>>>>>> which is >>>>>>>>>>>>>>>>>> only about 64 TB, or 10% of planned cluster size of 600TB. >>>>>>>>>>>>>>>>>> The end goal is to have a storage system that creates data >>>>>>>>>>>>>>>>>> safety, >>>>>>>>>>>>>>>>>> e.g. system may go down but data can not be lost. Our >>>>>>>>>>>>>>>>>> Front-End >>>>>>>>>>>>>>>>>> servers will continue to serve images from their own file >>>>>>>>>>>>>>>>>> system (we >>>>>>>>>>>>>>>>>> are serving about 16 Gbits at peak), however should we need >>>>>>>>>>>>>>>>>> to bring >>>>>>>>>>>>>>>>>> any of those down for maintenance, we will redirect all >>>>>>>>>>>>>>>>>> traffic to >>>>>>>>>>>>>>>>>> Hbase (should be no more than few hundred Mbps), while the >>>>>>>>>>>>>>>>>> front end >>>>>>>>>>>>>>>>>> server is repaired (for example having its disk replaced), >>>>>>>>>>>>>>>>>> after the >>>>>>>>>>>>>>>>>> repairs, we quickly repopulate it with missing files, while >>>>>>>>>>>>>>>>>> serving >>>>>>>>>>>>>>>>>> the missing remaining off Hbase. >>>>>>>>>>>>>>>>>> All in all should be very interesting project, and I am >>>>>>>>>>>>>>>>>> hoping not to >>>>>>>>>>>>>>>>>> run into any snags, however, should that happens, I am >>>>>>>>>>>>>>>>>> pleased to know >>>>>>>>>>>>>>>>>> that such a great and vibrant tech group exists that >>>>>>>>>>>>>>>>>> supports and uses >>>>>>>>>>>>>>>>>> HBASE :). >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> We're definetly interested in how your project progresses. >>>>>>>>>>>>>>>>> If you are >>>>>>>>>>>>>>>>> ever up in the city, you should drop by for a chat. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Cool. I'd like that. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> St.Ack >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> P.S. I'm also w/ Todd that you should move to 0.89 and blooms. >>>>>>>>>>>>>>>>> P.P.S I updated the wiki on stargate REST: >>>>>>>>>>>>>>>>> http://wiki.apache.org/hadoop/Hbase/Stargate >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Cool, I assume if we move to that it won't kill existing meta >>>>>>>>>>>>>>>> tables, >>>>>>>>>>>>>>>> and data? e.g. cross compatible? >>>>>>>>>>>>>>>> Is 0.89 ready for production environment? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -Jack >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
