Thanks for the responses lads. Sounds good. I took the tip of HBASE-10070 for a spin just now. It started up fine over existing dataset. At first it threw me off because it is missing trunk features but after I got over that, it looked fine. No strange errors in logs using my config from pre-HBASE-10070 startup. Thats good.
Running a pure randomread test on a small cluster I seem to see a slight slowdown (14.2k/second vs 13.7k/second out of bucketcache). You seen that? Just wondering. St.Ack On Tue, May 27, 2014 at 5:27 PM, Enis Söztutar <enis....@gmail.com> wrote: > Agreed with Devaraj. I do not think that these changes will destabilize the > code base much. All of the code committed to branch has been unit tested > and integration tests, both specific to region replicas and others have > been running for some time. > > Specifically, various large scale data ingestion + verification tests, bulk > load tests, hbck tests, shell tests, replication, mapreduce based tests, > snapshot tests, etc has been running on this code base. > > For testing the feature itself, HBASE-10572, HBASE-10616, and HBASE-10818 > adds tests for get/multi-get/scan and bulk load which has been running with > CM. HBASE-10817 adds a test for region replicas + replication. HBASE-10791 > changes PerformanceEvaluation to be able to work with region replicas. > Nicolas is also working on further perf testing and improvements. > > For 1.0, I think it should be fine to include this as an experimental > feature. Otherwise, it will be very hard to add this to the 1.x line. > > Are there any more concerns? If not, I would like to raise a VOTE soon. > > Enis > > > On Sat, May 24, 2014 at 10:40 AM, Devaraj Das <d...@hortonworks.com> > wrote: > > > Thanks Stack. On the testing, we have been adding unit tests for all > > patches. We have also tested the patches quite extensively on real > clusters > > via IT tests (existing and newly added) and issues discovered have been > > fixed. AFAICT, and from the tests run, this feature shouldn't destabilize > > the mainline. There are some limitations when region-replica is used - > > better integration with hbck (HBASE-10674) and support for split/merge > (and > > these are being worked on as part of phase 2). > > On May 23, 2014 9:56 PM, "Stack" <st...@duboce.net> wrote: > > > > > I think merge sooner rather than later. > > > > > > What sort of testing has been done on 10070 and what can you say about > > how > > > well the feature works? > > > > > > Where do you see risk, if any, of it destabilizing the mainline > > especially > > > what with us coming up to a 1.0. > > > > > > Thanks, > > > St.Ack > > > > > > > > > > > > On Wed, May 21, 2014 at 5:08 PM, Enis Söztutar <e...@apache.org> > wrote: > > > > > > > Hi, > > > > > > > > We would like to give an update on the status of HBASE-10070 work, > and > > > open > > > > up discussion for how we can do further development. > > > > > > > > We seem to be at a point where we have the core functionality of the > > > > region replica, as described in HBASE-10070 working. As pointed out > > > > under the section "Development Phases" in the design doc posted on > the > > > > jira HBASE-10070, this work was divided into two broad phases. The > > first > > > > phase introduces region replicas concept, the new consistency model, > > and > > > > corresponding RPC implementations. All of the issues for Phase 1 can > be > > > > found under [3]. Phase 2 is still in the works, and contains the > > proposed > > > > changes listed under [4]. > > > > > > > > With all the issues committed in HBASE-10070 branch in svn, we think > > that > > > > the "phase-1" is complete. The user documentation on HBASE-10513 > gives > > an > > > > accurate picture of what has been done in phase-1 and what the impact > > of > > > > using this feature is, APIs etc. We have added > > > > a couple of IT tests as part of this work and we have tested the work > > > > we did in "phase-1" of the project quite extensively in Hortonworks' > > > > infrastructure. > > > > > > > > In summary, with the code in branch, you can create tables with > region > > > > replicas, do gets / multi gets and scans using TIMELINE consistency > > with > > > > high availability. Region replicas periodically scan the files of the > > > > primary and pick up flushed / committed files. The RPC paths / > > > assignment, > > > > balancing etc are pretty stable. However some more performance > analysis > > > and > > > > tuning is needed. More information can be found in [1] and [2]. > > > > > > > > > > > > As a reminder, the development has been happening in the branch - > > > > hbase-10070 (https://github.com/apache/hbase/tree/hbase-10070). We > > have > > > > been pretty diligent about more than one committer's +1 on the branch > > > > commits and for almost all the subtasks in HBASE-10070 there is more > > than > > > > one +1 except for test fix issues or more trivial issues, where there > > > maybe > > > > only one +1. Enis/Nicolas/Sergey/Devaraj/Nick are the main > > contributors > > > > of code in the phase-1 and many patches have been reviewed already by > > > > people outside > > > > this group (mainly Stack, Jimmy) > > > > > > > > For Phase 2, we think that we can deliver on the proposed design > > > > incrementally over the next couple of months. However, we think that > it > > > > might be ok to merge the changes from phase 1 first, then do a > > > > commit-as-you-go approach for remaining items. Therefore, we would > like > > > to > > > > propose to merge the branch to trunk, and continue the work over > > there. > > > > This might also result in more reviews. > > > > > > > > Alternatively, we can continue the work in the branch, and do the > merge > > > at > > > > the end of Phase 2, but that will make the review process a bit more > > > > tricky, which is why we would like the merge sooner. > > > > > > > > What do you think? Comments, concerns? > > > > > > > > [1] > > > > > > > > > > > > > > https://issues.apache.org/jira/secure/attachment/12644237/hbase-10513_v1.patch > > > > [2] > > > > > > > > > > > > > > http://www.slideshare.net/enissoz/hbase-high-availability-for-reads-with-time > > > > [3] https://issues.apache.org/jira/browse/HBASE-10070 > > > > [4] https://issues.apache.org/jira/browse/HBASE-11183 > > > > > > > > Thanks, > > > > > > > > > > > -- > > CONFIDENTIALITY NOTICE > > NOTICE: This message is intended for the use of the individual or entity > to > > which it is addressed and may contain information that is confidential, > > privileged and exempt from disclosure under applicable law. If the reader > > of this message is not the intended recipient, you are hereby notified > that > > any printing, copying, dissemination, distribution, disclosure or > > forwarding of this communication is strictly prohibited. If you have > > received this communication in error, please contact the sender > immediately > > and delete it from your system. Thank You. > > >
