Nice work Enis. St.Ack
On Fri, Jun 27, 2014 at 7:32 PM, Enis Söztutar <[email protected]> wrote: > Update: > > I've completed the merge for branch hbase-10070 to master now. As a short > reminder the committed changes include phase 1 work for the new feature > called "read high availability using timeline consistent reads with region > replicas". Some more context can be found in [1] and vote thread here [2]. > > The merge contains 44 commits in total as follows. I've uploaded the > committed patches to relevant jiras, and set the fix versions for the jiras > as well (using git format-patch and the excellent jira-cli client). > > Applying: HBASE-10347 HRegionInfo changes for adding replicaId and > MetaEditor/MetaReader changes for region replicas > Applying: HBASE-10354 Add an API for defining consistency per request > Applying: HBASE-10348. HTableDescriptor changes for region replicas > Applying: HBASE-10517 NPE in MetaCache.clearCache() > Applying: HBASE-10354. Addendum commit. 10354.add.patch committed > Applying: HBASE-10356 Failover RPC's for multi-get > Applying: HBASE-10350. Master/AM/RegionStates changes to create and assign > region replicas (ddas) > Applying: HBASE-10359 Master/RS WebUI changes for region replicas > Applying: HBASE-10361. Enable/AlterTable support for region replicas (ddas) > Applying: HBASE-10352 Region and RegionServer changes for opening region > replicas, and refreshing store files > Applying: HBASE-10362. HBCK changes for supporting region replicas > Applying: HBASE-10351 LoadBalancer changes for supporting region replicas > Applying: HBASE-10630 NullPointerException in > ConnectionManager$HConnectionImplementation.locateRegionInMeta() due to > missing region info > Applying: HBASE-10633. StoreFileRefresherChore throws > ConcurrentModificationException sometimes > Applying: HBASE-10672. Table snapshot should handle tables whose > REGION_REPLICATION is greater than one > Applying: HBASE-10620. LoadBalancer.needsBalance() should check for > co-located region replicas as well > Applying: HBASE-10355 Failover RPC's from client using region replicas > Applying: HBASE-10704 BaseLoadBalancer#roundRobinAssignment() may add same > region to assignment plan multiple times > Applying: HBASE-10572 Create an IntegrationTest for region replicas > Applying: HBASE-10729. Enable table doesn't balance out replicas evenly if > the replicas were unassigned earlier > Applying: HBASE-10620 LoadBalancer.needsBalance() should check for > co-located region replicas as well (Addendum patch to fix return value from > balance() call) > Applying: HBASE-10734. Fix RegionStates.getRegionAssignments to not add > duplicate regions > Applying: HBASE-10616. Integration test for multi-get calls > Applying: HBASE-10743. Replica map update is problematic in RegionStates > Applying: HBASE-10616. Addendum that fixes the case of multiget_batchsize > == 1 > Applying: HBASE-10778 Unique keys accounting in MultiThreadedReader is > incorrect > Applying: HBASE-10858 TestRegionRebalancing is failing > Applying: HBASE-10817 Add some tests on a real cluster for replica: multi > master, replication > Applying: HBASE-10701 Cache invalidation improvements from client side > Applying: HBASE-10859 Use HFileLink in opening region files from > secondaries > Applying: HBASE-10791 Add integration test to demonstrate performance > improvement > Applying: HBASE-10634 Multiget doesn't fully work > Applying: HBASE-10794 multi-get should handle replica location missing from > cache > Applying: HBASE-10785 Metas own location should be cached > Applying: HBASE-10957 HMaster can abort with NPE in #rebuildUserRegions > (Nicolas Liochon) > Applying: HBASE-10661 > TestStochasticLoadBalancer.testRegionReplicationOnMidClusterWithRacks() is > flaky > Applying: HBASE-10513 Provide user documentation for region replicas > Applying: HBASE-10810 LoadTestTool should share the connection and > connection pool > Applying: HBASE-10357 Failover RPC's for scans (Devaraj Das) > Applying: HBASE-11214. Fixes for scans on a replicated table > Applying: HBASE-10818. Add integration test for bulkload with replicas > (Nick Dimiduk and Devaraj Das) > Applying: HBASE-10818. Addendum to reduce the number of times sleep happens > in the coprocessor. Also, puts back code to set consistency that got > deleted by accident in the last commit. > Applying: HBASE-11332 Fix for metas location cache from HBASE-10785 > Applying: Increase -XX:MaxPermSize for unit tests > > > Unit tests should be working now, but in case it fails on the apache infra, > I'll take a closer look at the jenkins build at > https://builds.apache.org/view/All/job/HBase-TRUNK/. We've been running > the > integration tests as well but ideally we want to spend some more time on > perf regression analysis and long tail latency improvements. We can iron > out the edges in time. Nicolas has already been doing work around this. > > Let me know if you see any issues related to this merge. > > > [1] > https://mail-archives.apache.org/mod_mbox/hbase-dev/201405.mbox/browser > [2] > > https://mail-archives.apache.org/mod_mbox/hbase-dev/201406.mbox/%3CCAMUu0w8uvb29%2BQYpZ8VwSO4qEpKRyKGNPzyj1kw7tLRhLPoZkA%40mail.gmail.com%3E > > > > > On Fri, Jun 27, 2014 at 7:03 PM, Ted Yu <[email protected]> wrote: > > > Good job. > > > > > > On Fri, Jun 27, 2014 at 5:45 PM, Enis Söztutar <[email protected]> wrote: > > > > > Pushed the rebased branch. You can go crazy on the master now. > > > > > > Enis > > > > > > > > > On Fri, Jun 27, 2014 at 4:37 PM, Enis Söztutar <[email protected]> > wrote: > > > > > > > Hi, > > > > > > > > I am merging the hbase-10070 branch to master. No checkins please. > > > > > > > > We were able to get unit tests working, and IT's working. There may > be > > a > > > > couple of edge cases after rebase, but we can resolve them in master. > > > > > > > > I'll send a detailed update once done. > > > > > > > > Enis > > > > > > > > > >
