Added: dev/hbase/2.0.6RC0/RELEASENOTES.md ============================================================================== --- dev/hbase/2.0.6RC0/RELEASENOTES.md (added) +++ dev/hbase/2.0.6RC0/RELEASENOTES.md Tue Aug 13 22:47:49 2019 @@ -0,0 +1,9100 @@ +# RELEASENOTES + +<!--- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + + +CHANGES.md and RELEASENOTES.md were generated using yetus releasedocmaker. + +First make sure what is in JIRA agrees with what is in git and vice-versa +(See HBASE-14025 for original exposition on how to do this followed by +the travails of various release managers. See also HBASE-18828 for a +deriviative makes the move to yetus releasedocmaker). + +Then make sure that anything in current release as noted in JIRA has +not made it out in earlier major/minor (see HBASE-14025 for how to +reconcile which). + +Obtain a yetus release (see HBASE-18828 for some help). To run +releasedocmaker, do as follows (below example is for hbase-2.0.0 +and yetus 0.9.0): + + $ ./bin/releasedocmaker -p HBASE --fileversions -v 2.0.5 -l \ + --sortorder=newer --skip-credits + +or on older yetus versions: + + $ ./release-doc-maker/releasedocmaker.py -p HBASE --fileversions \ + -v 2.0.0 -l --sortorder=newer --skip-credits + +Rename the output as CHANGES.md and RELEASENOTES.md. Edit both to put +document title above the apache license so markdown readers work. You +may have to bulk import old-style CHANGES.txt on to the end in a code +comment to preserve continuity of the CHANGELOG. + +DO NOT REMOVE THIS MARKER; FOR INTERPOLATING RNS!--> +# HBASE 2.0.6 Release Notes + +These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements. + + +--- + +* [HBASE-22777](https://issues.apache.org/jira/browse/HBASE-22777) | *Major* | **Add a multi-region merge (for fixing overlaps, etc.)** + +Changes merge so you can merge more than two regions at a time. Currently only available inside HBase. HBASE-22827, a follow-on, is about exposing the facility in the Admin API (and then via the shell). + + +--- + +* [HBASE-22539](https://issues.apache.org/jira/browse/HBASE-22539) | *Blocker* | **WAL corruption due to early DBBs re-use when Durability.ASYNC\_WAL is used** + +We found a critical bug which can lead to WAL corruption when Durability.ASYNC\_WAL is used. The reason is that we release a ByteBuffer before actually persist the content into WAL file. + +The problem maybe lead to several errors, for example, ArrayIndexOfOutBounds when replaying WAL. This is because that the ByteBuffer is reused by others. + +ERROR org.apache.hadoop.hbase.executor.EventHandler: Caught throwable while processing event RS\_LOG\_REPLAY +java.lang.ArrayIndexOutOfBoundsException: 18056 + at org.apache.hadoop.hbase.KeyValue.getFamilyLength(KeyValue.java:1365) + at org.apache.hadoop.hbase.KeyValue.getFamilyLength(KeyValue.java:1358) + at org.apache.hadoop.hbase.PrivateCellUtil.matchingFamily(PrivateCellUtil.java:735) + at org.apache.hadoop.hbase.CellUtil.matchingFamily(CellUtil.java:816) + at org.apache.hadoop.hbase.wal.WALEdit.isMetaEditFamily(WALEdit.java:143) + at org.apache.hadoop.hbase.wal.WALEdit.isMetaEdit(WALEdit.java:148) + at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:297) + at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:195) + at org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:100) + +And may even cause segmentation fault and crash the JVM directly. You will see a hs\_err\_pidXXX.log file and usually the problem is SIGSEGV. This is usually because that the ByteBuffer has already been returned to the OS and used for other purpose. + +The problem has been reported several times in the past and this time Wellington Ramos Chevreuil provided the full logs and deeply analyzed the logs so we can find the root cause. And Lijin Bin figured out that the problem may only happen when Durability.ASYNC\_WAL is used. Thanks to them. + +The problem only effects the 2.x releases, all users are highly recommand to upgrade to a release which has this fix in, especially that if you use Durability.ASYNC\_WAL. + + +--- + +* [HBASE-22771](https://issues.apache.org/jira/browse/HBASE-22771) | *Major* | **[HBCK2] fixMeta method and server-side support** + +Adds a fixMeta method to hbck Service. Fixes holes in hbase:meta. Follow-up to fix overlaps. + +Follow-on is adding a client-side to hbase-operator-tools that can exploit this new addition (HBASE-22825) + + +--- + +* [HBASE-22737](https://issues.apache.org/jira/browse/HBASE-22737) | *Major* | **Add a new admin method and shell cmd to trigger the hbck chore to run** + +Add a new method runHbckChore in Hbck interface and a new shell cmd hbck\_chore\_run to request HBCK chore to run at master side. + + +--- + +* [HBASE-22741](https://issues.apache.org/jira/browse/HBASE-22741) | *Major* | **Show catalogjanitor consistency complaints in new 'HBCK Report' page** + +Adds a "CatalogJanitor hbase:meta Consistency Issues" section to the new 'HBCK Report' page added by HBASE-22709. This section is empty unless the most recent CatalogJanitor scan turned up problems. If so, will show table of issues found. + + +--- + +* [HBASE-22723](https://issues.apache.org/jira/browse/HBASE-22723) | *Major* | **Have CatalogJanitor report holes and overlaps; i.e. problems it sees when doing its regular scan of hbase:meta** + +When CatalogJanitor runs, it now checks for holes, overlaps, empty info:regioninfo columns and bad servers. Dumps findings into log. Follow-up adds report to new 'HBCK Report' linked off the Master UI. + +NOTE: All features but the badserver check made it into branch-2.1 and branch-2.0 backports. + + +--- + +* [HBASE-22709](https://issues.apache.org/jira/browse/HBASE-22709) | *Major* | **Add a chore thread in master to do hbck checking and display results in 'HBCK Report' page** + +1. Add a new chore thread in master to do hbck checking +2. Add a new web ui "HBCK Report" page to display checking results. + +This feature is enabled by default. And the hbck chore run per 60 minutes by default. You can config "hbase.master.hbck.checker.interval" to a value lesser than or equal to 0 for disabling the chore. + +Notice: the config "hbase.master.hbck.checker.interval" was renamed to "hbase.master.hbck.chore.interval" in HBASE-22737. + + +--- + +* [HBASE-22722](https://issues.apache.org/jira/browse/HBASE-22722) | *Blocker* | **Upgrade jackson databind dependencies to 2.9.9.1** + +Upgrade jackson databind dependency to 2.9.9.1 due to CVEs + +https://nvd.nist.gov/vuln/detail/CVE-2019-12814 + +https://nvd.nist.gov/vuln/detail/CVE-2019-12384 + + +--- + +* [HBASE-22527](https://issues.apache.org/jira/browse/HBASE-22527) | *Major* | **[hbck2] Add a master web ui to show the problematic regions** + +Add a new master web UI to show the potentially problematic opened regions. There are three case: +1. Master thought this region opened, but no regionserver reported it. +2. Master thought this region opened on Server1, but regionserver reported Server2 +3. More than one regionservers reported opened this region + + +--- + +* [HBASE-22690](https://issues.apache.org/jira/browse/HBASE-22690) | *Major* | **Deprecate / Remove OfflineMetaRepair in hbase-2+** + +OfflineMetaRepair is no longer supported in HBase-2+. Please refer to https://hbase.apache.org/book.html#HBCK2 + +This tool is deprecated in 2.x and will be removed in 3.0. + + +--- + +* [HBASE-22617](https://issues.apache.org/jira/browse/HBASE-22617) | *Blocker* | **Recovered WAL directories not getting cleaned up** + +In HBASE-20734 we moved the recovered.edits onto the wal file system but when constructing the directory we missed the BASE\_NAMESPACE\_DIR('data'). So when using the default config, you will find that there are lots of new directories at the same level with the 'data' directory. + +In this issue, we add the BASE\_NAMESPACE\_DIR back, and also try our best to clean up the wrong directories. But we can only clean up the region level directories, so if you want a clean fs layout on HDFS you still need to manually delete the empty directories at the same level with 'data'. + +The effect versions are 2.2.0, 2.1.[1-5], 1.4.[8-10], 1.3.[3-5]. + + +--- + +* [HBASE-22148](https://issues.apache.org/jira/browse/HBASE-22148) | *Blocker* | **Provide an alternative to CellUtil.setTimestamp** + +<!-- markdown --> + +The `CellUtil.setTimestamp` method changes to be an API with audience `LimitedPrivate(COPROC)` in HBase 3.0. With that designation the API should remain stable within a given minor release line, but may change between minor releases. + +Previously, this method was deprecated in HBase 2.0 for removal in HBase 3.0. Deprecation messages in HBase 2.y releases have been updated to indicate the expected API audience change. + + +--- + +* [HBASE-22226](https://issues.apache.org/jira/browse/HBASE-22226) | *Trivial* | **Incorrect level for headings in asciidoc** + +Warnings for level headings are corrected in the book for the HBase Incompatibilities section. + + +--- + +* [HBASE-22384](https://issues.apache.org/jira/browse/HBASE-22384) | *Minor* | **Formatting issues in administration section of book** + +Fixes a formatting issue in the administration section of the book, where listing indentation were a little bit off. + + +--- + +* [HBASE-22392](https://issues.apache.org/jira/browse/HBASE-22392) | *Trivial* | **Remove extra/useless +** + +Removed extra + in HRegion, HStore and LoadIncrementalHFiles for branch-2 and HRegion and HStore for branch-1. + + +--- + +* [HBASE-20494](https://issues.apache.org/jira/browse/HBASE-20494) | *Major* | **Upgrade com.yammer.metrics dependency** + +Updated metrics core from 3.2.1 to 3.2.6. + + +--- + +* [HBASE-22358](https://issues.apache.org/jira/browse/HBASE-22358) | *Minor* | **Change rubocop configuration for method length** + +The rubocop definition for the maximum method length was set to 75. + + +--- + +* [HBASE-22379](https://issues.apache.org/jira/browse/HBASE-22379) | *Minor* | **Fix Markdown for "Voting on Release Candidates" in book** + +Fixes the formatting of the "Voting on Release Candidates" to actually show the quote and code formatting of the RAT check. + + +--- + +* [HBASE-20851](https://issues.apache.org/jira/browse/HBASE-20851) | *Minor* | **Change rubocop config for max line length of 100** + +The rubocop configuration in the hbase-shell module now allows a line length with 100 characters, instead of 80 as before. For everything before 2.1.5 this change introduces rubocop itself. + + +--- + +* [HBASE-22152](https://issues.apache.org/jira/browse/HBASE-22152) | *Major* | **Create a jenkins file for yetus to processing GitHub PR** + +Add a new jenkins file for running pre commit check for GitHub PR. + + +--- + +* [HBASE-22123](https://issues.apache.org/jira/browse/HBASE-22123) | *Minor* | **REST gateway reports Insufficient permissions exceptions as 404 Not Found** + +When insufficient permissions, you now get: + +HTTP/1.1 403 Forbidden + +on the HTTP side, and in the message + +Forbidden +org.apache.hadoop.hbase.security.AccessDeniedException: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions for user âmyuser',action: get, tableName:mytable, family:cf. +at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.authorizeAccess(RangerAuthorizationCoprocessor.java:547) +and the rest of the ADE stack + + +--- + +* [HBASE-22100](https://issues.apache.org/jira/browse/HBASE-22100) | *Minor* | **False positive for error prone warnings in pre commit job** + +Now we will sort the javac WARNING/ERROR before generating diff in pre-commit so we can get a stable output for the error prone. The downside is that we just sort the output lexicographically so the line number will also be sorted lexicographically, which is a bit strange to human. + + + +# HBASE 2.0.5 Release Notes + +These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements. + + +--- + +* [HBASE-21636](https://issues.apache.org/jira/browse/HBASE-21636) | *Major* | **Enhance the shell scan command to support missing scanner specifications like ReadType, IsolationLevel etc.** + +Allows shell to set Scan options previously not exposed. See additions as part of the scan help by typing following hbase shell: + +hbase\> help 'scan' + + +--- + +* [HBASE-21727](https://issues.apache.org/jira/browse/HBASE-21727) | *Minor* | **Simplify documentation around client timeout** + +Deprecated HBaseConfiguration#getInt(Configuration, String, String, int) method and removed it from 3.0.0 version. + + +--- + +* [HBASE-21764](https://issues.apache.org/jira/browse/HBASE-21764) | *Major* | **Size of in-memory compaction thread pool should be configurable** + +Introduced an new config key in this issue: hbase.regionserver.inmemory.compaction.pool.size. the default value would be 10. you can configure this to set the pool size of in-memory compaction pool. Note that all memstores in one region server will share the same pool, so if you have many regions in one region server, you need to set this larger to compact faster for better read performance. + + +--- + +* [HBASE-21684](https://issues.apache.org/jira/browse/HBASE-21684) | *Major* | **Throw DNRIOE when connection or rpc client is closed** + +Make StoppedRpcClientException extend DoNotRetryIOException. + + +--- + +* [HBASE-21791](https://issues.apache.org/jira/browse/HBASE-21791) | *Blocker* | **Upgrade thrift dependency to 0.12.0** + +IMPORTANT: Due to security issues, all users who use hbase thrift should avoid using releases which do not have this fix. + +The effect releases are: +2.1.x: 2.1.2 and below +2.0.x: 2.0.4 and below +1.x: 1.4.x and below + +If you are using the effect releases above, please consider upgrading to a newer release ASAP. + + +--- + +* [HBASE-21734](https://issues.apache.org/jira/browse/HBASE-21734) | *Major* | **Some optimization in FilterListWithOR** + +After HBASE-21620, the filterListWithOR has been a bit slow because we need to merge each sub-filter's RC , while before HBASE-21620, we will skip many RC merging, but the logic was wrong. So here we choose another way to optimaze the performance: removing the KeyValueUtil#toNewKeyCell. +Anoop Sam John suggested that the KeyValueUtil#toNewKeyCell can save some GC before because if we copy key part of cell into a single byte[], then the block the cell refering won't be refered by the filter list any more, the upper layer can GC the data block quickly. while after HBASE-21620, we will update the prevCellList for every encountered cell now, so the lifecycle of cell in prevCellList for FilterList will be quite shorter. so just use the cell ref for saving cpu. +BTW, we removed all the arrays streams usage in filter list, because it's also quite time-consuming in our test. + + +--- + +* [HBASE-21738](https://issues.apache.org/jira/browse/HBASE-21738) | *Critical* | **Remove all the CSLM#size operation in our memstore because it's an quite time consuming.** + +We found the memstore snapshotting would cost much time because of calling the time-consuming ConcurrentSkipListMap#Size, it would make the p999 latency spike happen. So in this issue, we remove all ConcurrentSkipListMap#size in memstore by counting the cellsCount in MemstoreSizeing. As the issue described, the p999 latency spike was mitigated. + + +--- + +* [HBASE-21732](https://issues.apache.org/jira/browse/HBASE-21732) | *Critical* | **Should call toUpperCase before using Enum.valueOf in some methods for ColumnFamilyDescriptor** + +Now all the Enum configs in ColumnFamilyDescriptor can accept lower case config value. + + +--- + +* [HBASE-21712](https://issues.apache.org/jira/browse/HBASE-21712) | *Minor* | **Make submit-patch.py python3 compatible** + +Python3 support was added to dev-support/submit-patch.py. To install newly required dependencies run \`pip install -r dev-support/python-requirements.txt\` command. + + +--- + +* [HBASE-18569](https://issues.apache.org/jira/browse/HBASE-18569) | *Major* | **Add prefetch support for async region locator** + +Add prefetch support for async region locator. The default value is 10. Set 'hbase.client.locate.prefetch.limit' in hbase-site.xml if you want to use another value for it. + + + +# HBASE 2.0.4 Release Notes + +These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements. + + +--- + +* [HBASE-21635](https://issues.apache.org/jira/browse/HBASE-21635) | *Major* | **Use maven enforcer to ban imports from illegal packages** + +Use de.skuzzle.enforcer.restrict-imports-enforcer-rule extension for maven enforcer plugin to ban illegal imports at compile time. Now if you use illegal imports, for example, import com.google.common.\*, there will be a compile error, instead of a checkstyle warning. + + +--- + +* [HBASE-21401](https://issues.apache.org/jira/browse/HBASE-21401) | *Critical* | **Sanity check when constructing the KeyValue** + +Add a sanity check when constructing KeyValue from a byte[]. we use the constructor when we're reading kv from socket or HFIle or WAL(replication). the santiy check isn't designed for discovering the bits corruption in network transferring or disk IO. It is designed to detect bugs inside HBase in advance. and HBASE-21459 indicated that there's extremely small performance loss for diff kinds of keyvalue. + + +--- + +* [HBASE-21551](https://issues.apache.org/jira/browse/HBASE-21551) | *Blocker* | **Memory leak when use scan with STREAM at server side** + +<!-- markdown --> + + + + +### Summary +HBase clusters will experience Region Server failures due to out of memory errors due to a leak given any of the following: + +* User initiates Scan operations set to use the STREAM reading type +* User initiates Scan operations set to use the default reading type that read more than 4 * the block size of column families involved in the scan (e.g. by default 4*64KiB) +* Compactions run + +### Root cause + +When there are long running scans the Region Server process attempts to optimize access by using a different API geared towards sequential access. Due to an error in HBASE-20704 for HBase 2.0+ the Region Server fails to release related resources when those scans finish. That same optimization path is always used for the HBase internal file compaction process. + +### Workaround + +Impact for this error can be minimized by setting the config value âhbase.storescanner.pread.max.bytesâ to MAX_INT to avoid the optimization for default user scans. Clients should also be checked to ensure they do not pass the STREAM read type to the Scan API. This will have a severe impact on performance for long scans. + +Compactions always use this sequential optimized reading mechanism so downstream users will need to periodically restart Region Server roles after compactions have happened. + + +--- + +* [HBASE-21146](https://issues.apache.org/jira/browse/HBASE-21146) | *Minor* | **(2.0) Add ability for HBase Canary to ignore a configurable number of ZooKeeper down nodes** + +Adds -permittedZookeeperFailures \<N\> + +Makes it so Canary will keep running reporting on downed zk ensemble members rather than exit. + + +--- + +* [HBASE-21557](https://issues.apache.org/jira/browse/HBASE-21557) | *Major* | **Set version to 2.0.4 on branch-2.0 so can cut an RC** + +Set project version to 2.0.4 from 2.0.4-SNAPSHOT + + +--- + +* [HBASE-21544](https://issues.apache.org/jira/browse/HBASE-21544) | *Major* | **Backport HBASE-20734 Colocate recovered edits directory with hbase.wal.dir** + +This change moves the recovered.edits files which are created by the WALSplitter from the default filesystem into the WAL filesystem. This better enables the separate filesystem for WAL and HFile deployment model, by avoiding a check which requires that the HFile filesystem provides the hflush capability. + + + +# HBASE 2.0.3 Release Notes + +These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements. + + +--- + +* [HBASE-21496](https://issues.apache.org/jira/browse/HBASE-21496) | *Major* | **Set version to 2.0.3 on branch-2.0 in prep for first RC** + +Changed version on branch-2.0 from 2.0.2 to 2.0.3. + + +--- + +* [HBASE-21423](https://issues.apache.org/jira/browse/HBASE-21423) | *Major* | **Procedures for meta table/region should be able to execute in separate workers** + +The procedure for meta table will be executed in a separate worker thread named 'Urgent Worker' to avoid stuck. A new config named 'hbase.master.urgent.procedure.threads' is added, the default value for it is 1. To disable the separate worker, set it to 0. + + +--- + +* [HBASE-21417](https://issues.apache.org/jira/browse/HBASE-21417) | *Critical* | **Pre commit build is broken due to surefire plugin crashes** + +Add -Djdk.net.URLClassPath.disableClassPathURLCheck=true when executing surefire plugin. + + +--- + +* [HBASE-21191](https://issues.apache.org/jira/browse/HBASE-21191) | *Major* | **Add a holding-pattern if no assign for meta or namespace (Can happen if masterprocwals have been cleared).** + +Puts master startup into holding pattern if meta is not assigned (previous it would exit). To make progress again, operator needs to inject an assign (Caveats and instruction can be found in HBASE-21035). + + +--- + +* [HBASE-21237](https://issues.apache.org/jira/browse/HBASE-21237) | *Blocker* | **Use CompatRemoteProcedureResolver to dispatch open/close region requests to RS** + +Use CompatRemoteProcedureResolver instead of ExecuteProceduresRemoteCall to dispatch region open/close requests to RS. Since ExecuteProceduresRemoteCall will group all the open/close operations in one call and execute them sequentially on the target RS. If one operation fails, all the operation will be marked as failure. + + +--- + +* [HBASE-21322](https://issues.apache.org/jira/browse/HBASE-21322) | *Major* | **Add a scheduleServerCrashProcedure() API to HbckService** + +Adds scheduleServerCrashProcedure to the HbckService. + + +--- + +* [HBASE-21215](https://issues.apache.org/jira/browse/HBASE-21215) | *Major* | **Figure how to invoke hbck2; make it easy to find** + +Adds to bin/hbase means of invoking hbck2. Pass the new '-j' option on the 'hbck' command with a value of the full path to the HBCK2.jar. + +E.g: + +$ ./bin/hbase hbck -j ~/checkouts/hbase-operator-tools/hbase-hbck2/target/hbase-hbck2-1.0.0-SNAPSHOT.jar setTableState x ENABLED + + +--- + +* [HBASE-21372](https://issues.apache.org/jira/browse/HBASE-21372) | *Major* | **Set hbase.assignment.maximum.attempts to Long.MAX** + +Retry assigns 'forever' (or until an intervention such as a ServerCrashProcedure). + +Previous retry was a maximum of ten times but on failure, handling was an indeterminate. + + +--- + +* [HBASE-21338](https://issues.apache.org/jira/browse/HBASE-21338) | *Major* | **[balancer] If balancer is an ill-fit for cluster size, it gives little indication** + +The description claims the balancer not dynamically configurable but this is an error; it is http://hbase.apache.org/book.html#dyn\_config + +Also, if balancer is seen to be cutting out too soon, try setting "hbase.master.balancer.stochastic.runMaxSteps" to true. + +Adds cleaner logging around balancer start. + + +--- + +* [HBASE-21073](https://issues.apache.org/jira/browse/HBASE-21073) | *Major* | **"Maintenance mode" master** + + Instead of being an ephemeral state set by hbck, maintenance mode is now + an explicit toggle set by either configuration property or environment + variable. In maintenance mode, master will host system tables and not + assign any user-space tables to RSs. This gives operators the ability to + affect repairs to meta table with fewer moving parts. + + +--- + +* [HBASE-21075](https://issues.apache.org/jira/browse/HBASE-21075) | *Blocker* | **Confirm that we can (rolling) upgrade from 2.0.x and 2.1.x to 2.2.x after HBASE-20881** + +hbase-2.2.x uses a new Procedure form assiging/unassigning/moving Regions; it does not process hbase-2.1.x and earlier Unassign/Assign Procedure types. Upgrade requires that we first drain the Master Procedure Store of old style Procedures before starting the new 2.2.x Master. The patch here facilitates the draining process. + +On your running hbase-2.1.1+ (or 2.0.3+ cluster), when upgrading: + +1. Shutdown both active and standby masters (Your cluster will continue to server reads and writes without interruption). +2. Set the property hbase.procedure.upgrade-to-2-2 to true in hbase-site.xml for the Master, and start only one Master, still using the 2.1.1+ (or 2.0.3+) binaries. +3. Wait until the Master quits. Confirm that there is a 'READY TO ROLLING UPGRADE' message in the master log as the cause of the shutdown. The Procedure Store is now empty. +4. Start new Masters with the new 2.2.0+ code. + + +--- + +* [HBASE-21335](https://issues.apache.org/jira/browse/HBASE-21335) | *Critical* | **Change the default wait time of HBCK2 tool** + +Changed waitTime parameter to lockWait on bypass. Changed default waitTime from 0 -- i.e. wait for ever -- to 1ms so if lock is held, we'll go past it and if override enforce bypass. + + +--- + +* [HBASE-21291](https://issues.apache.org/jira/browse/HBASE-21291) | *Major* | **Add a test for bypassing stuck state-machine procedures** + +bypass will now throw an Exception if passed a lockWait \<= 0; i.e bypass will prevent an operator getting stuck on an entity lock waiting forever (lockWait == 0) + + +--- + +* [HBASE-21320](https://issues.apache.org/jira/browse/HBASE-21320) | *Major* | **[canary] Cleanup of usage and add commentary** + +Cleans up usage and docs around Canary. Does not change command-line args (though we should -- smile). + + +--- + +* [HBASE-21158](https://issues.apache.org/jira/browse/HBASE-21158) | *Critical* | **Empty qualifier cell should not be returned if it does not match QualifierFilter** + +<!-- markdown --> + + +Scans that make use of `QualifierFilter` previously would erroneously return both columns with an empty qualifier along with those that matched. After this change that behavior has changed to only return those columns that match. + + +--- + +* [HBASE-21185](https://issues.apache.org/jira/browse/HBASE-21185) | *Minor* | **WALPrettyPrinter: Additional useful info to be printed by wal printer tool, for debugability purposes** + +This adds two extra features to WALPrettyPrinter tool: + +1) Output for each cell combined size of cell descriptors, plus the cell value itself, in a given WAL edit. This is printed on the results as "cell total size sum:" info by default; + +2) An optional -g/--goto argument, that allows to seek straight to that specific WAL file position, then sequentially reading the WAL from that point towards its end; + + +--- + +* [HBASE-21287](https://issues.apache.org/jira/browse/HBASE-21287) | *Major* | **JVMClusterUtil Master initialization wait time not configurable** + +Local HBase cluster (as used by unit tests) wait times on startup and initialization can be configured via \`hbase.master.start.timeout.localHBaseCluster\` and \`hbase.master.init.timeout.localHBaseCluster\` + + +--- + +* [HBASE-21280](https://issues.apache.org/jira/browse/HBASE-21280) | *Trivial* | **Add anchors for each heading in UI** + +Adds anchors #tables, #tasks, etc. + + +--- + +* [HBASE-21223](https://issues.apache.org/jira/browse/HBASE-21223) | *Critical* | **[amv2] Remove abort\_procedure from shell** + +Removed the abort\_procedure command from shell -- dangerous -- and deprecated abortProcedure in Admin API. + + + + + +# HBASE 2.0.2 Release Notes + +These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements. + +--- + +* [HBASE-20942](https://issues.apache.org/jira/browse/HBASE-20942) | *Major* | **Improve RpcServer TRACE logging** + +Allows configuration of the length of RPC messages printed to the log at TRACE level via "hbase.ipc.trace.param.size" in RpcServer. + + +--- + +* [HBASE-20649](https://issues.apache.org/jira/browse/HBASE-20649) | *Minor* | **Validate HFiles do not have PREFIX\_TREE DataBlockEncoding** + +<!-- markdown --> + + + +Users who have previously made use of prefix tree encoding can now check that their existing HFiles no longer contain data that uses it with an additional preupgrade check command. + +``` +hbase pre-upgrade validate-hfile +``` + +Please see the "HFile Content validation" section of the ref guide's coverage of the pre-upgrade validator tool for usage details. + + +--- + +* [HBASE-20941](https://issues.apache.org/jira/browse/HBASE-20941) | *Major* | **Create and implement HbckService in master** + +Adds an HBCK Service and a first method to force-change-in-table-state for use by an HBCK client effecting 'repair' to a malfunctioning HBase. + + +--- + +* [HBASE-21072](https://issues.apache.org/jira/browse/HBASE-21072) | *Major* | **Block out HBCK1 in hbase2** + +Fence out hbase-1.x hbck1 instances. Stop them making state changes on an hbase-2.x cluster; they could do damage. We do this by writing the hbck1 lock file into place on hbase-2.x Master start-up. + +To disable this new behavior, set hbase.write.hbck1.lock.file to false + + +--- + +* [HBASE-21012](https://issues.apache.org/jira/browse/HBASE-21012) | *Critical* | **Revert the change of serializing TimeRangeTracker** + +HFiles generated by 2.0.0, 2.0.1, 2.1.0 are not forward compatible to 1.4.6-, 1.3.2.1-, 1.2.6.1-, and other inactive releases. Why HFile lose compatability is hbase in new versions (2.0.0, 2.0.1, 2.1.0) use protobuf to serialize/deserialize TimeRangeTracker (TRT) while old versions use DataInput/DataOutput. To solve this, We have to put HBASE-21012 to 2.x and put HBASE-21013 in 1.x. For more information, please check HBASE-21008. + + +--- + +* [HBASE-20813](https://issues.apache.org/jira/browse/HBASE-20813) | *Minor* | **Remove RPC quotas when the associated table/Namespace is dropped off** + +In previous releases, when a Space Quota was configured on a table or namespace and that table or namespace was deleted, the Space Quota was also deleted. This change improves the implementation so that the same is also done for RPC Quotas. + + +--- + +* [HBASE-20856](https://issues.apache.org/jira/browse/HBASE-20856) | *Minor* | **PITA having to set WAL provider in two places** + +With this change if a WAL's meta provider (hbase.wal.meta\_provider) is not explicitly set, it now defaults to whatever hbase.wal.provider is set to. Previous, the two settings operated independently, each with its own default. + +This change is operationally incompatible with previous HBase versions because the default WAL meta provider no longer defaults to AsyncFSWALProvider but to hbase.wal.provider. + +The thought is that this is more in line with an operator's expectation, that a change in hbase.wal.provider is sufficient to change how WALs are written, especially given hbase.wal.meta\_provider is an obscure configuration and that the very idea that meta regions would have their own wal provider would likely come as a surprise. + + +--- + +* [HBASE-20538](https://issues.apache.org/jira/browse/HBASE-20538) | *Critical* | **Upgrade our hadoop versions to 2.7.7 and 3.0.3** + +Update hadoop-two.version to 2.7.7 and hadoop-three.version to 3.0.3 due to a JDK issue which is solved by HADOOP-15473. + + +--- + +* [HBASE-20884](https://issues.apache.org/jira/browse/HBASE-20884) | *Major* | **Replace usage of our Base64 implementation with java.util.Base64** + +Class org.apache.hadoop.hbase.util.Base64 has been removed in it's entirety from HBase 2+. In HBase 1, unused methods have been removed from the class and the audience was changed from Public to Private. This class was originally intended as an internal utility class that could be used externally but thinking since changed; these classes should not have been advertised as public to end-users. + +This represents an incompatible change for users who relied on this implementation. An alternative implementation for affected clients is available at java.util.Base64 when using Java 8 or newer; be aware, it may encode/decode differently. For clients seeking to restore this specific implementation, it is available in the public domain for download at http://iharder.sourceforge.net/current/java/base64/ + + +--- + +* [HBASE-20691](https://issues.apache.org/jira/browse/HBASE-20691) | *Blocker* | **Storage policy should allow deferring to HDFS** + +After HBASE-20691 we have changed the default setting of hbase.wal.storage.policy from "HOT" back to "NONE" which means we defer the policy to HDFS. This fixes the problem of release 2.0.0 that the storage policy of WAL directory will defer to HDFS and may not be "HOT" even if you explicitly set hbase.wal.storage.policy to "HOT" + + +--- + +* [HBASE-20839](https://issues.apache.org/jira/browse/HBASE-20839) | *Blocker* | **Fallback to FSHLog if we can not instantiated AsyncFSWAL when user does not specify AsyncFSWAL explicitly** + +As we hack into the internal of DFSClient when implementing AsyncFSWAL to get better performance, a patch release of hadoop can make it broken. + +So now, if user does not specify a wal provider, then we will first try to use 'asyncfs', i.e, the AsyncFSWALProvider. If we fail due to some compatible issues, we will fallback to 'filesystem', i.e, FSHLog. + + +--- + +* [HBASE-20244](https://issues.apache.org/jira/browse/HBASE-20244) | *Blocker* | **NoSuchMethodException when retrieving private method decryptEncryptedDataEncryptionKey from DFSClient** + +HDFS-12574 made an incompatible change to HdfsKMSUtil with different method signature. +This issue uses reflection to try known method signatures in order to work with hadoop releases with and without HDFS-12574 + + +--- + +* [HBASE-20642](https://issues.apache.org/jira/browse/HBASE-20642) | *Major* | **IntegrationTestDDLMasterFailover throws 'InvalidFamilyOperationException** + +This changes client-side nonce generation to use the same nonce for re-submissions of client RPC DDL operations. + + +--- + +* [HBASE-20745](https://issues.apache.org/jira/browse/HBASE-20745) | *Major* | **Log when master proc wal rolls** + +Log Master WAL Proc at INFO level so can tell where we transition; will help debugging/figuring accounting. Also change DEFAULT\_RIT\_CHORE\_INTERVAL\_MSEC from 5 to 60 seconds; makes it so we emit STUCK RIT notice once a minute only rather than 12 times a minute (latter was causing us to quickly roll-away the logging around problem 'events'). + + +--- + +* [HBASE-20681](https://issues.apache.org/jira/browse/HBASE-20681) | *Major* | **IntegrationTestDriver fails after HADOOP-15406 due to missing hamcrest-core** + +<!-- markdown --> + + + + +HBase now relies on an internal mechanism to determine when it is running a local hbase cluster meant for external interaction vs an encapsulated test. When created via the `HBaseTestingUtility`, ports for Master and RegionServer services and UIs will be set to random ports to allow for multiple parallel uses on a single machine. Normally when running a Standalone HBase Deployment (as described in the HBase Reference Guide) the ports will be picked according to the same defaults used in a full cluster set up. If you wish to instead use the random port assignment set `hbase.localcluster.assign.random.ports` to true. + + +--- + +* [HBASE-20004](https://issues.apache.org/jira/browse/HBASE-20004) | *Minor* | **Client is not able to execute REST queries in a secure cluster** + +Added 'hbase.rest.http.allow.options.method' configuration property to allow user to decide whether Rest Server HTTP should allow OPTIONS method or not. By default it is enabled in HBase 2.1.0+ versions and in other versions it is disabled. +Similarly 'hbase.thrift.http.allow.options.method' is added HBase 1.5, 2.1.0 and 3.0.0 versions. It is disabled by default. + + +--- + +* [HBASE-20327](https://issues.apache.org/jira/browse/HBASE-20327) | *Minor* | **When qualifier is not specified, append and incr operation do not work (shell)** + +This change will enable users to perform append and increment operation with null qualifier via hbase-shell. + + +--- + +* [HBASE-18842](https://issues.apache.org/jira/browse/HBASE-18842) | *Minor* | **The hbase shell clone\_snaphost command returns bad error message** + +<!-- markdown --> + + + +When attempting to clone a snapshot but using a namespace that does not exist, the HBase shell will now correctly report the exception as caused by the passed namespace. Previously, the shell would report that the problem was an unknown namespace but it would claim the user provided table name was not found as a namespace. Both before and after this change the shell properly used the passed namespace to attempt to handle the request. + + + + +# HBASE 2.0.0 Release Notes + + +These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements. + + +--- + +* [HBASE-20464](https://issues.apache.org/jira/browse/HBASE-20464) | *Major* | **Disable IMC** + +Change the default so that on creation of new tables, In-Memory Compaction BASIC is NOT enabled. + +This change is in branch-2.0 only, not in branch-2. + + +--- + +* [HBASE-20276](https://issues.apache.org/jira/browse/HBASE-20276) | *Blocker* | **[shell] Revert shell REPL change and document** + +<!-- markdown --> + + + +The HBase shell now behaves as it did prior to the changes that started in HBASE-15965. Namely, some shell commands return values that may be further manipulated within the shell's IRB session. + +The command line option `--return-values` is no longer acted on by the shell since it now always behaves as it did when passed this parameter. Passing the option results in a harmless warning about this change. + +Users who wish to maintain the behavior seen in the 1.4.0-1.4.2 releases of the HBase shell should refer to the section _irbrc_ in the reference guide for how to configure their IRB session to avoid echoing expression results to the console. + + +--- + +* [HBASE-18792](https://issues.apache.org/jira/browse/HBASE-18792) | *Blocker* | **hbase-2 needs to defend against hbck operations** + +As of HBase version 2.0, the hbck tool is significantly changed. In general, all Read-Only options are supported and can be be used safely. Most -fix/ -repair options are NOT supported. Please see usage below for details on which options are not supported: + + +Usage: fsck [opts] {only tables} + where [opts] are: + -help Display help options (this) + -details Display full report of all regions. + -timelag \<timeInSeconds\> Process only regions that have not experienced any metadata updates in the last \<timeInSeconds\> seconds. + -sleepBeforeRerun \<timeInSeconds\> Sleep this many seconds before checking if the fix worked if run with -fix + -summary Print only summary of the tables and status. + -metaonly Only check the state of the hbase:meta table. + -sidelineDir \<hdfs://\> HDFS path to backup existing meta. + -boundaries Verify that regions boundaries are the same between META and store files. + -exclusive Abort if another hbck is exclusive or fixing. + + Datafile Repair options: (expert features, use with caution!) + -checkCorruptHFiles Check all Hfiles by opening them to make sure they are valid + -sidelineCorruptHFiles Quarantine corrupted HFiles. implies -checkCorruptHFiles + + Replication options + -fixReplication Deletes replication queues for removed peers + + Metadata Repair options supported as of version 2.0: (expert features, use with caution!) + -fixVersionFile Try to fix missing hbase.version file in hdfs. + -fixReferenceFiles Try to offline lingering reference store files + -fixHFileLinks Try to offline lingering HFileLinks + -noHdfsChecking Don't load/check region info from HDFS. Assumes hbase:meta region info is good. Won't check/fix any HDFS issue, e.g. hole, orphan, or overlap + -ignorePreCheckPermission ignore filesystem permission pre-check + +NOTE: Following options are NOT supported as of HBase version 2.0+. + + UNSUPPORTED Metadata Repair options: (expert features, use with caution!) + -fix Try to fix region assignments. This is for backwards compatiblity + -fixAssignments Try to fix region assignments. Replaces the old -fix + -fixMeta Try to fix meta problems. This assumes HDFS region info is good. + -fixHdfsHoles Try to fix region holes in hdfs. + -fixHdfsOrphans Try to fix region dirs with no .regioninfo file in hdfs + -fixTableOrphans Try to fix table dirs with no .tableinfo file in hdfs (online mode only) + -fixHdfsOverlaps Try to fix region overlaps in hdfs. + -maxMerge \<n\> When fixing region overlaps, allow at most \<n\> regions to merge. (n=5 by default) + -sidelineBigOverlaps When fixing region overlaps, allow to sideline big overlaps + -maxOverlapsToSideline \<n\> When fixing region overlaps, allow at most \<n\> regions to sideline per group. (n=2 by default) + -fixSplitParents Try to force offline split parents to be online. + -removeParents Try to offline and sideline lingering parents and keep daughter regions. + -fixEmptyMetaCells Try to fix hbase:meta entries not referencing any region (empty REGIONINFO\_QUALIFIER rows) + + UNSUPPORTED Metadata Repair shortcuts + -repair Shortcut for -fixAssignments -fixMeta -fixHdfsHoles -fixHdfsOrphans -fixHdfsOverlaps -fixVersionFile -sidelineBigOverlaps -fixReferenceFiles-fixHFileLinks + -repairHoles Shortcut for -fixAssignments -fixMeta -fixHdfsHoles + + +--- + +* [HBASE-19994](https://issues.apache.org/jira/browse/HBASE-19994) | *Major* | **Create a new class for RPC throttling exception, make it retryable.** + +A new RpcThrottlingException deprecates ThrottlingException. The new RpcThrottlingException is a retryable Exception that clients will retry when Rpc throttling quota is exceeded. The deprecated ThrottlingException is a nonretryable Exception. + + +--- + +* [HBASE-20224](https://issues.apache.org/jira/browse/HBASE-20224) | *Blocker* | **Web UI is broken in standalone mode** + +Standalone webui was broken inadvertently by HBASE-20027. + + +--- + +* [HBASE-18784](https://issues.apache.org/jira/browse/HBASE-18784) | *Major* | **Use of filesystem that requires hflush / hsync / append / etc should query outputstream capabilities** + +<!-- markdown --> + + + +If HBase is run on top of Apache Hadoop libraries that support the needed APIs it will verify that underlying Filesystem implementations provide the needed durability mechanisms to safely operate. The needed APIs *should* be present in Hadoop 3 release and Hadoop 2 releases starting in the Hadoop 2.9 series. If the APIs are not available, HBase behaves as it has in previous releases (that is, it moves forward assuming such a check would pass). + +Where this check fails, it is unsafe to rely on HBase in a production setting. In the event of process or node failure, the HBase RegionServer process may fail to have access to all the data it previously wrote to its write ahead log, resulting in data loss. In the event of process or node failure, the HBase master process may lose all or part of the write ahead log that it relies on for cluster management operations, leaving the cluster in an inconsistent state that we aren't sure it could recover from. + +Notably, the LocalFileSystem implementation provided by Hadoop reports (accurately) via these new APIs that it can not provide the durability HBase needs to operate. As such, the current instructions for single-node HBase operation have been updated both with a) how to bypass this safety check and b) a strong warning about the dire consequences of doing so outside of a dev/test environment. + + +--- + +* [HBASE-20219](https://issues.apache.org/jira/browse/HBASE-20219) | *Critical* | **An error occurs when scanning with reversed=true and loadColumnFamiliesOnDemand=true** + +Throws DoNotRetryIOException when you ask for a reverse scan loading adjacent column families on demand. Previous it threw IllegalStateException + + +--- + +* [HBASE-20358](https://issues.apache.org/jira/browse/HBASE-20358) | *Minor* | **Fix bin/hbase thrift usage text** + +Cleanup usage message and command-line processing (no functional change). + + +--- + +* [HBASE-20182](https://issues.apache.org/jira/browse/HBASE-20182) | *Blocker* | **Can not locate region after split and merge** + +Now if we hit a split parent when locating a region, we will skip to the next row and try again until the region does not contain our row. So there will be no RegionOfflineException for a split parent any more, instead, if the split children have not been onlined yet, i.e, we finally arrive at a region which does not contain our row, an IOException will be thrown. + + +--- + +* [HBASE-20149](https://issues.apache.org/jira/browse/HBASE-20149) | *Critical* | **Purge dev javadoc from bin tarball (or make a separate tarball of javadoc)** + +We no longer include dev or dev test javadocs in our binary bundle. We still build them; they are just not included because they were half the size of the resultant tarball. + +Here is our story on javadoc as of this commit: + + \* apidocs - user facing main api javadocs. currently for a release line, published on website and linked from menu. included in the bin tarball + \* devapidocs - hbase internal javadocs. currently for a release line, published on the website but not linked from the menu. no longer included in the bin tarball. + \* testapidocs - user facing test scope api javadocs. currently for a release line, not published. included in the bin tarball. + \* testdevapidocs - hbase internal test scope javadocs. currently for a release line, not published. no longer included in the bin tarball + + +--- + +* [HBASE-18828](https://issues.apache.org/jira/browse/HBASE-18828) | *Blocker* | **[2.0] Generate CHANGES.txt** + +Moves us over to yetus releasedocmaker tooling generating CHANGES. CHANGES is not markdown (CHANGES.md) as opposed to CHANGES.txt. We've also added a new RELEASENOTES.md that lists JIRA release notes (courtesy of releasedocmaker). + +CHANGES/RELEASENOTES are current as of now. Will need a 'freshening' when we cut the RC. + + +--- + +* [HBASE-14175](https://issues.apache.org/jira/browse/HBASE-14175) | *Critical* | **Adopt releasedocmaker for better generated release notes** + +We will use yetus releasedocmaker to make our changes doc from here on out. A CHANGELOG.md will replace our current CHANGES.txt. Adjacent, we'll keep up a RELEASENOTES.md doc courtesy of releasedocmaker. + +Over in HBASE-18828 is where we are working through steps for the RM integrating this new tooling. + + +--- + +* [HBASE-16499](https://issues.apache.org/jira/browse/HBASE-16499) | *Critical* | **slow replication for small HBase clusters** + +Changed the default value for replication.source.ratio from 0.1 to 0.5. Which means now by default 50% of the total RegionServers in peer cluster(s) will participate in replication. + + +--- + +* [HBASE-16459](https://issues.apache.org/jira/browse/HBASE-16459) | *Trivial* | **Remove unused hbase shell --format option** + +<!-- markdown --> + + + + +The HBase `shell` command no longer recognizes the option `--format`. Previously this option only recognized the default value of 'console'. The default value is now always used. + + +--- + +* [HBASE-20259](https://issues.apache.org/jira/browse/HBASE-20259) | *Critical* | **Doc configs for in-memory-compaction and add detail to in-memory-compaction logging** + +Disables in-memory compaction as default. + +Adds logging of in-memory compaction configuration on creation. + +Adds a chapter to the refguide on this new feature. + + +--- + +* [HBASE-20282](https://issues.apache.org/jira/browse/HBASE-20282) | *Major* | **Provide short name invocations for useful tools** + +\`hbase regionsplitter\` is a new short invocation for \`hbase org.apache.hadoop.hbase.util.RegionSplitter\` + + +--- + +* [HBASE-20314](https://issues.apache.org/jira/browse/HBASE-20314) | *Major* | **Precommit build for master branch fails because of surefire fork fails** + +Upgrade surefire plugin to 2.21.0. + + +--- + +* [HBASE-20130](https://issues.apache.org/jira/browse/HBASE-20130) | *Critical* | **Use defaults (16020 & 16030) as base ports when the RS is bound to localhost** + +<!-- markdown --> + + + +When region servers bind to localhost (mostly in pseudo distributed mode), default ports (16020 & 16030) are used as base ports. This will support up to 9 instances of region servers by default with `local-regionservers.sh` script. If additional instances are needed, see the reference guide on how to deploy with a different range using the environment variables `HBASE_RS_BASE_PORT` and `HBASE_RS_INFO_BASE_PORT`. + + +--- + +* [HBASE-20111](https://issues.apache.org/jira/browse/HBASE-20111) | *Critical* | **Able to split region explicitly even on shouldSplit return false from split policy** + +When a split is requested on a Region, the RegionServer hosting that Region will now consult the configured SplitPolicy for that table when determining if a split of that Region is allowed. When a split is disallowed (due to the Region not being OPEN or the SplitPolicy denying the request), the operation will \*not\* be implicitly retried as it has previously done. Users will need to guard against and explicitly retry region split requests which are denied by the system. + + +--- + +* [HBASE-20223](https://issues.apache.org/jira/browse/HBASE-20223) | *Blocker* | **Use hbase-thirdparty 2.1.0** + +Moves commons-cli and commons-collections4 into the HBase thirdparty shaded jar which means that these are no longer generally available for users on the classpath. + + +--- + +* [HBASE-19128](https://issues.apache.org/jira/browse/HBASE-19128) | *Major* | **Purge Distributed Log Replay from codebase, configurations, text; mark the feature as unsupported, broken.** + +Removes Distributed Log Replay feature. Disable the feature before upgrading. + + +--- + +* [HBASE-19504](https://issues.apache.org/jira/browse/HBASE-19504) | *Major* | **Add TimeRange support into checkAndMutate** + +1) checkAndMutate accept a TimeRange to query the specified cell +2) remove writeToWAL flag from Region#checkAndMutate since it is useless (this is a incompatible change) + + +--- + +* [HBASE-20237](https://issues.apache.org/jira/browse/HBASE-20237) | *Critical* | **Put back getClosestRowBefore and throw UnknownProtocolException instead... for asynchbase client** + +Throw UnknownProtocolException if a client connects and tries to invoke the old getClosestRowOrBefore method. Pre-hbase-1.0.0 or asynchbase do this instead of using its replacement, the reverse Scan. + +getClosestRowOrBefore was implemented as a flag on Get. Before this patch though the flag was set, hbase2 were ignoring it. This made it look like a pre-1.0.0 client was 'working' but then it'd fail finding the appropriate Region for a client-specified row doing lookups into hbase:meta. + + +--- + +* [HBASE-20247](https://issues.apache.org/jira/browse/HBASE-20247) | *Major* | **Set version as 2.0.0 in branch-2.0 in prep for first RC** + +Set version as 2.0.0 on branch-2.0. + + +--- + +* [HBASE-20090](https://issues.apache.org/jira/browse/HBASE-20090) | *Major* | **Properly handle Preconditions check failure in MemStoreFlusher$FlushHandler.run** + +When there is concurrent region split, MemStoreFlusher may not find flushable region if the only candidate region left hasn't received writes (resulting in 0 data size). +After this JIRA, such scenario wouldn't trigger Precondition assertion (replaced by an if statement to see whether there is any flushable region). +If there is no flushable region, a DEBUG log would appear in region server log, saying "Above memory mark but there is no flushable region". + + +--- + +* [HBASE-19552](https://issues.apache.org/jira/browse/HBASE-19552) | *Major* | **update hbase to use new thirdparty libs** + +hbase-thirdparty libs have moved to o.a.h.thirdparty offset. Netty shading system property is no longer necessary. + + +--- + +* [HBASE-20119](https://issues.apache.org/jira/browse/HBASE-20119) | *Minor* | **Introduce a pojo class to carry coprocessor information in order to make TableDescriptorBuilder accept multiple cp at once** + +1) Make all methods in TableDescriptorBuilder be setter pattern. +addCoprocessor -\> setCoprocessor +addColumnFamily -\> setColumnFamily +(addCoprocessor and addColumnFamily are still in branch-2 but they are marked as deprecated) +2) add CoprocessorDescriptor to carry cp information +3) add CoprocessorDescriptorBuilder to build CoprocessorDescriptor +4) TD disallow user to set negative priority to coprocessor since parsing the negative value will cause a exception + + +--- + +* [HBASE-17165](https://issues.apache.org/jira/browse/HBASE-17165) | *Critical* | **Add retry to LoadIncrementalHFiles tool** + +Adds retry to load of incremental hfiles. Pertinent key is HConstants.HBASE\_CLIENT\_RETRIES\_NUMBER. Default is HConstants.DEFAULT\_HBASE\_CLIENT\_RETRIES\_NUMBER. + + +--- + +* [HBASE-20108](https://issues.apache.org/jira/browse/HBASE-20108) | *Critical* | **\`hbase zkcli\` falls into a non-interactive prompt after HBASE-15199** + +This issue fixes a runtime dependency issues where JLine is not made available on the classpath which causes the ZooKeeper CLI to appear non-interactive. JLine was being made available unintentionally via the JRuby jar file on the classpath for the HBase shell. While the JRuby jar is not always present, the fix made here was to selectively include the JLine dependency on the zkcli command's classpath. + + +--- + +* [HBASE-8770](https://issues.apache.org/jira/browse/HBASE-8770) | *Blocker* | **deletes and puts with the same ts should be resolved according to mvcc/seqNum** + +This behavior is available as a new feature. See HBASE-15968 release note. + +This issue is just about adding to the refguide documentation on the HBASE\_15968 feature. + + +--- + +* [HBASE-19114](https://issues.apache.org/jira/browse/HBASE-19114) | *Major* | **Split out o.a.h.h.zookeeper from hbase-server and hbase-client** + +Splits out most of ZooKeeper related code into a separate new module: hbase-zookeeper. +Also, renames some ZooKeeper related classes to follow a common naming pattern - "ZK" prefix - as compared to many different styles earlier. + + +--- + +* [HBASE-19437](https://issues.apache.org/jira/browse/HBASE-19437) | *Critical* | **Batch operation can't handle the null result for Append/Increment** + +The result from server is changed from null to Result.EMPTY\_RESULT when Append/Increment operation can't retrieve any data from server, + + +--- + +* [HBASE-17448](https://issues.apache.org/jira/browse/HBASE-17448) | *Major* | **Export metrics from RecoverableZooKeeper** + +Committed to master and branch-1 + + +--- + +* [HBASE-19400](https://issues.apache.org/jira/browse/HBASE-19400) | *Major* | **Add missing security checks in MasterRpcServices** + +Added ACL check to following Admin functions: +enableCatalogJanitor, runCatalogJanitor, cleanerChoreSwitch, runCleanerChore, execProcedure, execProcedureWithReturn, normalize, normalizerSwitch, coprocessorService. +When ACL is enabled, only those with ADMIN rights will be able to invoke these operations successfully. + + +--- + +* [HBASE-20048](https://issues.apache.org/jira/browse/HBASE-20048) | *Blocker* | **Revert serial replication feature** + +Revert the serial replication feature from all branches. Plan to reimplement it soon and land onto 2.1 release line. + + +--- + +* [HBASE-19166](https://issues.apache.org/jira/browse/HBASE-19166) | *Blocker* | **AsyncProtobufLogWriter persists ProtobufLogWriter as class name for backward compatibility** + +For backward compatibility, AsyncProtobufLogWriter uses "ProtobufLogWriter" as writer class name and SecureAsyncProtobufLogWriter uses "SecureProtobufLogWriter" as writer class name. + + +--- + +* [HBASE-18596](https://issues.apache.org/jira/browse/HBASE-18596) | *Blocker* | **[TEST] A hbase1 cluster should be able to replicate to a hbase2 cluster; verify** + +Replication between versions verified as basically working. 0.98.25-SNAPSHOT to beta-2 hbase2 and a 1.2-ish version tried. + + +--- + +* [HBASE-20017](https://issues.apache.org/jira/browse/HBASE-20017) | *Blocker* | **BufferedMutatorImpl submit the same mutation repeatedly** + +This change fixes multithreading issues in the implementation of BufferedMutator. BufferedMutator should not be used with 1.4 releases prior to 1.4.2. + + +--- + +* [HBASE-20032](https://issues.apache.org/jira/browse/HBASE-20032) | *Minor* | **Receving multiple warnings for missing reporting.plugins.plugin.version** + +Add (latest) version elements missing from reporting plugins in top-level pom. + + +--- + +* [HBASE-19954](https://issues.apache.org/jira/browse/HBASE-19954) | *Major* | **Separate TestBlockReorder into individual tests to avoid ShutdownHook suppression error against hadoop3** + +hadoop3 minidfscluster removes all shutdown handlers when the cluster goes down which made this test that does FS-stuff fail (Fix was to break up the test so each test method ran with an unadulterated FS). + + +--- + +* [HBASE-20014](https://issues.apache.org/jira/browse/HBASE-20014) | *Major* | **TestAdmin1 Times out** + +Ups the overall test timeout from 10 minutes to 13minutes. 15minutes is the surefire timeout. + + +--- + +* [HBASE-20020](https://issues.apache.org/jira/browse/HBASE-20020) | *Critical* | **Make sure we throw DoNotRetryIOException when ConnectionImplementation is closed** + +Add checkClosed to core Client methods. Avoid unnecessary retry. + + +--- + +* [HBASE-19978](https://issues.apache.org/jira/browse/HBASE-19978) | *Major* | **The keepalive logic is incomplete in ProcedureExecutor** + +Completes keep-alive logic and then enables it; ProcedureExecutor Workers will spin up more threads when need settling back to the core count after the burst in demand has passed. Default keep-alive is one minute. Default core-count is CPUs/4 or 16, which ever is greater. Maximum is an arbitrary core-count \* 10 (a limit that should never be hit and if it is, there is something else very wrong). + + +--- + +* [HBASE-19950](https://issues.apache.org/jira/browse/HBASE-19950) | *Minor* | **Introduce a ColumnValueFilter** + +ColumnValueFilter provides a way to fetch matched cells only by providing specified column, value and a comparator, which is different from SingleValueFilter, fetching an entire row as soon as a matched cell found. + + +--- + +* [HBASE-18294](https://issues.apache.org/jira/browse/HBASE-18294) | *Major* | **Reduce global heap pressure: flush based on heap occupancy** + +A region is flushed if its memory component exceeds the region flush threshold. +A flush policy decides which stores to flush by comparing the size of the store to a column-family-flush threshold. +If the overall size of all memstores in the machine exceeds the bounds defined by the administrator (denoted global pressure) a region is selected and flushed. +HBASE-18294 changes flush decisions to be based on heap-occupancy and not data (key-value) size, consistently across levels. This rolls back some of the changes by HBASE-16747. Specifically, +(1) RSs, Regions and stores track their overall on-heap and off-heap occupancy, +(2) A region is flushed when its on-heap+off-heap size exceeds the region flush threshold specified in hbase.hregion.memstore.flush.size, +(3) The store to be flushed is chosen based on its on-heap+off-heap size +(4) At the RS level, a flush is triggered when the overall on-heap exceeds the on-heap limit, or when the overall off-heap size exceeds the off-heap limit (low/high water marks). + +Note that when the region flush size is set to XXmb a region flush may be triggered even before writing keys and values of size XX because the total heap occupancy of the region which includes additional metadata exceeded the threshold. + + +--- + +* [HBASE-19116](https://issues.apache.org/jira/browse/HBASE-19116) | *Critical* | **Currently the tail of hfiles with CellComparator\* classname makes it so hbase1 can't open hbase2 written hfiles; fix** + +hbase-2.x sets KeyValue Comparators into the tail of hfiles rather than CellComparator, what it uses internally, just so hbase-1.x can continue to read hbase-2.x written hfiles. + + +--- + +* [HBASE-19948](https://issues.apache.org/jira/browse/HBASE-19948) | *Major* | **Since HBASE-19873, HBaseClassTestRule, Small/Medium/Large has different semantic** + +In subtask, fixed doc and annotations to be more explicit that test timings are for the whole Test Fixture/Test Class/Test Suite NOT the test method only as we'd measuring up to this (tother subtasks untethered Categorization and test timeout such that all categories now have a ten minute timeout -- no test can run longer than ten minutes or it gets killed/timedout). + + +--- + +* [HBASE-16060](https://issues.apache.org/jira/browse/HBASE-16060) | *Blocker* | **1.x clients cannot access table state talking to 2.0 cluster** + +By default, we mirror table state to zookeeper so hbase-1.x clients will work against an hbase-2 cluster (With this patch, hbase-1.x clients can do most Admin functions including table create; hbase-1.x clients can do all Table/DML against hbase-2 cluster). + +Flag to disable mirroring is hbase.mirror.table.state.to.zookeeper; set it to false in Configuration. + +Related, Master on startup will look to see if there are table state znodes left over by an hbase-1 instance. If any found, it will migrate the table state to hbase-2 setting the state into the hbase:meta table where table state is now kept. We will do this check on every Master start. Notion is that this will be overall beneficial with low impediment. To disable the migration check, set hbase.migrate.table.state.from.zookeeper to false. + + +--- + +* [HBASE-19900](https://issues.apache.org/jira/browse/HBASE-19900) | *Critical* | **Region-level exception destroy the result of batch** + +This fix makes the following changes to how client handle the both of action result and region exception. +1) honor the action result rather than region exception. If the action have both of true result and region exception, the action is fine as the exception is caused by other actions which are in the same region. +2) honor the action exception rather than region exception. If the action have both of action exception and region exception, we deal with the action exception only. If we also handle the region exception for the same action, it will introduce the negative count of actions in progress. The AsyncRequestFuture#waitUntilDone will block forever. + + +--- + +* [HBASE-19841](https://issues.apache.org/jira/browse/HBASE-19841) | *Major* | **Tests against hadoop3 fail with StreamLacksCapabilityException** + +HBaseTestingUtility now assumes that all clusters will use local storage until a MiniDFSCluster is started or assigned. + + +--- + +* [HBASE-19528](https://issues.apache.org/jira/browse/HBASE-19528) | *Major* | **Major Compaction Tool** + +Tool allows you to compact a cluster with given concurrency of regionservers compacting at a given time. If tool completes successfully everything requested for compaction will be compacted, regardless of region moves, splits and merges. + + +--- + +* [HBASE-19919](https://issues.apache.org/jira/browse/HBASE-19919) | *Major* | **Tidying up logging** + +(I thought this change innocuous but I made work for a co-worker when I upped interval between log cleaner runs -- meant a smoke test failed because we were slow doing an expected cleanup). + +Edit of log lines removing redundancy. Shorten thread names shown in log. Made some log TRACE instead of DEBUG. Capitalizations. + +Upped log cleaner interval from every minute to every ten minutes. hbase.master.cleaner.interval + +Lowered default count of threads started by Procedure Executor from count of CPUs to 1/4 of count of CPUs. + + +--- + +* [HBASE-19901](https://issues.apache.org/jira/browse/HBASE-19901) | *Major* | **Up yetus proclimit on nightlies** + +Pass to yetus a dockermemlimit of 20G and a proclimit of 10000. Defaults are 4G and 1G respectively. + + +--- + +* [HBASE-19912](https://issues.apache.org/jira/browse/HBASE-19912) | *Minor* | **The flag "writeToWAL" of Region#checkAndRowMutate is useless** + +Remove useless 'writeToWAL' flag of Region#checkAndRowMutate & related class + + +--- + +* [HBASE-19911](https://issues.apache.org/jira/browse/HBASE-19911) | *Major* | **Convert some tests from small to medium because they are timing out: TestNettyRpcServer, TestClientClusterStatus, TestCheckTestClasses** + +Changed a few tests so they are medium sized rather than small size. + +Also, upped the time we wait on small tests to 60seconds from 30seconds. Small tests are tests that run in 15seconds or less. What we changed was the timeout watcher. It is now more lax, more tolerant of dodgy infrastructure that might be running tests slowly. + + +--- + +* [HBASE-19892](https://issues.apache.org/jira/browse/HBASE-19892) | *Major* | **Checking 'patch attach' and yetus 0.7.0 and move to Yetus 0.7.0** + +Moved our internal yetus reference from 0.6.0 to 0.7.0. Concurrently, I changed hadoopqa to run with 0.7.0 (by editing the config in jenkins). + + +--- + +* [HBASE-19873](https://issues.apache.org/jira/browse/HBASE-19873) | *Major* | **Add a CategoryBasedTimeout ClassRule for all UTs** + +Along with @category -- small, medium, large -- all hbase tests must now carry a ClassRule as follows: + ++ @ClassRule ++ public static final HBaseClassTestRule CLASS\_RULE = ++ HBaseClassTestRule.forClass(TestInterfaceAudienceAnnotations.class); + +where the class changes by test. + +Currently the classrule enforces timeout for the whole test suite -- i.e. if a SmallTest Category then all the tests in the TestSuite must complete inside 60seconds, the timeout we set on SmallTest Category test suite -- but is meant to be a repository for general, runtime, hbase test facility. + + +--- + +* [HBASE-19770](https://issues.apache.org/jira/browse/HBASE-19770) | *Critical* | **Add '--return-values' option to Shell to print return values of commands in interactive mode** + +Introduces a new option to the HBase shell: -r, --return-values. When the shell is in "interactive" mode (default), the return value of shell commands are not returned to the user as they dirty the console output. For those who desire this functionality, the "--return-values" option restores the old functionality of the commands passing their return value to the user. + + +--- + +* [HBASE-15321](https://issues.apache.org/jira/browse/HBASE-15321) | *Major* | **Ability to open a HRegion from hdfs snapshot.** + +HRegion.openReadOnlyFileSystemHRegion() provides the ability to open HRegion from a read-only hdfs snapshot. Because hdfs snapshots are read-only, no cleanup happens when using this API. + + +--- + +* [HBASE-17513](https://issues.apache.org/jira/browse/HBASE-17513) | *Critical* | **Thrift Server 1 uses different QOP settings than RPC and Thrift Server 2 and can easily be misconfigured so there is no encryption when the operator expects it.** + +This change fixes an issue where users could have unintentionally configured the HBase Thrift1 server to run without wire-encryption, when they believed they had configured the Thrift1 server to do so. + + +--- + +* [HBASE-19828](https://issues.apache.org/jira/browse/HBASE-19828) | *Major* | **Flakey TestRegionsOnMasterOptions.testRegionsOnAllServers** + +Disables TestRegionsOnMasterOptions because Regions on Master does not work reliably; see HBASE-19831. + + +--- + +* [HBASE-18963](https://issues.apache.org/jira/browse/HBASE-18963) | *Major* | **Remove MultiRowMutationProcessor and implement mutateRows... methods using batchMutate()** + +Modified HRegion.mutateRow() APIs to use batchMutate() instead of processRowsWithLocks() with MultiRowMutationProcessor. MultiRowMutationProcessor is removed to have single write path that uses batchMutate(). + + +--- + +* [HBASE-19163](https://issues.apache.org/jira/browse/HBASE-19163) | *Major* | **"Maximum lock count exceeded" from region server's batch processing** + +When there are many mutations against the same row in a batch, as each mutation will acquire a shared row lock, it will exceed the maximum shared lock count the java ReadWritelock supports (64k). Along with other optimization, the batch is divided into multiple possible minibatches. A new config is added to limit the maximum number of mutations in the minibatch. + + \<property\> + \<name\>hbase.regionserver.minibatch.size\</name\> + \<value\>20000\</value\> + \</property\> +The default value is 20000. + + +--- + +* [HBASE-19739](https://issues.apache.org/jira/browse/HBASE-19739) | *Minor* | **Include thrift IDL files in HBase binary distribution** + +Thrift IDLs are now shipped, bundled up in the respective hbase-\*thrift.jars (look for files ending in .thrift). + + +--- + +* [HBASE-11409](https://issues.apache.org/jira/browse/HBASE-11409) | *Major* | **Add more flexibility for input directory structure to LoadIncrementalHFiles** + +Allows for users to bulk load entire tables from hdfs by specifying the parameter -loadTable. This allows you to pass in a table level directory and have all regions column families bulk loaded, if you do not specify the -loadTable parameter LoadIncrementalHFiles will work as before. Note: you must have a pre-created table to run with -loadTable it will not create one for you. + + +--- + +* [HBASE-19769](https://issues.apache.org/jira/browse/HBASE-19769) | *Critical* | **IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce jobs** + +Client-side ZooKeeper metrics which were added to 2.0.0 alpha/beta releases cause issues when launching MapReduce jobs via {{yarn jar}} on the command line. This stems from ClassLoader separation issues that YARN implements. It was chosen that the easiest solution was to remove these ZooKeeper metrics entirely. + + +--- + +* [HBASE-19783](https://issues.apache.org/jira/browse/HBASE-19783) | *Minor* | **Change replication peer cluster key/endpoint from a not-null value to null is not allowed** + +To reduce the confusing behavior, now when you call updatePeerConfig with empty ClusterKey or ReplicationEndpointImpl, but the value of field of the to-be-updated ReplicationPeerConfig is not null, we will throw exception instead of ignoring them. + + +--- + +* [HBASE-19483](https://issues.apache.org/jira/browse/HBASE-19483) | *Major* | **Add proper privilege check for rsgroup commands** + +This JIRA aims at refactoring AccessController, using ACL as core library in CPs. +1. Stripping out a public class AccessChecker from AccessController, using ACL as core library in CPs. AccessChecker don't have any dependency on anything CP related. Create it's instance from other CPS. +2. Change the default value of hbase.security.authorization to false. +3. Don't use CP hooks to check access in RSGroup. Use the access checker instance directly in functions of RSGroupAdminServiceImpl. + + +--- + +* [HBASE-19358](https://issues.apache.org/jira/browse/HBASE-19358) | *Major* | **Improve the stability of splitting log when do fail over** + +After HBASE-19358 we introduced a new property hbase.split.writer.creation.bounded to limit the opening writers for each WALSplitter. If set to true, we won't open any writer for recovered.edits until the entries accumulated in memory reaching hbase.regionserver.hlog.splitlog.buffersize (which defaults at 128M) and will write and close the file in one go instead of keeping the writer open. It's false by default and we recommend to set it to true if your cluster has a high region load (like more than 300 regions per RS), especially when you observed obvious NN/HDFS slow down during hbase (single RS or cluster) failover. + + +--- + +* [HBASE-19651](https://issues.apache.org/jira/browse/HBASE-19651) | *Minor* | **Remove LimitInputStream** + +HBase had copied from guava the file LmiitedInputStream. This commit removes the copied file in favor of (our internal, shaded) guava's ByteStreams.limit. Guava 14.0's LIS noted: "Use ByteStreams.limit(java.io.InputStream, long) instead. This class is scheduled to be removed in Guava release 15.0." + + +--- + +* [HBASE-19691](https://issues.apache.org/jira/browse/HBASE-19691) | *Critical* | **Do not require ADMIN permission for obtaining ClusterStatus** + +This change reverts an unintentional requirement for global ADMIN permission to obtain cluster status from the active HMaster. + + +--- + +* [HBASE-19486](https://issues.apache.org/jira/browse/HBASE-19486) | *Major* | ** Periodically ensure records are not buffered too long by BufferedMutator** + +The BufferedMutator now supports two settings that are used to ensure records do not stay too long in the buffer of a BufferedMutator. For periodically flushing the BufferedMutator there is now a "Timeout": "How old may the oldest record in the buffer be before we force a flush" and a "TimerTick": How often do we check if the timeout has been exceeded. Using these settings you can make the BufferedMutator automatically flush the write buffer if after the specified number of milliseconds no flush has occurred. + +This is mainly useful in streaming scenarios (i.e. writing data into HBase using Apache Flink/Beam/Storm) where it is common (especially in a test/development situation) to see small unpredictable bursts of data that need to be written into HBase. When using the BufferedMutator till now the effect was that records would remain in the write buffer until the buffer was full or an explicit flush was triggered. In practice this would mean that the 'last few records' of a burst would remain in the write buffer until the next burst arrives filling the buffer to capacity and thus triggering a flush. + + +--- + +* [HBASE-19670](https://issues.apache.org/jira/browse/HBASE-19670) | *Major* | **Workaround: Purge User API building from branch-2 so can make a beta-1** + +Disable filtering of User API based off yetus annotation done in doclet. See parent issue for build failure currently being worked on but not done in time for a beta-1. + + +--- + +* [HBASE-19282](https://issues.apache.org/jira/browse/HBASE-19282) | *Major* | **CellChunkMap Benchmarking and User Interface** + +When MSLAB is in use (that is the default config) , we will always use the CellChunkMap indexing variant for in memory flushed Immutable segments. When MSLAB is turned off, we will use CellAraryMap. These can not be changed with any configs. The in memory flush threshold been made to be default to 10% of region flush size. This can be turned using 'hbase.memstore.inmemoryflush.threshold.factor'. + + +--- + +* [HBASE-19628](https://issues.apache.org/jira/browse/HBASE-19628) | *Major* | **ByteBufferCell should extend ExtendedCell** + +ByteBufferCell â ByteBufferExtendedCell +MapReduceCell â MapReduceExtendedCell +ByteBufferChunkCell â ByteBufferChunkKeyValue +NoTagByteBufferChunkCell â NoTagByteBufferChunkKeyValue +KeyOnlyByteBufferCell â KeyOnlyByteBufferExtendedCell +TagRewriteByteBufferCell â TagRewriteByteBufferExtendedCell +ValueAndTagRewriteByteBufferCell â ValueAndTagRewriteByteBufferExtendedCell +EmptyByteBufferCell â EmptyByteBufferExtendedCell +FirstOnRowByteBufferCell â FirstOnRowByteBufferExtendedCell +LastOnRowByteBufferCell â LastOnRowByteBufferExtendedCell +FirstOnRowColByteBufferCell â FirstOnRowColByteBufferExtendedCell +FirstOnRowColTSByteBufferCell â FirstOnRowColTSByteBufferExtendedCell +LastOnRowColByteBufferCell â LastOnRowColByteBufferCell +OffheapDecodedCell â OffheapDecodedExtendedCell + + +--- + +* [HBASE-19576](https://issues.apache.org/jira/browse/HBASE-19576) | *Major* | **Introduce builder for ReplicationPeerConfig and make it immutable** + +Add a ReplicationPeerConfigBuilder to create ReplicationPeerConfig and make ReplicationPeerConfig immutable. Meanwhile, deprecated set\* methods in ReplicationPeerConfig. + + +--- + +* [HBASE-10092](https://issues.apache.org/jira/browse/HBASE-10092) | *Critical* | **Move to slf4j** + +We now have slf4j as our front-end. Be careful adding logging from here on out; make sure it slf4j. + +From here on out, as us devs go, we need to convert log messages from being 'guarded' -- i.e. surrounded by if (LOG.isDebugEnabled...) -- to instead being parameterized log messages. e.g. the latter rather than the former in the below: + +logger.debug("The new entry is "+entry+"."); +logger.debug("The new entry is {}.", entry); + +See [1] for background on perf benefits. + +Note, FATAL log level is not present in slf4j. It is noted as a Marker but won't show in logs as a LEVEL. + +1. https://www.slf4j.org/faq.html#logging\_performance + + +--- + +* [HBASE-19148](https://issues.apache.org/jira/browse/HBASE-19148) | *Blocker* | **Reevaluate default values of configurations** + +Removed unused hbase.fs.tmp.dir from hbase-default.xml. + +Upped hbase.master.fileSplitTimeout from 30s to 10minutes (suggested by production experience) + +Added note that handler-count should be ~CPU count. + +hbase.regionserver.logroll.multiplier has been changed from 0.95 to 0.5 AND the default block size has been doubled. + +A few of the core configs are now dumped to the log on startup. + + +--- + +* [HBASE-19492](https://issues.apache.org/jira/browse/HBASE-19492) | *Major* | **Add EXCLUDE\_NAMESPACE and EXCLUDE\_TABLECFS support to replication peer config** + +Add two new field: EXCLUDE\_NAMESPACE and EXCLUDE\_TABLECFS to replication peer config. + +If replicate\_all flag is true, it means all user tables will be replicated to peer cluster. Then allow config exclude namespaces or exclude table-cfs which can't be replicated to peer cluster. + +If replicate\_all flag is false, it means all user tables can't be replicated to peer cluster. Then allow to config namespaces or table-cfs which will be replicated to peer cluster. + + +--- + +* [HBASE-19494](https://issues.apache.org/jira/browse/HBASE-19494) | *Major* | **Create simple WALKey filter that can be plugged in on the Replication Sink** + +Adds means of adding very basic filter on the sink side of replication. We already have a means of installing filter source-side, which is better place to filter edits before they are shipped over the network, but this facility is needed by hbase-indexer. + +Set hbase.replication.sink.walentrysinkfilter with a no-param Constructor implementation. See test in patch for example. + + +--- + +* [HBASE-19112](https://issues.apache.org/jira/browse/HBASE-19112) | *Blocker* | **Suspect methods on Cell to be deprecated** + +Adds method Cell#getType which returns enum describing Cell Type. + +Deprecates the following Cell methods: + + getTypeByte + getSequenceId + getTagsArray + getTagsOffset + getTagsLength + +CPs trying to build cells should use RawCellBuilderFactory that supports building cells with tags. + + +--- + +* [HBASE-14790](https://issues.apache.org/jira/browse/HBASE-14790) | *Major* | **Implement a new DFSOutputStream for logging WAL only** + +Implement a FanOutOneBlockAsyncDFSOutput for writing WAL only, the WAL provider which uses this class is AsyncFSWALProvider. + +It is based on netty, and will write to 3 DNs at the same time concurrently(fan-out) so generally it will lead to a lower latency. And it is also fail-fast, the stream will become unwritable immediately after there are any read/write errors, no pipeline recovery. You need to call recoverLease to force close the output for this case. And it only supports to write a file with a single block. For WAL this is a good behavior as we can always open a new file when the old one is broken. The performance analysis in HBASE-16890 shows that it has a better performance. + +Behavior changes: +1. As now we write to 3 DNs concurrently, according to the visibility guarantee of HDFS, the data will be available immediately when arriving at DN since all the DNs will be considered as the last one in pipeline. This means replication may read uncommitted data and replicate it to the remote cluster and cause data inconsistency. HBASE-14004 is used to solve the problem. +2. There will be no sync failure. When the output is broken, we will open a new file and write all the unacked wal entries to the new file. This means that we may have duplicated entries in wal files. HBASE-14949 is used to solve this problem. + + +--- + +* [HBASE-15536](https://issues.apache.org/jira/browse/HBASE-15536) | *Critical* | **Make AsyncFSWAL as our default WAL** + +Now the default WALProvider is AsyncFSWALProvider, i.e. 'asyncfs'. +If you want to change back to use FSHLog, please add this in hbase-site.xml +{code} +\<property\> +\<name\>hbase.wal.provider\</name\> +\<value\>filesystem\</value\> +\</property\> +{code} +If you want to use FSHLog with multiwal, please add this in hbase-site.xml +{code} +\<property\> +\<name\>hbase.wal.regiongrouping.delegate.provider\</name\> +\<value\>filesystem\</value\> +\</property\> +{code} + +This patch also sets hbase.wal.async.use-shared-event-loop to false so WAL has its own netty event group. + + +--- + +* [HBASE-19462](https://issues.apache.org/jira/browse/HBASE-19462) | *Major* | **Deprecate all addImmutable methods in Put** + +Deprecates Put#addImmutable as of release 2.0.0, this will be removed in HBase 3.0.0. Use {@link #add(Cell)} and {@link org.apache.hadoop.hbase.CellBuilder} instead + + +--- + +* [HBASE-19213](https://issues.apache.org/jira/browse/HBASE-19213) | *Minor* | **Align check and mutate operations in Table and AsyncTable** + +In Table interface deprecate checkAndPut, checkAndDelete and checkAndMutate methods. +Similarly to AsyncTable a new method was added to replace the deprecated ones: CheckAndMutateBuilder checkAndMutate(byte[] row, byte[] family) with CheckAndMutateBuilder interface which can be used to construct the checkAnd\*() operations. + + +--- + +* [HBASE-19134](https://issues.apache.org/jira/browse/HBASE-19134) | *Major* | **Make WALKey an Interface; expose Read-Only version to CPs** + +Made WALKey an Interface and added a WALKeyImpl implementation. WALKey comes through to Coprocessors. WALKey is read-only. + + +--- + +* [HBASE-18169](https://issues.apache.org/jira/browse/HBASE-18169) | *Blocker* | **Coprocessor fix and cleanup before 2.0.0 release** + +Refactor of Coprocessor API for hbase2. Purged methods that exposed too much of our internals. Other hooks were recast so they no longer took or returned internal classes; instead we pass Interfaces or read-only versions of implementations. + +Here is some overview doc on changes in hbase2 for Coprocessors including detail on why the change was made: +https://github.com/apache/hbase/blob/branch-2.0/dev-support/design-docs/Coprocessor\_Design\_Improvements-Use\_composition\_instead\_of\_inheritance-HBASE-17732.adoc + + +--- + +* [HBASE-19301](https://issues.apache.org/jira/browse/HBASE-19301) | *Major* | **Provide way for CPs to create short circuited connection with custom configurations** + +Provided a way for the CP users to create a short circuitable connection with custom configs. + +createConnection(Configuration) is added to MasterCoprocessorEnvironment, RegionServerCoprocessorEnvironment and RegionCoprocessorEnvironment. + +The getConnection() method already available in these Env interfaces returns the cluster connection used by the server (which the server also uses) where as this new method will create a new connection on request. The difference from connection created using ConnectionFactory APIs is that this connection can short circuit the calls to same server avoiding the RPC paths. The connection will NOT be cached/maintained by server. That should be done the CPs. + +Be careful creating Connections out of a Coprocessor. See the javadoc on these createConnection and getConnection. + + +--- + +* [HBASE-19357](https://issues.apache.org/jira/browse/HBASE-19357) | *Major* | **Bucket cache no longer L2 for LRU cache** + +Removed cacheDataInL1 option for HCD +BucketCache is no longer the L2 for LRU on heap cache. When BC is used, data blocks will be strictly on BC only where as index/bloom blocks are on LRU L1 cache. +Config 'hbase.bucketcache.combinedcache.enabled' is removed. There is no way set combined mode = false. Means make BC as victim handler for LRU cache. +This will be one more noticeable change when one uses BucketCache in File mode. Then the system table's data block(Including the META table) will be cached in Bucket Cache files only. Plain scan from META files alone test reveal that the throughput of file mode BC is almost half only. But for META entries we have RegionLocation cache at client side connections. So this would not be a big concern in a real cluster usage. Will check more on this and probably fix even when we do tiered BucketCache. + + +--- + +* [HBASE-19430](https://issues.apache.org/jira/browse/HBASE-19430) | *Major* | **Remove the SettableTimestamp and SettableSequenceId** + +All the cells which are used in server side are of ExtendedCell now. + + +--- + +* [HBASE-19295](https://issues.apache.org/jira/browse/HBASE-19295) | *Major* | **The Configuration returned by CPEnv should be read-only.** + +CoprocessorEnvironment#getConfiguration returns a READ-ONLY Configuration. Attempts at altering the returned Configuration -- whether setting or adding resources -- will result in an IllegalStateException warning of the Read-only condition of the returned Configuration. + + +--- + +* [HBASE-19410](https://issues.apache.org/jira/browse/HBASE-19410) | *Major* | **Move zookeeper related UTs to hbase-zookeeper and mark them as ZKTests** + +There is a new HBaseZKTestingUtility which can only start a mini zookeeper cluster. And we will publish sources for test-jar for all modules. + + +--- + +* [HBASE-19323](https://issues.apache.org/jira/browse/HBASE-19323) | *Major* | **Make netty engine default in hbase2** + +NettyRpcServer is now our default RPC server replacing SimpleRpcServer. + + +--- + +* [HBASE-19426](https://issues.apache.org/jira/browse/HBASE-19426) | *Major* | **Move has() and setTimestamp() to Mutation** + +Moves #has and #setTimestamp back up to Mutation from the subclass Put so available to other Mutation implementations. + + +--- + +* [HBASE-19384](https://issues.apache.org/jira/browse/HBASE-19384) | *Critical* | **Results returned by preAppend hook in a coprocessor are replaced with null from other coprocessor even on bypass** + +When a coprocessor sets 'bypass', we will skip calling subsequent Coprocessors that may be stacked-up on the method invocation; e.g. if a prePut has three coprocessors hooked up, if the first coprocessor decides to set 'bypass', we will not call the two subsequent coprocessors (this is similar to the 'complete' functionality that was in hbase1, removed in hbase2). + + +--- + +* [HBASE-19408](https://issues.apache.org/jira/browse/HBASE-19408) | *Trivial* | **Remove WALActionsListener.Base** + +1) remove the WALActionsListener.Base +2) provide default method implementation to WALActionsListener +The person who want to receive the notification of WAL events should implements the WALActionsListener rather than WALActionsListener.Base. + + +--- + +* [HBASE-19339](https://issues.apache.org/jira/browse/HBASE-19339) | *Critical* | **Eager policy results in the negative size of memstore** + +Enable TestAcidGuaranteesWithEagerPolicy and TestAcidGuaranteesWithAdaptivePolicy + + +--- + +* [HBASE-19336](https://issues.apache.org/jira/browse/HBASE-19336) | *Major* | **Improve rsgroup to allow assign all tables within a specified namespace by only writing namespace** + +Add two new shell cmd. +move\_namespaces\_rsgroup is used to reassign tables of specified namespaces from one RegionServer group to another. +move\_servers\_namespaces\_rsgroup is used to reassign regionServers and tables of specified namespaces from one group to another. + + +--- + +* [HBASE-19285](https://issues.apache.org/jira/browse/HBASE-19285) | *Critical* | **Add per-table latency histograms** + +Per-RegionServer table latency histograms have been returned to HBase (after being removed due to impacting performance). These metrics are exposed via a new JMX bean "TableLatencies" with the typical naming conventions: namespace, table, and histogram component. + + +--- + +* [HBASE-19359](https://issues.apache.org/jira/browse/HBASE-19359) | *Major* | **Revisit the default config of hbase client retries number** + +The default value of hbase.client.retries.number was 35. It is now 10.
[... 7408 lines stripped ...]
