Added: dev/hbase/3.0.0-beta-1RC0/RELEASENOTES.md ============================================================================== --- dev/hbase/3.0.0-beta-1RC0/RELEASENOTES.md (added) +++ dev/hbase/3.0.0-beta-1RC0/RELEASENOTES.md Tue Dec 26 08:22:57 2023 @@ -0,0 +1,7425 @@ +# RELEASENOTES +<!--- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +--> +# HBASE 3.0.0-beta-1 Release Notes + +These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements. + + +--- + +* [HBASE-27855](https://issues.apache.org/jira/browse/HBASE-27855) | *Minor* | **Support dynamic adjustment of flusher count** + +Support dynamic adjustment of flusher count + + +--- + +* [HBASE-27888](https://issues.apache.org/jira/browse/HBASE-27888) | *Minor* | **Record readBlock message in log when it takes too long time** + +Add a configuration parameter,which control to record read block slow in logs. +\<property\> + \<name\>hbase.fs.reader.warn.time.ms\</name\> + \<value\>-1\</value\> +\</property\> +If reading block cost time in milliseconds more than the threshold, a warning will be logged,the default value is -1, it means skipping record the read block slow warning log. + + +--- + +* [HBASE-27798](https://issues.apache.org/jira/browse/HBASE-27798) | *Major* | **Client side should back off based on wait interval in RpcThrottlingException** + +When throttling quotas are enabled, clients will now respect the wait interval sent along from the server in the RpcThrottlingException. As a result, in these cases retries will backoff differently from your normally configured pause time/schedule to respect the throttling dictated by the server. + + +--- + +* [HBASE-27657](https://issues.apache.org/jira/browse/HBASE-27657) | *Major* | **Connection and Request Attributes** + +It is now possible to send Connection and Request attributes to the server. + +Connection attributes are passed in via new method overloads in ConnectionFactory. These attributes are only sent once per connection, during connection handshake. They will remain in memory on the server side for the duration of the connection. + +Request attributes can be configured via a new setRequestAttributes on TableBuilder and AsyncTableBuilder. Requests sent through the built table will all send along the configured attributes. Each request sent to a RegionServer will include the request attributes in the header. This differs from existing Operation attributes for Scan (which doesn't accept operation attributes) and Multi (which each operation in a batch has its own attributes). + +These new attributes can be retrieved in server side plugins by inspecting RpcServer.getCurrentCall(). The returned RpcCall has a getHeader(), which returns the RequestHeader which has a getAttributeList() containing request attributes. We also added a getConnectionHeader() to RpcCall, which also has a getAttributeList() for getting connection attributes. + +Note: This involved addition of a new Map\<String, byte[]\> argument to ConnectionImplementation, AsyncConnectionImpl, NettyRpcClient, and BlockingRpcClient. Users who have custom implementations of these Private and LimitedPrivate classes will need to update their constructors accordingly to match so that reflection continues to work. + + +--- + +* [HBASE-26874](https://issues.apache.org/jira/browse/HBASE-26874) | *Major* | **VerifyReplication recompare async** + +VerifyReplication can now do multiple recompares (rather than 1 previously), with exponential backoff. In order to speed up the job when there are many discrepancies, the recompares can be run in background threads. New options added: --recompareTries (default 1 if --recompareSleep is \> 0, otherwise 0), --recompareThreads (default 0), --recompareBackoffExponent (default 1). + + +--- + +* [HBASE-27896](https://issues.apache.org/jira/browse/HBASE-27896) | *Major* | **Disable hdfs readahead for pread reads** + +PREAD reads will no longer do HDFS readahead by default. This should save substantial disk and network IO for random read workloads, but one can re-enable it if desired by setting "hbase.store.reader.no-readahead" to false. + + +--- + +* [HBASE-27956](https://issues.apache.org/jira/browse/HBASE-27956) | *Major* | **Support wall clock profiling in ProfilerServlet** + +You can now do wall clock profiling with async-profiler by specifying ?event=wall query param on the profiler servlet (/prof) + + +--- + +* [HBASE-27947](https://issues.apache.org/jira/browse/HBASE-27947) | *Critical* | **RegionServer OOM under load when TLS is enabled** + +When a slow client is not able to read responses from the server fast enough, the server side channel outbound buffer will grow. This can eventually lead to an OOM under extreme cases, and here we add new configurations to protect against that: +- hbase.server.netty.writable.watermark.low +- hbase.server.netty.writable.watermark.high +- hbase.server.netty.writable.watermark.fatal + +When high watermark is exceeded, server will stop accepting new requests from the client. When outbound bytes drops below the low watermark, it will start again. This does not stop the server from processing already enqueued requests, so if those requests continue to grow the outbound bytes beyond the fatal threshold, the connection will be forcibly closed. + +Also added new metrics for monitoring this situation in bean "Hadoop:service=HBase,name=RegionServer,sub=IPC": + - UnwritableTime\_\* - histogram of time periods between when the high watermark was exceeded and when it eventually drops below low watermark. +- nettyTotalPendingOutboundBytes - as the name suggests, for all channels the total amount of bytes waiting to be written to sockets +- nettyMaxPendingOutboundBytes - the number of bytes waiting on the most backed up channel across all channels + + +--- + +* [HBASE-26867](https://issues.apache.org/jira/browse/HBASE-26867) | *Minor* | **Introduce a FlushProcedure** + +Introduce a flush procedure to flush tables. The flush procedure is used to perform flush-related operations by default. Set hbase.flush.procedure.enabled to false to disable flush procedure. + + +--- + +* [HBASE-28008](https://issues.apache.org/jira/browse/HBASE-28008) | *Major* | **Add support for tcnative** + +If a properly shaded netty-tcnative is found on the classpath, hbase will automatically pick it up for use in accelerating TLS handling. Properly shaded means relocated to prefix with org.apache.hbase.thirdparty + + +--- + +* [HBASE-28068](https://issues.apache.org/jira/browse/HBASE-28068) | *Minor* | **Add hbase.normalizer.merge.merge\_request\_max\_number\_of\_regions property to limit max number of regions in a merge request for merge normalization** + +Added a new property "hbase.normalizer.merge.merge\_request\_max\_number\_of\_regions" to limit the max number of region to be processed for merge request in a single merge normalisation. Defaults to 100 + + +--- + +* [HBASE-27970](https://issues.apache.org/jira/browse/HBASE-27970) | *Major* | **Make sync replication also work with FSHLog** + +Now sync replication could also work with FSHLog. It requires an extra thread for performing concurrent append/sync, so it will be a bit costly compared to AsyncFSWAL. + + +--- + +* [HBASE-28168](https://issues.apache.org/jira/browse/HBASE-28168) | *Minor* | **Add option in RegionMover.java to isolate one or more regions on the RegionSever** + +This adds a new "isolate\_regions" operation to RegionMover, which allows operators to pass a list of region encoded ids to be "isolated" in the passed RegionServer. +Regions currently deployed in the RegionServer that are not in the passed list of regions would be moved to other RegionServers. Regions in the passed list that are currently on other RegionServers would be moved to the passed RegionServer. + +Please refer to the command help for further information. + + +--- + +* [HBASE-28043](https://issues.apache.org/jira/browse/HBASE-28043) | *Major* | **Reduce seeks from beginning of block in StoreFileScanner.seekToPreviousRow** + +Optimizes StoreFileScanner.seekToPreviousRow to use keep track of a hint which allows us to eliminate one seek per call, resulting in 40% or more throughput increase for reverse scans. External users (Phoenix) of this LimitedPrivate method should be aware of the change in semantics. It is expected that seekToPreviousRow is used for scanning backwards in the StoreFile. Calling with an originalKey greater than the previously passed key (i.e. forward in the StoreFile), the originalKey will not be honored. Instead use seek for this case. + + +--- + +* [HBASE-28222](https://issues.apache.org/jira/browse/HBASE-28222) | *Major* | **Leak in ExportSnapshot during verifySnapshot on S3A** + +ExportSnapshot now uses FileSystems from the global FileSystem cache, and as such does not close those FileSystems when it finishes. If users plan to run ExportSnapshot over and over in a single process for different FileSystem urls, they should run FileSystem.closeAll() between runs. See JIRA for details. + + +--- + +* [HBASE-25549](https://issues.apache.org/jira/browse/HBASE-25549) | *Major* | **Provide a switch that allows avoiding reopening all regions when modifying a table to prevent RIT storms.** + +New APIs are added to Admin, AsyncAdmin, and hbase shell to allow modifying a table without reopening all regions. Care should be used in using this API, as regions will be in an inconsistent state until they are all reopened. Whether this matters depends on the change, and some changes are disallowed (such as enabling region replication or adding/removing a column family). + + +--- + +* [HBASE-28215](https://issues.apache.org/jira/browse/HBASE-28215) | *Major* | **Region reopen procedure should support some sort of throttling** + +Adds new configurations to control the speed and batching of region reopens after modifying a table: +- hbase.reopen.table.regions.progressive.batch.size.max - When set, the HMaster will progressively reopen regions, starting with one region and then doubling until it reaches the specified max. After reaching the max, it will continue reopening at that batch size until all regions are reopened. +- hbase.reopen.table.regions.progressive.batch.backoff.ms - When set, the HMaster will back off for this amount of time between each batch. + + +--- + +* [HBASE-27532](https://issues.apache.org/jira/browse/HBASE-27532) | *Major* | **Add block bytes scanned metrics** + +Adds blockBytesScannedCount metrics for the regionserver, table, and operations in JMX. Adds a countOfBlockBytesScanned to ScanMetrics. + + +--- + +* [HBASE-28196](https://issues.apache.org/jira/browse/HBASE-28196) | *Major* | **Yield SCP and TRSP when they are blocked** + +For ServerCrashProcedure, TransitRegionStateProcedure and related region assignment procedures like OpenRegionProcedure/CloseRegionProcedure, now we will try our best to release the procedure worker when doing time consuming operations. Including: + +1. When updating region state to hbase:meta. We will use async client to get a CompletableFuture, and then suspend the procedure. When the CompletableFuture is done, we add the procedure back. +2. When acquiring region state node lock, if we can not get the lock immediately, we will also suspend the procedure and release the procedure worker. Once it is our turn to get the lock, we will add the procedure back. + +In this way, if meta is offline, it is not likely that the assign procedures will consume all the procedure workers and cause dead lock. You can see HBASE-19976 for more detailed description about this problem. + + + +# HBASE 3.0.0-alpha-4 Release Notes + +These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements. + + +--- + +* [HBASE-26945](https://issues.apache.org/jira/browse/HBASE-26945) | *Minor* | **Quotas causes too much load on meta for large clusters** + +The quotas chore no longer scans meta to get information about region counts. Instead it uses the Admin API, which reads in-memory state in the HMaster and should scale well for large clusters. + + +--- + +* [HBASE-13126](https://issues.apache.org/jira/browse/HBASE-13126) | *Critical* | **Provide alternate mini cluster classes other than HBTU for downstream users to write unit tests** + +Introduce a TestingHBaseCluster for users to implement integration test with mini hbase cluster. +See this section https://hbase.apache.org/book.html#\_integration\_testing\_with\_an\_hbase\_mini\_cluster in the ref guide for more details on how to use it. +TestingHBaseCluster also allowes you to start a mini hbase cluster based on external HDFS cluster and zookeeper cluster, please see the release note of HBASE-26167 for more details. +HBaseTestingUtility is marked as deprecated and will be 'removed' in the future. + + +--- + +* [HBASE-26790](https://issues.apache.org/jira/browse/HBASE-26790) | *Major* | **getAllRegionLocations can cache locations with null hostname** + +getAllRegionLocations will no longer add locations to the MetaCache which have no server name. Region locations without a server name can occur for brief periods during region splits and merges, but the MetaCache relies on every cached location having a server name. + + +--- + +* [HBASE-27060](https://issues.apache.org/jira/browse/HBASE-27060) | *Major* | **Allow sharing connections between AggregationClient instances** + +Added a new no-arg constructor for AggregationClient which allows lightweight usage of just the methods accepting a Table. Also added a new constructor which takes a Connection, allowing the user to pass their own externally managed connection which will be used when methods accepting a TableName are used. + + +--- + +* [HBASE-27078](https://issues.apache.org/jira/browse/HBASE-27078) | *Major* | **Allow configuring a separate timeout for meta scans** + +Similar to hbase.read.rpc.timeout and hbase.client.scanner.timeout.period for normal scans, this issue adds two new configs for meta scans: hbase.client.meta.read.rpc.timeout and hbase.client.meta.scanner.timeout.period. Each meta scan RPC call will be limited by hbase.client.meta.read.rpc.timeout, while hbase.client.meta.scanner.timeout.period acts as an overall operation timeout. + +Additionally, for 2.5.0, normal Table-based scan RPCs will now be limited by hbase.read.rpc.timeout if configured, instead of hbase.rpc.timeout. This behavior already existed for AsyncTable scans. + + +--- + +* [HBASE-27148](https://issues.apache.org/jira/browse/HBASE-27148) | *Major* | **Move minimum hadoop 3 support version to 3.2.3** + +Bump the minimum hadoop 3 dependency to 3.2.3. + +Also upgrade apache-avro to 1.11.0 and exclude all jackson 1.x dependencies since all jackson 1.x versions have vulnerabilities. + +Notice that for hadoop 2 dependency we will need to include jackson 1.x because hadoop directly depend on it. + + +--- + +* [HBASE-27048](https://issues.apache.org/jira/browse/HBASE-27048) | *Major* | **Server side scanner time limit should account for time in queue** + +Server will now account for queue time when determining how long a scanner can run before heartbeat should be returned. This should help avoid timeouts when server is overloaded. + + +--- + +* [HBASE-27219](https://issues.apache.org/jira/browse/HBASE-27219) | *Minor* | **Change JONI encoding in RegexStringComparator** + +In RegexStringComparator an infinite loop can occur if an invalid UTF8 is encountered. We now use joni's NonStrictUTF8Encoding instead of UTF8Encoding to avoid the issue. + + +--- + +* [HBASE-27230](https://issues.apache.org/jira/browse/HBASE-27230) | *Major* | **RegionServer should be aborted when WAL.sync throws TimeoutIOException** + +This changes add additional logic for WAL.sync: +If WAL.sync get a timeout exception, we wrap TimeoutIOException as a special WALSyncTimeoutIOException. When upper layer such as HRegion.doMiniBatchMutate called by HRegion.batchMutation catches this special exception, we abort the region server. + + +--- + +* [HBASE-27204](https://issues.apache.org/jira/browse/HBASE-27204) | *Critical* | **BlockingRpcClient will hang for 20 seconds when SASL is enabled after finishing negotiation** + +When Kerberos authentication succeeds, on the server side, after receiving the final SASL token from the client, we simply wait for the client to continue by sending the connection header. After HBASE-24579, on the client side, an additional readStatus() was added, which assumed that after negotiation has completed a status code will be sent. However when authentication has succeeded the server will not send one. As a result the client would hang and only throw an exception when the configured read timeout is reached, which is 20 seconds by default. This was especially noticeable when using BlockingRpcClient as the client implementation. HBASE-24579 was reverted to correct this issue. + + +--- + +* [HBASE-27232](https://issues.apache.org/jira/browse/HBASE-27232) | *Major* | **Fix checking for encoded block size when deciding if block should be closed** + +This changed behaviour of "hbase.writer.unified.encoded.blocksize.ratio" property: + +Previous behaviour: Checks if the encoded block size \>= ("hbase.writer.unified.encoded.blocksize.ratio" \* BLOCK\_SIZE) \|\| (non-encoded block size \>= BLOCK\_SIZE) when delimiting hfile blocks. As most often (non-encoded block size \>= BLOCK\_SIZE) will be reached, setting "hbase.writer.unified.encoded.blocksize.ratio" usually had no effect. +The default value for "hbase.writer.unified.encoded.blocksize.ratio" was "1". + +New behaviour: If "hbase.writer.unified.encoded.blocksize.ratio" is set to anything different from "0", it will check if encoded block size \>= ("hbase.writer.unified.encoded.blocksize.ratio" \* BLOCK\_SIZE) when delimiting an hfile block. If "hbase.writer.unified.encoded.blocksize.ratio" is not set, it will check if encoded block size \>= BLOCK\_SIZE \|\| non-encoded block size \>= BLOCK\_SIZE when delimiting an hfile block. + + +--- + +* [HBASE-27225](https://issues.apache.org/jira/browse/HBASE-27225) | *Major* | **Add BucketAllocator bucket size statistic logging** + +When bucket cache is enabled, setting loglevel to DEBUG for the org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocator logger will result in a periodic summary of bucket usage for the bucket cache. This log output can give insight into block distribution of the cluster, where bucket cache memory is being allocated, and how much waste/fragmentation there is in the bucket cache. See Javadoc in BucketAllocate for more explanation of the logging. + + +--- + +* [HBASE-27229](https://issues.apache.org/jira/browse/HBASE-27229) | *Major* | **BucketCache statistics should not count evictions by hfile** + +The eviction metric for the BucketCache has been updated to only count evictions triggered by the eviction process (i.e responding to cache pressure). This brings it in-line with the other cache implementations, with the goal of this metric being to give operators insight into cache pressure. Other evictions by hfile or drain to storage engine failure, etc, no longer count towards the eviction rate. + + +--- + +* [HBASE-27144](https://issues.apache.org/jira/browse/HBASE-27144) | *Minor* | **Add special rpc handlers for bulkload operations** + +Bulkload will consume a lot of resources in the cluster. We try to reduce the impact of bulkload on online services and do simple resource isolation for bulkload. +The bulkload RpcExecutor is disabled by default. Enable by setting "hbase.regionserver.bulkload.handler.count" greater than 0. +Online HBASE cluster "hbase.regionserver.bulkload.handler.count" is recommended to be set to 1. +Offline HBASE clusters can be appropriately increased. +"hbase.regionserver.bulkload.handler.count" works with "hbase.ipc.server.bulkload.max.callqueue.length". + + +--- + +* [HBASE-26666](https://issues.apache.org/jira/browse/HBASE-26666) | *Major* | **Add native TLS encryption support to RPC server/client** + +Full support for TLS/SSL encryption in Netty RPC client and server. The implementation is based on Netty's built-in SSL handler and capable of using JDK or OpenSSL implementation whichever is available on the classpath. The feature also supports side-by-side plaintext and encrypted communication which enables upgrading existing clusters with zero downtime. + + +--- + +* [HBASE-27241](https://issues.apache.org/jira/browse/HBASE-27241) | *Major* | **Add metrics for evaluating cost and effectiveness of bloom filters** + +Adds 3 new metrics, which are available per-table and per-regionserver: +- bloomFilterRequestsCount -- how many checks against a bloom filter occurred. Note that a single read might increment this multiple times, one for each storefile it needs to check. +- bloomFilterNegativeResultsCount -- how many of those requests came back negative, indicating the bloom filter helped to avoid opening a storefile which did not include the row. If this value is low, it might be worth disabling bloom filters on a table +- bloomFilterEligibleRequestsCount -- increments in the same way as bloomFilterRequestsCount, but only when bloom filters are \_not\_ enabled for a table. If this value is high, it might be worth enabling bloom filters on the table. + +Additionally makes 2 existing metrics available on a per-table basis: +- staticBloomSize -- uncompressed size of the bloom filters for a table +- staticIndexSize -- uncompressed size of the storefile indexes for a table + +You can combine bloom filter request and result counts with staticBloomSize to determine if the cost (size) is worth the effectiveness (result/request count) of the bloom. + + +--- + +* [HBASE-27281](https://issues.apache.org/jira/browse/HBASE-27281) | *Critical* | **Add default implementation for Connection$getClusterId** + +Adds a default null implementation for Connection$getClusterId. Downstream applications should implement this method. + + +--- + +* [HBASE-27264](https://issues.apache.org/jira/browse/HBASE-27264) | *Major* | **Add options to consider compressed size when delimiting blocks during hfile writes** + +This modified the block write logic for deciding on block size boundaries. When checking for block size limit, block writer now delegates this decision to implementations of the BlockCompressedSizePredicator interface. The aim here is to allow for pluggable implementations of how block size limit is defined when compression is in use. By default, it assumes the UncompressedBlockSizePredicator implementation, which doesn't modify the current behaviour of closing the block based on its uncompressed size. The alternative PreviousBlockCompressionRatePredicator implementation compares the uncompressed size against an adjusted block size, calculated using uncompressed/compressed sizes from the previous block as: BLOCK\_SIZE \* (uncompressed/compressed). +Implementations of BlockCompressedSizePredicator should be defined by the hbase.block.compressed.size.predicator config property. + +To avoid very larger block sizes in the case of large compression factor, this also defines the hbase.block.max.size.uncompressed config property. If not set, this max size defaults to BLOCK\_SIZE \* 10. + + +--- + +* [HBASE-27089](https://issues.apache.org/jira/browse/HBASE-27089) | *Minor* | **Add âcommons.crypto.stream.buffer.sizeâ configuration** + +Add a 'commons.crypto.stream.buffer.size' config for setting the buffer size when doing AES crypto for RPC. + + +--- + +* [HBASE-27129](https://issues.apache.org/jira/browse/HBASE-27129) | *Major* | **Add a config that allows us to configure region-level storage policies** + +Add a 'hbase.hregion.block.storage.policy' so you can config storage policy at region level. This is useful when you want to control the storage policy for the directories other than CF directories, such as .splits, .recovered.edits, etc. + + +--- + +* [HBASE-27104](https://issues.apache.org/jira/browse/HBASE-27104) | *Major* | **Add a tool command list\_unknownservers** + +Introduce a shell command 'list\_unknownservers' to list unknown servers. + + +--- + +* [HBASE-27305](https://issues.apache.org/jira/browse/HBASE-27305) | *Minor* | **add an option to skip file splitting when bulkload hfiles** + +Add a 'hbase.loadincremental.fail.if.need.split.hfile' configuration. If set to true, th bulk load operation will fail immediately if we need to split the hfiles. This can be used to prevent unexpected time consuming bulk load operation. + + +--- + +* [HBASE-20904](https://issues.apache.org/jira/browse/HBASE-20904) | *Major* | **Prometheus metrics http endpoint for monitoring integration** + +HBase metrics can be published in JSON and now Prometheus friendly formats via servlets. Any of these servlets can be enabled or disabled by the configuration property 'hbase.http.metrics.servlets' in the 'hbase-default.xml' file. + +The value for the property should be a comma separated list of the servlet aliases which are '{jmx, metrics, prometheus}'. +The '/jmx', '/metrics', '/prometheus' servlets are enabled by default. + +To get metrics using these servlets access the URL 'http://SERVER\_HOSTNAME:SERVER\_WEB\_UI\_PORT/endpoint'. Where endpoint is one of {'/jmx', '/metrics', '/prometheus'}. + + +--- + +* [HBASE-27320](https://issues.apache.org/jira/browse/HBASE-27320) | *Minor* | **hide some sensitive configuration information in the UI** + +hide superuser and password related settings in the configuration UI + + +--- + +* [HBASE-27340](https://issues.apache.org/jira/browse/HBASE-27340) | *Minor* | **Artifacts with resolved profiles** + +Published poms now contain runtime dependencies only; build and test time dependencies are stripped. Profiles are also now resolved and in-lined at publish time. This removes the need/ability of downstreamers shaping hbase dependencies via enable/disable of hbase profile settings (Implication is that now the hbase project publishes artifacts for hadoop2 and for hadoop3, and so on). + + +--- + +* [HBASE-27224](https://issues.apache.org/jira/browse/HBASE-27224) | *Major* | **HFile tool statistic sampling produces misleading results** + +Fixes HFilePrettyPrinter's calculation of min and max size for an HFile so that it will truly be the min and max for the whole file. Previously was based on just a sampling, as with the histograms. Additionally adds a new argument to the tool '-d' which prints detailed range counts for each summary. The range counts give you the exact count of rows/cells that fall within the pre-defined ranges, useful for giving more detailed insight into outliers. + + +--- + +* [HBASE-27371](https://issues.apache.org/jira/browse/HBASE-27371) | *Major* | **Bump spotbugs version** + +Bump spotbugs version from 4.2.2 to 4.7.2. Also bump maven spotbugs plugin version from 4.2.0 to 4.7.2.0. + + +--- + +* [HBASE-27372](https://issues.apache.org/jira/browse/HBASE-27372) | *Major* | **Update java versions in our Dockerfiles** + +Upgrade java version to 11.0.16.1 and 8u345b01 in the docker files which are used in our pre commit and nightly jobs. +Remove JDK7 in these docker files as we do not support JDK7 any more. + + +--- + +* [HBASE-27280](https://issues.apache.org/jira/browse/HBASE-27280) | *Major* | **Add mutual authentication support to TLS** + +By default, when TLS is enabled, we will also enable mutual authentication of certificates. This means, during handshake, the client will authenticate the server's certificate (as is usual) and also the server will authenticate the client's certificate. Additionally, each side will validate that the hostname presented by the certificate matches the address of the connection. These default settings can be customized with new properties "hbase.server.netty.tls.client.auth.mode" (default NEED, possibly values NEED, WANT, NONE), "hbase.server.netty.tls.verify.client.hostname" (default true), and "hbase.client.netty.tls.verify.server.hostname" (default true). Additionally, during hostname verification, if necessary we will fallback on reverse lookup. The reverse lookup can be disabled via "hbase.rpc.tls.host-verification.reverse-dns.enabled" (default true) + + +--- + +* [HBASE-27304](https://issues.apache.org/jira/browse/HBASE-27304) | *Minor* | **Support using IP to expose master/rs servers for some special scenarios** + +In some scenarios, such as the elastic scaling scenario on the cloud, the HBase client may not be able to resolve the hostname of the newly added node. If the network is interconnected, the client can actually access the HBase cluster nodes through ip. However, since the HBase client obtains the Master/RS address info from or the ZK or the meta table, so the Master/RS of the HBase cluster needs to expose the service with ip instead of the hostname. Therefore, We can use hostname by default, but at the same time, we can also provide a config âhbase.server.useip.enabledâ +to support whether to use ip for Master/RS service. + + +--- + +* [HBASE-27392](https://issues.apache.org/jira/browse/HBASE-27392) | *Major* | **Add a new procedure type for implementing some global operations such as migration** + +Add a GlobalProcedureInterface for implementing some global operations. It has a special queue in MasterProcedureScheduler. + + +--- + +* [HBASE-27381](https://issues.apache.org/jira/browse/HBASE-27381) | *Major* | **Still seeing 'Stuck' in static initialization creating RegionInfo instance** + +Static constant UNDEFINED has been removed from public interface RegionInfo. This is a breaking change, but resolves a critical deadlock bug. This constant was never meant to be exposed and has been deprecated since version 2.3.2 with no replacement. + + +--- + +* [HBASE-27434](https://issues.apache.org/jira/browse/HBASE-27434) | *Major* | **Use $revision as placeholder for maven version to make it easier to control the version from command line** + +Use ${revision} as placeholder for maven version in pom, so later you can use 'mvn install -Drevision=xxx' to specify the version at build time. +After this change, you can not use mvn versions:set to bump the version, instead. you should just modify the parent pom to change the value of the 'revision' property in the properties section. + + +--- + +* [HBASE-27091](https://issues.apache.org/jira/browse/HBASE-27091) | *Minor* | **Speed up the loading of table descriptor from filesystem** + +Loading table descriptors with multiple threads is supported after HBASE-27091 (but turned off by default). The thread number could be configured via \`hbase.tabledescriptor.parallel.load.threads\`, and setting it back to zero will turn off the feature. + + +--- + +* [HBASE-27472](https://issues.apache.org/jira/browse/HBASE-27472) | *Major* | **The personality script set wrong hadoop2 check version for branch-2** + +This only affects branch-2 but for aliging the personality scripts across all active branches, we apply it to all active branches. + + +--- + +* [HBASE-27477](https://issues.apache.org/jira/browse/HBASE-27477) | *Major* | **Tweak hbase-vote.sh to support hbase-thirdparty** + +\`dev-support/hbase-vote.sh\` can now be used for evaluating release candidates from the hbase-thirdparty repo. + + +--- + +* [HBASE-27443](https://issues.apache.org/jira/browse/HBASE-27443) | *Major* | **Use java11 in the general check of our jenkins job** + +Change to use java 11 in nightly and pre commit jobs. + +Bump error prone to 2.16 and force using jdk11 when error prone is enabled. + + +--- + +* [HBASE-27444](https://issues.apache.org/jira/browse/HBASE-27444) | *Minor* | **Add tool commands list\_enabled\_tables and list\_disabled\_tables** + +Introduce two shell commands 'list\_enabled\_tables' and 'list\_disabled\_tables' to list enabled or disabled tables. + + +--- + +* [HBASE-27506](https://issues.apache.org/jira/browse/HBASE-27506) | *Minor* | **Optionally disable sorting directories by size in CleanerChore** + +Added \`hbase.cleaner.directory.sorting\` configuration to enable the CleanerChore to sort the subdirectories by consumed space and start the cleaning with the largest subdirectory. Enabled by default. + + +--- + +* [HBASE-27513](https://issues.apache.org/jira/browse/HBASE-27513) | *Major* | **Modify README.txt to mention how to contribue** + +Remove README.txt and replace it with README.md. +Add a 'How to Contribute' section to tell contributors how to acquire a jira account. + + +--- + +* [HBASE-27474](https://issues.apache.org/jira/browse/HBASE-27474) | *Major* | **Evict blocks on split/merge; Avoid caching reference/hlinks if compaction is enabled** + +This modifies behaviour of block cache management as follows: +1) Always evict blocks for the files from parent split region once it's closed, regardless of the "hbase.rs.evictblocksonclose" configured value; +2) If compactions are enabled, doesn't cache blocks for the refs/link files under split daughters once these regions are opened; + +For #1 above, an additional evict\_cache property has been added to the CloseRegionRequest protobuf message. It's default to false. Rolling upgrading cluster would retain the previous behaviour on RSes not yet upgraded. + + +--- + +* [HBASE-27233](https://issues.apache.org/jira/browse/HBASE-27233) | *Major* | **Read blocks into off-heap if caching is disabled for read** + +Using Scan.setCacheBlocks(false) with on-heap LRUBlockCache will now result in significantly less heap allocations for those scans if hbase.server.allocator.pool.enabled is enabled. Previously all allocations went to on-heap if LRUBlockCache was used, but now it will go to the off-heap pool if cache blocks is enabled. + + +--- + +* [HBASE-27514](https://issues.apache.org/jira/browse/HBASE-27514) | *Major* | **Move some persistent states from zookeeper to master region** + +Moved the follow states from zookeeper to master local region's 'state' family. + +The on/off state for load balancer +The on/off state for region normalizer +The enable/disable state for snapshot cleanup task +The enable/disable for split/merge + + +--- + +* [HBASE-27565](https://issues.apache.org/jira/browse/HBASE-27565) | *Major* | **Make the initial corePoolSize configurable for ChoreService** + +Add 'hbase.choreservice.initial.pool.size' configuration property to set the initial number of threads for the ChoreService. + + +--- + +* [HBASE-27529](https://issues.apache.org/jira/browse/HBASE-27529) | *Major* | **Provide RS coproc ability to attach WAL extended attributes to mutations at replication sink** + +New regionserver coproc endpoints that can be used by coproc at the replication sink cluster if WAL has extended attributes. +Using the new endpoints, WAL extended attributes can be transferred to Mutation attributes at the replication sink cluster. + + +--- + +* [HBASE-27493](https://issues.apache.org/jira/browse/HBASE-27493) | *Major* | **Allow namespace admins to clone snapshots created by them** + +Allow namespace admins to clone snapshots created by them to any table inside their namespace, not just re-create the old table + + +--- + +* [HBASE-27575](https://issues.apache.org/jira/browse/HBASE-27575) | *Minor* | **Bump future from 0.18.2 to 0.18.3 in /dev-support** + +pushed to 2.4, 2.5, branch-2, and master + + +--- + +* [HBASE-27541](https://issues.apache.org/jira/browse/HBASE-27541) | *Minor* | **Backups should be able to be restored to a separate filesystem** + +Adds a new withRestoreRootDir method to RestoreRequest. When specified, the bulk output will be sent to that directory rather than the value of hbase.fs.tmp.dir (which defaults to current user's home dir on current FS). You can pass a fully qualified URI to withRestoreRootDir, which allows the RestoreJob to save the output to a remote cluster if you've loaded the job's Configuration with the appropriate hdfs configs to connect to that cluster. + + +--- + +* [HBASE-27551](https://issues.apache.org/jira/browse/HBASE-27551) | *Major* | **Add config options to delay assignment to retain last region location** + +This change introduces a boolean hbase.master.scp.retain.assignment.force property with default value of false to the AssignmentManager. +AssignmentManager already defines a hbase.master.scp.retain.assignment property, which enables AssignmentManager to prioritise the previous RegionServer the region was online when coming up with an assignment plan. This, however, does not guarantee the assignment retainment, in case the SCP triggers the TransitRegionStateProcedure (TRSP) before the given RegionServer is online. +To forcibly "honour" the retainment, hbase.master.scp.retain.assignment.force property should be also set to true. +Note that this could delay the region assignment until the given RegionServer reports itself as online to the master, and RITs may be reported on master UI or by HBCK. +The amount of time the TRSP will try to open the region on the given RS is determined by hbase.master.scp.retain.assignment.force.retries (default to 600). Between each retry, the TRSP will sleep for an exponential factor of the value defined in hbase.master.scp.retain.assignment.force.wait-interval (default to 50) in millis. + + +--- + +* [HBASE-27558](https://issues.apache.org/jira/browse/HBASE-27558) | *Major* | **Scan quotas and limits should account for total block IO** + +Scan quotas and hbase.server.scanner.max.result.size will now be enforced against the total bytes of actual blocks scanned by the request. Block bytes scanned will typically be much higher for heavily filtered scans, since the cost will include the size of every block read rather than just the size of the returned cells. If you have heavily filtered scans, you may need to account for that in your defined quotas. + + +--- + +* [HBASE-27570](https://issues.apache.org/jira/browse/HBASE-27570) | *Major* | **Unify tracking of block IO across all read request types** + +Unifies block IO accounting code across all request types. As a result, the behavior of MultiActionResultTooLarge handling may change slightly. In certain circumstances, a multiget might trip the "too large" threshold sooner than it used to in the past. This should largely be transparent to users since the client automatically retries, and the behavior change should only be noticeable for very large multigets that fetch many rows from the same block. + + +--- + +* [HBASE-27534](https://issues.apache.org/jira/browse/HBASE-27534) | *Major* | **Determine too large requests by response block size rather than cell size** + +In addition to response size, requests will now show up in the slow log if the block bytes scanned to retrieve the cells exceeds the hbase.ipc.warn.response.size. This will cause heavily filtered scans which scan lots of data but don't return much to start showing up. + + +--- + +* [HBASE-27689](https://issues.apache.org/jira/browse/HBASE-27689) | *Major* | **Update README.md about how to request a jira account** + +Guide contributors to https://selfserve.apache.org/jira-account.html for requesting a jira account in README.md + + +--- + +* [HBASE-27250](https://issues.apache.org/jira/browse/HBASE-27250) | *Minor* | **MasterRpcService#setRegionStateInMeta does not support replica region encodedNames or region names** + +MasterRpcServices#setRegionStateInMeta can now work with both primary and timeline-consistent replica regions. + + +--- + +* [HBASE-27681](https://issues.apache.org/jira/browse/HBASE-27681) | *Major* | **Refactor Table Latency Metrics** + +Table request metrics (table latency histograms and TableReadQueryMeter/TableWriteQueryMeter) have been reorganized into a new metric bean TableRequests. The new TableRequests bean is published separately for each table on the regionserver, with the bean name including the table name and metric names within the bean simplified. Beans will be cleaned up after tables leave the regionserver. + +Users who collect TableLatencies or table query meters will have to update metric collection to work with the new structure. For example: + +Old: +{ + "name": "Hadoop:service=HBase,name=RegionServer,sub=TableLatencies", + "modelerType": "RegionServer,sub=TableLatencies", + "tag.Context": "regionserver", + "tag.Hostname": "hostname", + "Namespace\_default\_table\_usertable\_metric\_deleteTime\_num\_ops": 0, + "Namespace\_default\_table\_usertable\_metric\_deleteTime\_min": 0, + "Namespace\_default\_table\_usertable\_metric\_deleteTime\_max": 0, + "Namespace\_default\_table\_usertable\_metric\_deleteTime\_mean": 0, + ... +} + +New: +{ + "name": "Hadoop:service=HBase,name=RegionServer,sub=TableRequests\_Namespace\_default\_table\_usertable", + "modelerType": "RegionServer,sub=TableRequests\_Namespace\_default\_table\_usertable", + "tag.Context": "regionserver", + "tag.Hostname": "hostname", + "DeleteTime\_num\_ops": 0, + "DeleteTime\_min": 0, + "DeleteTime\_max": 0, + "DeleteTime\_mean": 0, + ... +} + + +--- + +* [HBASE-27632](https://issues.apache.org/jira/browse/HBASE-27632) | *Major* | **Refactor WAL.Reader implementation so we can better support WAL splitting and replication** + +Introduced two types of WAL reader, WALStreamReader and WALTailingReader. The former one is for most use cases where we only need to read closed WAL files, such as splitting and replaying, and the second one is for replication, where we need to tail a WAL file which is being written currently. + +The config 'hbase.regionserver.hlog.reader.impl' is removed, as now we have two types of WAL reader. And starting from at least 0.98, we do not support sequence file format WAL file any more, the only accepted format is protobuf. And for encrypted WAL files, now you do not need to specify SecureProtobufLogReader either, as the new implementation will choose to use SecureWALCellCodec based on the properties in WALHeader. +There is still a config called 'hbase.regionserver.wal.stream.reader.impl', for specifying the implementation for WALStreamReader. This is only designed to be used in HBase itself, for injecting some error when writing UTs, downstream users should not use it as it is unstable. + + +--- + +* [HBASE-27702](https://issues.apache.org/jira/browse/HBASE-27702) | *Major* | **Remove 'hbase.regionserver.hlog.writer.impl' config** + +Changed the configuration names to 'hbase.regionserver.wal.writer.impl' and 'hbase.regionserver.wal.async.writer.impl', which are mainly used for unit testing. End users should not use them. + +Removed 'SecureProtobufLogWriter' and 'SecureAsyncProtobufLogWriter'. You can now use ProtobufLogWriter and AsyncProtobufLogWriter to write encrypted data directly. Since the configuration names have changed, there will be no problems if usersâ old configuration files contain SecureProtobufLogWriter and SecureAsyncProtobufLogWriter, as they will never be loaded. But we still suggest that you remove these unused configurations from your configuration file. + + +--- + +* [HBASE-27651](https://issues.apache.org/jira/browse/HBASE-27651) | *Minor* | **hbase-daemon.sh foreground\_start should propagate SIGHUP and SIGTERM** + +<!-- markdown --> +Introduce separate `trap`s for SIGHUP vs. the rest. Treat `SIGINT`, `SIGKILL`, and `EXIT` identically, as before. Use the signal name without `SIG` prefix for increased portability, as per the POSIX man page for `trap`. + +`SIGTERM` handler will now honor `HBASE_STOP_TIMEOUT` as described in the file header. + + +--- + +* [HBASE-27748](https://issues.apache.org/jira/browse/HBASE-27748) | *Major* | **Bump jettison from 1.5.2 to 1.5.4** + +Bump jettison from 1.5.2 to 1.5.4 for CVE-2023-1436. + + +--- + +* [HBASE-27765](https://issues.apache.org/jira/browse/HBASE-27765) | *Major* | **Add biggest cell related info into web ui** + +Save len and key of the biggest cell into fileinfo when generate hfile, and shows them on webui for better monitor. + + +--- + +* [HBASE-27110](https://issues.apache.org/jira/browse/HBASE-27110) | *Major* | **Move replication peer storage from zookeeper to other storage systems** + +Introduced a âhbase.replication.peer.storage.implâ config to support using different replication peer storage implementation. For now there are two built in implementations, âzookeeperâ and âfilesystemâ. The default implementation is still zookeeper as users do not need to specify this configuration in the past so if we change the default implementation, after upgrading, the implementation will be changed and cause data loss. +For filesystem replication peer storage, there is a config âhbase.replication.peers.directoryâ for specifying the directory. The value is relative to the hbase root directory and the default value is âpeersâ. +We also implemented a tool for migrating replication peer data across different replication peer storage implementations. Use + +./bin/hbase copyreppeers \<SRC\_REPLICATION\_PEER\_STORAGE\> \<DST\_REPLICATION\_PEER\_STORAGE\> + +to copy the replication peer data. Notice that we will not delete the data in the src storage, for supporting online migration. You need to delete the old storage manually after copying. +For supporting online migration, we implemented a shell command to disable replication peer modification. Use + +peer\_modification\_switch \<disableOrEnable\>, \<drainProcedures\> + +to enable or disable replication peer modification. The \`drainProcedures\` parameter means whether you want to wait until all the existing peer modification procedures to finish before returning when disabling peer modification. We also implemented a shell command âpeer\_modification\_enabledâ to query whether replication peer modification is enabled currently. + +So when you want to migrate replication peer storage online, you can disable replication peer modification first, then run the copyreppeers tool to copy the replication data from the old storage to new storage, then update all the config files in the cluster, then trigger a online configuration update to load the configuration. + + +--- + +* [HBASE-27808](https://issues.apache.org/jira/browse/HBASE-27808) | *Major* | **Change flatten mode for oss in our pom file** + +Changed the flatten mode from default to oss. It will include these extra section in the published pom files: + +name, description, url, developers, scm, inceptionYear, organization, mailingLists, issueManagement, distributionManagement. + + +--- + +* [HBASE-27693](https://issues.apache.org/jira/browse/HBASE-27693) | *Major* | **Support for Hadoop's LDAP Authentication mechanism (Web UI only)** + +Support for LDAP Authentication in the HBase Web UI via Hadoop's LdapAuthenticationFilter. + + +--- + +* [HBASE-27762](https://issues.apache.org/jira/browse/HBASE-27762) | *Major* | **Include EventType and ProcedureV2 pid in logging via MDC** + +<!-- markdown --> +Log the `o.a.h.hbase.executor.EventType` and ProcedureV2 pid in log messages via MDC. PatternLayouts on master and branch-2 have been updated to make use of the MDC variables. Note that due to LOG4J2-3660, log lines for which the MDC is empty will have extraneous characters. To opt-in on branch-2.5 or branch-2.4, make an appropriate change to `conf/log4j2.properties`. + + +--- + +* [HBASE-27799](https://issues.apache.org/jira/browse/HBASE-27799) | *Major* | **RpcThrottlingException wait interval message is misleading between 0-1s** + +The RpcThrottleException now includes millis in the message + + +--- + +* [HBASE-27838](https://issues.apache.org/jira/browse/HBASE-27838) | *Minor* | **Update zstd-jni from version 1.5.4-2 -\> 1.5.5-2** + +Bump zstd-jni from 1.5.4-2 to 1.5.5-2, which fixed a critical issue on s390x. + + +--- + +* [HBASE-27109](https://issues.apache.org/jira/browse/HBASE-27109) | *Major* | **Move replication queue storage from zookeeper to a separated HBase table** + +We introduced a table based replication queue storage in this issue. The queue data will be stored in hbase:replication table. This is the last piece of persistent data on zookeeper. So after this change, we are OK to clean up all the data on zookeeper, as now they are all transient, a cluster restarting can fix everything. + +The data structure has been changed a bit as now we only support an offset for a WAL group instead of storing all the WAL files for a WAL group. Please see the replication internals section in our ref guide for more details. + +To break the cyclic dependency issue, i.e, creating a new WAL writer requires writing to replication queue storage first but with table based replication queue storage, you first need a WAL writer when you want to update to table, now we will not record a queue when creating a new WAL writer instance. The downside for this change is that, the logic for claiming queue and WAL cleaner are much more complicated. See AssignReplicationQueuesProcedure and ReplicationLogCleaner for more details if you have interest. + +Notice that, we will use a separate WAL provider for hbase:replication table, so you will see a new WAL file for the region server which holds the hbase:replication table. If we do not do this, the update to hbase:replication table will also generate some WAL edits in the WAL file we need to track in replication, and then lead to more updates to hbase:replication table since we have advanced the replication offset. In this way we will generate a lot of garbage in our WAL file, even if we write nothing to the cluster. So a separated WAL provider which is not tracked by replication is necessary here. + +The data migration will be done automatically during rolling upgrading, of course the migration via a full cluster restart is also supported, but please make sure you restart master with new code first. The replication peers will be disabled during the migration and no claiming queue will be scheduled at the same time. So you may see a lot of unfinished SCPs during the migration but do not worry, it will not block the normal failover, all regions will be assigned. The replication peers will be enabled again after the migration is done, no manual operations needed. + +The ReplicationSyncUp tool is also affected. The goal of this tool is to replicate data to peer cluster while the source cluster is down. But if we store the replication queue data in a hbase table, it is impossible for us to get the newest data if the source cluster is down. So here we choose to read from the region directory directly to load all the replication queue data in memory, and do the sync up work. We may lose the newest data so in this way we need to replicate more data but it will not affect correctness. + + + +# HBASE 3.0.0-alpha-3 Release Notes + +These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements. + + +--- + +* [HBASE-26956](https://issues.apache.org/jira/browse/HBASE-26956) | *Major* | **ExportSnapshot tool supports removing TTL** + +ExportSnapshot tool support removing TTL of snapshot. If we use the ExportSnapshot tool to recover snapshot with TTL from cold storage to hbase cluster, we can set \`-reset-ttl\` to prevent snapshot from being deleted immediately. + + +--- + +* [HBASE-26167](https://issues.apache.org/jira/browse/HBASE-26167) | *Major* | **Allow users to not start zookeeper and dfs cluster when using TestingHBaseCluster** + +Introduce two new methods when creating a TestingHBaseClusterOption. + +public Builder useExternalDfs(String uri) +public Builder useExternalZooKeeper(String connectString) + +Users can use these two methods to specify external zookeeper or HDFS cluster to be used by the TestingHBaseCluster. + + +--- + +* [HBASE-27108](https://issues.apache.org/jira/browse/HBASE-27108) | *Blocker* | **Revert HBASE-25709** + +HBASE-25709 caused a regression for scans that result in a large number of rows and has been reverted in this release. + + +--- + +* [HBASE-15519](https://issues.apache.org/jira/browse/HBASE-15519) | *Major* | **Add per-user metrics** + +Adds per-user metrics for reads/writes to each RegionServer. These metrics are exported by default. hbase.regionserver.user.metrics.enabled can be used to disable the feature if desired for any reason. + + +--- + +* [HBASE-26923](https://issues.apache.org/jira/browse/HBASE-26923) | *Minor* | **PerformanceEvaluation support encryption option** + +Add a new command line argument: --encryption to enable encryptopn in PerformanceEvaluation tool. + +Usage: + encryption Encryption type to use (AES, ...). Default: 'NONE'" + +Examples: + To run a AES encryption sequentialWrite: + $ bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --table=xxx --encryption='AES' sequentialWrite 10 + + +--- + +* [HBASE-26983](https://issues.apache.org/jira/browse/HBASE-26983) | *Major* | **Upgrade JRuby to 9.3.4.0** + +Updates the version of JRuby that ships with HBase for the HBase shell to 9.3.4.0. + +Note that this changes the supported version of Ruby for HBase shell integration to Ruby 2.6. + + +--- + +* [HBASE-26993](https://issues.apache.org/jira/browse/HBASE-26993) | *Major* | **Make the new framework for region replication could work for SKIP\_WAL** + +In this issue we make the new region replication framework introduced by HBASE-26233 could also replicate for table which DURABILITY is Durability.SKIP\_WAL or for individual Mutation which is Durability.SKIP\_WAL. +It is implemented entirely on the basis of HBASE-26233 and requires no additional configuration. + + +--- + +* [HBASE-26649](https://issues.apache.org/jira/browse/HBASE-26649) | *Major* | **Support meta replica LoadBalance mode for RegionLocator#getAllRegionLocations()** + +When setting 'hbase.locator.meta.replicas.mode' to "LoadBalance" at HBase client, RegionLocator#getAllRegionLocations() now load balances across all Meta Replica Regions. Please note, results from non-primary meta replica regions may contain stale data. + + +--- + +* [HBASE-27055](https://issues.apache.org/jira/browse/HBASE-27055) | *Minor* | **Add additional comments when using HBASE\_TRACE\_OPTS with standalone mode** + +hbase-env.sh has been updated with an optional configuration HBASE\_OPTS for standalone mode + +# export HBASE\_OPTS="${HBASE\_OPTS} ${HBASE\_TRACE\_OPTS} -Dotel.resource.attributes=service.name=hbase-standalone" + + +--- + +* [HBASE-26342](https://issues.apache.org/jira/browse/HBASE-26342) | *Major* | **Support custom paths of independent configuration and pool for hfile cleaner** + +Configure the custom hifle paths (under archive directory), e.g. data/default/testTable1,data/default/testTable2 by "hbase.master.hfile.cleaner.custom.paths". +Configure hfile cleaner classes for the custom paths by "hbase.master.hfilecleaner.custom.paths.plugins". +Configure the shared pool size of custom hfile cleaner paths by "hbase.cleaner.custom.hfiles.pool.size". + + +--- + +* [HBASE-27047](https://issues.apache.org/jira/browse/HBASE-27047) | *Minor* | **Fix typo for metric drainingRegionServers** + +Fix typo for metric drainingRegionServers. Change metric name from draininigRegionServers to drainingRegionServers. + + +--- + +* [HBASE-25465](https://issues.apache.org/jira/browse/HBASE-25465) | *Minor* | **Use javac --release option for supporting cross version compilation** + +When compiling with java 11 and above, we will use --release 8 to maintain java 8 compatibility. +Also upgrade jackson to 2.13.1 because in hbase-thirdparty 4.1.0 we shade jackson 2.13.1. + + +--- + +* [HBASE-27024](https://issues.apache.org/jira/browse/HBASE-27024) | *Major* | **The User API and Developer API links are broken on hbase.apache.org** + +Upgrade maven-site-plugin to 3.12.0, maven-javadoc-plugin to 3.4.0. + + +--- + +* [HBASE-26986](https://issues.apache.org/jira/browse/HBASE-26986) | *Major* | **Trace a one-shot execution of a Master procedure** + +Individual executions of procedures are now wrapped in a tracing span. No effort is made to coordinate multiple executions back to a common PID. + + +--- + +* [HBASE-27013](https://issues.apache.org/jira/browse/HBASE-27013) | *Major* | **Introduce read all bytes when using pread for prefetch** + +Introduce optional flag hfile.pread.all.bytes.enabled for pread that must read full bytes with the next block header. This feature is specially helpful when users are running HBase with Blob storage like S3 and Azure Blob storage. Especially when using HBase with S3A and set fs.s3a.experimental.input.fadvise=sequential, it can save input stream from seeking backward that spent more time on storage connection reset time. + + +--- + +* [HBASE-26899](https://issues.apache.org/jira/browse/HBASE-26899) | *Major* | **Run spotless:apply** + +Run spotless:apply to format our code base. +When viewing 'git blame', you may find a file has a large amount lines are modified by the commit of HBASE-26899, so you need to go back to the commit before this commit, and viewing 'git blame' again. + + +--- + +* [HBASE-26617](https://issues.apache.org/jira/browse/HBASE-26617) | *Major* | **Use spotless to reduce the pain on fixing checkstyle issues** + +Use spotless to format our java file and pom file, using the hbase\_eclipse\_formatter.xml and eclipse.importerorder file under our dev-support directory. +On all branches, the ratchetFrom is set the commit just before the commit which introduces the spotless plugin, so we will only format the files which are touched in later commits. +From now on, you should type mvn spotless:apply before generating a PR, the spotless plugin will fix most of the format issues for you. + + +--- + +* [HBASE-22349](https://issues.apache.org/jira/browse/HBASE-22349) | *Major* | **Stochastic Load Balancer skips balancing when node is replaced in cluster** + +StochasticLoadBalancer now respects the hbase.regions.slop configuration value as another factor in determining whether to attempt a balancer run. If any regionserver has a region count outside of the target range, the balancer will attempt to balance. Using the default 0.2 value, the target range is 80%-120% of the average (mean) region count per server. Whether the balancer will ultimately move regions will still depend on the weights of StochasticLoadBalancer's cost functions. + + +--- + +* [HBASE-26581](https://issues.apache.org/jira/browse/HBASE-26581) | *Minor* | **Add metrics around failed replication edits** + +Adds new metrics sink.failedBatches and source.failedBatches to replication metrics, allowing one to track failures on both sides of a replication stream. As with other source metrics, the new source.failedBatches is reported globally and by peer id. + + +--- + +* [HBASE-26807](https://issues.apache.org/jira/browse/HBASE-26807) | *Major* | **Unify CallQueueTooBigException special pause with CallDroppedException** + +Introduces a new config "hbase.client.pause.server.overloaded", deprecating old "hbase.client.pause.cqtbe". The new config specifies a special pause time to use when the client receives an exception from the server indicating that it is overloaded. Currently this applies to CallQueueTooBigException and CallDroppedException. + + +--- + +* [HBASE-26942](https://issues.apache.org/jira/browse/HBASE-26942) | *Minor* | **cache region locations when getAllRegionLocations()** + +Calling AsyncRegionLocator.getAllRegionLocations() will now add all of those returned region locations into the meta cache for the client. + + +--- + +* [HBASE-26891](https://issues.apache.org/jira/browse/HBASE-26891) | *Minor* | **Make MetricsConnection scope configurable** + +Adds a new "hbase.client.metrics.scope" config which allows users to define a custom scope for each Connection's metric instance. The default scope has also been changed to include the clusterId of the Connection, which should help differentiate metrics for processes connecting to multiple clusters. The scope is added to the ObjectName for JMX beans, so can be used to query for metrics for a particular connection. Using a custom scope might be useful in cases where you maintain separate Connections for writes vs reads. In that case you can set the scope appropriately and differentiate metrics for each. + + +--- + +* [HBASE-26067](https://issues.apache.org/jira/browse/HBASE-26067) | *Major* | **Change the way on how we track store file list** + +Introduces the StoreFileTracker interface to HBase. This is a server-side interface which abstracts how a Store (column family) knows what files should be included in that Store. Previously, HBase relied on a listing the directory a Store used for storage to determine the files which should make up that Store. + +After this feature, there are two implementations of StoreFileTrackers. The first (and default) implementation is listing the Store directory. The second is a new implementation which records files which belong to a Store within each Store. Whenever the list of files that make up a Store change, this metadata file will be updated. + +This feature is notable in that it better enables HBase to function on storage systems which do not provide the typical posix filesystem semantics, most importantly, those which do not implement a file rename operation which is atomic. Storage systems which do not implement atomic renames often implement a rename as a copy and delete operation which amplifies the I/O costs by 2x. + +At scale, this feature should have a 2x reduction in I/O costs when using storage systems that do not provide atomic renames, most importantly in HBase compactions and memstore flushes. See the corresponding section, "Store File Tracking", in the HBase book for more information on how to use this feature. + + +--- + +* [HBASE-26618](https://issues.apache.org/jira/browse/HBASE-26618) | *Minor* | **Involving primary meta region in meta scan with CatalogReplicaLoadBalanceSimpleSelector** + +When META replica LoadBalance mode is enabled at client-side, clients will try to read from one META region first. If META location is from any non-primary META regions, in case of errors, it will fall back to the primary META region. + + +--- + +* [HBASE-26245](https://issues.apache.org/jira/browse/HBASE-26245) | *Major* | **Store region server list in master local region** + +A typical HBase deployment on cloud is to store the data other than WAL on OSS, and store the WAL data on a special HDFS cluster. A common operation is to rebuild the cluster with fresh new zk cluster and HDFS cluster, with only the old rootdir on OSS. But it requires extra manual steps since we rely on the WAL directory to find out previous live region servers, so we can schedule SCP to bring regions online. +After this issue, now it is possible to rebuild the cluster without extra manual steps as we will also store the previous live region servers in master local region. +But notice that you'd better stop masters first and then region servers when rebuilding, as some tests show that if there are some pending procedures, the new clusters may still hang. + + +--- + +* [HBASE-26810](https://issues.apache.org/jira/browse/HBASE-26810) | *Major* | **Add dynamic configuration support for system coprocessors** + +Now dynamic configuration, configurign hbase-site and then call update\_all\_config, can also applied for the following properties. + +hbase.coprocessor.master.classes +hbase.coprocessor.region.classes +hbase.coprocessor.regionserver.classes +hbase.coprocessor.user.region.classes + + +--- + +* [HBASE-25895](https://issues.apache.org/jira/browse/HBASE-25895) | *Major* | **Implement a Cluster Metrics JSON endpoint** + +Introduces a REST+JSON endpoint on the master info server port, which is used to render the "Region Visualizer" section of the Master Status page. When enabled, access to this API is gated by authentication only, just like the Master Status page. This API is considered InterfaceAudience.Private, can change or disappear without notice. + + +--- + +* [HBASE-26323](https://issues.apache.org/jira/browse/HBASE-26323) | *Major* | **Introduce a SnapshotProcedure** + +Introduce a snapshot procedure to snapshot tables. Set hbase.snapshot.procedure.enabled to false to disable snapshot procedure. + + +--- + +* [HBASE-26822](https://issues.apache.org/jira/browse/HBASE-26822) | *Major* | **Revert the changes on hbase-daemon.sh after switching to log4j2 properties file** + +Remove the HBASE\_ROOT\_LOGGER\_LEVEL, HBASE\_ROOT\_LOGGER\_APPENDER, HBASE\_SECURITY\_LOGGER\_LEVEL and HBASE\_SECURITY\_LOGGER\_APPENDER environment variables as log4j 2.17.2 supports set level and appenders at once. +You can still use HBASE\_ROOT\_LOGGER and HBASE\_SECURITY\_LOGGER environment variables. + + +--- + +* [HBASE-26552](https://issues.apache.org/jira/browse/HBASE-26552) | *Major* | **Introduce retry to logroller to avoid abort** + +For retrying to roll log, the wait timeout is limited by "hbase.regionserver.logroll.wait.timeout.ms", +and the max retry time is limited by "hbase.regionserver.logroll.retries". +Do not retry to roll log is the default behavior. + + +--- + +* [HBASE-26723](https://issues.apache.org/jira/browse/HBASE-26723) | *Major* | **Switch to use log4j2.properties file to configure log4j2** + +Use log4j2.properties file instead of log4j2.xml. + +With LOG4J2-3341 in place, we could still make use of the old syntax where we could specify level and appenders at once, so we do not need to do splitting in our bash script and also make it less hurt for our users to upgrade. + + +--- + +* [HBASE-26640](https://issues.apache.org/jira/browse/HBASE-26640) | *Major* | **Reimplement master local region initialization to better work with SFT** + +Introduced a 'hbase.master.store.region.file-tracker.impl' config to specify the store file tracker implementation for master local region. + +If not present, master local region will use the cluster level store file tracker implementation. + + +--- + +* [HBASE-26673](https://issues.apache.org/jira/browse/HBASE-26673) | *Major* | **Implement a shell command for change SFT implementation** + +Introduced two shell commands for change table's or family's sft: + +change\_sft: + Change table's or table column family's sft. Examples: + hbase\> change\_sft 't1','FILE' + hbase\> change\_sft 't2','cf1','FILE' + +change\_sft\_all: + Change all of the tables's sft matching the given regex: + hbase\> change\_sft\_all 't.\*','FILE' + hbase\> change\_sft\_all 'ns:.\*','FILE' + hbase\> change\_sft\_all 'ns:t.\*','FILE' + + +--- + +* [HBASE-26742](https://issues.apache.org/jira/browse/HBASE-26742) | *Major* | **Comparator of NOT\_EQUAL NULL is invalid for checkAndMutate** + +The semantics of checkAndPut for null(or empty) value comparator is changed, the old match is always true. +But we should consider that EQUAL or NOT\_EQUAL for null check is a common usage, so the semantics of checkAndPut for matching null is correct now. +There is rare use of LESS or GREATER null, so keep the semantics for them. + + +--- + +* [HBASE-26688](https://issues.apache.org/jira/browse/HBASE-26688) | *Major* | **Threads shared EMPTY\_RESULT may lead to unexpected client job down.** + +Result#advance with empty cell list will always return false but not raise NoSuchElementException when called multiple times. +This is a behavior change so it is an 'incompatible change', but since it will not introduce any compile error and the old behavior is 'broken', so we also fix it for current release branches. + + +--- + +* [HBASE-26714](https://issues.apache.org/jira/browse/HBASE-26714) | *Major* | **Introduce path configuration for system coprocessors** + +After HBASE-26714, users of hbase can now set system coprocessor with a path of the JAR file other than the files in default classpath. E.g. when setting hbase.coprocessor.region.classes/hbase.coprocessor.regionserver.classes/hbase.coprocessor.user.region.classes/hbase.coprocessor.master.classes, user can set a comma-separated list with the format of className\|priority\|path, where the path could be a file path of the supported filesystem. + + +--- + +* [HBASE-26473](https://issues.apache.org/jira/browse/HBASE-26473) | *Major* | **Introduce \`db.hbase.container\_operations\` span attribute** + +<!-- markdown --> Introduces an HBase-specific tracing attribute called `db.hbase.container_operations`. This attribute contains a list of table operations contained in the batch/list/envelope operation, allowing an operator to seek, for example, all `PUT` operations, even they occur inside of a `BATCH` or `COMPARE_AND_SET`. + + +--- + +* [HBASE-26587](https://issues.apache.org/jira/browse/HBASE-26587) | *Major* | **Introduce a new Admin API to change SFT implementation** + +Introduced two admin methods: + +void modifyTableStoreFileTracker(TableName, String) throws IOException; +void modifyColumnFamilyStoreFileTracker(TableName, byte[], String) throws IOException; + +You can use these two methods to change the store file tracker implementation for a table or a family, where you just need to specify the final store file tracker implementation, and do not need to take care of the 'migration' intermediate state, at master side we will take care of it for you. + + +--- + +* [HBASE-26661](https://issues.apache.org/jira/browse/HBASE-26661) | *Major* | **Remove deprecated methods in MasterObserver** + +Removed the below methods in MasterObserver + +void preModifyTable(ObserverContext\<MasterCoprocessorEnvironment\>, TableName, TableDescriptor) throws IOException; + +void postModifyTable(ObserverContext\<MasterCoprocessorEnvironment\>, TableName, TableDescriptor) throws IOException; + +void preModifyTableAction(ObserverContext\<MasterCoprocessorEnvironment\>, TableName, TableDescriptor) throws IOException; + +void postCompletedModifyTableAction(ObserverContext\<MasterCoprocessorEnvironment\>, TableName, TableDescriptor) throws IOException; + +void preModifyNamespace(ObserverContext\<MasterCoprocessorEnvironment\>, NamespaceDescriptor) throws IOException; + +void postModifyNamespace(ObserverContext\<MasterCoprocessorEnvironment\>, NamespaceDescriptor) throws IOException; + + +--- + +* [HBASE-26469](https://issues.apache.org/jira/browse/HBASE-26469) | *Critical* | **correct HBase shell exit behavior to match code passed to exit** + +<!-- markdown --> +User input handling has been refactored to make use of IRB sessions directly and the HBase shell attempts to ensure user provided calls to exit are able to convey failure and success. + +Those scripting use of the HBase shell should be aware that the exit code may have changed: + * a 0 code, or no code, passed to a call to exit from stdin in non-interactive mode will now exit cleanly. in prior versions this would have exited with an error and non-zero exit code. (note that in HBase 2.4.x this call will still result in a non-zero exit code) + * for other combinations of passing in an initialization script or reading from stdin with using the non-interactive flag, the exit code being 0 or non-0 should now line up with releases prior to 2.4, which is a change in behavior compared to versions 2.4.0 - 2.4.9. + +Please see the issue details for a table of expected exit codes. + + +--- + +* [HBASE-26631](https://issues.apache.org/jira/browse/HBASE-26631) | *Major* | **Upgrade junit to 4.13.2** + +Upgrade junit to 4.13.2 for addressing CVE-2020-15250. + + +--- + +* [HBASE-26233](https://issues.apache.org/jira/browse/HBASE-26233) | *Major* | **The region replication framework should not be built upon the general replication framework** + +In this issue we re-implement the 'Async WAL Replication' mechanism fo region replication, to decouple with the general replication framwork. + +Now, we extend the MVCC write entry to be able to carry an action once it is completed, so we can attach the WAL edits to this action and send them out directly after the write entry is completed, without touch the actual WAL system. So the special 'region\_replica\_replication' peer is also useless. + +We also introduce a new replicateToReplica method at region server side for receiving the WAL edits, which has a much simpler logic than the old replay method. + +After this issue, we do not need to treat META table specially in region replication, as said above, we do not depend on the general replication framework any more. But to avoid effecting the usage too much, the 'config hbase.region.replica.replication.catalog.enabled' is kept, you still need to enable this flag if you want to enable 'Async WAL Replication' support for META table. + +Rolling upgrading with 'Async WAL Replication enabled is supported. During rolling upgrading, the 'region\_replica\_replication' will be removed automatically after master is upgraded. And at region server side, if the new replicateToReplica method is not available when it tries to replicate to secondary replicas, it will fallback to use replay method automatically. + +Please related sections in our ref guide are: + +http://hbase.apache.org/book.html#async.wal.replication +http://hbase.apache.org/book.html#async.wal.replication.meta +http://hbase.apache.org/book.html#\_async\_wal\_replication\_for\_meta\_table\_as\_of\_hbase\_3\_0\_0 + + +--- + +* [HBASE-26347](https://issues.apache.org/jira/browse/HBASE-26347) | *Major* | **Support detect and exclude slow DNs in fan-out of WAL** + +This issue provides the method to detect slow datanodes by checking the packets processing time of each datanode connected by the WAL. When a datanode is considered slow, the datanode will be added to an exclude cache on the regionserver, and every stream created will exclude all the cached slow datanodes in a configured period. The exclude logic cooperate with the log rolling logic, will react more sensitively to the lower slow datanodes, whatever there is hardware failure or hotspots. + +hbase.regionserver.async.wal.max.exclude.datanode.countï¼default 3ï¼and hbase.regionserver.async.wal.exclude.datanode.info.ttl.hour (default 6) means no more than 3 slow datanodes will be excluded on one regionserver, and the exclude cache for the slow datanodes is valid in 6 hours. + +There are two conditions used to determine whether a datanode is slow, +1. For small packet, we just have a simple time limit(configured by hbase.regionserver.async.wal.datanode.slow.packet.process.time.millis, default 6s), without considering the size of the packet. + +2. For large packet, we will calculate the speed, and check if the speed (configured by hbase.regionserver.async.wal.datanode.slow.packet.speed.min.kbs, default 20KB/s) is too slow. + +The large and small split point is configured by hbase.regionserver.async.wal.datanode.slow.check.speed.packet.data.length.min (default 64KB). + + +--- + +* [HBASE-26079](https://issues.apache.org/jira/browse/HBASE-26079) | *Major* | **Use StoreFileTracker when splitting and merging** + +Changes MergeTableRegionProcedure and SplitTableRegionProcedure to use the underlying StoreFileTracker implementation to select the valid files on regions to be merged/split, rather than direct listing those from file system. It also changes merge/split commit logic in HRegionFileSystem, to add the links/refs on resulting split/merged regions to the StoreFileTracker. + + + +# HBASE 3.0.0-alpha-2 Release Notes + +These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements. + + +--- + +* [HBASE-26606](https://issues.apache.org/jira/browse/HBASE-26606) | *Major* | **Upgrade log4j2 to 2.17.0** + +Upgrade log4j2 to 2.17.0 to address CVE-2021-45105. + + +--- + +* [HBASE-26542](https://issues.apache.org/jira/browse/HBASE-26542) | *Minor* | **Apply a \`package\` to test protobuf files** + +The protobuf structures used in test are all now scoped by the package name \`hbase.test.pb\`. + + +--- + +* [HBASE-26572](https://issues.apache.org/jira/browse/HBASE-26572) | *Major* | **Upgrade to log4j 2.16.0** + +Upgrade log4j2 to 2.16.0 to address CVE-2021-45046. + + +--- + +* [HBASE-26554](https://issues.apache.org/jira/browse/HBASE-26554) | *Minor* | **Introduce a new parameter in jmx servlet to exclude the specific mbean** + +introduce a new parameter 'excl' in jmx servlet to exclude the specific mbean + + +--- + +* [HBASE-26557](https://issues.apache.org/jira/browse/HBASE-26557) | *Major* | **log4j2 has a critical RCE vulnerability** + +Upgrade log4j2 to 2.15.0 for addressing CVE-2021-44228. + + +--- + +* [HBASE-26524](https://issues.apache.org/jira/browse/HBASE-26524) | *Major* | **Support remove coprocessor by class name via alter table command** + +This is an incompatible change for TableDescriptorBuilder#removeCoprocessor, if coprocessor does not exist and removeCoprocessor is being called, the new behavior is emitting an IllegalArgumentException instead of previously do nothing + +From now on, alter command introduces a new method table\_remove\_coprocessor that operator can remove table coprocessor by class name. Please see the usage as below + +\* remove a single table coprocessor +alter 't1', METHOD =\> 'table\_remove\_coprocessor', CLASSNAME =\> 'org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver' + +\* remove multiple coprocessors +alter 't1', METHOD =\> 'table\_remove\_coprocessor', CLASSNAME =\> ['org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver', 'org.apache.hadoop.hbase.coprocessor.Export'] + + +--- + +* [HBASE-26529](https://issues.apache.org/jira/browse/HBASE-26529) | *Minor* | **Document HBASE-26524 to section of Dynamic Unloading** + +added the basic command usage of \`table\_remove\_coprocessor\` to the definition guide at src/main/asciidoc/\_chapters/cp.adoc. + + +hbase\> alter 'users', METHOD =\> 'table\_remove\_coprocessor', CLASSNAME =\> + 'org.myname.hbase.Coprocessor.RegionObserverExample' + + +--- + +* [HBASE-26512](https://issues.apache.org/jira/browse/HBASE-26512) | *Major* | **Make timestamp format configurable in HBase shell scan output** + +HBASE-23930 changed the formatting of the timestamp attribute on each Cell as displayed by the HBase shell to be formatted as an ISO-8601 string rather that milliseconds since the epoch. Some users may have logic which expects the timestamp to be displayed as milliseconds since the epoch. This change introduces the configuration property hbase.shell.timestamp.format.epoch which controls whether the shell will print an ISO-8601 formatted timestamp (the default "false") or milliseconds since the epoch ("true"). + + +--- + +* [HBASE-26484](https://issues.apache.org/jira/browse/HBASE-26484) | *Major* | **Update vote.tmpl in our create-release scripts to link KEYS from downloads.a.o instead of dist.a.o** + +Use downloads.a.o instead of dist.a.o in our vote.tmpl for releasing. + + +--- + +* [HBASE-26426](https://issues.apache.org/jira/browse/HBASE-26426) | *Major* | **Remove 'Review Board' section from the site** + +HBase now uses github PR for submitting and reviewing patches, so remove Review board link on our website. + + +--- + +* [HBASE-26363](https://issues.apache.org/jira/browse/HBASE-26363) | *Major* | **OpenTelemetry configuration support for per-process service names** + +<!-- markdown -->Each HBase process can have its own `service.name`, a value that can be completely customized by the operator. See the comment and examples in conf/hbase-env.sh. + + +--- + +* [HBASE-26362](https://issues.apache.org/jira/browse/HBASE-26362) | *Major* | **Upload mvn site artifacts for nightly build to nightlies** + +Now we will upload the site artifacts to nightlies for nightly build as well as pre commit build. + + +--- + +* [HBASE-26316](https://issues.apache.org/jira/browse/HBASE-26316) | *Minor* | **Per-table or per-CF compression codec setting overrides** + +It is now possible to specify codec configuration options as part of table or column family schema definitions. The configuration options will only apply to the defined scope. For example: + + hbase\> create 'sometable', \\ + { NAME =\> 'somefamily', COMPRESSION =\> 'ZSTD' }, \\ + CONFIGURATION =\> { 'hbase.io.compress.zstd.level' =\> '9' } + + +--- + +* [HBASE-26329](https://issues.apache.org/jira/browse/HBASE-26329) | *Major* | **Upgrade commons-io to 2.11.0** + +Upgraded commons-io to 2.11.0. + + +--- + +* [HBASE-26186](https://issues.apache.org/jira/browse/HBASE-26186) | *Major* | **jenkins script for caching artifacts should verify cached file before relying on it** + +Add a '--verify-tar-gz' option to cache-apache-project-artifact.sh for verifying whether the cached file can be parsed as a gzipped tarball. +Use this option in our nightly job to avoid failures on broken cached hadoop tarballs. + + +--- + +* [HBASE-26339](https://issues.apache.org/jira/browse/HBASE-26339) | *Major* | **SshPublisher will skip uploading artifacts if the build is failure** + +Now we will mark build as unstable instead of failure when the yetus script returns error. This is used to solve the problem that the SshPublisher jenkins plugin will skip uploading artifacts if the build is marked as failure. In fact, the test output will be more important when there are UT failures. + + +--- + +* [HBASE-26317](https://issues.apache.org/jira/browse/HBASE-26317) | *Major* | **Publish the test logs for pre commit jenkins job to nightlies** + +Now we will upload test\_logs.zip for our pre commit jobs to nightlies to save space on jenkins node. You can see the test\_logs.txt to get the actual url of the test\_logs.zip, or visit https://nightlies.apache.org/hbase directly to find the artifacts. + + +--- + +* [HBASE-26313](https://issues.apache.org/jira/browse/HBASE-26313) | *Major* | **Publish the test logs for our nightly jobs to nightlies.apache.org** + +Now we will upload test\_logs.zip for our nightly jobs to nightlies to save space on jenkins node. You can see the test\_logs.txt to get the actual url of the test\_logs.zip, or visit https://nightlies.apache.org/hbase directly to find the artifacts. + + +--- + +* [HBASE-26318](https://issues.apache.org/jira/browse/HBASE-26318) | *Major* | **Publish test logs for flaky jobs to nightlies** + +Now we will upload the surefire output for our flaky test jobs to nightlies to save space on jenkins node. You can see the test\_logs.txt to get the actual url of the surefire output, or visit https://nightlies.apache.org/hbase directly to find the artifacts. + + +--- + +* [HBASE-26259](https://issues.apache.org/jira/browse/HBASE-26259) | *Major* | **Fallback support to pure Java compression** + +This change introduces provided compression codecs to HBase as + new Maven modules. Each module provides compression codec support that formerly required Hadoop native codecs, which in turn relies on native code integration, which may or may not be available on a given hardware platform or in an operational environment. We now provide codecs in the HBase distribution for users whom for whatever reason cannot or do not wish to deploy the Hadoop native codecs. + + +--- + +* [HBASE-26321](https://issues.apache.org/jira/browse/HBASE-26321) | *Major* | **Post blog to hbase.apache.org on SCR cache sizing** + +Pushed blog at https://blogs.apache.org/hbase/entry/an-hbase-hdfs-short-circuit + + +--- + +* [HBASE-26270](https://issues.apache.org/jira/browse/HBASE-26270) | *Minor* | **Provide getConfiguration method for Region and Store interface** + +Provide 'getReadOnlyConfiguration' method for Store and Region interface + + +--- + +* [HBASE-26273](https://issues.apache.org/jira/browse/HBASE-26273) | *Major* | **TableSnapshotInputFormat/TableSnapshotInputFormatImpl should use ReadType.STREAM for scanning HFiles** + +HBase's MapReduce API which can operate over HBase snapshots will now default to using ReadType.STREAM instead of ReadType.DEFAULT (which is PREAD) as a result of this change. HBase developers expect that STREAM will perform significantly better for average Snapshot-based batch jobs. Users can restore the previous functionality (using PREAD) by updating their code to explicitly set a value of \`ReadType.PREAD\` on the \`Scan\` object they provide to TableSnapshotInputFormat, or by setting the configuration property "hbase.TableSnapshotInputFormat.scanner.readtype" to "PREAD" in hbase-site.xml. + + +--- + +* [HBASE-25288](https://issues.apache.org/jira/browse/HBASE-25288) | *Major* | **Make MasterRpcServices not extends RSRpcServices and also HMaster not extends HRegionServer** + +HMaster is not a sub class for HRegionServer now, and also MasterRpcServices is not a sub class for RSRpcServices. + +MasterRpcServices still implements the AdminService interface, but only updateConfiguration, getRegionInfo, clearSlowLogsResponses and getLogEntries can be called, for other methods in AdminService, you will get a DoNotRetryIOException when calling against master. + + +--- + +* [HBASE-25891](https://issues.apache.org/jira/browse/HBASE-25891) | *Major* | **Remove dependence on storing WAL filenames for backup** + +\* Remove dependence on storing WAL filenames for backup in backup:system meta table + + +--- + +* [HBASE-26276](https://issues.apache.org/jira/browse/HBASE-26276) | *Major* | **Allow HashTable/SyncTable to perform rawScan when comparing cells** + +Added --rawScan option to HashTable job, which allows HashTable/SyncTable to perform raw scans. If this property is omitted, it defaults to false. When used together with --versions set to a high value, SyncTable will fabricate delete markers to all old versions still hanging (not cleaned yet by major compaction), avoiding the inconsistencies reported in HBASE-21596. + + +--- + +* [HBASE-26230](https://issues.apache.org/jira/browse/HBASE-26230) | *Major* | **Start an in process HRegionServer in maintenance mode** + +When enabling maintenance mode, now we will start an in process region server instead of letting HMaster carry the system regions. + + +--- + +* [HBASE-26147](https://issues.apache.org/jira/browse/HBASE-26147) | *Major* | **Add dry run mode to hbase balancer** + +This change adds new API to the Admin interface for triggering Region balancing on a cluster. A new BalanceRequest object was introduced which allows for configuring a dry run of the balancer (compute a plan without enacting it) and running the balancer in the presence of RITs. Corresponding API was added to the HBase shell as well. + + +--- + +* [HBASE-26103](https://issues.apache.org/jira/browse/HBASE-26103) | *Major* | **conn.getBufferedMutator(tableName) leaks thread executors and other problems (for master branch)** + +Deprecate (unused) BufferedMutatorParams#pool and BufferedMutatorParams#getPool + + +--- + +* [HBASE-26204](https://issues.apache.org/jira/browse/HBASE-26204) | *Major* | **VerifyReplication should obtain token for peerQuorumAddress too** + +VerifyReplication obtains tokens even if the peer quorum parameter is used. VerifyReplication with peer quorum can be used for secure clusters also. + + +--- + +* [HBASE-26180](https://issues.apache.org/jira/browse/HBASE-26180) | *Major* | **Introduce a initial refresh interval for RpcConnectionRegistry** + +Introduced a 'hbase.client.bootstrap.initial\_refresh\_delay\_secs' config to control the first refresh delay for bootstrap nodes. The default value is 1/10 of periodic refresh interval. + + +--- + +* [HBASE-26173](https://issues.apache.org/jira/browse/HBASE-26173) | *Major* | **Return only a sub set of region servers as bootstrap nodes** + +Introduced a 'hbase.client.bootstrap.node.limit' config to limit the max number of bootstrap nodes we return to client. The default value is 10. + + +--- + +* [HBASE-26182](https://issues.apache.org/jira/browse/HBASE-26182) | *Major* | **Allow disabling refresh of connection registry endpoint** + +Set 'hbase.client.bootstrap.refresh\_interval\_secs' to -1 can disable refresh of connection registry endpoint. + + +--- + +* [HBASE-26212](https://issues.apache.org/jira/browse/HBASE-26212) | *Minor* | **Allow AuthUtil automatic renewal to be disabled** + +This change introduces a configuration property "hbase.client.keytab.automatic.renewal" to control AuthUtil, the class which automatically tries to perform Kerberos ticket renewal in client applications. This configuration property defaults to "true", meaning that AuthUtil will automatically attempt to renew Kerberos tickets per its capabilities. Those who want AuthUtil to not renew client Kerberos tickets can set this property to be "false". + + +--- + +* [HBASE-26174](https://issues.apache.org/jira/browse/HBASE-26174) | *Major* | **Make rpc connection registry the default registry on 3.0.0** + +Now RpcConnectionRegistry is the default ConnectionRegistry. + + +--- + +* [HBASE-26172](https://issues.apache.org/jira/browse/HBASE-26172) | *Major* | **Deprecate MasterRegistry** + +MasterRegistry is deprecated. Please use RpcConnectionRegistry instead. + + +--- + +* [HBASE-26193](https://issues.apache.org/jira/browse/HBASE-26193) | *Major* | **Do not store meta region location as permanent state on zookeeper** + +Introduce a new 'info' family in master local region for storing the location of meta regions.
[... 5943 lines stripped ...]
