Copilot commented on code in PR #8459: URL: https://github.com/apache/hbase/pull/8459#discussion_r3550987469
########## hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/read-replica-cluster.mdx: ########## @@ -0,0 +1,201 @@ +--- +title: "Read Replica Cluster" +description: "Running a secondary HBase cluster in read-only mode against shared cloud storage to scale read workloads." +--- + +## Background [#read-replica-cluster-background] + +A _Read Replica Cluster_ is an entire HBase cluster running in global read-only mode against the same shared +storage (`hbase.rootdir`) as an active read-write cluster. Both clusters list the same HFiles in the same +HDFS / cloud-object-store location; no data is copied. Reads can be served from either cluster, letting the +read workload be fanned out across multiple clusters without doubling storage cost. + +Typical use cases: + +- Fan out heavy scan / analytical workloads off the primary cluster. +- Add cross-availability-zone read capacity backed by a single shared bucket. +- Stand up an isolated cluster for read-mostly experiments without copying data. + +<Callout type="info"> + **Eventual consistency.** A replica only sees data once (a) the active cluster has flushed the data to + HFiles in shared storage, and (b) the replica has been told to re-read shared storage via the + `refresh_meta` and `refresh_hfiles` commands. MemStore data on the active cluster is invisible to the + replica until flushed. +</Callout> + +The parent design lives on [HBASE-29081](https://issues.apache.org/jira/browse/HBASE-29081). + +## Design + +The feature has three parts. + +### Custom `hbase:meta` per cluster + +Every cluster sharing a `hbase.rootdir` needs its own `hbase:meta`, because region assignments are +node-scoped and cannot be shared. Other system tables (`hbase:namespace`, `hbase:acl`, `hbase:replication`) +_are_ safe to share because their contents are storage-wide and the replica never writes to them. Review Comment: The doc says system tables like `hbase:namespace` are safe to share, but namespace table data has been folded into `hbase:meta` (see `TableName.NAMESPACE_TABLE_NAME` deprecation), so listing `hbase:namespace` here is misleading. Also, read-only mode still allows writes to `master:store` (master local region) in addition to the per-cluster `hbase:meta`, so it’s useful to call that out when explaining what must be unique per cluster. ########## hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/read-replica-cluster.mdx: ########## @@ -0,0 +1,201 @@ +--- +title: "Read Replica Cluster" +description: "Running a secondary HBase cluster in read-only mode against shared cloud storage to scale read workloads." +--- + +## Background [#read-replica-cluster-background] + +A _Read Replica Cluster_ is an entire HBase cluster running in global read-only mode against the same shared +storage (`hbase.rootdir`) as an active read-write cluster. Both clusters list the same HFiles in the same +HDFS / cloud-object-store location; no data is copied. Reads can be served from either cluster, letting the +read workload be fanned out across multiple clusters without doubling storage cost. + +Typical use cases: + +- Fan out heavy scan / analytical workloads off the primary cluster. +- Add cross-availability-zone read capacity backed by a single shared bucket. +- Stand up an isolated cluster for read-mostly experiments without copying data. + +<Callout type="info"> + **Eventual consistency.** A replica only sees data once (a) the active cluster has flushed the data to + HFiles in shared storage, and (b) the replica has been told to re-read shared storage via the + `refresh_meta` and `refresh_hfiles` commands. MemStore data on the active cluster is invisible to the + replica until flushed. +</Callout> + +The parent design lives on [HBASE-29081](https://issues.apache.org/jira/browse/HBASE-29081). + +## Design + +The feature has three parts. + +### Custom `hbase:meta` per cluster + +Every cluster sharing a `hbase.rootdir` needs its own `hbase:meta`, because region assignments are +node-scoped and cannot be shared. Other system tables (`hbase:namespace`, `hbase:acl`, `hbase:replication`) +_are_ safe to share because their contents are storage-wide and the replica never writes to them. + +The configuration key `hbase.meta.table.suffix` selects a per-cluster suffix; the meta table becomes +`hbase:meta_<suffix>` and the master's local store directory becomes `MasterData_<suffix>`. The suffix must +match `[a-zA-Z0-9]+`. + +### Global read-only mode + +`hbase.global.readonly.enabled=true` puts a cluster into read-only mode. Five coprocessor controllers under +`org.apache.hadoop.hbase.security.access` intercept every user-table mutation path and throw +`WriteAttemptedOnReadOnlyClusterException` (a `DoNotRetryIOException`) with the message +`Operation not allowed in Read-Only Mode`: + +| Class | Coprocessor host | Blocks | +| -------------------------------- | ---------------- | ----------------------------------------------------------------------------------------------------- | +| `MasterReadOnlyController` | Master | DDL, snapshots, splits, merges, namespace ops, ACL/quota ops, replication-peer ops | +| `RegionReadOnlyController` | Region | put, delete, batchMutate, checkAnd\*, append, increment, flush, compaction, WAL append, commit/replay | +| `RegionServerReadOnlyController` | RegionServer | WAL roll, replication sink mutations, log replay | +| `BulkLoadReadOnlyController` | Region | bulk-load prepare/cleanup | +| `EndpointReadOnlyController` | Region | all coprocessor endpoint invocations | + +Operators do not load these classes manually. `CoprocessorConfigurationUtil.syncReadOnlyConfigurations` +adds them to `hbase.coprocessor.master.classes`, `hbase.coprocessor.regionserver.classes`, and +`hbase.coprocessor.region.classes` at startup and on every dynamic +`ConfigurationManager.notifyAllObservers` event — so the flag can be flipped at runtime with +`update_all_config` (see Case 3). + +### Preventing Multiple Active Clusters (active.cluster.suffix.id) + +Two clusters writing to the same `hbase.rootdir` would corrupt shared storage. To enforce a single writer, an +active master creates a protobuf-serialized sentinel at `<hbase.rootdir>/active.cluster.suffix.id` recording +its cluster ID and meta suffix. `MasterFileSystem.negotiateActiveClusterSuffixFile` runs at master startup: + +- An **active** cluster (`hbase.global.readonly.enabled=false`) creates the file if absent, or verifies its + contents match its own identity. If the file belongs to another cluster, startup aborts with an + `IOException`. +- A **replica** cluster (`hbase.global.readonly.enabled=true`) does not read or write the file; it logs + `[Read-replica feature] Replica cluster is being started in Read Only Mode` and continues. + +`AbstractReadOnlyController.manageActiveClusterIdFile` handles the dynamic toggle: switching to read-only +deletes the file if this cluster owns it, and switching back to read-write creates the file if absent. + +## Configuration + +On every node of the **read replica cluster**, add the following to `hbase-site.xml`: + +```xml +<property> + <name>hbase.global.readonly.enabled</name> + <value>true</value> + <description> + Put this cluster into global read-only mode. All user-table writes, flushes, + compactions, splits, and merges are blocked. The five ReadOnly coprocessor + controllers are loaded automatically. + </description> +</property> +<property> + <name>hbase.meta.table.suffix</name> + <value>shared</value> + <description> + Optional. If set, the meta table is named hbase:meta_<suffix> and the + master's local store directory is MasterData_<suffix>. Value must match + [a-zA-Z0-9]+. The same value MUST be set on the active cluster and every + read replica that shares the same hbase.rootdir. + </description> Review Comment: The `hbase.meta.table.suffix` description says “The same value MUST be set on the active cluster and every read replica that shares the same hbase.rootdir”, but earlier this page correctly states each cluster needs its own `hbase:meta`. If clusters share the same suffix they will share the same meta table and master local region dir, which defeats the isolation and can break the deployment. ########## hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/read-replica-cluster.mdx: ########## @@ -0,0 +1,201 @@ +--- +title: "Read Replica Cluster" +description: "Running a secondary HBase cluster in read-only mode against shared cloud storage to scale read workloads." +--- + +## Background [#read-replica-cluster-background] + +A _Read Replica Cluster_ is an entire HBase cluster running in global read-only mode against the same shared +storage (`hbase.rootdir`) as an active read-write cluster. Both clusters list the same HFiles in the same +HDFS / cloud-object-store location; no data is copied. Reads can be served from either cluster, letting the +read workload be fanned out across multiple clusters without doubling storage cost. + +Typical use cases: + +- Fan out heavy scan / analytical workloads off the primary cluster. +- Add cross-availability-zone read capacity backed by a single shared bucket. +- Stand up an isolated cluster for read-mostly experiments without copying data. + +<Callout type="info"> + **Eventual consistency.** A replica only sees data once (a) the active cluster has flushed the data to + HFiles in shared storage, and (b) the replica has been told to re-read shared storage via the + `refresh_meta` and `refresh_hfiles` commands. MemStore data on the active cluster is invisible to the + replica until flushed. +</Callout> + +The parent design lives on [HBASE-29081](https://issues.apache.org/jira/browse/HBASE-29081). + +## Design + +The feature has three parts. + +### Custom `hbase:meta` per cluster + +Every cluster sharing a `hbase.rootdir` needs its own `hbase:meta`, because region assignments are +node-scoped and cannot be shared. Other system tables (`hbase:namespace`, `hbase:acl`, `hbase:replication`) +_are_ safe to share because their contents are storage-wide and the replica never writes to them. + +The configuration key `hbase.meta.table.suffix` selects a per-cluster suffix; the meta table becomes +`hbase:meta_<suffix>` and the master's local store directory becomes `MasterData_<suffix>`. The suffix must +match `[a-zA-Z0-9]+`. + +### Global read-only mode + +`hbase.global.readonly.enabled=true` puts a cluster into read-only mode. Five coprocessor controllers under +`org.apache.hadoop.hbase.security.access` intercept every user-table mutation path and throw +`WriteAttemptedOnReadOnlyClusterException` (a `DoNotRetryIOException`) with the message +`Operation not allowed in Read-Only Mode`: + +| Class | Coprocessor host | Blocks | +| -------------------------------- | ---------------- | ----------------------------------------------------------------------------------------------------- | +| `MasterReadOnlyController` | Master | DDL, snapshots, splits, merges, namespace ops, ACL/quota ops, replication-peer ops | +| `RegionReadOnlyController` | Region | put, delete, batchMutate, checkAnd\*, append, increment, flush, compaction, WAL append, commit/replay | +| `RegionServerReadOnlyController` | RegionServer | WAL roll, replication sink mutations, log replay | +| `BulkLoadReadOnlyController` | Region | bulk-load prepare/cleanup | +| `EndpointReadOnlyController` | Region | all coprocessor endpoint invocations | + +Operators do not load these classes manually. `CoprocessorConfigurationUtil.syncReadOnlyConfigurations` +adds them to `hbase.coprocessor.master.classes`, `hbase.coprocessor.regionserver.classes`, and +`hbase.coprocessor.region.classes` at startup and on every dynamic +`ConfigurationManager.notifyAllObservers` event — so the flag can be flipped at runtime with +`update_all_config` (see Case 3). + +### Preventing Multiple Active Clusters (active.cluster.suffix.id) + +Two clusters writing to the same `hbase.rootdir` would corrupt shared storage. To enforce a single writer, an +active master creates a protobuf-serialized sentinel at `<hbase.rootdir>/active.cluster.suffix.id` recording +its cluster ID and meta suffix. `MasterFileSystem.negotiateActiveClusterSuffixFile` runs at master startup: + +- An **active** cluster (`hbase.global.readonly.enabled=false`) creates the file if absent, or verifies its + contents match its own identity. If the file belongs to another cluster, startup aborts with an + `IOException`. +- A **replica** cluster (`hbase.global.readonly.enabled=true`) does not read or write the file; it logs + `[Read-replica feature] Replica cluster is being started in Read Only Mode` and continues. + +`AbstractReadOnlyController.manageActiveClusterIdFile` handles the dynamic toggle: switching to read-only +deletes the file if this cluster owns it, and switching back to read-write creates the file if absent. + +## Configuration + +On every node of the **read replica cluster**, add the following to `hbase-site.xml`: + +```xml +<property> + <name>hbase.global.readonly.enabled</name> + <value>true</value> + <description> + Put this cluster into global read-only mode. All user-table writes, flushes, + compactions, splits, and merges are blocked. The five ReadOnly coprocessor + controllers are loaded automatically. + </description> +</property> +<property> + <name>hbase.meta.table.suffix</name> + <value>shared</value> + <description> + Optional. If set, the meta table is named hbase:meta_<suffix> and the + master's local store directory is MasterData_<suffix>. Value must match + [a-zA-Z0-9]+. The same value MUST be set on the active cluster and every + read replica that shares the same hbase.rootdir. + </description> +</property> +``` + +The **active cluster** uses the same `hbase.rootdir` and the same `hbase.meta.table.suffix` (if any) but +leaves `hbase.global.readonly.enabled` unset or `false`. + +`hbase.global.readonly.enabled` is a dynamic configuration — a config-change event reloads the read-only +coprocessors without restarting the process. All nodes must agree on the value; operators are responsible for +keeping every `hbase-site.xml` in sync before issuing `update_all_config`. + +## Operation and maintenance + +### Case 1. Bring up a new read replica cluster + +1. Provision the replica cluster on hardware that can reach the active cluster's `hbase.rootdir` (typically + the same HDFS or object store). +2. Set `hbase.global.readonly.enabled=true` in the replica's `hbase-site.xml`. And set up + `hbase.meta.table.suffix`, to distinguish the replica cluster's meta table on the shared storage. +3. Start the cluster and verify if the Master log shows + `[Read-replica feature] Replica cluster is being started in Read Only Mode`. +4. From the replica shell, run `refresh_meta` and then `refresh_hfiles` to materialize the active cluster's + current state on the replica. + +### Case 2. Routine sync after writes on the active cluster + +```ruby +# On the active cluster +hbase> flush 'my_namespace:my_table' +``` + +```ruby +# On the read replica cluster +hbase> refresh_meta +hbase> refresh_hfiles 'TABLE_NAME' => 'my_namespace:my_table' +``` + +Always run `refresh_meta` first, then `refresh_hfiles`. `refresh_hfiles` only refreshes regions that are open +on the replica, so newly discovered regions must be in meta (and assigned) before their HFiles can be picked +up. `refresh_hfiles` supports three scopes: + +```ruby +hbase> refresh_hfiles # all user tables +hbase> refresh_hfiles 'TABLE_NAME' => 'ns:table' # one table +hbase> refresh_hfiles 'NAMESPACE' => 'ns' # one namespace +``` + +Passing both `TABLE_NAME` and `NAMESPACE` to `refresh_hfiles` is rejected. Both commands return a procedure +ID that can be tracked through the master UI or `Admin.getProcedures()`. + +If the replica's block cache holds stale entries for a table that has just been refreshed, evict them with +the pre-existing `clear_block_cache 'my_namespace:my_table'` shell command. + +`Admin` and `AsyncAdmin` expose the same operations programmatically: + +```java +long pid = admin.refreshMeta(); +long pid = admin.refreshHFiles(); // all user tables +long pid = admin.refreshHFiles(TableName.valueOf("ns:table")); // one table +long pid = admin.refreshHFiles("ns"); // one namespace Review Comment: The Java snippet redeclares `pid` on every line, which won’t compile if copied verbatim. Use one declaration and then reassign (or use distinct variable names). ########## hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/read-replica-cluster.mdx: ########## @@ -0,0 +1,201 @@ +--- +title: "Read Replica Cluster" +description: "Running a secondary HBase cluster in read-only mode against shared cloud storage to scale read workloads." +--- + +## Background [#read-replica-cluster-background] + +A _Read Replica Cluster_ is an entire HBase cluster running in global read-only mode against the same shared +storage (`hbase.rootdir`) as an active read-write cluster. Both clusters list the same HFiles in the same +HDFS / cloud-object-store location; no data is copied. Reads can be served from either cluster, letting the +read workload be fanned out across multiple clusters without doubling storage cost. + +Typical use cases: + +- Fan out heavy scan / analytical workloads off the primary cluster. +- Add cross-availability-zone read capacity backed by a single shared bucket. +- Stand up an isolated cluster for read-mostly experiments without copying data. + +<Callout type="info"> + **Eventual consistency.** A replica only sees data once (a) the active cluster has flushed the data to + HFiles in shared storage, and (b) the replica has been told to re-read shared storage via the + `refresh_meta` and `refresh_hfiles` commands. MemStore data on the active cluster is invisible to the + replica until flushed. +</Callout> + +The parent design lives on [HBASE-29081](https://issues.apache.org/jira/browse/HBASE-29081). + +## Design + +The feature has three parts. + +### Custom `hbase:meta` per cluster + +Every cluster sharing a `hbase.rootdir` needs its own `hbase:meta`, because region assignments are +node-scoped and cannot be shared. Other system tables (`hbase:namespace`, `hbase:acl`, `hbase:replication`) +_are_ safe to share because their contents are storage-wide and the replica never writes to them. + +The configuration key `hbase.meta.table.suffix` selects a per-cluster suffix; the meta table becomes +`hbase:meta_<suffix>` and the master's local store directory becomes `MasterData_<suffix>`. The suffix must +match `[a-zA-Z0-9]+`. + +### Global read-only mode + +`hbase.global.readonly.enabled=true` puts a cluster into read-only mode. Five coprocessor controllers under +`org.apache.hadoop.hbase.security.access` intercept every user-table mutation path and throw +`WriteAttemptedOnReadOnlyClusterException` (a `DoNotRetryIOException`) with the message +`Operation not allowed in Read-Only Mode`: + +| Class | Coprocessor host | Blocks | +| -------------------------------- | ---------------- | ----------------------------------------------------------------------------------------------------- | +| `MasterReadOnlyController` | Master | DDL, snapshots, splits, merges, namespace ops, ACL/quota ops, replication-peer ops | +| `RegionReadOnlyController` | Region | put, delete, batchMutate, checkAnd\*, append, increment, flush, compaction, WAL append, commit/replay | +| `RegionServerReadOnlyController` | RegionServer | WAL roll, replication sink mutations, log replay | +| `BulkLoadReadOnlyController` | Region | bulk-load prepare/cleanup | +| `EndpointReadOnlyController` | Region | all coprocessor endpoint invocations | + +Operators do not load these classes manually. `CoprocessorConfigurationUtil.syncReadOnlyConfigurations` +adds them to `hbase.coprocessor.master.classes`, `hbase.coprocessor.regionserver.classes`, and +`hbase.coprocessor.region.classes` at startup and on every dynamic +`ConfigurationManager.notifyAllObservers` event — so the flag can be flipped at runtime with +`update_all_config` (see Case 3). + +### Preventing Multiple Active Clusters (active.cluster.suffix.id) + +Two clusters writing to the same `hbase.rootdir` would corrupt shared storage. To enforce a single writer, an +active master creates a protobuf-serialized sentinel at `<hbase.rootdir>/active.cluster.suffix.id` recording +its cluster ID and meta suffix. `MasterFileSystem.negotiateActiveClusterSuffixFile` runs at master startup: + +- An **active** cluster (`hbase.global.readonly.enabled=false`) creates the file if absent, or verifies its + contents match its own identity. If the file belongs to another cluster, startup aborts with an + `IOException`. +- A **replica** cluster (`hbase.global.readonly.enabled=true`) does not read or write the file; it logs + `[Read-replica feature] Replica cluster is being started in Read Only Mode` and continues. + +`AbstractReadOnlyController.manageActiveClusterIdFile` handles the dynamic toggle: switching to read-only +deletes the file if this cluster owns it, and switching back to read-write creates the file if absent. + +## Configuration + +On every node of the **read replica cluster**, add the following to `hbase-site.xml`: + +```xml +<property> + <name>hbase.global.readonly.enabled</name> + <value>true</value> + <description> + Put this cluster into global read-only mode. All user-table writes, flushes, + compactions, splits, and merges are blocked. The five ReadOnly coprocessor + controllers are loaded automatically. + </description> +</property> +<property> + <name>hbase.meta.table.suffix</name> + <value>shared</value> + <description> + Optional. If set, the meta table is named hbase:meta_<suffix> and the + master's local store directory is MasterData_<suffix>. Value must match + [a-zA-Z0-9]+. The same value MUST be set on the active cluster and every + read replica that shares the same hbase.rootdir. + </description> +</property> +``` + +The **active cluster** uses the same `hbase.rootdir` and the same `hbase.meta.table.suffix` (if any) but +leaves `hbase.global.readonly.enabled` unset or `false`. Review Comment: This says the active cluster uses the same `hbase.meta.table.suffix` as the replica. For a shared `hbase.rootdir` deployment the active and replica clusters must have different meta suffixes (each cluster needs its own `hbase:meta`/`MasterData_*`). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
