kgeisz opened a new pull request, #8517: URL: https://github.com/apache/hbase/pull/8517
[HBASE-30087: Create GitHub Actions for testing Read-Replica feature](https://issues.apache.org/jira/browse/HBASE-30087) # Introduction - [Apache HBase Read Replica Cluster Feature](https://docs.google.com/document/d/1EI0lsURX1BZhv3DYgMvZCl4EUy-ADJRkHUc1PjzZtj0/edit?usp=sharing) - [Read Replica Test Plan Document](https://docs.google.com/document/d/1liMDJd8xtgu9ExzfjnY6mY6hYNd4Y_7H-MpVZ5qWZJg/edit?usp=sharing) - [HBase Website Documentation for Read Replica Cluster](https://hbase.apache.org/docs/read-replica-cluster) HBase's Read-Replica feature was merged into master in PR #8364. This pull request introduces integration tests for Read-Replica via GitHub Actions and Docker containers. GitHub Actions and Docker containers have been useful for testing this feature because it works around the `META_TABLE_NAME` issue mentioned in [HBASE-29691](https://issues.apache.org/jira/browse/HBASE-29691) and PR #7730. In a Read-Replica setup, clusters share the same storage location but need to have different `META_TABLE_NAME`s. The table names are distinguished using the `hbase.meta.table.suffix` configuration property. However, this still leads to problems when using `MiniHBaseCluster` to test a multi-cluster Read-Replica setup because these clusters run in the same JVM and end up sharing the same `static META_TABLE_NAME` variable. # How It Works - The `.github/workflows/read-replica-integration-tests.yml` file is the main driver of the integration tests. This file defines the workflow and steps performed during the integration test. - The `dev-support/integration-test/read-replica` directory contains various files for building a Docker image, running containers with HBase procecces, and running integration test scripts written in Python. It also contains files and directories used by HBase, such as `conf` directories. - The `hbase_docker_client.py` file is the most important Python file. Every test script uses this file to communicate with each hbase-docker container. It does so by using `docker exec` to run commands in a container's HBase shell. The tests are triggered when a someone submits a pull request with the `read-replica` label. Once the label is added, the test begins by checking out the HBase repo. This repo contains the pull request's code, so any changes to `read-replica-integration-tests.yml` and `dev-support/integration-test/read-replica` can be tested immediately. After, some environment setup is done. Next, the test checks out the submitted pull request's HBase repo again, except the repo's directory is placed in `dev-support/integration-test/read-replica`. Here, the repo can be used to build a Docker image for running HBase. After the image has been built, two Docker containers are started, each of which are running HBase in a Read-Replica setup. One container starts as the active cluster (read-write mode), while the other container starts as a replica cluster (read-only mode). These clusters share a `data-store` directory that contain the `hbase.rootdir`. It is a mounted volume between each container and the local filesystem, and it is created by the Python scripts before container startup and given `777` permissions in order to avoid HBase startup failures. There are other volumes as well for easy access, such as each container's `conf` directory and a `utils` directory for bulkloading data. Once the containers are up and running, a series of Python scripts are run as integration tests. They test expected behavior for a Read-Replica cluster setup, such as verifying valid/invalid startup, blocking writes on replica clusters, being able to refresh meta and HFiles on replica clusters to make them consistent with the active cluster, read-only flag flipping (`hbase.global.readonly.enabled`), verifying bug fixes, etc. The containers are shut down when the test is complete. If a failure occurs at any point, the workflow is stopped and the HBase logs for each container are dumped. # Other Information - The test is re-triggered when a new commit is pushed. If the test is already running when a new commit is pushed, then that test is stopped and a new one is started using the latest commit. The test is not run for draft PRs. - `docker-compose.yml` defines and configures two hbase-docker containers capable of running in a read-replica setup. - `build_images.sh` uses `Dockerfile` to build an hbase-docker image. This script assumes there is an `hbase` repo in the same directory. - `.env` defines environment variables used by `docker-compose.yml`, `build_images.sh`, and Python scripts. - `proto_compiler.py` copies `ActiveClusterSuffix.proto` from the hbase repo and compiles it. The generated output is used for verifying the `active.cluster.suffix.id` file. - `utils` is a directory containing scripts for bulkloading data into HBase - `conf1` and `conf2` are the active and replica cluster's configuration directories, respectively. They each contain their own `hbase-site.xml`, `log4j2.properties`, and `zoo.cfg` files. The directories are defined as mounted volumes in `docker-compose.yml`. This is especially useful for changing the value of `hbase.global.readonly.enabled` in order to change a cluster's read-only mode. # Example Test Runs - Successful test run. - This test run is from an example pull request that makes changes to resolve [HBASE-30301](https://issues.apache.org/jira/browse/HBASE-30301). This test run builds a Docker image that has the bug fix, resulting in all tests passing. - https://github.com/kgeisz/hbase/actions/runs/30566311529/job/90954447739 - Failed test run. - The test run from this example PR does not include changes for [HBASE-30301](https://issues.apache.org/jira/browse/HBASE-30301). As a result, the Docker image that was built did not contain the bug fix, and one of the integration tests failed. The HBase logs were dumped after the failure. - https://github.com/kgeisz/hbase/actions/runs/30374441518/job/90326363151 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
