This is an automated email from the ASF dual-hosted git repository.
maoling pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/zookeeper.git
The following commit(s) were added to refs/heads/master by this push:
new 5c10229 ZOOKEEPER-3601: introduce the fault injection framework:
Byteman for ZooKeeper
5c10229 is described below
commit 5c102298f8a160ea996be7b6d6f95189d4ff2f41
Author: maoling <[email protected]>
AuthorDate: Sat May 29 20:53:04 2021 +0800
ZOOKEEPER-3601: introduce the fault injection framework: Byteman for
ZooKeeper
- **Byteman** is a powerful tool to inject faults during runtime,
especially for distributed system
- In the future, we can also introduce it into our unit test infra, just as
hadoop, cassandra did. [PR-123](https://github.com/apache/zookeeper/pull/123)
is a good starting.
- more details in the
[ZOOKEEPER-3601](https://issues.apache.org/jira/browse/ZOOKEEPER-3601)
Author: maoling <[email protected]>
Reviewers: Andor Molnar <[email protected]>
Closes #1135 from maoling/ZOOKEEPER-3601
---
.../src/main/resources/markdown/zookeeperTools.md | 83 ++++++++++++++++++++++
1 file changed, 83 insertions(+)
diff --git a/zookeeper-docs/src/main/resources/markdown/zookeeperTools.md
b/zookeeper-docs/src/main/resources/markdown/zookeeperTools.md
index 7e67db6..cd8b441 100644
--- a/zookeeper-docs/src/main/resources/markdown/zookeeperTools.md
+++ b/zookeeper-docs/src/main/resources/markdown/zookeeperTools.md
@@ -30,6 +30,8 @@ limitations under the License.
* [zk-smoketest](#zk-smoketest)
* [Testing](#Testing)
+ * [Fault Injection Framework](#fault-injection)
+ * [Byteman](#Byteman)
* [Jepsen Test](#jepsen-test)
<a name="Scripts"></a>
@@ -526,6 +528,87 @@ existing installations. More details are
[here](https://github.com/phunt/zk-smok
## Testing
+<a name="fault-injection"></a>
+
+### Fault Injection Framework
+
+<a name="Byteman"></a>
+
+#### Byteman
+
+- **Byteman** is a tool which makes it easy to trace, monitor and test the
behaviour of Java application and JDK runtime code.
+It injects Java code into your application methods or into Java runtime
methods without the need for you to recompile, repackage or even redeploy your
application.
+Injection can be performed at JVM startup or after startup while the
application is still running.
+- Visit the official [website](https://byteman.jboss.org/) to download the
latest release
+- A brief tutorial can be found
[here](https://developer.jboss.org/wiki/ABytemanTutorial)
+
+ ```bash
+ Preparations:
+ # attach the byteman to 3 zk servers during runtime
+ # 55001,55002,55003 is byteman binding port; 714,740,758 is the zk server
pid
+ ./bminstall.sh -b -Dorg.jboss.byteman.transform.all
-Dorg.jboss.byteman.verbose -p 55001 714
+ ./bminstall.sh -b -Dorg.jboss.byteman.transform.all
-Dorg.jboss.byteman.verbose -p 55002 740
+ ./bminstall.sh -b -Dorg.jboss.byteman.transform.all
-Dorg.jboss.byteman.verbose -p 55003 758
+
+ # load the fault injection script
+ ./bmsubmit.sh -p 55002 -l my_zk_fault_injection.btm
+ # unload the fault injection script
+ ./bmsubmit.sh -p 55002 -u my_zk_fault_injectionr.btm
+ ```
+
+Look at the below examples to customize your byteman fault injection script
+
+Example 1: This script makes leader's zxid roll over, to force re-election.
+
+```bash
+cat zk_leader_zxid_roll_over.btm
+
+RULE trace zk_leader_zxid_roll_over
+CLASS org.apache.zookeeper.server.quorum.Leader
+METHOD propose
+IF true
+DO
+ traceln("*** Leader zxid has rolled over, forcing re-election ***");
+ $1.zxid = 4294967295L
+ENDRULE
+```
+
+Example 2: This script makes the leader drop the ping packet to a specific
follower.
+The leader will close the **LearnerHandler** with that follower, and the
follower will enter the state:LOOKING
+then re-enter the quorum with the state:FOLLOWING
+
+```bash
+cat zk_leader_drop_ping_packet.btm
+
+RULE trace zk_leader_drop_ping_packet
+CLASS org.apache.zookeeper.server.quorum.LearnerHandler
+METHOD ping
+AT ENTRY
+IF $0.sid == 2
+DO
+ traceln("*** Leader drops ping packet to sid: 2 ***");
+ return;
+ENDRULE
+```
+
+Example 3: This script makes one follower drop ACK packet which has no big
effect in the broadcast phrase, since after receiving
+the majority of ACKs from the followers, the leader can commit that proposal
+
+```bash
+cat zk_leader_drop_ping_packet.btm
+
+RULE trace zk.follower_drop_ack_packet
+CLASS org.apache.zookeeper.server.quorum.SendAckRequestProcessor
+METHOD processRequest
+AT ENTRY
+IF true
+DO
+ traceln("*** Follower drops ACK packet ***");
+ return;
+ENDRULE
+```
+
+
<a name="jepsen-test"></a>
### Jepsen Test
