Hi, We are currently setting up and executing a testbench to analyze and validate the resilience and robustness of the SEPATH hypervisor running across the three nodes of a cluster (SEAPATH - LF Energy Wiki<https://lf-energy.atlassian.net/wiki/spaces/SEAP/overview>). As part of these tests, we evaluated the behavior of the Ceph service in a simulated failure scenario. Using tc and netem, we introduced approximately 40% random packet loss on the cluster network of one node, which resulted in quorum instability.
The main observations were as follows: * Continuous increase in election_epoch, indicating frequent monitor re-elections. * Intermittent quorum formation, with transitions between (1,2), (1,3), (2,3), and (1,2,3). * Repeated changes in quorum_leader_name, alternating mainly between node 1 and node 2. * A tendency for hypervisor1 (rank 0) to regain leadership whenever it restored connectivity with a majority of nodes. At the time of the test, we found that election_strategy was configured with value 1. Based on our understanding, this behavior is consistent with the classic election mechanism, where all monitors are treated equally and the monitor with the lowest rank is typically selected as leader. To further analyze this behavior, we compared the two monitor election strategies under the same induced failure condition. In both cases, we captured TCP traffic on port 3300 over a 10-minute period using tshark. With the classic strategy, repeated tests showed between 68 and 74 leader changes, continuous quorum leave-and-rejoin events for the affected node, and approximately double the traffic compared to the lossless scenario (636Kbits/s versus 340Kbits/s). By contrast, with the connectivity-based strategy, repeated tests showed only 1 to 4 leader changes, leadership limited to node 2 and node 3, no rejoin of the affected node to the quorum, and only an approximately 10% increase in TCP traffic compared to the lossless case (437 Kbits/s versus 400 Kbits/s) >From a traffic perspective, this approach consumes less bandwidth and >significantly improves system stability during cluster network failures, since >node 1 is no longer repeatedly re-elected as leader while experiencing network >issues. Based on the measurements we have collected and our comparison of the election algorithms, strategy 3 appears to be the most suitable option for our use case. We would therefore appreciate understanding whether there is any additional consideration or technical reason for preferring strategy 1 over strategy 3, which may explain why it is the default configuration. Thanks in advance. Best regards, Christian Narváez A. _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
