sergey-chugunov-1985 commented on code in PR #12740: URL: https://github.com/apache/ignite/pull/12740#discussion_r3666233169
########## modules/ducktests/tests/ignitetest/tests/mdc/transactional_partition_test.py: ########## @@ -0,0 +1,147 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +MDC transactional load through a cross-DC network partition. + +A TRANSACTIONAL cache spans both data centers: with backups=1 and the +MdcAffinityBackupFilter every partition owns exactly one copy per DC, so every +explicit-transaction write (a plain put on a transactional cache) must reach a +node in the other DC. A continuous single-threaded insert load runs from the main +DC; the instant the DCs are partitioned the next commit cannot reach all partition +copies and fails with a cache exception. The load cuts itself off on that first +exception and records how many inserts had succeeded. + +After the split settles the test asserts that the cluster really split-brained, +that the load stopped because of the partition (not because it ran out of work), +and - the point of the scenario - that the aborted explicit transactions left +nothing hanging on either half-ring and no suspicious entries in the server logs. + +Data accessibility during the split is deliberately NOT checked: a transactional +cache needs all partition copies available, which a split-brained half-ring cannot +offer. +""" +from time import sleep + +from ducktape.mark import parametrize + +from ignitetest.services.mdc.mdc_cluster import MdcCluster, cross_dc_network, DC_1, DC_2 +from ignitetest.utils import cluster, ignite_versions +from ignitetest.utils.ignite_test import IgniteTest +from ignitetest.utils.version import DEV_BRANCH + +CACHE_NAME = "mdc-tx-load" + +BACKUPS = 1 + +# Fresh, disjoint key range for the continuous insert load: the load advances the key +# on every success, so [LOAD_KEY_FROM, LOAD_KEY_FROM + successfulInserts) gets inserted. +LOAD_KEY_FROM_DC_1 = 1_000_000 +LOAD_KEY_TO_DC_1 = 10_000_000 + +LOAD_KEY_FROM_DC_2 = 10_000_000 +LOAD_KEY_TO_DC_2 = 20_000_000 + +# Let the load accumulate successful inserts before the DCs are cut apart. +LOAD_WARMUP_SECS = 15 + +# Time for discovery to detect the partition and for both half-rings to complete PME +# and account for the split. +SPLIT_SETTLE_SECS = 15 + + +class MdcTransactionalPartitionTest(IgniteTest): + """ + Transactional load resilience to a cross-DC network partition. + """ + @cluster(num_nodes=7) + @ignite_versions(str(DEV_BRANCH)) + @parametrize(cross_dc_latency_ms=100) + def test_transactional_load_cut_on_partition(self, ignite_version, cross_dc_latency_ms): + """ + Continuous explicit-transaction insert load from the main DC is cut off by the first + cache exception the cross-DC partition triggers; afterwards no transaction is left + hanging on either half-ring and the server logs are clean. + """ + mdc = MdcCluster(self, ignite_version, srv_per_dc=2, runners_per_dc=1, loaders_per_dc={DC_2: 1}, + network_timeout=20_000, tcp_connect_timeout=10_000) + + with cross_dc_network(self.logger, mdc, delay_ms=cross_dc_latency_ms) as net: + mdc.start_servers() + + # Continuous single-threaded explicit-transaction insert load from the backup DC. Review Comment: This comment contradicts a high-level comment on the test. The high-level comment says that the load runs in the main DC while from what I see in the code the actual runner is started in the backup DC. ########## modules/ducktests/tests/ignitetest/tests/mdc/transactional_partition_test.py: ########## @@ -0,0 +1,147 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +MDC transactional load through a cross-DC network partition. + +A TRANSACTIONAL cache spans both data centers: with backups=1 and the +MdcAffinityBackupFilter every partition owns exactly one copy per DC, so every +explicit-transaction write (a plain put on a transactional cache) must reach a +node in the other DC. A continuous single-threaded insert load runs from the main +DC; the instant the DCs are partitioned the next commit cannot reach all partition +copies and fails with a cache exception. The load cuts itself off on that first +exception and records how many inserts had succeeded. + +After the split settles the test asserts that the cluster really split-brained, +that the load stopped because of the partition (not because it ran out of work), +and - the point of the scenario - that the aborted explicit transactions left +nothing hanging on either half-ring and no suspicious entries in the server logs. + +Data accessibility during the split is deliberately NOT checked: a transactional +cache needs all partition copies available, which a split-brained half-ring cannot Review Comment: I suspect this statement isn't true: a transactional cache can handle data modification requests if all nodes which backup copies are assigned to are declared FAILED by the discovery component. So I think we could add a data accessibility check and even a check that data is possible to modify in this scenario as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
