hubcio commented on code in PR #3985:
URL: https://github.com/apache/iggy/pull/3985#discussion_r3879790347
##########
core/server/src/partition_helpers.rs:
##########
@@ -613,7 +619,18 @@ pub async fn build_partition_fresh(
durable_view: recovered_state
.as_ref()
.map(|state| (state.view, state.log_view)),
- view_fallback: None,
+ // Both planes pick their primary as `view % replica_count` from
+ // their OWN view counter. A group left at view 0 while the
+ // metadata plane sits elsewhere therefore names a different node
+ // than the roster advertises as leader, and nothing routes a
+ // partition write across that gap: the client is sent to the
+ // metadata leader and refused there for the whole budget. Seeding
+ // from the metadata view keeps the two congruent for a group born
+ // after a metadata election. Peers materialise the same namespace
+ // from the same committed event and read the same published view,
+ // so they agree; a seed that races an election lands one replica a
+ // view behind, which its `StartView` catch-up already covers.
+ view_fallback: view_seed,
Review Comment:
`view_fallback` sets view but not `log_view`, so this primary runs at view V
with `log_view` 0 and fails the `log_view != view` guard in
`handle_request_start_view` - it answers no probes. restarts then elect to V+1,
undoing the alignment.
##########
core/server/src/partition_helpers.rs:
##########
@@ -613,7 +619,18 @@ pub async fn build_partition_fresh(
durable_view: recovered_state
.as_ref()
.map(|state| (state.view, state.log_view)),
- view_fallback: None,
+ // Both planes pick their primary as `view % replica_count` from
+ // their OWN view counter. A group left at view 0 while the
+ // metadata plane sits elsewhere therefore names a different node
+ // than the roster advertises as leader, and nothing routes a
+ // partition write across that gap: the client is sent to the
+ // metadata leader and refused there for the whole budget. Seeding
+ // from the metadata view keeps the two congruent for a group born
+ // after a metadata election. Peers materialise the same namespace
+ // from the same committed event and read the same published view,
+ // so they agree; a seed that races an election lands one replica a
+ // view behind, which its `StartView` catch-up already covers.
+ view_fallback: view_seed,
incarnation: None,
join,
},
Review Comment:
a seeded group now needs a superblock write at creation, since view !=
view_durable trips the persist gate - and that gate withholds every view-scoped
send until it lands. fresh partitions never wrote one before.
##########
core/integration/tests/cluster/partition_primary_routing.rs:
##########
@@ -0,0 +1,201 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! The node a client is told is "the leader" must be the node that accepts a
+//! partition write.
+//!
+//! `get_cluster_metadata` marks a node `Leader` from the METADATA plane's
+//! `primary_index` alone, while a partition write is only accepted by the
+//! primary of that partition's OWN consensus group. Both planes pick their
+//! primary as `view % replica_count`, but their views are independent
+//! counters, so the two answers agree only while the views are congruent mod
+//! the replica count. Every other 3-node test happens to run with both planes
+//! at view 0, where node 0 is leader and partition primary at once, so none of
+//! them can see the split.
+//!
+//! This test forces the views apart: it moves the metadata plane off view 0,
+//! brings every node back, then creates a topic whose partition group is brand
+//! new and therefore still at view 0. The two assertions are deliberately
+//! separate, because they distinguish the two candidate defects:
+//!
+//! - Node 0 accepts the write. A partition primary EXISTS; "partition groups
+//! never elect" is refuted.
+//! - The advertised leader accepts the write. If this fails while the first
+//! passes, the defect is purely that the roster advertises the metadata
+//! leader as the destination for partition traffic.
+
+use std::str::FromStr;
+use std::time::Duration;
+
+use iggy::prelude::*;
+use integration::harness::TestHarness;
+use integration::iggy_harness;
+use tokio::time::sleep;
+
+const STREAM_NAME: &str = "partition-routing-stream";
+const TOPIC_NAME: &str = "partition-routing-topic";
+const PARTITION_ID: u32 = 0;
+
+/// Long enough for the backups to miss `cluster.heartbeat_timeout` (5s by
+/// default) and conclude an election.
+const ELECTION_SETTLE: Duration = Duration::from_secs(15);
+/// Long enough for the restarted node 0 to rejoin at the new view.
+const REJOIN_SETTLE: Duration = Duration::from_secs(10);
+/// Well past a healthy send and past the SDK's own transient replay window, so
+/// exceeding it means the client is not going to recover on its own.
+const SEND_BUDGET: Duration = Duration::from_secs(45);
Review Comment:
`send_raw` is already capped at 30s by `RESPONSE_READ_TIMEOUT`, so 45s never
fires and there is no unbounded send to guard against. drop it or set it under
30s.
##########
core/integration/tests/cluster/partition_primary_routing.rs:
##########
@@ -0,0 +1,201 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! The node a client is told is "the leader" must be the node that accepts a
+//! partition write.
+//!
+//! `get_cluster_metadata` marks a node `Leader` from the METADATA plane's
+//! `primary_index` alone, while a partition write is only accepted by the
+//! primary of that partition's OWN consensus group. Both planes pick their
+//! primary as `view % replica_count`, but their views are independent
+//! counters, so the two answers agree only while the views are congruent mod
+//! the replica count. Every other 3-node test happens to run with both planes
+//! at view 0, where node 0 is leader and partition primary at once, so none of
+//! them can see the split.
+//!
+//! This test forces the views apart: it moves the metadata plane off view 0,
+//! brings every node back, then creates a topic whose partition group is brand
+//! new and therefore still at view 0. The two assertions are deliberately
+//! separate, because they distinguish the two candidate defects:
+//!
+//! - Node 0 accepts the write. A partition primary EXISTS; "partition groups
+//! never elect" is refuted.
+//! - The advertised leader accepts the write. If this fails while the first
+//! passes, the defect is purely that the roster advertises the metadata
+//! leader as the destination for partition traffic.
+
+use std::str::FromStr;
+use std::time::Duration;
+
+use iggy::prelude::*;
+use integration::harness::TestHarness;
+use integration::iggy_harness;
+use tokio::time::sleep;
+
+const STREAM_NAME: &str = "partition-routing-stream";
+const TOPIC_NAME: &str = "partition-routing-topic";
+const PARTITION_ID: u32 = 0;
+
+/// Long enough for the backups to miss `cluster.heartbeat_timeout` (5s by
+/// default) and conclude an election.
+const ELECTION_SETTLE: Duration = Duration::from_secs(15);
+/// Long enough for the restarted node 0 to rejoin at the new view.
+const REJOIN_SETTLE: Duration = Duration::from_secs(10);
+/// Well past a healthy send and past the SDK's own transient replay window, so
+/// exceeding it means the client is not going to recover on its own.
+const SEND_BUDGET: Duration = Duration::from_secs(45);
+
+fn message(payload: &str) -> IggyMessage {
+ IggyMessage::from_str(payload).expect("build message")
+}
+
+/// The roster's current leader as a node index, read through an already
+/// connected client so no new connection is attempted while the cluster may
+/// still be settling.
+async fn read_leader_index(harness: &TestHarness, client: &IggyClient) ->
Option<usize> {
+ let metadata = client.get_cluster_metadata().await.ok()?;
+ let leader_port = metadata
+ .nodes
+ .iter()
+ .find(|node| node.role == ClusterNodeRole::Leader)?
+ .endpoints
+ .tcp;
+ (0..harness.cluster_size()).find(|index| {
+ harness
+ .node(*index)
+ .tcp_addr()
+ .is_some_and(|address| address.port() == leader_port)
+ })
+}
+
+#[iggy_harness(cluster_nodes = 3)]
+async fn
given_metadata_view_moved_when_producing_to_a_fresh_topic_should_reach_the_advertised_leader(
+ harness: &mut TestHarness,
+) {
+ // Kill node 0: it is the view-0 primary of BOTH planes, so the metadata
+ // plane must elect someone else. Nothing has been written yet, so no
+ // partition group exists to move with it. Fixed waits rather than polling:
+ // dialing a leaderless cluster blocks for the SDK's own budget, and a poll
+ // loop that opens a fresh connection each round never converges.
+ harness.kill_node(0).expect("kill node 0");
+ sleep(ELECTION_SETTLE).await;
+ harness.restart_node(0).expect("restart node 0");
+ sleep(REJOIN_SETTLE).await;
+
+ let probe = harness
+ .root_client_for_node(1)
+ .await
+ .expect("root client on node 1 after the election");
+ let leader = read_leader_index(harness, &probe)
+ .await
+ .expect("the cluster must name a leader once the election settled");
+ assert_ne!(
+ leader, 0,
+ "killing node 0 must have moved the metadata plane off view 0; \
+ with the leader back at node 0 the two planes agree and the split
cannot show"
+ );
+
+ // A brand-new topic: its partition consensus group starts at view 0, so
+ // its primary is replica 0, while the metadata leader is not node 0.
+ let setup = harness
+ .root_client_for_node(leader)
+ .await
+ .expect("root client on the metadata leader");
+ setup
+ .create_stream(STREAM_NAME)
+ .await
+ .expect("create stream");
+ let stream_id = Identifier::named(STREAM_NAME).expect("stream identifier");
+ setup
+ .create_topic(
+ &stream_id,
+ TOPIC_NAME,
+ &TopicCreateOptions {
+ partitions_count: Some(1),
+ message_expiry: Some(IggyExpiry::NeverExpire),
+ ..TopicCreateOptions::default()
+ },
+ )
+ .await
+ .expect("create topic");
+ let topic_id = Identifier::named(TOPIC_NAME).expect("topic identifier");
+ let partitioning = Partitioning::partition_id(PARTITION_ID);
+
+ // Where a client dialing node 0 actually ends up. If the SDK's leader
+ // check moves it to the metadata leader, that redirect is itself the
+ // defect: it walks the client off the only node that can accept the write.
+ let node_zero_address = harness
+ .node(0)
+ .tcp_addr()
+ .expect("node 0 exposes a TCP endpoint")
+ .to_string();
+ let on_node_zero = harness
+ .root_client_for_node(0)
+ .await
+ .expect("root client on node 0");
+ let landed_on = on_node_zero.get_connection_info().await.server_address;
Review Comment:
`landed_on` only gets printed. `assert_eq!(landed_on, node_zero_address)` is
the guard that would catch the redirect.
##########
core/integration/tests/cluster/partition_primary_routing.rs:
##########
@@ -0,0 +1,201 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! The node a client is told is "the leader" must be the node that accepts a
+//! partition write.
+//!
+//! `get_cluster_metadata` marks a node `Leader` from the METADATA plane's
+//! `primary_index` alone, while a partition write is only accepted by the
+//! primary of that partition's OWN consensus group. Both planes pick their
+//! primary as `view % replica_count`, but their views are independent
+//! counters, so the two answers agree only while the views are congruent mod
+//! the replica count. Every other 3-node test happens to run with both planes
+//! at view 0, where node 0 is leader and partition primary at once, so none of
+//! them can see the split.
+//!
+//! This test forces the views apart: it moves the metadata plane off view 0,
+//! brings every node back, then creates a topic whose partition group is brand
+//! new and therefore still at view 0. The two assertions are deliberately
+//! separate, because they distinguish the two candidate defects:
+//!
+//! - Node 0 accepts the write. A partition primary EXISTS; "partition groups
+//! never elect" is refuted.
+//! - The advertised leader accepts the write. If this fails while the first
+//! passes, the defect is purely that the roster advertises the metadata
+//! leader as the destination for partition traffic.
+
+use std::str::FromStr;
+use std::time::Duration;
+
+use iggy::prelude::*;
+use integration::harness::TestHarness;
+use integration::iggy_harness;
+use tokio::time::sleep;
+
+const STREAM_NAME: &str = "partition-routing-stream";
+const TOPIC_NAME: &str = "partition-routing-topic";
+const PARTITION_ID: u32 = 0;
+
+/// Long enough for the backups to miss `cluster.heartbeat_timeout` (5s by
+/// default) and conclude an election.
+const ELECTION_SETTLE: Duration = Duration::from_secs(15);
+/// Long enough for the restarted node 0 to rejoin at the new view.
+const REJOIN_SETTLE: Duration = Duration::from_secs(10);
+/// Well past a healthy send and past the SDK's own transient replay window, so
+/// exceeding it means the client is not going to recover on its own.
+const SEND_BUDGET: Duration = Duration::from_secs(45);
+
+fn message(payload: &str) -> IggyMessage {
+ IggyMessage::from_str(payload).expect("build message")
+}
+
+/// The roster's current leader as a node index, read through an already
+/// connected client so no new connection is attempted while the cluster may
+/// still be settling.
+async fn read_leader_index(harness: &TestHarness, client: &IggyClient) ->
Option<usize> {
+ let metadata = client.get_cluster_metadata().await.ok()?;
+ let leader_port = metadata
+ .nodes
+ .iter()
+ .find(|node| node.role == ClusterNodeRole::Leader)?
+ .endpoints
+ .tcp;
+ (0..harness.cluster_size()).find(|index| {
+ harness
+ .node(*index)
+ .tcp_addr()
+ .is_some_and(|address| address.port() == leader_port)
+ })
+}
+
+#[iggy_harness(cluster_nodes = 3)]
+async fn
given_metadata_view_moved_when_producing_to_a_fresh_topic_should_reach_the_advertised_leader(
+ harness: &mut TestHarness,
+) {
+ // Kill node 0: it is the view-0 primary of BOTH planes, so the metadata
+ // plane must elect someone else. Nothing has been written yet, so no
+ // partition group exists to move with it. Fixed waits rather than polling:
+ // dialing a leaderless cluster blocks for the SDK's own budget, and a poll
+ // loop that opens a fresh connection each round never converges.
+ harness.kill_node(0).expect("kill node 0");
+ sleep(ELECTION_SETTLE).await;
+ harness.restart_node(0).expect("restart node 0");
+ sleep(REJOIN_SETTLE).await;
+
+ let probe = harness
+ .root_client_for_node(1)
+ .await
+ .expect("root client on node 1 after the election");
+ let leader = read_leader_index(harness, &probe)
+ .await
+ .expect("the cluster must name a leader once the election settled");
+ assert_ne!(
Review Comment:
this is a setup precondition, not an invariant - `primary_index` has no
`Status::Normal` gate, so a node mid view-change at view 3 advertises node 0
again and this fails a healthy cluster.
##########
core/server/src/partition_reconciler.rs:
##########
@@ -252,12 +260,23 @@ impl ReconcilerCtx {
cluster_id,
self_replica_id,
replica_count,
+ metadata_view,
failure_state: RefCell::new(AHashMap::new()),
last_revision: Cell::new(None),
last_pass_noop: Cell::new(false),
}
}
+ /// The view a partition group materialised now should start in, or `None`
+ /// while the metadata plane has published no view yet.
+ fn partition_view_seed(&self) -> Option<u32> {
+ let view = self.metadata_view.load(Ordering::Relaxed);
+ if view == METADATA_VIEW_UNKNOWN {
+ return None;
+ }
+ u32::try_from(view).ok()
Review Comment:
`.ok()` would silently map an out-of-range view back to the view-0 start
this change removes. the sentinel is handled above, so make this an `expect` or
a `debug_assert`.
##########
core/server/src/partition_helpers.rs:
##########
@@ -613,7 +619,18 @@ pub async fn build_partition_fresh(
durable_view: recovered_state
.as_ref()
.map(|state| (state.view, state.log_view)),
- view_fallback: None,
+ // Both planes pick their primary as `view % replica_count` from
+ // their OWN view counter. A group left at view 0 while the
+ // metadata plane sits elsewhere therefore names a different node
+ // than the roster advertises as leader, and nothing routes a
+ // partition write across that gap: the client is sent to the
+ // metadata leader and refused there for the whole budget. Seeding
+ // from the metadata view keeps the two congruent for a group born
+ // after a metadata election. Peers materialise the same namespace
+ // from the same committed event and read the same published view,
Review Comment:
not true - each node publishes its own metadata view on a 100ms poll, and a
restarted node publishes `METADATA_VIEW_UNKNOWN` while it has ceded. a create
right after an election seeds different views on different nodes.
##########
core/server/src/partition_helpers.rs:
##########
@@ -534,6 +539,7 @@ pub async fn build_partition_fresh(
cluster_id: u128,
self_replica_id: u8,
replica_count: u8,
+ view_seed: Option<u32>,
Review Comment:
the seed also lands on the `ProbeAsBackup` branch when there's no superblock
record. a probing replica then sits above the group's real view and drops the
primary's `StartView`. only pass it for `JoinMode::Init`.
##########
core/integration/tests/cluster/partition_primary_routing.rs:
##########
@@ -0,0 +1,201 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! The node a client is told is "the leader" must be the node that accepts a
+//! partition write.
+//!
+//! `get_cluster_metadata` marks a node `Leader` from the METADATA plane's
+//! `primary_index` alone, while a partition write is only accepted by the
+//! primary of that partition's OWN consensus group. Both planes pick their
+//! primary as `view % replica_count`, but their views are independent
+//! counters, so the two answers agree only while the views are congruent mod
+//! the replica count. Every other 3-node test happens to run with both planes
+//! at view 0, where node 0 is leader and partition primary at once, so none of
+//! them can see the split.
+//!
+//! This test forces the views apart: it moves the metadata plane off view 0,
+//! brings every node back, then creates a topic whose partition group is brand
+//! new and therefore still at view 0. The two assertions are deliberately
+//! separate, because they distinguish the two candidate defects:
+//!
+//! - Node 0 accepts the write. A partition primary EXISTS; "partition groups
+//! never elect" is refuted.
+//! - The advertised leader accepts the write. If this fails while the first
+//! passes, the defect is purely that the roster advertises the metadata
+//! leader as the destination for partition traffic.
+
+use std::str::FromStr;
+use std::time::Duration;
+
+use iggy::prelude::*;
+use integration::harness::TestHarness;
+use integration::iggy_harness;
+use tokio::time::sleep;
+
+const STREAM_NAME: &str = "partition-routing-stream";
+const TOPIC_NAME: &str = "partition-routing-topic";
+const PARTITION_ID: u32 = 0;
+
+/// Long enough for the backups to miss `cluster.heartbeat_timeout` (5s by
+/// default) and conclude an election.
+const ELECTION_SETTLE: Duration = Duration::from_secs(15);
+/// Long enough for the restarted node 0 to rejoin at the new view.
+const REJOIN_SETTLE: Duration = Duration::from_secs(10);
+/// Well past a healthy send and past the SDK's own transient replay window, so
+/// exceeding it means the client is not going to recover on its own.
+const SEND_BUDGET: Duration = Duration::from_secs(45);
+
+fn message(payload: &str) -> IggyMessage {
+ IggyMessage::from_str(payload).expect("build message")
+}
+
+/// The roster's current leader as a node index, read through an already
+/// connected client so no new connection is attempted while the cluster may
+/// still be settling.
+async fn read_leader_index(harness: &TestHarness, client: &IggyClient) ->
Option<usize> {
+ let metadata = client.get_cluster_metadata().await.ok()?;
+ let leader_port = metadata
+ .nodes
+ .iter()
+ .find(|node| node.role == ClusterNodeRole::Leader)?
+ .endpoints
+ .tcp;
+ (0..harness.cluster_size()).find(|index| {
+ harness
+ .node(*index)
+ .tcp_addr()
+ .is_some_and(|address| address.port() == leader_port)
+ })
+}
+
+#[iggy_harness(cluster_nodes = 3)]
+async fn
given_metadata_view_moved_when_producing_to_a_fresh_topic_should_reach_the_advertised_leader(
+ harness: &mut TestHarness,
+) {
+ // Kill node 0: it is the view-0 primary of BOTH planes, so the metadata
+ // plane must elect someone else. Nothing has been written yet, so no
+ // partition group exists to move with it. Fixed waits rather than polling:
+ // dialing a leaderless cluster blocks for the SDK's own budget, and a poll
+ // loop that opens a fresh connection each round never converges.
+ harness.kill_node(0).expect("kill node 0");
+ sleep(ELECTION_SETTLE).await;
+ harness.restart_node(0).expect("restart node 0");
+ sleep(REJOIN_SETTLE).await;
+
+ let probe = harness
+ .root_client_for_node(1)
+ .await
+ .expect("root client on node 1 after the election");
+ let leader = read_leader_index(harness, &probe)
+ .await
+ .expect("the cluster must name a leader once the election settled");
+ assert_ne!(
+ leader, 0,
+ "killing node 0 must have moved the metadata plane off view 0; \
+ with the leader back at node 0 the two planes agree and the split
cannot show"
+ );
+
+ // A brand-new topic: its partition consensus group starts at view 0, so
+ // its primary is replica 0, while the metadata leader is not node 0.
+ let setup = harness
+ .root_client_for_node(leader)
+ .await
+ .expect("root client on the metadata leader");
+ setup
+ .create_stream(STREAM_NAME)
+ .await
+ .expect("create stream");
+ let stream_id = Identifier::named(STREAM_NAME).expect("stream identifier");
+ setup
+ .create_topic(
+ &stream_id,
+ TOPIC_NAME,
+ &TopicCreateOptions {
+ partitions_count: Some(1),
+ message_expiry: Some(IggyExpiry::NeverExpire),
+ ..TopicCreateOptions::default()
+ },
+ )
+ .await
+ .expect("create topic");
+ let topic_id = Identifier::named(TOPIC_NAME).expect("topic identifier");
+ let partitioning = Partitioning::partition_id(PARTITION_ID);
+
+ // Where a client dialing node 0 actually ends up. If the SDK's leader
+ // check moves it to the metadata leader, that redirect is itself the
+ // defect: it walks the client off the only node that can accept the write.
+ let node_zero_address = harness
+ .node(0)
+ .tcp_addr()
+ .expect("node 0 exposes a TCP endpoint")
+ .to_string();
+ let on_node_zero = harness
Review Comment:
`root_client_for_node` logs in, and `login_user` follows the leader redirect
- so this client is on `leader`, not node 0. both assertions hit the same
connection and neither tests node 0. use a raw `TcpClient` with no login.
##########
core/integration/tests/cluster/partition_primary_routing.rs:
##########
@@ -0,0 +1,201 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! The node a client is told is "the leader" must be the node that accepts a
+//! partition write.
+//!
+//! `get_cluster_metadata` marks a node `Leader` from the METADATA plane's
+//! `primary_index` alone, while a partition write is only accepted by the
+//! primary of that partition's OWN consensus group. Both planes pick their
+//! primary as `view % replica_count`, but their views are independent
+//! counters, so the two answers agree only while the views are congruent mod
+//! the replica count. Every other 3-node test happens to run with both planes
+//! at view 0, where node 0 is leader and partition primary at once, so none of
+//! them can see the split.
+//!
+//! This test forces the views apart: it moves the metadata plane off view 0,
+//! brings every node back, then creates a topic whose partition group is brand
+//! new and therefore still at view 0. The two assertions are deliberately
+//! separate, because they distinguish the two candidate defects:
+//!
+//! - Node 0 accepts the write. A partition primary EXISTS; "partition groups
+//! never elect" is refuted.
+//! - The advertised leader accepts the write. If this fails while the first
+//! passes, the defect is purely that the roster advertises the metadata
+//! leader as the destination for partition traffic.
+
+use std::str::FromStr;
+use std::time::Duration;
+
+use iggy::prelude::*;
+use integration::harness::TestHarness;
+use integration::iggy_harness;
+use tokio::time::sleep;
+
+const STREAM_NAME: &str = "partition-routing-stream";
+const TOPIC_NAME: &str = "partition-routing-topic";
+const PARTITION_ID: u32 = 0;
+
+/// Long enough for the backups to miss `cluster.heartbeat_timeout` (5s by
+/// default) and conclude an election.
+const ELECTION_SETTLE: Duration = Duration::from_secs(15);
+/// Long enough for the restarted node 0 to rejoin at the new view.
+const REJOIN_SETTLE: Duration = Duration::from_secs(10);
+/// Well past a healthy send and past the SDK's own transient replay window, so
+/// exceeding it means the client is not going to recover on its own.
+const SEND_BUDGET: Duration = Duration::from_secs(45);
+
+fn message(payload: &str) -> IggyMessage {
+ IggyMessage::from_str(payload).expect("build message")
+}
+
+/// The roster's current leader as a node index, read through an already
+/// connected client so no new connection is attempted while the cluster may
+/// still be settling.
+async fn read_leader_index(harness: &TestHarness, client: &IggyClient) ->
Option<usize> {
Review Comment:
this is `disk::leader_node_index` with a different client and an `Option`
return. add a node-index parameter there instead of forking it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
