bitflicker64 commented on code in PR #3376: URL: https://github.com/apache/hugegraph/pull/3376#discussion_r4329337404
########## hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/raft/TxnPeerGate.java: ########## @@ -0,0 +1,204 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.pd.raft; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.LinkedHashMap; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.TimeUnit; +import java.util.function.Function; +import java.util.function.LongSupplier; +import java.util.function.Supplier; + +import org.apache.hugegraph.pd.common.PDException; +import org.apache.hugegraph.pd.grpc.Pdpb; + +import lombok.extern.slf4j.Slf4j; + +/** + * Keeps a TXN from being proposed while a peer of the raft group may not apply its opcode. A + * PD from before the TXN entry logs an unknown op and goes on, so a leader that proposed a + * TXN to such a peer would commit an entry the peer never applies, and the peer would serve + * stale metadata once it leads. Every peer of the current configuration (voters and learners, + * not this node) must have answered the member probe with a {@code maxKvOp} at or above + * {@link KVOperation#TXN} once; that answer is remembered for the life of this process (a + * release cannot go down, and a PD downgrade after the first TXN is unsupported). A peer that + * has not been confirmed, or that answered with a lower opcode, blocks every TXN with + * {@code TXN_NOT_PROPOSED} naming it; a confirmed peer that is unreachable later does not + * block. Unconfirmed peers are probed again on a TXN attempt at most once per + * {@code probeIntervalMs}, and at once when the configuration changed, so a peer added to the + * group starts unconfirmed. After a restart everything starts unconfirmed, and the first TXN + * probes every peer once. + * <p> + * A membership change goes through {@link #confirm(Collection)} first: every peer it adds is + * probed at once and an unconfirmed or older one refuses the change. A peer that joins later + * catches up by applying the log, TXN entries included, so it must apply them; checking it at + * join time also closes the window between a TXN's check and its proposal. Peers the change + * keeps are covered by {@link #check()} before every TXN, so they are not probed here. + */ +@Slf4j +public class TxnPeerGate { + + private final Supplier<Collection<String>> peers; + private final Function<String, CompletableFuture<RaftRpcProcessor.GetMemberResponse>> probe; + private final long probeIntervalMs; + private final long probeTimeoutMs; + private final LongSupplier clock; + + private final Set<String> confirmed = ConcurrentHashMap.newKeySet(); Review Comment: Fixed in 171760d25. The leader now records the confirmed endpoints, itself included, through raft: `RaftKVStore` implements `TxnPeerGate.Record` on a raw key, `__TXN_CONFIRMED_PEERS`, proposed as one PUT without waiting, and reads it from its own store. A new leader skips every recorded peer and probes only the rest, so in your case B proposes with A down; a peer no leader confirmed still blocks. `RaftEngine` adds the leader itself to each write, since a leader never probes itself. Tests: `TxnPeerGateTest.testConfirmationsSurviveALeaderChange` (failover with one peer down) and `KvTxnApplierTest.testConfirmedPeersAreRecordedThroughRaft`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
