Thank Wiliam & Tzs Wo! Yeah I just realized that it's somehow a "client-side" or "connection-aspect" issue.
The issue of thenApply is the same as blocking IO that we cannot pipeline requests but only send the next read after the pioneer write return from the server. We can resolve this issue if we use Ratis as IoTDB that wraps the RaftServer with a facade and handles server-side connection customizedly. That is, the client pipelines requests, but the server sequences the requests and `submitClientRequest` with thenApply chain. This is like how ZooKeeper's CommitProcessor chains its read/write requests[1]. So here we may have another perspective on the question: what are the expected methods of application based on Ratis? If we want users to write applications with RaftClient in App's client-side and only customized StateMachine on App's server side, even use RaftServer as the server facade, then the current query/applyTransaction, and even the fixed one-shot RPC[2] can be a limitation. If we encourage users to wrap RaftServer with their own server facade and use RaftClient in the App's server-side on demand like IoTDB does, then maybe the experience of submitting requests on the server side can be improved; or we provide some examples to do so. (Generally, it should be fine to always use RaftClient in the App's server-side, while it can cause one extra codec, but we already have to send the request to other peers on writes.) Best, tison. [1] https://github.com/apache/zookeeper/blob/bc0e61854ca362261d899295422ac14dd5d82e59/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/CommitProcessor.java#L251-L259 [2] https://lists.apache.org/thread/rg65qoph54hlpdhmoc3p80sd0z6wzwjm Tsz Wo Sze <[email protected]> 于2023年8月29日周二 02:36写道： > HI tison, William, > > Thanks for pointing out the problem! > > For the long term, we should support Read-After-Write Consistency. It > seems easy to modify the read-index algorithm to support it: use the index > of the previous request as the read index, instead of the commit index. > > As a work around, we may use the async API with thenApply: > - Client 1: async put k1 as v1 and get future f1 > - Client 1: f1.thenApply(reply -> if (reply.isSucces()) {async get k1}). > > Tsz-Wo > > > On Mon, Aug 28, 2023 at 8:28 AM William Song <[email protected]> wrote: > >> Hi tison, >> >> The readIndex provides only the linearizable consistency guarantee. A >> linearizable read returned by RaftServer at least reflects any changes >> already committed and replied to the client. If you require a more strict >> consistency guarantee, like read reflecting the latest write, you may need >> to add extra synchronization logic. I think blocking IO or dependencies on >> completionStage will both work. >> >> Regards, >> William >> >> >> 2023年8月27日 12:20，tison <[email protected]> 写道： >> >> Hi, >> >> I have asked a related question before[1] about syncing better query >> and applytransaction. >> >> Now, when I dive deeper into the Raft-based system, I notice a new case >> for consistency. >> >> Take the following executions as an example (supposed we implement a Map >> statemachine): >> >> Client 1 put k1 as v1 >> Client 1 get k1 >> >> Generally, we expect the get requests gets v1. But if these two requests >> are issued in async, it's not always true. >> >> Even the client TCP connection can guarantee that "get k1" goes after >> "put k1", and we do read index for "get k1", the read index returned can be >> lower then "put k1" if it's not yet committed. Then the "get k1" request >> gets an empty value. >> >> I'm not sure what is the best practices to implement read my write for a >> Ratis-based system. >> >> 1. If we're using blocking IO, it won't be an issue cause "get k1" must >> be after "put k1" committed and even applied. >> 2. If we build an extra facade over Ratis like IoTDB does, we can check >> the read conditions as I did at [2]. >> 3. But what if we only customize the statemachine and use the Ratis >> client? I don't know which happens before relations I can depend on for >> concurrent query and applytransaction. >> >> Also, in [2] for raft-rs there is one more subtle issue that their >> ReadIndex request can get lost or silently ignored, which causes the caller >> to retry that can even get a much later read index. I'm curious if Ratis >> has the same issue (lose ReadIndex or silently ignore and the caller >> doesn't get response forever). >> >> Best, >> tison. >> >> [1] https://lists.apache.org/thread/wh9cf9456y1k3pt9v0qs9o3wychl937s >> [2] >> https://github.com/tikv/raft-rs/discussions/525#discussioncomment-6819686 >> >> >>
