Ivan Andika created HDDS-17872:
----------------------------------

             Summary: Replace readOnlyAsync API with readIndexAsync
                 Key: HDDS-17872
                 URL: https://issues.apache.org/jira/browse/HDDS-17872
             Project: Apache Ozone
          Issue Type: Improvement
            Reporter: Ivan Andika
            Assignee: Ivan Andika


RATIS-2509 introduced the readOnlyAsync to remove the serde cost by allowing 
the server implementation to directly submit reads to the server without 
needing to construct RaftClientRequest and ser/deserializing the payload.

However, while developing the ReadIndex batching implementation (RATIS-2524) we 
found an issue while benchmarking where the actual reads are serialized instead 
of being executed by the threads calling the reads (e.g. the IPC threads). 

The main issue is that the current readOnlyAsync couple both the ReadIndex 
barrier (i.e. readIndexAsync to send ReadIndex and waitForAdvance to wait for 
the current applied index to reach the ReadIndex) and the actual read 
implementation (i.e. StateMachine#query).

We use the synchronous "thenCompose" callback to couple the two phases. The 
issue is that the read can actually be executed on the thread that calls 
`CompletableFuture.complete` (which is usually a lot lower than the IPC 
threads). Currently, the completion thread will be called on the gRPC workers 
since it's called in StreamObserver.onNext in GrpcService.readIndexAsync. In 
the ReadIndex batching the issue is even worse since the ReadIndex future is 
batched into a single future of the batch and CompletableFuture#complete for 
all these reads is called a single thread. So this can cause multiple reads to 
be called one by one which kills performance (we see that linearizable read 
with ReadIndex batching performs worse than linearizable read without ReadIndex 
batching).

One proposed idea is to have replace the readOnlyAsync with readIndexAsync that 
only does the first barrier phase. The actual reads can be done by the server 
threads itself. It is then the caller responsibility to call readIndexAsync 
before calling the actual read implementation. In Ozone, it will look something 
like

 
{code:java}
division.readIndexAsync(clientId, readRequestType).get();
return stateMachine.query(request).get();{code}
 

Since the readOnlyAsync is not in any Ratis release, I propose to simply 
replace it.

There is a precedent for this, in etcd 
[https://github.com/etcd-io/etcd/blob/v3.7.4/server/etcdserver/read/read.go] , 
the etcd server implementation will call LinearizableReadNotify as a way to 
trigger and wait for the readindex only, and then the subsequent read is done 
in the caller thread itself.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to