fapifta commented on a change in pull request #2979: URL: https://github.com/apache/ozone/pull/2979#discussion_r790890624
########## File path: hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/README.gRPC.md ########## @@ -0,0 +1,253 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# The gRPC client internals +This document is about the current gRPC client internals. It aims to give enough +context about gRPC and the Ratis client with which it shares its interface to +understand the current behaviour, and implementation better. +Besides this, at the end one can find information about possibilities that gRPC +can provide, and why we are not using it at the moment. + + +## Responsibilities of client code using the client +There are two types of requests in the client, sync and async. These requests +are considered to be synced from the caller's point of view, and they just block +the thread in which the caller sends the request. +If the client code is using the same client from multiple threads, the requests +are sent in via multiple message streams on the same channel. + +### When client code uses synced calls +There is no additional responsibilities on the client code about single +requests, the requests will be sent and the thread is blocked until the response +arrives. +However, if there are multiple dependent calls that are sent from multiple +threads, the client code has to externally synchronize the dependent calls in +between its threads. + +### When client code uses async calls +There are two async calls `WriteChunk` and `PutBlock` that can be sent via +the `ContainerProtocolCalls` though nothing prevents a client code to send +other type of commands in an asynchronous fashion by calling the gRPC client's +sendCommandAsync directly. +When using async calls, the client code has to ensure ordering between requests +if they need to be ordered. +For example `PutBlock` requires all related `WriteChunk` to finish successfully +before it is called, in order to have consistent data and metadata stored on the +DataNode in all the cases. The gRPC client itself does not do any measures to +guarantee such an ordering. More details about the reasons will be shared later +in this doc. In a nutshell, it brings in performance and complexity penalties +with no real gains. + +Retries are not handled nor provided for async calls, for these calls, the +client code has to implement the retry logic on its own, as that may involve +re-sending multiple requests, and may affect request ordering as well. + + +## How clients can be created +The `XCeiverClientManager` class is managing creation of the client instances, +as defined in the `XCeiverClientFactory` interface. +A user of this class can acquire a client with the help of acquire methods, and +then release it when it is not using it anymore. + +It is important to note that the lifecycle of the clients are up to the user, +and if a client is not released it may leak, as the clients are reference +counted and evicted from the cache if they are not references for more than +`scm.container.client.idle.threshold` milliseconds ago. The default idle +threshold is 10 seconds. + +The client is getting connected to the DataNode upon creation, and at every +request it checks if the connection is still up and ready. In case the +connection was terminated for any reason, the client simply tries to reconnect +to the DataNode. + + +## Client to server communication in a nutshell +In gRPC there are 3 things that is part of the communication, stub, channel, and +messages. + +Stub provides tha server side API on the client side. +Channel manages the communication properties, and the socket level connection. +Messages in our case are sent via bi-directional streaming with the help of +StreamObservers. + +When we initiate a connection we create a stub, and a channel, and when we send +a request (a message) we create the bi-directional stream, we send in a +request proto message, and we get back a response proto message, then we close +the stream. +We never send multiple requests via the same stream, although we can if we want +to, there are reasons that will be discussed with more details later on. + +The server side of communication processes the requests in the same thread in +which they arrive, via `HDDSDispatcher` in `GrpcXceiverService`, and this also +means that the requests are processed in more threads as the gRPC server side +provides multi-threaded processing per streams. + + +## Client to server communication in more detail + +### How requests are handled on client and server side +The gRPC client is also known as the standalone client, is used only for +reads in the current environment, but during the Erasure Coded storage +development we already selected the standalone client for writes. That effort +initiated the analysis of the client side, and this writeup. +The plan is to continue to use the gRPC client only for reads, and for EC writes +in the future, if that changes, then this document might as well need an update. Review comment: In #3012 I have referred the file from the class' API doc, and also I removed the sentence starting as "The plan is..." this plan is documented in the JIRA tasks, and will be visible as the code evolves, when changes to be made, hopefully one does not forget to update this doc, if it is necessary. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
