This is an automated email from the ASF dual-hosted git repository. elek pushed a commit to branch HDDS-4440 in repository https://gitbox.apache.org/repos/asf/ozone.git
commit af946408c560e1ccbb5a1e6711775eb7e140113b Author: Elek Márton <[email protected]> AuthorDate: Mon Dec 7 16:01:28 2020 +0100 diagram about the old a new approach --- hadoop-hdds/docs/content/design/s3-performance.md | 33 ++++++++++++++++++++- .../docs/static/design/s3-performance-new.png | Bin 0 -> 33688 bytes .../docs/static/design/s3-performance-old.png | Bin 0 -> 34927 bytes 3 files changed, 32 insertions(+), 1 deletion(-) diff --git a/hadoop-hdds/docs/content/design/s3-performance.md b/hadoop-hdds/docs/content/design/s3-performance.md index 8687563..7ce5683 100644 --- a/hadoop-hdds/docs/content/design/s3-performance.md +++ b/hadoop-hdds/docs/content/design/s3-performance.md @@ -184,7 +184,7 @@ public interface OmTransport { ... ``` - The proposal is to create a new **additional** transport, based on GRPC, which can do the per-request authentication. **Existing Hadoop clients will use the well-known Hadoop RPC client**, but S3g can start to use this specific transport to achieve better performance. +The proposal is to create a new **additional** transport, based on GRPC, which can do the per-request authentication. **Existing Hadoop clients will use the well-known Hadoop RPC client**, but S3g can start to use this specific transport to achieve better performance. As this is nothing more, just a transport: exactly the same messages (`OmRequest`) will be used, it's not a new RPC interface. @@ -196,6 +196,37 @@ A new GRPC service should be started in Ozone Manager, which receives `OMRequest With this approach the `OzoneClient` instances can be cached on S3g side (with persistent GRPC connections) as the authentication information is not part of the OzoneClient any more (added by the `OmTransport` implementation per request (in case of GRPC) or per connection (in case of HadoopRPC)). +To make it easier to understand the implementation of this approach, let's compare the old (existing) and new (proposed) approach. + +### Old approach + +1. OzoneClientProvider creates a new OzoneClient for each of the HTTP requests (@RequestScope) + +2. OzoneToken is created based on the authentication header (signature, string to sign) + +3. OzoneClient creates new OM connection (Hadoop RPC) with the new OzoneToken + +4. OM extracts the information from OzoneToken and validates the signature + +5. OzoneClient is injected to the REST endpoints + +6. OzoneClient is used for any client calls + +7. When HTTP request is handled, the OzoneClient is closed + +  + +### New approach + +1. OzoneClientProvider creates client with @ApplicationScope. Connection is always open to the OM (clients can be pooled) +2. OM doesn't authentication the connection, each of the requests should be authenticated one-by-one +3. OzoneClients are always injected to the REST endpoints +4. For each new HTTP request the authorization header is extracted and added to the outgoing GRPC request as a token +5. OM authenticate each of the request and calls the same request handler as before +6. OzoneClient can be open long-term + + + # Possible alternatives * It's possible to use pure Hadoop RPC client instead of Ozone Client which would make the client connection slightly cheaper (service discovery call is not required) but it's still require to create new connections for each requests (and downloading data without OzoneClient may have own challenges). diff --git a/hadoop-hdds/docs/static/design/s3-performance-new.png b/hadoop-hdds/docs/static/design/s3-performance-new.png new file mode 100644 index 0000000..f2cb405 Binary files /dev/null and b/hadoop-hdds/docs/static/design/s3-performance-new.png differ diff --git a/hadoop-hdds/docs/static/design/s3-performance-old.png b/hadoop-hdds/docs/static/design/s3-performance-old.png new file mode 100644 index 0000000..20ebfc2 Binary files /dev/null and b/hadoop-hdds/docs/static/design/s3-performance-old.png differ --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
