priyeshkaratha commented on code in PR #10742: URL: https://github.com/apache/ozone/pull/10742#discussion_r3664679814
########## hadoop-hdds/docs/content/design/common-auth-service.md: ########## @@ -0,0 +1,727 @@ +--- +title: Custos — Protocol-Neutral Pluggable Authentication for Ozone +summary: Custos, a shared service that validates client identity for all Ozone access paths on the server side +date: 2026-07-13 +jira: HDDS-15845 +status: proposed +author: Abhishek Pal +--- +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# Custos — Ozone Auth Service Design Doc + +## Table of Contents +1. [Problem Statement](#1-problem-statement) +2. [Scope](#2-scope) +3. [Architecture Overview](#3-architecture-overview) + * [3.1 Request flow](#31-request-flow) + * [3.2 Module structure](#32-module-structure) + * [3.3 Deployment and scaling](#33-deployment-and-scaling) +4. [Background: Existing Authentication Mechanisms](#4-background-existing-authentication-mechanisms) + * [4.1 OzoneFS delegation token path](#41-ozonefs-delegation-token-path) + * [4.2 S3 static-key authentication path](#42-s3-static-key-authentication-path) +5. [Client Use Cases](#5-client-use-cases) + * [5.1 Spark job with many executors (OzoneFS)](#51-spark-job-with-many-executors-ozonefs) + * [5.2 Hive / HiveServer2](#52-hive--hiveserver2) + * [5.3 S3 clients with static keys](#53-s3-clients-with-static-keys) +6. [Custos Provider Interface](#6-custos-provider-interface) +7. [CustosToken and UserInfo](#7-custostoken-and-userinfo) + * [7.1 Token format](#71-token-format) + * [7.2 Token field semantics](#72-token-field-semantics) + * [7.3 Token size](#73-token-size) + * [7.4 UserInfo change and UGI construction](#74-userinfo-change-and-ugi-construction) + * [7.5 OMRequest wire extension](#75-omrequest-wire-extension) +8. [Token Signing and Verification](#8-token-signing-and-verification) + * [8.1 Signing algorithm and key source](#81-signing-algorithm-and-key-source) + * [8.2 Signing input](#82-signing-input) + * [8.3 Verification at OM](#83-verification-at-om) + * [8.4 SCM secret-key integration](#84-scm-secret-key-integration) +9. [Phase 1 — OzoneFS / Hadoop-RPC Clients](#9-phase-1--ozonefs--hadoop-rpc-clients) +10. [Phase 2 — S3 Gateway](#10-phase-2--s3-gateway) +11. [Edge Cases and Security Considerations](#11-edge-cases-and-security-considerations) +12. [Compatibility and Upgrade](#12-compatibility-and-upgrade) +13. [Future Work](#13-future-work) + +--- + +> *Custos* is Latin for "guardian" or "watchman". + +## 1. Problem Statement + +Ozone has several ways for a client to prove who it is. +Each one is written separately, and there is no shared place to plug in a new one. +Three concrete problems follow: + +**(a) Non-Kerberos credentials cannot get a token from OM.** +When security is on, OM only issues a delegation token if the RPC connection used Kerberos or a certificate. +This check lives in `OzoneManager.isAllowedDelegationTokenOp()`: + +```java +private boolean isAllowedDelegationTokenOp() throws IOException { + AuthenticationMethod authMethod = getConnectionAuthenticationMethod(); + return !UserGroupInformation.isSecurityEnabled() + || (authMethod == AuthenticationMethod.KERBEROS) + || (authMethod == AuthenticationMethod.KERBEROS_SSL) + || (authMethod == AuthenticationMethod.CERTIFICATE); +} +``` + +A client that authenticated with an OIDC token (from a cloud identity provider) has no Kerberos ticket. +So it cannot obtain a delegation token and cannot use the OzoneFS path at all. + +**(b) Every credential type is coded on its own, with no shared hook.** +Adding a new credential type today means editing several unrelated places: `OMClientRequest.getUserInfo()`, `OzoneManager.isAllowedDelegationTokenOp()`, and the S3 validation path. +There is no single interface where a new provider can be registered. + +**(c) The identity carried to OM has no group information.** +`OMClientRequest.createUGI()` builds a `UserGroupInformation` from the user name only: + +```java +userGroupInformation = UserGroupInformation.createRemoteUser( + getUserInfo().getUserName()); +``` + +The `UserInfo` protobuf message has no `groups` field. +When an authorization check needs group membership, it must be resolved out of band (for example by Ranger's own group provider). +The token the client presents already knows the groups, but that information is dropped before it reaches OM. + +--- + +## 2. Scope + +Custos is a server-side component that validates client credentials through pluggable providers and issues a signed `CustosToken`. +OM verifies the token locally. +This design covers **authentication only**. +Authorization (policy evaluation) is future work — see [Section 13](#13-future-work). +Custos does **not** replace or skip OM's existing ACL checks; a request verified from a `CustosToken` still runs through the same `checkAcls()` path as any other request. +This keeps authorization exactly where it is today and avoids introducing a second policy decision point in this design. + +Custos is meant to be the single authentication entry point for every Ozone client access path. +This design works two paths through concretely: OzoneFS / Hadoop-RPC clients (Phase 1) and the S3 gateway (Phase 2). +Other access paths follow the same provider model and are not designed here. + +The work is split into two phases. + +**Phase 1 — OzoneFS / Hadoop-RPC clients:** +- Let a driver that authenticated with OIDC (or any configured provider) get a session-scoped `CustosToken` in place of the Kerberos-gated delegation token. +- Keep the executor path unchanged. Executors carry the token in YARN `Credentials` and never contact Custos. +- Keep legacy Kerberos clients working with no change. +- Carry groups inside the token so OM can build a full `UserGroupInformation`. + +**Phase 2 — S3 Gateway:** +- Make Custos the single validation point for S3 credentials, starting with long-lived static keys. +- Route static-key validation through the same provider contract so any other credential type can plug in later without new gateway code. + +**Out of scope:** +- Authorization policy evaluation (Ranger, native ACLs). +- Temporary (session-token) credential validation — see [Section 13](#13-future-work). +- Inter-service authentication (SCM ↔ OM ↔ Datanode). + +--- + +## 3. Architecture Overview + +### 3.1 Request flow + +A client presents a credential to Custos. +Custos validates it through the matching provider and returns a signed `CustosToken`. +The client carries that token to OM on each request. +OM verifies the token locally — no network call — and then runs its existing ACL checks. + +``` Review Comment: Could you create a detailed architecture/design diagram that illustrates the following 1. The client interaction flow with OM, SCM and DN through Custos authentication. 2. How the Recon authenticates while connecting to different services (OM, SCM, DN). 4. The complete request flow, including token generation, validation, service-to-service authentication, and communication between the components. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
