spacemonkd commented on code in PR #10742: URL: https://github.com/apache/ozone/pull/10742#discussion_r3655123066
########## hadoop-hdds/docs/content/design/common-auth-service.md: ########## @@ -0,0 +1,727 @@ +--- +title: Custos — Protocol-Neutral Pluggable Authentication for Ozone +summary: Custos, a shared service that validates client identity for all Ozone access paths on the server side +date: 2026-07-13 +jira: HDDS-15845 +status: proposed +author: Abhishek Pal +--- +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# Custos — Ozone Auth Service Design Doc + +## Table of Contents +1. [Problem Statement](#1-problem-statement) +2. [Scope](#2-scope) +3. [Architecture Overview](#3-architecture-overview) + * [3.1 Request flow](#31-request-flow) + * [3.2 Module structure](#32-module-structure) + * [3.3 Deployment and scaling](#33-deployment-and-scaling) +4. [Background: Existing Authentication Mechanisms](#4-background-existing-authentication-mechanisms) + * [4.1 OzoneFS delegation token path](#41-ozonefs-delegation-token-path) + * [4.2 S3 static-key authentication path](#42-s3-static-key-authentication-path) +5. [Client Use Cases](#5-client-use-cases) + * [5.1 Spark job with many executors (OzoneFS)](#51-spark-job-with-many-executors-ozonefs) + * [5.2 Hive / HiveServer2](#52-hive--hiveserver2) + * [5.3 S3 clients with static keys](#53-s3-clients-with-static-keys) +6. [Custos Provider Interface](#6-custos-provider-interface) +7. [CustosToken and UserInfo](#7-custostoken-and-userinfo) + * [7.1 Token format](#71-token-format) + * [7.2 Token field semantics](#72-token-field-semantics) + * [7.3 Token size](#73-token-size) + * [7.4 UserInfo change and UGI construction](#74-userinfo-change-and-ugi-construction) + * [7.5 OMRequest wire extension](#75-omrequest-wire-extension) +8. [Token Signing and Verification](#8-token-signing-and-verification) + * [8.1 Signing algorithm and key source](#81-signing-algorithm-and-key-source) + * [8.2 Signing input](#82-signing-input) + * [8.3 Verification at OM](#83-verification-at-om) + * [8.4 SCM secret-key integration](#84-scm-secret-key-integration) +9. [Phase 1 — OzoneFS / Hadoop-RPC Clients](#9-phase-1--ozonefs--hadoop-rpc-clients) +10. [Phase 2 — S3 Gateway](#10-phase-2--s3-gateway) +11. [Edge Cases and Security Considerations](#11-edge-cases-and-security-considerations) +12. [Compatibility and Upgrade](#12-compatibility-and-upgrade) +13. [Future Work](#13-future-work) + +--- + +> *Custos* is Latin for "guardian" or "watchman". + +## 1. Problem Statement + +Ozone has several ways for a client to prove who it is. +Each one is written separately, and there is no shared place to plug in a new one. +Three concrete problems follow: + +**(a) Non-Kerberos credentials cannot get a token from OM.** +When security is on, OM only issues a delegation token if the RPC connection used Kerberos or a certificate. +This check lives in `OzoneManager.isAllowedDelegationTokenOp()`: + +```java +private boolean isAllowedDelegationTokenOp() throws IOException { + AuthenticationMethod authMethod = getConnectionAuthenticationMethod(); + return !UserGroupInformation.isSecurityEnabled() + || (authMethod == AuthenticationMethod.KERBEROS) + || (authMethod == AuthenticationMethod.KERBEROS_SSL) + || (authMethod == AuthenticationMethod.CERTIFICATE); +} +``` + +A client that authenticated with an OIDC token (from a cloud identity provider) has no Kerberos ticket. +So it cannot obtain a delegation token and cannot use the OzoneFS path at all. + +**(b) Every credential type is coded on its own, with no shared hook.** +Adding a new credential type today means editing several unrelated places: `OMClientRequest.getUserInfo()`, `OzoneManager.isAllowedDelegationTokenOp()`, and the S3 validation path. +There is no single interface where a new provider can be registered. + +**(c) The identity carried to OM has no group information.** +`OMClientRequest.createUGI()` builds a `UserGroupInformation` from the user name only: + +```java +userGroupInformation = UserGroupInformation.createRemoteUser( + getUserInfo().getUserName()); +``` + +The `UserInfo` protobuf message has no `groups` field. +When an authorization check needs group membership, it must be resolved out of band (for example by Ranger's own group provider). +The token the client presents already knows the groups, but that information is dropped before it reaches OM. + +--- + +## 2. Scope + +Custos is a server-side component that validates client credentials through pluggable providers and issues a signed `CustosToken`. +OM verifies the token locally. +This design covers **authentication only**. +Authorization (policy evaluation) is future work — see [Section 13](#13-future-work). +Custos does **not** replace or skip OM's existing ACL checks; a request verified from a `CustosToken` still runs through the same `checkAcls()` path as any other request. +This keeps authorization exactly where it is today and avoids introducing a second policy decision point in this design. + +Custos is meant to be the single authentication entry point for every Ozone client access path. +This design works two paths through concretely: OzoneFS / Hadoop-RPC clients (Phase 1) and the S3 gateway (Phase 2). +Other access paths follow the same provider model and are not designed here. + +The work is split into two phases. + +**Phase 1 — OzoneFS / Hadoop-RPC clients:** +- Let a driver that authenticated with OIDC (or any configured provider) get a session-scoped `CustosToken` in place of the Kerberos-gated delegation token. +- Keep the executor path unchanged. Executors carry the token in YARN `Credentials` and never contact Custos. +- Keep legacy Kerberos clients working with no change. +- Carry groups inside the token so OM can build a full `UserGroupInformation`. + +**Phase 2 — S3 Gateway:** +- Make Custos the single validation point for S3 credentials, starting with long-lived static keys. +- Route static-key validation through the same provider contract so any other credential type can plug in later without new gateway code. + +**Out of scope:** +- Authorization policy evaluation (Ranger, native ACLs). +- Temporary (session-token) credential validation — see [Section 13](#13-future-work). +- Inter-service authentication (SCM ↔ OM ↔ Datanode). + +--- + +## 3. Architecture Overview + +### 3.1 Request flow + +A client presents a credential to Custos. +Custos validates it through the matching provider and returns a signed `CustosToken`. +The client carries that token to OM on each request. +OM verifies the token locally — no network call — and then runs its existing ACL checks. + +``` +┌────────────┐ ┌───────────────┐ ┌────────────────┐ +│ Client │────▶│ Custos │────▶│ Ozone Manager │ +│ │ │ │ │ │ +│ Presents │ │ 1. Pick │ │ 1. Verify │ +│ credential │ │ provider │ │ token sig │ +│ │ │ 2. Validate │ │ 2. Check │ +│ │ │ credential │ │ audience │ +│ │ │ 3. Issue │ │ + expiry │ +│ │ │ CustosToken│ │ 3. Build UGI │ +│ │ │ │ │ + run ACLs │ +└────────────┘ └───────────────┘ └────────────────┘ +``` + +The token is a bearer of a verified identity, not a proxy for an authorization decision. +OM trusts the *identity* the token asserts (subject and groups) because the signature proves Custos issued it, but OM still evaluates its own ACLs against that identity. + +### 3.2 Module structure + +- **`hadoop-ozone/ozone-custos-common`**: shared library containing the `CustosTokenProto` definition, the `CustosTokenSigner` interface (with the shared `computeSigningInput` used by both signing and verification), and the provider plugin interfaces (`CustosProvider`, `IdentityProvider`, `CustosCredential`, `CustosIdentity`, `CustosException`). Depended on by both Custos and OM. +- **`hadoop-ozone/ozone-custos`**: the standalone Custos service process — the gRPC server, the provider and identity-provider implementations, provider loading, the SCM-backed token signer, token issuance, audit, and metrics. Depends on `hadoop-hdds/framework` for the SCM `SecretKeyClient` (see [Section 8.4](#84-scm-secret-key-integration)). +- **OM** owns its own `CustosTokenVerifier` (in the `ozone-manager` module). OM links only against `ozone-custos-common` for the token type and `computeSigningInput`, and verifies with the SCM `SecretKeyClient` it already holds — so the Custos provider dependencies (for example a JWKS/OIDC client) never enter the OM classpath. + +### 3.3 Deployment and scaling + +Custos is stateless. +It holds no per-client session state: a token is self-describing and is verified from its signature alone. +All signing and verification use the symmetric key that SCM distributes through `SecretKeyClient` / `ManagedSecretKey` ([Section 8.4](#84-scm-secret-key-integration)), so any Custos instance can issue a token that any OM can verify, and OM never has to reach a *specific* Custos instance. + Review Comment: The currently plan is to implement this to target only filesystem clients (whether we merge STS into this or not would be a future decision to make, so that the current STS work isn't affected and folks can go ahead without blocking on other tasks) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
