errose28 commented on a change in pull request #3095: URL: https://github.com/apache/ozone/pull/3095#discussion_r814250311
########## File path: hadoop-hdds/docs/content/feature/S3-Multi-Tenancy-Setup.md ########## @@ -0,0 +1,107 @@ +--- +title: "Setup" +weight: 1 +menu: + main: + parent: "S3 Multi-Tenancy" +summary: Preparing Ozone clusters to enable Multi-Tenancy feature +--- +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Steps to enable S3 Multi-Tenancy feature in Ozone clusters. + + +## Setting up S3 Multi-Tenancy for production + +### Secure the cluster + +Follow [this guide](< ref "security/SecureOzone.md" >) to Kerberize (secure) the cluster if the cluster is not Kerberized yet. Review comment: I'm getting 404 on this. I think it needs {{ }} around it. ########## File path: hadoop-hdds/docs/content/interface/S3-Tenant.md ########## @@ -0,0 +1,324 @@ +--- +title: "Tenant commands" +menu: + main: + parent: "Command Line Interface" +summary: Ozone subcommands for S3 tenant management +--- +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +For a higher level understanding of multi-tenancy architecture, see [Multi-Tenancy feature]({{< ref "feature/S3-Multi-Tenancy.md" >}}). + +All Multi-Tenancy subcommands are located under CLI `ozone tenant`. + +The commands below assume a Kerberized Ozone cluster with Ranger install. Enabling HTTPS on S3 Gateway is optional but recommended. + +## Quick Start + +### Setup + +Follow the [Multi-Tenancy Setup]({{< ref "feature/S3-Multi-Tenancy-Setup.md" >}}) guide if you haven't done so. + +If the OzoneManagers are running in HA, append `--om-service-id=` accordingly to the commands. + +### Create a tenant + +Create a new tenant in the current Ozone cluster. +This operation requires Ozone cluster administrator privilege. + +Apart from adding new OM DB entries, creating a tenant also does the following in the background: +1. Creates a volume of the exact same name. Therefore, volume name restrictions apply to the tenant name as well. Specifying a custom volume name during tenant creation is not supported yet. Tenant volume cannot be changed once the tenant is created. +2. Creates two new Ranger roles, `tenantName-UserRole` and `tenantName-AdminRole`. +3. Creates new Ranger policies that allows all tenant users to list and create buckets by default under the tenant volume, but only bucket owners and tenant admins are allowed to access the bucket contents. + +```shell +ozone tenant create <TENANT_NAME> +``` + +Example: + +```shell +bash-4.2$ kinit -kt /etc/security/keytabs/om.keytab om/[email protected] +bash-4.2$ ozone tenant create tenantone +2022-02-16 00:00:00,000 [main] INFO rpc.RpcClient: Creating Tenant: 'tenantone', with new volume: 'tenantone' +Created tenant 'tenantone'. +``` + + +### List tenants + +List all tenants in an Ozone cluster. + +```shell +ozone tenant list +``` + +Example: + +```shell +bash-4.2$ ozone tenant list +tenantone +``` + + +### Assign a user to a tenant + +The first user in a tenant must be assigned by an Ozone cluster administrator. + +By default when user `testuser` is assigned to tenant `tenantone`, the generated Access ID for the user in this tenant is `tenantone$testuser`. + +- Be sure to enclose the Access ID in single quotes in Bash when using it so it doesn't get auto-translated into environment variables. + +It is possible to assign a user to multiple tenants. + +```shell +ozone tenant user assign <USER_NAME> --tenant=<TENANT_NAME> +``` + +`<USER_NAME>` should be a short user name for a Kerberos principal, e.g. `testuser` when the Kerberos principal is `testuser/[email protected]` + +Example: + +```shell +bash-4.2$ ozone tenant user assign testuser --tenant=tenantone +Assigned 'testuser' to 'tenantone' with accessId 'tenantone$testuser'. +export AWS_ACCESS_KEY_ID='tenantone$testuser' +export AWS_SECRET_ACCESS_KEY='<GENERATED_SECRET>' +``` + + +### Assign a user as a tenant admin + +The first user in a tenant must be assigned by an Ozone cluster administrator. + +Both delegated and non-delegated tenant admin can assign and revoke **regular** tenant users. + +The only difference between delegated tenant admin and non-delegated tenant admin is that delegated tenant admin can assign and revoke tenant **admins** in the tenant, +while non-delegated tenant admin can't. + +Unless `--delegated=false` is specified, `ozone tenant assignadmin` assigns **delegated** tenant admins by default. + +It is possible to assign a user to be tenant admins in multiple tenants. + +```shell +ozone tenant user assignadmin <ACCESS_ID> --delegated=true --tenant=<TENANT_NAME> +``` + +Example: + +```shell +bash-4.2$ ozone tenant user assignadmin 'tenantone$testuser' --tenant=tenantone +Assigned admin to 'tenantone$testuser' in tenant 'tenantone' +``` + +Once `testuser` becomes a tenant admin of `tenantone`, one can kinit as `testuser` and assign new users to the tenant, +even new tenant admins (if delegated). Example commands for illustration: + +```shell +kinit -kt /etc/security/keytabs/testuser.keytab testuser/[email protected] +ozone tenant user assign testuser2 --tenant=tenantone +ozone tenant user assignadmin 'tenantone$testuser2' --tenant=tenantone +``` + + +### List users in a tenant + +```shell +ozone tenant user list --tenant=<TENANT_NAME> +``` + +Example: + +```shell +bash-4.2$ ozone tenant user list --tenant=tenantone +- User 'testuser' with accessId 'tenantone$testuser' +``` + + +### Get tenant user info + +This command lists all tenants a user is assigned to. + +```shell +ozone tenant user info <USER_NAME> +``` + +Example: + +```shell +bash-4.2$ ozone tenant user info testuser +User 'testuser' is assigned to: +- Tenant 'tenantone' delegated admin with accessId 'tenantone$testuser' +``` + + +### Revoke a tenant admin + +```shell +ozone tenant user revokeadmin <ACCESS_ID> +``` + +Example: + +```shell +bash-4.2$ ozone tenant user revokeadmin 'tenantone$testuser' +Revoked admin role of 'tenantone$testuser'. +``` + + +### Revoke user access from a tenant + +```shell +ozone tenant user revoke <ACCESS_ID> +``` + +Example: + +```shell +bash-4.2$ ozone tenant user revoke 'tenantone$testuser' +Revoked accessId 'tenantone$testuser'. +``` + + +### Delete a tenant + +In order to be able to delete a tenant, the tenant has to be empty. i.e. All users need to be revoked before a tenant can be deleted. +Otherwise OM will throw `TENANT_NOT_EMPTY` exception and refuse to delete the tenant. + +Note that it is intentional by design that the volume created and associated with the tenant during tenant creation is not removed. +An admin has to remove the volume manually as prompt in the CLI, if deemed necessary. + +```shell +ozone tenant delete <TENANT_NAME> +``` + +Example: + +```shell +bash-4.2$ ozone tenant delete tenantone +Deleted tenant 'tenantone'. +But the associated volume 'tenantone' is not removed. To delete it, run + ozone sh volume delete tenantone +``` + +If an Ozone cluster admin (or whoever has the permission to delete the volume in Ranger) tries delete a volume before the tenant is deleted using the command above, +the `ozone sh volume delete` command would fail because the volume reference count is not zero: + +```shell +bash-4.2$ ozone sh volume delete tenantone +VOLUME_IS_REFERENCED Volume reference count is not zero (1). Ozone features are enabled on this volume. Try `ozone tenant delete <tenantId>` first. +``` + + +## Bonus: Accessing a bucket in a tenant volume via S3 Gateway using S3 API + Review comment: Might be good to clarify here that the example here is still operating on tenantone with testuser. ########## File path: hadoop-hdds/docs/content/feature/S3-Multi-Tenancy-Setup.md ########## @@ -0,0 +1,107 @@ +--- +title: "Setup" +weight: 1 +menu: + main: + parent: "S3 Multi-Tenancy" +summary: Preparing Ozone clusters to enable Multi-Tenancy feature +--- +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Steps to enable S3 Multi-Tenancy feature in Ozone clusters. + + +## Setting up S3 Multi-Tenancy for production + +### Secure the cluster + +Follow [this guide](< ref "security/SecureOzone.md" >) to Kerberize (secure) the cluster if the cluster is not Kerberized yet. + +### Set up S3 Gateway + +Follow [this guide](< ref "interface/S3.md" >) the cluster to set up at least one S3 Gateway if the cluster doesn't have a S3 Gateway yet. Review comment: Same issue as above. ########## File path: hadoop-hdds/docs/content/feature/S3-Multi-Tenancy.md ########## @@ -0,0 +1,94 @@ +--- +title: "S3 Multi-Tenancy" +menu: + main: + parent: Features +summary: Ozone Multi-Tenancy that allows multiple tenants to share the same Ozone cluster. Compatible with S3 API. +--- +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Before Ozone multi-tenancy, all S3 access to Ozone (via [S3 Gateway]({{< ref "interface/S3.md" >}})) are +confined to a **single** designated S3 volume (that is volume `s3v`, by default). + +Ozone multi-tenancy allows **multiple** S3-accessible volumes to be created. +Each volume can be managed separately by their own tenant admins via CLI for user operations, and via Apache Ranger for access control. + +The concept **tenant** is introduced to Ozone by multi-tenancy. +Each tenant has its own designated volume. +Each user assigned to a tenant will be able to access the associated volume with an _Access ID & Secret Key_ pair +generated when an Ozone cluster admin or tenant admin assigns the user to the tenant using CLI. + +> This multi-tenant feature allows Ozone resources to be compartmentalized in different isolation zones (tenants). + +> This multi-tenant support will also allow users to access Ozone volumes over AWS S3 APIs (without any modifications to the APIs). + +## Basics + +1. Initial tenant creation has to be done by an Ozone cluster admin under the CLI. +2. The Ozone cluster admin will have to assign the first user of a tenant. Once assigned, an _Access ID & Secret Key pair_ (key pair) will be generated for that user for access via S3 Gateway. + - The key pair serves to authenticate the end user to the Ozone Manager (via client requests from S3 Gateway). Tenant volume is selected based on the Access ID. + - After successful authentication, Ozone Manager uses the underlying user name (not the Access ID) to identify the user. The user name is used to perform authorization checks in Apache Ranger. + - A key pair is tied to the user name it is assigned to. If a user is assigned key pairs in multiple tenants, all key pairs point to the same user name internally in Ozone Manager. + - A user can be only assigned one key pair in a same tenant. Ozone Manager rejects the tenant user assign request if a user is already assigned to the same tenant (i.e. when the user has already been assigned an Access ID in this tenant). + - Key pairs assigned to a user for different tenants are used to access buckets under different tenants. For instance, `testuser` uses `tenantone$testuser` to access `tenantone` buckets, and uses `tenanttwo$testuser` to access `tenanttwo` buckets via S3 Gateway. Review comment: I think this might be easier to understand here: ```suggestion - One user can be assigned to multiple tenants. The user will have a different key pair to access each tenant. For instance, `testuser` could use `tenantone$testuser` to access `tenantone` buckets, and use `tenanttwo$testuser` to access `tenanttwo` buckets via S3 Gateway. ``` ########## File path: hadoop-hdds/docs/content/feature/S3-Multi-Tenancy.md ########## @@ -0,0 +1,94 @@ +--- +title: "S3 Multi-Tenancy" +menu: + main: + parent: Features +summary: Ozone Multi-Tenancy that allows multiple tenants to share the same Ozone cluster. Compatible with S3 API. +--- +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Before Ozone multi-tenancy, all S3 access to Ozone (via [S3 Gateway]({{< ref "interface/S3.md" >}})) are +confined to a **single** designated S3 volume (that is volume `s3v`, by default). + +Ozone multi-tenancy allows **multiple** S3-accessible volumes to be created. +Each volume can be managed separately by their own tenant admins via CLI for user operations, and via Apache Ranger for access control. + +The concept **tenant** is introduced to Ozone by multi-tenancy. +Each tenant has its own designated volume. +Each user assigned to a tenant will be able to access the associated volume with an _Access ID & Secret Key_ pair +generated when an Ozone cluster admin or tenant admin assigns the user to the tenant using CLI. + +> This multi-tenant feature allows Ozone resources to be compartmentalized in different isolation zones (tenants). + +> This multi-tenant support will also allow users to access Ozone volumes over AWS S3 APIs (without any modifications to the APIs). + +## Basics + +1. Initial tenant creation has to be done by an Ozone cluster admin under the CLI. +2. The Ozone cluster admin will have to assign the first user of a tenant. Once assigned, an _Access ID & Secret Key pair_ (key pair) will be generated for that user for access via S3 Gateway. + - The key pair serves to authenticate the end user to the Ozone Manager (via client requests from S3 Gateway). Tenant volume is selected based on the Access ID. + - After successful authentication, Ozone Manager uses the underlying user name (not the Access ID) to identify the user. The user name is used to perform authorization checks in Apache Ranger. + - A key pair is tied to the user name it is assigned to. If a user is assigned key pairs in multiple tenants, all key pairs point to the same user name internally in Ozone Manager. + - A user can be only assigned one key pair in a same tenant. Ozone Manager rejects the tenant user assign request if a user is already assigned to the same tenant (i.e. when the user has already been assigned an Access ID in this tenant). + - Key pairs assigned to a user for different tenants are used to access buckets under different tenants. For instance, `testuser` uses `tenantone$testuser` to access `tenantone` buckets, and uses `tenanttwo$testuser` to access `tenanttwo` buckets via S3 Gateway. + - A bucket link can be set up if cross-tenant (cross-volume) access is desired. Review comment: We should probably add the tenant bucket link command to the tenant CLI docs, and then link that here. ########## File path: hadoop-hdds/docs/content/interface/S3-Tenant.md ########## @@ -0,0 +1,324 @@ +--- +title: "Tenant commands" +menu: + main: + parent: "Command Line Interface" Review comment: I still think this would fit better as a subsection under the multi-tenancy header. I think it disrupts the flow having the docs split under two separate headers. Also, pretty much every feature adds new CLIs but they put them with their feature, not in this section, so I think we should do the same. ########## File path: hadoop-hdds/docs/content/feature/S3-Multi-Tenancy.md ########## @@ -0,0 +1,94 @@ +--- +title: "S3 Multi-Tenancy" +menu: + main: + parent: Features +summary: Ozone Multi-Tenancy that allows multiple tenants to share the same Ozone cluster. Compatible with S3 API. +--- +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Before Ozone multi-tenancy, all S3 access to Ozone (via [S3 Gateway]({{< ref "interface/S3.md" >}})) are +confined to a **single** designated S3 volume (that is volume `s3v`, by default). + +Ozone multi-tenancy allows **multiple** S3-accessible volumes to be created. +Each volume can be managed separately by their own tenant admins via CLI for user operations, and via Apache Ranger for access control. + +The concept **tenant** is introduced to Ozone by multi-tenancy. +Each tenant has its own designated volume. +Each user assigned to a tenant will be able to access the associated volume with an _Access ID & Secret Key_ pair +generated when an Ozone cluster admin or tenant admin assigns the user to the tenant using CLI. + +> This multi-tenant feature allows Ozone resources to be compartmentalized in different isolation zones (tenants). + +> This multi-tenant support will also allow users to access Ozone volumes over AWS S3 APIs (without any modifications to the APIs). + +## Basics + +1. Initial tenant creation has to be done by an Ozone cluster admin under the CLI. +2. The Ozone cluster admin will have to assign the first user of a tenant. Once assigned, an _Access ID & Secret Key pair_ (key pair) will be generated for that user for access via S3 Gateway. + - The key pair serves to authenticate the end user to the Ozone Manager (via client requests from S3 Gateway). Tenant volume is selected based on the Access ID. + - After successful authentication, Ozone Manager uses the underlying user name (not the Access ID) to identify the user. The user name is used to perform authorization checks in Apache Ranger. + - A key pair is tied to the user name it is assigned to. If a user is assigned key pairs in multiple tenants, all key pairs point to the same user name internally in Ozone Manager. + - A user can be only assigned one key pair in a same tenant. Ozone Manager rejects the tenant user assign request if a user is already assigned to the same tenant (i.e. when the user has already been assigned an Access ID in this tenant). + - Key pairs assigned to a user for different tenants are used to access buckets under different tenants. For instance, `testuser` uses `tenantone$testuser` to access `tenantone` buckets, and uses `tenanttwo$testuser` to access `tenanttwo` buckets via S3 Gateway. + - A bucket link can be set up if cross-tenant (cross-volume) access is desired. +3. The Ozone cluster admin can then assign tenant admin roles to that user. +4. Tenant admin are able to assign new users to the tenant. + - They can even assign new tenant admins in their tenant, if they are delegated tenant admins, which is the default. See the usage below for more details. Review comment: Can probably just link to the CLI docs here instead of referring to the usage section that then refers to the CLI docs. If possible a link that focuses the section on assigning admin would be good, but not I'm not sure if that is supported. ########## File path: hadoop-hdds/docs/content/feature/S3-Multi-Tenancy.md ########## @@ -0,0 +1,94 @@ +--- +title: "S3 Multi-Tenancy" +menu: + main: + parent: Features +summary: Ozone Multi-Tenancy that allows multiple tenants to share the same Ozone cluster. Compatible with S3 API. +--- +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Before Ozone multi-tenancy, all S3 access to Ozone (via [S3 Gateway]({{< ref "interface/S3.md" >}})) are +confined to a **single** designated S3 volume (that is volume `s3v`, by default). + +Ozone multi-tenancy allows **multiple** S3-accessible volumes to be created. +Each volume can be managed separately by their own tenant admins via CLI for user operations, and via Apache Ranger for access control. + +The concept **tenant** is introduced to Ozone by multi-tenancy. +Each tenant has its own designated volume. +Each user assigned to a tenant will be able to access the associated volume with an _Access ID & Secret Key_ pair +generated when an Ozone cluster admin or tenant admin assigns the user to the tenant using CLI. + +> This multi-tenant feature allows Ozone resources to be compartmentalized in different isolation zones (tenants). + +> This multi-tenant support will also allow users to access Ozone volumes over AWS S3 APIs (without any modifications to the APIs). + +## Basics + +1. Initial tenant creation has to be done by an Ozone cluster admin under the CLI. +2. The Ozone cluster admin will have to assign the first user of a tenant. Once assigned, an _Access ID & Secret Key pair_ (key pair) will be generated for that user for access via S3 Gateway. + - The key pair serves to authenticate the end user to the Ozone Manager (via client requests from S3 Gateway). Tenant volume is selected based on the Access ID. + - After successful authentication, Ozone Manager uses the underlying user name (not the Access ID) to identify the user. The user name is used to perform authorization checks in Apache Ranger. + - A key pair is tied to the user name it is assigned to. If a user is assigned key pairs in multiple tenants, all key pairs point to the same user name internally in Ozone Manager. + - A user can be only assigned one key pair in a same tenant. Ozone Manager rejects the tenant user assign request if a user is already assigned to the same tenant (i.e. when the user has already been assigned an Access ID in this tenant). + - Key pairs assigned to a user for different tenants are used to access buckets under different tenants. For instance, `testuser` uses `tenantone$testuser` to access `tenantone` buckets, and uses `tenanttwo$testuser` to access `tenanttwo` buckets via S3 Gateway. + - A bucket link can be set up if cross-tenant (cross-volume) access is desired. +3. The Ozone cluster admin can then assign tenant admin roles to that user. +4. Tenant admin are able to assign new users to the tenant. + - They can even assign new tenant admins in their tenant, if they are delegated tenant admins, which is the default. See the usage below for more details. + - Note that tenant admins still need to use Ozone tenant CLI to assign new users to the tenant. + - Once tenant admins get the Kerberos TGT (via `kinit`), they can run `user assign` command to assign new users. Ozone Manager will recognize that they are the tenant admins and allow the user to do so in their tenants. +5. After that, users can use any S3-compatible client (awscli, Python boto3 library, etc.) to access the buckets in the tenant volume via S3 Gateway using the generated key pairs. + + +## Access Control + +Ozone multi-tenancy relies on [Apache Ranger]({{< ref "security/SecurityWithRanger.md" >}}) to enforce access control to resources. + +By default, a group of Ranger policies are created when a tenant is created on the tenant volume: + +1. All users are able to create new buckets; +2. Only the bucket owner (i.e. the user that creates the bucket) and tenant admins can access the bucket content. Review comment: What if the bucket owner or admin wants to allow more users to access the bucket? We discussed this offline with @prashantpogde and it would need to be handled via Ranger, but we need to make sure the background sync doesn't overwrite the change. Once we have a solution it should be documented here. ########## File path: hadoop-hdds/docs/content/feature/S3-Multi-Tenancy.md ########## @@ -0,0 +1,94 @@ +--- +title: "S3 Multi-Tenancy" +menu: + main: + parent: Features +summary: Ozone Multi-Tenancy that allows multiple tenants to share the same Ozone cluster. Compatible with S3 API. +--- +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Before Ozone multi-tenancy, all S3 access to Ozone (via [S3 Gateway]({{< ref "interface/S3.md" >}})) are +confined to a **single** designated S3 volume (that is volume `s3v`, by default). + +Ozone multi-tenancy allows **multiple** S3-accessible volumes to be created. +Each volume can be managed separately by their own tenant admins via CLI for user operations, and via Apache Ranger for access control. + +The concept **tenant** is introduced to Ozone by multi-tenancy. +Each tenant has its own designated volume. +Each user assigned to a tenant will be able to access the associated volume with an _Access ID & Secret Key_ pair +generated when an Ozone cluster admin or tenant admin assigns the user to the tenant using CLI. + +> This multi-tenant feature allows Ozone resources to be compartmentalized in different isolation zones (tenants). + +> This multi-tenant support will also allow users to access Ozone volumes over AWS S3 APIs (without any modifications to the APIs). + +## Basics + +1. Initial tenant creation has to be done by an Ozone cluster admin under the CLI. +2. The Ozone cluster admin will have to assign the first user of a tenant. Once assigned, an _Access ID & Secret Key pair_ (key pair) will be generated for that user for access via S3 Gateway. + - The key pair serves to authenticate the end user to the Ozone Manager (via client requests from S3 Gateway). Tenant volume is selected based on the Access ID. + - After successful authentication, Ozone Manager uses the underlying user name (not the Access ID) to identify the user. The user name is used to perform authorization checks in Apache Ranger. + - A key pair is tied to the user name it is assigned to. If a user is assigned key pairs in multiple tenants, all key pairs point to the same user name internally in Ozone Manager. + - A user can be only assigned one key pair in a same tenant. Ozone Manager rejects the tenant user assign request if a user is already assigned to the same tenant (i.e. when the user has already been assigned an Access ID in this tenant). + - Key pairs assigned to a user for different tenants are used to access buckets under different tenants. For instance, `testuser` uses `tenantone$testuser` to access `tenantone` buckets, and uses `tenanttwo$testuser` to access `tenanttwo` buckets via S3 Gateway. + - A bucket link can be set up if cross-tenant (cross-volume) access is desired. +3. The Ozone cluster admin can then assign tenant admin roles to that user. +4. Tenant admin are able to assign new users to the tenant. + - They can even assign new tenant admins in their tenant, if they are delegated tenant admins, which is the default. See the usage below for more details. + - Note that tenant admins still need to use Ozone tenant CLI to assign new users to the tenant. + - Once tenant admins get the Kerberos TGT (via `kinit`), they can run `user assign` command to assign new users. Ozone Manager will recognize that they are the tenant admins and allow the user to do so in their tenants. +5. After that, users can use any S3-compatible client (awscli, Python boto3 library, etc.) to access the buckets in the tenant volume via S3 Gateway using the generated key pairs. + + +## Access Control + +Ozone multi-tenancy relies on [Apache Ranger]({{< ref "security/SecurityWithRanger.md" >}}) to enforce access control to resources. + +By default, a group of Ranger policies are created when a tenant is created on the tenant volume: Review comment: ```suggestion When a tenant is created, Ozone will create a set of policies on the tenant's volume which allow the following: ``` ########## File path: hadoop-hdds/docs/content/feature/S3-Multi-Tenancy.md ########## @@ -0,0 +1,94 @@ +--- +title: "S3 Multi-Tenancy" +menu: + main: + parent: Features +summary: Ozone Multi-Tenancy that allows multiple tenants to share the same Ozone cluster. Compatible with S3 API. +--- +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Before Ozone multi-tenancy, all S3 access to Ozone (via [S3 Gateway]({{< ref "interface/S3.md" >}})) are +confined to a **single** designated S3 volume (that is volume `s3v`, by default). + +Ozone multi-tenancy allows **multiple** S3-accessible volumes to be created. +Each volume can be managed separately by their own tenant admins via CLI for user operations, and via Apache Ranger for access control. + +The concept **tenant** is introduced to Ozone by multi-tenancy. +Each tenant has its own designated volume. +Each user assigned to a tenant will be able to access the associated volume with an _Access ID & Secret Key_ pair +generated when an Ozone cluster admin or tenant admin assigns the user to the tenant using CLI. + +> This multi-tenant feature allows Ozone resources to be compartmentalized in different isolation zones (tenants). + +> This multi-tenant support will also allow users to access Ozone volumes over AWS S3 APIs (without any modifications to the APIs). + +## Basics + +1. Initial tenant creation has to be done by an Ozone cluster admin under the CLI. +2. The Ozone cluster admin will have to assign the first user of a tenant. Once assigned, an _Access ID & Secret Key pair_ (key pair) will be generated for that user for access via S3 Gateway. + - The key pair serves to authenticate the end user to the Ozone Manager (via client requests from S3 Gateway). Tenant volume is selected based on the Access ID. + - After successful authentication, Ozone Manager uses the underlying user name (not the Access ID) to identify the user. The user name is used to perform authorization checks in Apache Ranger. + - A key pair is tied to the user name it is assigned to. If a user is assigned key pairs in multiple tenants, all key pairs point to the same user name internally in Ozone Manager. + - A user can be only assigned one key pair in a same tenant. Ozone Manager rejects the tenant user assign request if a user is already assigned to the same tenant (i.e. when the user has already been assigned an Access ID in this tenant). + - Key pairs assigned to a user for different tenants are used to access buckets under different tenants. For instance, `testuser` uses `tenantone$testuser` to access `tenantone` buckets, and uses `tenanttwo$testuser` to access `tenanttwo` buckets via S3 Gateway. + - A bucket link can be set up if cross-tenant (cross-volume) access is desired. +3. The Ozone cluster admin can then assign tenant admin roles to that user. +4. Tenant admin are able to assign new users to the tenant. + - They can even assign new tenant admins in their tenant, if they are delegated tenant admins, which is the default. See the usage below for more details. + - Note that tenant admins still need to use Ozone tenant CLI to assign new users to the tenant. + - Once tenant admins get the Kerberos TGT (via `kinit`), they can run `user assign` command to assign new users. Ozone Manager will recognize that they are the tenant admins and allow the user to do so in their tenants. +5. After that, users can use any S3-compatible client (awscli, Python boto3 library, etc.) to access the buckets in the tenant volume via S3 Gateway using the generated key pairs. + + +## Access Control Review comment: I think the ranger docs should be moved to their own multi-tenancy subsection/page. We can include this section there, but also missing information right now like which parts of the ranger policies/roles can you edit without the background sync overwriting them, and explaining that access control may not immediately take effect. This is one of the most complicated parts of the multi-tenancy feature so I think it warrants its own section. ########## File path: hadoop-hdds/docs/content/interface/S3-Tenant.md ########## @@ -0,0 +1,324 @@ +--- +title: "Tenant commands" +menu: + main: + parent: "Command Line Interface" +summary: Ozone subcommands for S3 tenant management +--- +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +For a higher level understanding of multi-tenancy architecture, see [Multi-Tenancy feature]({{< ref "feature/S3-Multi-Tenancy.md" >}}). + +All Multi-Tenancy subcommands are located under CLI `ozone tenant`. + +The commands below assume a Kerberized Ozone cluster with Ranger install. Enabling HTTPS on S3 Gateway is optional but recommended. + +## Quick Start + +### Setup + +Follow the [Multi-Tenancy Setup]({{< ref "feature/S3-Multi-Tenancy-Setup.md" >}}) guide if you haven't done so. + +If the OzoneManagers are running in HA, append `--om-service-id=` accordingly to the commands. + +### Create a tenant + +Create a new tenant in the current Ozone cluster. +This operation requires Ozone cluster administrator privilege. + +Apart from adding new OM DB entries, creating a tenant also does the following in the background: +1. Creates a volume of the exact same name. Therefore, volume name restrictions apply to the tenant name as well. Specifying a custom volume name during tenant creation is not supported yet. Tenant volume cannot be changed once the tenant is created. +2. Creates two new Ranger roles, `tenantName-UserRole` and `tenantName-AdminRole`. +3. Creates new Ranger policies that allows all tenant users to list and create buckets by default under the tenant volume, but only bucket owners and tenant admins are allowed to access the bucket contents. + +```shell +ozone tenant create <TENANT_NAME> +``` + +Example: + +```shell +bash-4.2$ kinit -kt /etc/security/keytabs/om.keytab om/[email protected] +bash-4.2$ ozone tenant create tenantone +2022-02-16 00:00:00,000 [main] INFO rpc.RpcClient: Creating Tenant: 'tenantone', with new volume: 'tenantone' +Created tenant 'tenantone'. +``` + + +### List tenants + +List all tenants in an Ozone cluster. + +```shell +ozone tenant list +``` + +Example: + +```shell +bash-4.2$ ozone tenant list +tenantone +``` + + +### Assign a user to a tenant + +The first user in a tenant must be assigned by an Ozone cluster administrator. + +By default when user `testuser` is assigned to tenant `tenantone`, the generated Access ID for the user in this tenant is `tenantone$testuser`. + +- Be sure to enclose the Access ID in single quotes in Bash when using it so it doesn't get auto-translated into environment variables. + +It is possible to assign a user to multiple tenants. + +```shell +ozone tenant user assign <USER_NAME> --tenant=<TENANT_NAME> +``` + +`<USER_NAME>` should be a short user name for a Kerberos principal, e.g. `testuser` when the Kerberos principal is `testuser/[email protected]` + +Example: + +```shell +bash-4.2$ ozone tenant user assign testuser --tenant=tenantone +Assigned 'testuser' to 'tenantone' with accessId 'tenantone$testuser'. +export AWS_ACCESS_KEY_ID='tenantone$testuser' +export AWS_SECRET_ACCESS_KEY='<GENERATED_SECRET>' +``` + + +### Assign a user as a tenant admin + +The first user in a tenant must be assigned by an Ozone cluster administrator. + +Both delegated and non-delegated tenant admin can assign and revoke **regular** tenant users. + +The only difference between delegated tenant admin and non-delegated tenant admin is that delegated tenant admin can assign and revoke tenant **admins** in the tenant, +while non-delegated tenant admin can't. + +Unless `--delegated=false` is specified, `ozone tenant assignadmin` assigns **delegated** tenant admins by default. + +It is possible to assign a user to be tenant admins in multiple tenants. + +```shell +ozone tenant user assignadmin <ACCESS_ID> --delegated=true --tenant=<TENANT_NAME> +``` + +Example: + +```shell +bash-4.2$ ozone tenant user assignadmin 'tenantone$testuser' --tenant=tenantone +Assigned admin to 'tenantone$testuser' in tenant 'tenantone' +``` + +Once `testuser` becomes a tenant admin of `tenantone`, one can kinit as `testuser` and assign new users to the tenant, +even new tenant admins (if delegated). Example commands for illustration: + +```shell +kinit -kt /etc/security/keytabs/testuser.keytab testuser/[email protected] +ozone tenant user assign testuser2 --tenant=tenantone +ozone tenant user assignadmin 'tenantone$testuser2' --tenant=tenantone +``` + + +### List users in a tenant + +```shell +ozone tenant user list --tenant=<TENANT_NAME> +``` + +Example: + +```shell +bash-4.2$ ozone tenant user list --tenant=tenantone +- User 'testuser' with accessId 'tenantone$testuser' +``` + + +### Get tenant user info + +This command lists all tenants a user is assigned to. + +```shell +ozone tenant user info <USER_NAME> +``` + +Example: + +```shell +bash-4.2$ ozone tenant user info testuser +User 'testuser' is assigned to: +- Tenant 'tenantone' delegated admin with accessId 'tenantone$testuser' +``` + + +### Revoke a tenant admin + +```shell +ozone tenant user revokeadmin <ACCESS_ID> +``` + +Example: + +```shell +bash-4.2$ ozone tenant user revokeadmin 'tenantone$testuser' +Revoked admin role of 'tenantone$testuser'. +``` + + +### Revoke user access from a tenant + +```shell +ozone tenant user revoke <ACCESS_ID> +``` + +Example: + +```shell +bash-4.2$ ozone tenant user revoke 'tenantone$testuser' +Revoked accessId 'tenantone$testuser'. +``` + + +### Delete a tenant + +In order to be able to delete a tenant, the tenant has to be empty. i.e. All users need to be revoked before a tenant can be deleted. +Otherwise OM will throw `TENANT_NOT_EMPTY` exception and refuse to delete the tenant. + +Note that it is intentional by design that the volume created and associated with the tenant during tenant creation is not removed. +An admin has to remove the volume manually as prompt in the CLI, if deemed necessary. + +```shell +ozone tenant delete <TENANT_NAME> +``` + +Example: + +```shell +bash-4.2$ ozone tenant delete tenantone +Deleted tenant 'tenantone'. +But the associated volume 'tenantone' is not removed. To delete it, run + ozone sh volume delete tenantone +``` + +If an Ozone cluster admin (or whoever has the permission to delete the volume in Ranger) tries delete a volume before the tenant is deleted using the command above, +the `ozone sh volume delete` command would fail because the volume reference count is not zero: + +```shell +bash-4.2$ ozone sh volume delete tenantone +VOLUME_IS_REFERENCED Volume reference count is not zero (1). Ozone features are enabled on this volume. Try `ozone tenant delete <tenantId>` first. +``` + + +## Bonus: Accessing a bucket in a tenant volume via S3 Gateway using S3 API + +### Configure AWS CLI + +```shell +bash-4.2$ aws configure +AWS Access Key ID [****************fslf]: tenantone$testuser +AWS Secret Access Key [****************fslf]: <GENERATED_SECRET> +Default region name [us-west-1]: +Default output format [None]: +``` + +### List buckets, create a bucket + +```shell +bash-4.2$ aws s3api --endpoint-url http://s3g:9878 list-buckets +{ + "Buckets": [] +} +bash-4.2$ aws s3api --endpoint-url http://s3g:9878 create-bucket --bucket bucket-test1 +{ + "Location": "http://s3g:9878/bucket-test1"; +} +bash-4.2$ aws s3api --endpoint-url http://s3g:9878 list-buckets +{ + "Buckets": [ + { + "Name": "bucket-test1", + "CreationDate": "2022-02-16T00:05:00.000Z" + } + ] +} +``` + +If aws cli reports `AccessDenied`, check if the user associated with the Access ID has the permission to access the volume. Review comment: Why would the tenancy operations not handle this automatically? Will this only happen in docker with the mock server? If so I think we should clarify that in the docker section of the setup page. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
