[
https://issues.apache.org/jira/browse/HDDS-4944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17319786#comment-17319786
]
István Fajth commented on HDDS-4944:
------------------------------------
Hi [~ppogde],
thank you for addressing my concerns here, and in the call as well. May I
suggest to have a link for the discussion recoding here as well, if you have it
uploaded somewhere where anyone can access?
To continue the discussion, I think my concerns related to cross-tenancy is
addressed during the discussion, and it is left out from the recording. I
understood that the basis of the cross-tenancy access will be in so called
"bucket policies" so that the authorization part can be done there, and it
seems we will do similarly for simple tenant user accesses as well according to
the plan.
Aggregates are to be delegated to background threads or separate processes that
should not interfere with realtime operations.
This part sounds good.
I also understood that tenant as an abstraction might be necessary, for two
reasons, one is to have a naming convention that resonates with the general
terminology used in the industry, and second is to have something that helps to
manage together all the things that helps us provide the notion of
multi-tenancy.
But I share Marton's concern about lack of discussion of other possibilities,
and the lack of clarity on why we choose to implement it as a layer on top of
what we have, I also think that it would be nice to somehow shed more light on
the pros and cons regarding these fundamental decisions.
I tried to push some suggestions and make a clear point on why I push for
decoupling certain aspects of the problem, but I am not sure if it went
through, so let me re-iterate a few things that I felt dismissed a bit.
As Marton already suggested, at the end of the day if we want to implement
account namespace isolation, we still would need a unique identifier for all
entities that are reaching the cluster. At the moment as we discussed we have
the AWS access key id coupled to a kerberos principal, and we map the two
together via a mapping of UGI to an AWS style access key id. This tight
coupling seems to be a problem, but as I see, the coupling of authentication,
groupping, authorization, and a few other things is our problem here. Hence I
suggested to decouple these, but I am unsure if this was clear for everyone,
and I felt it was not. So I try to rephrase the suggestion.
I think it is beneficial to think about authentication as something that
provides us a mapping from externally defined entities to an internal user
entity with all necessary information related to grouping of these users, and
after authenticating the client we should stick to this internal identifier and
group representation for anything beyond. With that we can rely on our internal
id to authorize, audit, limit (as part of QoS) the user. The way I imagine this
is very similar to what we have now, just decoupled from kerberos and UGI or
anything else. When a request arrives, the role of authentication is to verify
that the user is who them say they are, and when this is verified, we can
assign/attach the internal representation of the user to the request, and
proceed. Rest of the system can rely on the internal representation. For this
we at every request should rely on an authenticator implementation, that does
this verification, and chooses its own way to map the user to our internal
representation. Our internal representation should provide the external user id
(just to help report it in debug/logging), the internal user id, and any
groupping information that identifies the groups/roles/tenants the user belongs
to. With that we can provide all this data to any authorization system which
may or may not support grouping as it is. We can provide this to auditing, and
QoS susbsystems as well. And we can let the authentication implementation to
provide this as an interface implementation, which we can use.
With this, as I see, we can decouple ourselves even from Hadoop UGI, and the
implementation of an authentication scheme can implement the authentication
itself, along with the user mapping as provided in that system.
The default implementation can still rely on Hadoop UGI with kerberos, but we
can easily implement other solutions to any type of authentication scheme, we
can still rely on any authorization, while we can provide audit data, and QoS
logic.
I certainly lack a lot of internal detail about the current system in this
regard, and it is possible that your thoughts are similar, but it is not
visible in the current plans, that is why I would like to understand this part
more and push it a bit more. It is possible that I am ignorant, and this whole
stuff I envision is there or almost there, and you do not want to do anything
else but what I suggest, but still it is unclear for me at this point.
During the cross tenancy access discussion part, I had a concern coming up but
could not get to discuss it online
You said during the meeting that for maybe everyone, but for at least tenant
admins, all users are visible, did I misunderstand or isn't this contradictory
with the requirements? I mean, if tenant admins can see just the tenant
administered by themselves, then seeing all users possibly with a tenant id is
something that goes beyond this limit, as based on the list of users all
tenants become visible. This can be a problem in case where tenants are not
different sub-organizations within the same company, but strictly segregated
companies renting cluster space at some provider, or in case where there is a
tenant which is just hosted for a project, sharing limited information, but
possibly not the list of all users.
Back to parts of your reaction in this comment flow we did not talk about:
{quote}
>> There is one problem, 3/c in the features and limitations section, global
>> bucket namespace... If we have a well separated account
>> namespace isolation that does not get in our way we can use it on top of one
>> volume probably.
I am not sure I understand what is the disconnect here. Can you please
elaborate ?
{quote}
There is no disconnect here, 3/c in current features and limitation section
brings up the case when a user does not want a bucket namespace isolation and
would go with the S3 way, I wanted to express that one volume can be a perfect
fit for these cases, after re-reading 3/c, I think, I probably misunderstood
the sentence, and it says the same. Sorry about the confusion.
{quote}
This proposal is already separating out BucketNamespace and AccountNamespace
requirements. The proposal is already talking about leveraging the volume
abstraction to meet the requirements. I don't see how things that you/Marton
are suggesting are different. Perhaps we are saying the same thing.
{quote}
For me it is still not clear how the Tenant abstraction will manifest itself in
the architecture, probably this is why you feel we are talking about the same,
while we still are not sure. As we discussed, please try to elaborate as much
as possible about this in the PoC work, and we can get back to this after, I
also hope that we (me and Marton) will figure out also that we are on the same
page. I still feel that we need an authentication agnostic internal user
representation which is provided by the authentication plugin used, and
contains at least external user id, internal user id, and group's associated
with the user by the systems we use for authentication. If this, or something
even better is what we will use and we use it with the volume abstraction I may
be more than happy.
{quote}
Regarding the AccountNamespace isolation, It's just a way of grouping users at
a high level and ability to treat these groups as an isolation unit. I think it
should not be mixed/confused with authentication schemes. This proposal does
not limit or extend any existing authentication scheme that we have in Ozone.
{quote}
This is an interesting question, as my whole suggestion and most of my problems
is around this topic... At the moment, we are coupling the authentication
scheme to kerberos via Hadoop UGI even in the S3 gateway by mapping the access
key id to a UGI as we discussed. If this proposal does not limit or extend
this, then we will have a problem later on as I feel, and I don't see how it
will be solved. What I understood so far is that we are using UGI, and the
Hadoop groups mapping internally, I think, if we do not decouple ourselves from
these, and an auth plugin that provide us a unified view of user, and its
groups/roles/tenancy then we will end up patching and workaround UGI internals
to conform with our needs. That's is why I push to separate these from our
internal representation and push it to a pluggable authenticator, that provides
us the external user, based on that our internal user, and also its
groups/roles/tenancy information. I think, this is an important prerequisite
before we can talk about tenancy and that abstraction. Also it is really
unclear, how an external auth system like LDAP will be configured to provide us
the tenant identification for example? Will we have an internal map from user
to tenant? How exactly you are envisioning this? Describing alternatives and
reasons for selecting the actually implemented one, would be good to understand
here as well, based on that I hope to be as optimistic as you are and I hope to
think about this as just a high level grouping of users.
{quote}
I think the user ask here is the ability to isolate users/resources so that
quotas/QOS/usage-consolidation and enforcements can be placed based on these
isolation groups. It's not an immediate ask and this can be built over time.
There could be lazy implementation modules (e.g. recon) or an
inline-implementation module. I think we are on the same page about this being
a pluggable module.
{quote}
This brings us back to the grouping problem, where from these subsystems will
Ozone figure out what groups are there? For this to integrate smoothly we will
need a subsystem/plugin that handles the account namespace, and maps to
internally understood entities as I can think of it know. I would couple this
with authentication as proposed, because that is the point we can and probably
do map the user to tenant/roles/groups. Again I am maybe ignorant, and
something similar is in the making, but I would like to understand what is in
the making, and understand what are the benefits if something else is the plan.
------------
Finally as I went back to check the docs, I realized a few things we might want
to also discuss before getting too deep into the implementation.
One is the thing we mentioned in the call, namely how a user will access a
bucket in its own tenant, which access points will be used? How we will figure
out which resource to serve?
The simple case when a user belongs to one tenant, and accesses a resource in
its tenant I can imagine also in a multitude of ways... but I can not come up
with and have not seen an easy user friendly way to access tenant b's foo
bucket (even if access is granted for me) if I am in a tenant where there is a
foo bucket as well. Are there any ideas on this already?
An other question is whether and how much we want to segregate tenants data, I
guess volumes and buckets are already providing some segregation on the
container/block level, but how we will enforce that access remains within
boundaries when the user reads data from the DNs? Should we have concerns about
accessing other tenants data via DN api's? How we can enforce access on that
level? [~elek] Do you have any idea if we potentially have to think about
exploits here, or in the SCM layer, as we mostly talked about things that fell
into OM territory.
The proposal does not mention, but I guess, if a system is configured with a
certain mix of isolation styles, then reconfiguring it to an other style is not
possible, or is it? (I think of for example going from having just account
namespace isolation to account and bucket namespace isolation, or vice versa
for example)
In a multi-tenant word, why do we want to still have the S3V volume as
something everyone has access to? If admins want to configure it that way, this
seems to be ok, but if we provide that by default, then how users will tell if
they want to use that? Should users be enabled to use that by def? What if we
do not have anything there, or what if we want to migrate some stuff there when
multi-tenancy is turned on?
How we think about sub-groups within a tenant? What if later on we want to spin
off a tenant from one of these sub-groups? How this possibility -if we want to
provide a way - affects our current thinking?
Are there any memorable, but loosely coupled ideas or thoughts that were coming
up during design, that had an effect on certain decisions that were made? (to
add some color and more reasons on the selected way of thinking)
> Multi-Tenant Support in Ozone
> -----------------------------
>
> Key: HDDS-4944
> URL: https://issues.apache.org/jira/browse/HDDS-4944
> Project: Apache Ozone
> Issue Type: New Feature
> Components: Ozone CLI, Ozone Datanode, Ozone Manager, Ozone Recon,
> S3, SCM, Security
> Affects Versions: 1.2.0
> Reporter: Prashant Pogde
> Assignee: Prashant Pogde
> Priority: Major
> Labels: pull-request-available
> Attachments: Apache-S3-compatible-Multi-Tenant-Ozone-short.pdf.gz,
> Ozone MultiTenant Feature _ Requirements and Abstractions-3.pdf, Ozone,
> Multi-tenancy, S3, Kerberos....pdf, UseCaseAWSCompatibility.pdf,
> UseCaseCephCompatibility.pdf, UseCaseConfigureMultiTenancy.png,
> UseCaseCurrentOzoneS3BackwardCompatibility.pdf, VariousActorsInteractions.png
>
>
> This Jira will be used to track a new feature for Multi-Tenant support in
> Ozone. Initially Multi-Tenant feature would be limited to ozone-users
> accessing Ozone over S3 interface.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
