I'm don't agree. At least not completely ;)
Pluggable authorization (or hierarchies with the ability to define ACLs
at each level) would allow you to isolate different groups of users.
However each user will still be "aware" of one another's presence in the
topic namespace - and could also be a "noisy neighbor". For example:
say I configure Kafka with an authorization plugin that only allows
access to topics that start with the user's security principal. That
partitions up the topic namespace so that one user can't clash with
another - but has some limitations:
1) Applications need to be "aware" of the scheme. They can't just use
'topic1' - they need to know/find out their principal and prefix their
topics appropriately.
2) There's no quotas (or similar scheme) that stop one user from
impacting on another's available bandwidth.
3) Consumer groups would still be a shared namespace that users could
clash on.
How to move this forward? I agree with Gari that KIPs 11, 12 (and 13)
are still in-flight - so it might be difficult to write another related
KIP and keep it in step. The shared document approach that Ashish
suggested sounds like a good fit. We can record our intent - and work
out whether this needs to be written up as a KIP, or raised as
improvement JIRAs, once the dust settles on 11, 12 and 13.
- Adrian.
On 05/05/15 20:33, Gari Singh wrote:
I agree. If we can really do pluggable authorization (and even pluggable
authentication), it would not actually be hard to effectively implement a
multi-tenant solution. I am hoping to attempt to implement something like
this once there is code / patches for KIP-11 and KIP-12. So I wonder if we
would actually need a KIP for this versus requirements for an authorization
module (and possibly an authentication module) and then document a best
practice for setting up a multi-tenant deployment.
I am a bit worried that KIPs 11/12 might not give us everything we need,
but I think we can work through that as the implementation of both
progresses
On Tue, May 5, 2015 at 3:23 PM, Ashish Singh <asi...@cloudera.com> wrote:
Jay,
I agree with you. This will kind of depend on how KIP-11 and KIP-12 shapes
up, but we can definitely start putting together the ideas. Would you
suggest starting a kip or just a generic shared document that can be later
translated into a KIP?
On Tuesday, May 5, 2015, Jay Kreps <jay.kr...@gmail.com> wrote:
Hey guys,
It would be nice to get a design around this. Though there are currently
so
many big things in flight I do wonder if we should start another parallel
thing...? But working out a design can't hurt.
Personally I think since one of the goals of Kafka is data integration we
really want to support a central cluster with many users. So I think we
really want to have full hierarchies with all users being part of the
same
"tree" of streams. This way granting permissions on a new stream just
means
changing permissions on what "topic paths" a client has access to rather
than needing a new client instance that connects to that separate
namespace
(a la zookeeper). I think this will be a nice way to share config and acl
defaults too.
-Jay
On Tue, May 5, 2015 at 10:36 AM, Ashish Singh <asi...@cloudera.com
<javascript:;>> wrote:
Adrian,
Trying to follow up the discussion here. Is my understanding correct
that
if we have topic hierarchies then we can do without namespaces. To me
namespace is an abstraction, it can be implemented with topic
hierarchies
as well, would you agree? If so I guess topic hierarchies is the way to
go.
One more thing I would like to put forward as an advantage of having
topic
hierarchy is that we can support acls/ permissions inheritance in topic
hierarchies. This will avoid bootstrapping acls for each new topic.
On Wed, Apr 29, 2015 at 7:42 AM, Adrian Preston <prest...@uk.ibm.com
<javascript:;>>
wrote:
Thanks for your response.
I agree. I think it would be useful to get consensus on how
namespaces and topic-hierarchies relate to one another. To seed the
discussion - here's my viewpoint, which I hope others will challenge:
I see namespaces as something the broker provides to ensure that
two tenants can never interact with one another - even if,
for example, they both choose to use a topic called 'TOPIC1'. I
imagine that this would be achieved by having the broker silently
add a per tenant prefix to the topic name in each request, and
strip it off in the response. So, for example, if 'TENANT1' sends
a message to 'TOPIC1', this would be re-written so that the send is
to 'TENANT1-TOPIC1'.
If topic-hierarchies were available, then I think the prefix,
added/removed to implement namespaces, would be the first level of
qualification in the hierarchy. So, for example, if 'TENANT1' sends
a message to 'TOPIC1', this would be re-written so that the send is
to
'TENANT1/TOPIC1'.
Extrapolating from what's currently in KIP-21 (dynamic configuration)
I guess that topic-hierarchies might result in the possibility for
even finer grain topic configuration - e.g. a ZNode structure of:
'/config/topics/<topic_level1>/<topic_level2>/...'. This would work
best with an implementation of namespaces that was based on-top of
topic-hierarchies, allowing configuration to be applied at the scope
of: all tenants, one tenant, or one tenant's topics.
So in summary: I think that namespaces can be usefully implemented
independently of topic-hierarchies, and when topic-hierarchies are
implemented would be easily integrated.
Regards
- Adrian
-----Gwen Shapira <gshap...@cloudera.com <javascript:;>> wrote:
-----
To: "dev@kafka.apache.org <javascript:;>" <dev@kafka.apache.org
<javascript:;>>
From: Gwen Shapira <gshap...@cloudera.com <javascript:;>>
Date: 04/28/2015 06:54PM
Subject: Re: Adding multi-tenancy capabilities to Kafka
I think recent discussion showed some need for topic namespaces - for
example, Jun's use case for reserving topic names for specific users
discussed under authorization.
I think some discussion should happen on namespaces vs more
full-fledged
topic-hierarchy.
I like the simplicity of namespaces, but there may be other
requirements
(such as inheriting configuration).
Gwen
On Tue, Apr 28, 2015 at 10:42 AM, Adrian Preston <
prest...@uk.ibm.com
<javascript:;>>
wrote:
Hi all,
I've been looking at how a Kafka cluster could be used be deployed
so
that
it can be used by multiple tenants. Specifically: a scheme where
the
clients belonging to each tenant receive the impression they are
operating
against their own cluster. The ongoing security and quota work
looks
like
it might provide a lot of the isolation requirements, but each
tenant
would
currently share the same namespace for topics and consumer groups.
So
the
illusion of "it is my own cluster" is broken as soon as two tenants
try
independent to use the same topic name.
I wondered if other people on this list are interested in being
able
to
support multiple tenants in this way? And / or if the ability to
avoid
clashes in the topic namespace would be useful? I am considering
submitting a KIP in this area - but first wanted to get a feeling
for
whether these kinds of capabilities are of interest to others.
Thanks in advance,
- Adrian
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with
number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire
PO6
3AU
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with
number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire
PO6
3AU
--
Regards,
Ashish
--
Ashish 🎤h
