Technically I absolutely agree with you, this would indeed create issues. If we were just talking about this KIP I think I'd argue that it is not too harsh of a requirement for users to refrain from using new features until they have fully upgraded their entire cluster. I think in that case it could have been solved in the release notes - similarly to the way a binary protocol change is handled. However looking at the discussion on KIP-290 and thinking ahead to potential other changes on ACLs it would really just mean putting off a proper solution which is a versioning system for ACLs makes sense.
At least from the point of view of this KIP versioning should be a separate KIP as otherwise we don't solve the issue you mentioned above - not sure about 290.. I thought about this for a little while, would something like the following make sense? ACLs are either stored in a separate Zookeeper node or get a version stored with them (separate node is probably easier). So current ACLs would default to v0 and post-KIP252 would be an explicit v1 for example. Authorizers declare which versions they are compatible with (though I'd say i backwards compatibility is what we shoud shoot for) and load ACLs of those versions. Introduce a new parameter authorizer.acl.maxversion which controls which ACLs are loaded by the authorizer - nothing with a version higher than specified here gets loaded, even if the Authorizer would be able to. So the process for a cluster update would be similar to a binary protocol change, set authorizer.acl.maxversion to new_version - 1. Upgrade brokers one by one. Once you are done, change/remove parameter and restart cluster. I'm sure I missed something, but sound good in principle? Best regards, Sönke On Thu, May 3, 2018 at 8:15 PM, Colin McCabe <co...@cmccabe.xyz> wrote: > There are still some problems with compatibility here, right? > > One example is if we construct a DENY ACL with an IP range and then install > it. If all of our brokers have been upgraded, it will work. But if there > are some that still haven't been upgraded, they will not honor the DENY ACL, > possibly causing a security issue. > > In general, it seems like we need some kind of versioning system in ACLs to > handle these cases. > > best, > Colin > > On Thu, May 3, 2018, at 08:11, Sönke Liebau wrote: >> Hi all, >> >> I'd like to readopt this KIP, I got a bit sidetracked by other stuff >> after posting the initial version and discussion, sorry for that. >> >> I've added IPv6 to the KIP, but decided to forego the other scope >> extensions that I mentioned in my previous mail, as there are other >> efforts underway in KIP-290 that cover most of the suggestions >> already. >> >> Does anybody have any other objections to starting a vote on this KIP? >> >> Regards, >> Sönke >> >> On Fri, Feb 2, 2018 at 5:11 PM, Sönke Liebau <soenke.lie...@opencore.com> >> wrote: >> > Hi Manikumar, >> > >> > you are right, 5713 is a bit ambiguous about which fields are considered in >> > scope, but I agree that wildcards for Ips are not necessary when we have >> > ranges. >> > >> > I am wondering though, if we might want to extend the scope of this KIP a >> > bit while we are changing acl and authorizer classes anyway. >> > >> > After considering this a bit on a flihht with no wifi yesterday I came up >> > with the following: >> > >> > * wildcards or regular expressions for principals, groups and topics >> > * extend the KafkaPrincipal object to allow adding custom key-value pairs >> > in >> > principalbuilder implementations >> > * extend SimpleAclAuthorizer and the ACL tools to authorize on these >> > key/value pairs >> > >> > The second and third bullet points would allow easy creation of for example >> > a principalbuilder that adds groups the user belongs to in the active >> > directory to its principal, without requiring the user to also extend the >> > authorizer and create custom ACL storage. This would significantly lower >> > the >> > technical debt incurred by custom authorizer mechanisms I think. >> > >> > There are a few issues to hash out of course, but I'd think in general this >> > should work work nicely and be a step towards meeting corporate >> > authorization requirements. >> > >> > Best regards, >> > Sönke >> > >> > Am 01.02.2018 18:46 schrieb "Manikumar" <manikumar.re...@gmail.com>: >> > >> > Hi, >> > >> > They are few deployments using IPv6. It is good to support IPv6 also. >> > >> > I think KAFKA-5713 is about adding regular expression support to resource >> > names (topic. consumer etc..). >> > Yes, wildcards (*) in hostname doesn't makes sense. Range and subnet >> > support will give us the flexibility. >> > >> > On Thu, Feb 1, 2018 at 5:56 PM, Sönke Liebau < >> > soenke.lie...@opencore.com.invalid> wrote: >> > >> >> Hi Manikumar, >> >> >> >> the current proposal indeed leaves out IPv6 addresses, as I was unsure >> >> whether Kafka fully supports that yet to be honest. But it would be >> >> fairly easy to add these to the proposal - I'll update it over the >> >> weekend. >> >> >> >> Regarding KAFKA-5713, I simply listed it as related, since it is >> >> similar in spirit, if not exact wording. Parts of that issue >> >> (wildcards in hosts) would be covered by this kip - just in a slightly >> >> different way. Do we really need wildcard support in IP addresses if >> >> we can specify ranges and subnets? I considered it, but only came up >> >> with scenarios that seemed fairly academic to me, like allowing the >> >> same host from multiple subnets (10.0.*.1) for example. >> >> >> >> Allowing wildcards has the potential to make the code more complex, >> >> depending on how we decide to implement this feature, hance I decided >> >> to leave wildcards out for now. >> >> >> >> What do you think? >> >> >> >> Best regards, >> >> Sönke >> >> >> >> On Thu, Feb 1, 2018 at 10:14 AM, Manikumar <manikumar.re...@gmail.com> >> >> wrote: >> >> > Hi, >> >> > >> >> > 1. Do we support IPv6 CIDR/ranges? >> >> > >> >> > 2. KAFKA-5713 is mentioned in Related JIRAs section. But there is no >> >> > mention of wildcard support in the KIP. >> >> > >> >> > >> >> > Thanks, >> >> > >> >> > On Thu, Feb 1, 2018 at 4:05 AM, Sönke Liebau < >> >> > soenke.lie...@opencore.com.invalid> wrote: >> >> > >> >> >> Hey everybody, >> >> >> >> >> >> following a brief inital discussion a couple of days ago on this list >> >> >> I'd like to get a discussion going on KIP-252 which would allow >> >> >> specifying ip ranges and subnets for the -allow-host and --deny-host >> >> >> parameters of the acl tool. >> >> >> >> >> >> The KIP can be found at >> >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP- >> >> >> 252+-+Extend+ACLs+to+allow+filtering+based+on+ip+ranges+and+subnets >> >> >> >> >> >> Best regards, >> >> >> Sönke >> >> >> >> >> >> >> >> >> >> >> -- >> >> Sönke Liebau >> >> Partner >> >> Tel. +49 179 7940878 >> >> OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany >> >> >> > >> > >> >> >> >> -- >> Sönke Liebau >> Partner >> Tel. +49 179 7940878 >> OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
