Hi Ron,
Thanks for the explanation. I guess thinking about it a little bit more, we
should just add --zk-tls-config-file to all of these commands.
We will be removing this option (plus ZK in general) from these commands in the
next major release, but ZK is still supported in 2.5, so we should just do the
logical thing. (The exception is ZkSecurityMigrator, which will stay).
best,
Colin
On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
> Hi Colin.
>
> <<< It seems like this [--zk-tls-config-file information] could just appear
> in a configuration file, which all of these tools already accept (I think)
>
> ZkSecurityMigrator has no such property file facility; adding a
> "--zk-tls-config-file" parameter is exactly for this purpose. If we add
> that to ZkSecurityMigrator then it is trivial to add it to other commands
> (the same code is simply reused; it ends up being just a few extra lines).
> I do not see any parameter in the other two commands to adjust the ZK
> connection; ConfigCommand accepts a "--command-config" flag, but according
> to the code "This is used only with --bootstrap-server option for
> describing and altering broker configs."
>
> I do agree there would be no need to add "--zk-tls-config-file" to
> ReassignPartitionsCommand if its direct ZK connectivity is replaced in time
> for the next release.
>
> ConfigCommand supports the "--bootstrap-server" option and will have its
> direct ZooKeeper access formally deprecated as per KIP-555, but the special
> use case of bootstrapping a ZooKeeper ensemble with encrypted credentials
> prior to starting Kafka will still exist, so it feels like while
> "--zk-tls-config-file" would never be used except for this use case it
> could probably still be added for this particular situation.
>
> Ron
>
> P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I
> sent) did not go through. I am trying to get emails through now to move
> this discussion forward.
>
> On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <cmcc...@apache.org> wrote:
>
> > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > > Hi everyone. I would like to make the following changes to the KIP.
> > >
> > > MOTIVATION:
> > > Include a statement that it will be difficult in the short term to
> > > deprecate direct Zookeeper communication in kafka-configs.{sh, bat}
> > (which
> > > invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka cluster
> > > with encrypted passwords in Zookeeper is an explicitly-supported use
> > case;
> > > therefore it is in scope to be able to securely configure the CLI tools
> > > that still leverage non-deprecated direct Zookeeper communication for TLS
> > > (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
> > > zookeeper-security-migration.sh).
> >
> > Hi Ron,
> >
> > Thanks for the KIP.
> >
> > About deprecations:
> >
> > * zookeeper-security-migration: clearly, deprecating ZK access in this one
> > would not make sense, since it would defeat the whole point of the tool :)
> >
> > * kafka-reassign-partitions: ZK access should be deprecated here. KIP-455
> > implementation has dragged a bit, but this should get done soon. Certainly
> > before the next release.
> >
> > * kafka-configs: I think ZK access should be deprecated here as well. I
> > agree there is a super-special bootstrapping case here, but that should
> > have its own tool, not use kafka-configs.
> >
> > I will post a separate KIP for this, though.
> >
> > >
> > > GOALS:
> > > Support the secure configuration of TLS-encrypted communication between
> > > Zookeeper and:
> > > a) Kafka brokers
> > > b) The three CLI tools mentioned above that still support direct,
> > > non-deprecated communication to Zookeeper
> > > It is explicitly out-of-scope to deprecate any direct Zookeeper
> > > communication in CLI tools as part of this KIP; such work will occur in
> > > future KIPs instead.
> > >
> > > PUBLIC INTERFACES:
> > > 1) The following new broker configurations will be recognized.
> > > zookeeper.client.secure (default value = false, for backwards
> > > compatibility)
> > > zookeeper.clientCnxnSocket
> > > zookeeper.ssl.keyStore.location
> > > zookeeper.ssl.keyStore.password
> > > zookeeper.ssl.trustStore.location
> > > zookeeper.ssl.trustStore.password
> > > It will be an error for any of the last 5 values to be left unspecified
> > if
> > > zookeeper.client.secure is explicitly set to true.
> > >
> > > 2) In addition, the kafka.security.authorizer.AclAuthorizer class
> > supports
> > > the ability to connect to a different Zookeeper instance than the one the
> > > brokers use. We therefore also add the following optional configs, which
> > > override the corresponding ones from above when present:
> > > authorizer.zookeeper.client.secure
> > > authorizer.zookeeper.clientCnxnSocket
> > > authorizer.zookeeper.ssl.keyStore.location
> > > authorizer.zookeeper.ssl.keyStore.password
> > > authorizer.zookeeper.ssl.trustStore.location
> > > authorizer.zookeeper.ssl.trustStore.password
> > >
> > > 3) The three CLI tools mentioned above will support a new
> > --zk-tls-config-file
> > > <String: Zookeeper TLS configuration file path>" option. The following
> > > properties will be recognized in that file, and unrecognized properties
> > > will be ignored to allow the possibility of pointing zk-tls-config-file
> > at
> > > the broker's config file.
> > > zookeeper.client.secure (default value = false)
> > > zookeeper.clientCnxnSocket
> > > zookeeper.ssl.keyStore.location
> > > zookeeper.ssl.keyStore.password
> > > zookeeper.ssl.trustStore.location
> > > zookeeper.ssl.trustStore.password
> > > It will be an error for any of the last 5 values to be left unspecified
> > if
> > > zookeeper.client.secure is explicitly set to true.
> >
> > Do we really need a --zk-tls-config-file flag? It seems like this could
> > just appear in a configuration file, which all of these tools already
> > accept (I think).
> >
> > best,
> > Colin
> >
> >
> > >
> > > Ron
> > >
> > > On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <rndg...@gmail.com> wrote:
> > >
> > > > Hi everyone. Let's get this discussion going again now that Kafka 2.4
> > > > with Zookeeper 3.5.6 is out.
> > > >
> > > > First, regarding the KIP number, the other KIP that was using this
> > number
> > > > moved to KIP 534, so KIP 515 remains the correct number for this
> > > > discussion. I've updated the Kafka Improvement Proposal page to list
> > this
> > > > KIP in the 515 slot, so we're all set there.
> > > >
> > > > Regarding the actual issues under discussion, I think there are some
> > > > things we should clarify.
> > > >
> > > > 1) It is possible to use TLS connectivity to Zookeeper from Apache
> > Kafka
> > > > 2.4 -- the problem is that configuration information has to be passed
> > via
> > > > system properties as "-D" command line options on the java invocation
> > of
> > > > the broker, and those are not secure (anyone with access to the box
> > can see
> > > > the command line used to invoke the broker); the configuration includes
> > > > sensitive information (e.g. a keystore password), so we need a secure
> > > > mechanism for passing the configuration values. I believe the real
> > > > motivation for this KIP is to harden the configuration mechanism for
> > > > Zookeeper TLS connectivity.
> > > >
> > > > 2) I believe the list of CLI tools that continue to use direct
> > Zookeeper
> > > > connectivity in a non-deprecated fashion is:
> > > > a) zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
> > > > b)
> > > >
> > kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
> > > > c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand
> > > >
> > > > 3) I believe Kafka.admin.ConfigCommand presents a conundrum because it
> > > > explicitly states in a comment that a supported use case is
> > bootstrapping a
> > > > Kafka cluster with encrypted passwords in Zookeeper (see
> > > >
> > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65
> > ).
> > > > This means it will be especially difficult to deprecate this particular
> > > > direct Zookeeper connectivity without a different storage mechanism for
> > > > dynamic configuration values being available (i.e. the self-managed
> > quorum
> > > > referred to in KIP-500).
> > > >
> > > > I think it would be easier and simpler to harden the Zookeeper TLS
> > > > configuration in both Kafka and in CLI tools compared to trying to
> > > > deprecate the direct Zookeeper connectivity in the above 3 CLI tools --
> > > > especially when ConfigCommand has no obvious short-term path to
> > deprecation.
> > > >
> > > > Regarding how to harden the TLS configuration, adding the appropriate
> > > > configs to Kafka is an obvious choice for the brokers. Not that these
> > > > values cannot be dynamically reconfigurable because the values would be
> > > > required to connect to Zookeeper via TLS in the first place.
> > > >
> > > > Regarding how to harden the TLS configuration for the 3 remaining CLI
> > > > tools, it would be relatively straightforward to add support for a
> > > > "--zk-tls-config-file <String: Zookeeper TLS configuration system
> > > > properties file path>" option to each one.
> > > >
> > > > Thoughts?
> > > >
> > > > Ron
> > > >
> > > > On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes <pere.ur...@gmail.com
> > >
> > > > wrote:
> > > >
> > > >> True, the number is duplicated.
> > > >>
> > > >> do you know how can we get the problem fix? I was not aware, I
> > missed,
> > > >> sorry the need to add the KIP to the table.
> > > >>
> > > >> -- Pere
> > > >>
> > > >> Missatge de Jorge Esteban Quilcate Otoya <quilcate.jo...@gmail.com>
> > del
> > > >> dia
> > > >> dt., 3 de set. 2019 a les 13:43:
> > > >>
> > > >> > Hi Pere,
> > > >> >
> > > >> > Have you add your KIP to the list here
> > > >> >
> > > >>
> > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > >> ?
> > > >> >
> > > >> > I found the KIP number assigned to another.
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes <
> > pere.ur...@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> >> Thanks for your time Harsha,
> > > >> >> anyone else with comments? looking forward to hearing from you.
> > > >> >>
> > > >> >> Stupid question: when do you move from discussion to vote?
> > > >> >>
> > > >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia dv., 30
> > > >> d’ag.
> > > >> >> 2019 a les 21:59:
> > > >> >>
> > > >> >> > Thanks Pere. KIP looks good to me.
> > > >> >> > -Harsha
> > > >> >> >
> > > >> >> >
> > > >> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes <
> > > >> >> pere.ur...@gmail.com>
> > > >> >> > wrote:
> > > >> >> >
> > > >> >> >> Not really,
> > > >> >> >> my idea is to keep the JAAS parameter, so people don't see
> > major
> > > >> >> >> changes. But if you pass a properties file, then this takes
> > > >> precedence
> > > >> >> over
> > > >> >> >> the other, with the idea that you can do sasl as well with the
> > > >> >> properties
> > > >> >> >> files.
> > > >> >> >>
> > > >> >> >> Makes sense?
> > > >> >> >>
> > > >> >> >> -- Pere
> > > >> >> >>
> > > >> >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia dv.,
> > 30
> > > >> >> d’ag.
> > > >> >> >> 2019 a les 19:00:
> > > >> >> >>
> > > >> >> >>> Hi Pere,
> > > >> >> >>> Thanks for the KIP. Enabling SSL for zookeeper for
> > Kafka
> > > >> >> makes
> > > >> >> >>> sense.
> > > >> >> >>> "The changes are planned to be introduced in a compatible way,
> > by
> > > >> >> >>> keeping the current JAAS variable precedence."
> > > >> >> >>> Can you elaborate a bit here. If the user configures a JAAS
> > file
> > > >> with
> > > >> >> >>> Client section it will take precedence over zookeeper SSL
> > configs?
> > > >> >> >>>
> > > >> >> >>> Thanks,
> > > >> >> >>> Harsha
> > > >> >> >>>
> > > >> >> >>>
> > > >> >> >>>
> > > >> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes <
> > > >> >> pere.ur...@gmail.com>
> > > >> >> >>> wrote:
> > > >> >> >>>
> > > >> >> >>>> Hi,
> > > >> >> >>>> quick question, I saw in another mail that 2.4 release is
> > planned
> > > >> for
> > > >> >> >>>> September. I think it would be really awesome to have this for
> > > >> this
> > > >> >> >>>> release, do you think we can make it?
> > > >> >> >>>>
> > > >> >> >>>> -- Pere
> > > >> >> >>>>
> > > >> >> >>>> Missatge de Pere Urbón Bayes <pere.ur...@gmail.com> del dia
> > dj.,
> > > >> 29
> > > >> >> >>>> d’ag. 2019 a les 20:10:
> > > >> >> >>>>
> > > >> >> >>>> Hi,
> > > >> >> >>>> this is my first KIP for a change in Apache Kafka, so I'm
> > really
> > > >> need
> > > >> >> >>>> to the process. Looking forward to hearing from you and learn
> > the
> > > >> >> best
> > > >> >> >>>> ropes here.
> > > >> >> >>>>
> > > >> >> >>>> I would like to propose this KIP-515 to enable the
> > > >> ZookeeperClients
> > > >> >> to
> > > >> >> >>>> take full advantage of the TLS communication in the new
> > Zookeeper
> > > >> >> 3.5.5.
> > > >> >> >>>> Specially interesting it the Zookeeper Security Migration,
> > that
> > > >> >> without
> > > >> >> >>>> this change will not work with TLS, disabling users to use
> > ACLs
> > > >> when
> > > >> >> the
> > > >> >> >>>> Zookeeper cluster use TLS.
> > > >> >> >>>>
> > > >> >> >>>> link:
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >>
> > > >>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
> > > >> >> >>>>
> > > >> >> >>>> Looking forward to hearing from you on this,
> > > >> >> >>>>
> > > >> >> >>>> /cheers
> > > >> >> >>>>
> > > >> >> >>>> --
> > > >> >> >>>> Pere Urbon-Bayes
> > > >> >> >>>> Software Architect
> > > >> >> >>>> http://www.purbon.com
> > > >> >> >>>> https://twitter.com/purbon
> > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > >> >> >>>>
> > > >> >> >>>> --
> > > >> >> >>>> Pere Urbon-Bayes
> > > >> >> >>>> Software Architect
> > > >> >> >>>> http://www.purbon.com
> > > >> >> >>>> https://twitter.com/purbon
> > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > >> >> >>>>
> > > >> >> >>>
> > > >> >> >>>
> > > >> >> >>
> > > >> >> >> --
> > > >> >> >> Pere Urbon-Bayes
> > > >> >> >> Software Architect
> > > >> >> >> http://www.purbon.com
> > > >> >> >> https://twitter.com/purbon
> > > >> >> >> https://www.linkedin.com/in/purbon/
> > > >> >> >>
> > > >> >> >
> > > >> >> >
> > > >> >>
> > > >> >> --
> > > >> >> Pere Urbon-Bayes
> > > >> >> Software Architect
> > > >> >> http://www.purbon.com
> > > >> >> https://twitter.com/purbon
> > > >> >> https://www.linkedin.com/in/purbon/
> > > >> >>
> > > >> >
> > > >>
> > > >> --
> > > >> Pere Urbon-Bayes
> > > >> Software Architect
> > > >> http://www.purbon.com
> > > >> https://twitter.com/purbon
> > > >> https://www.linkedin.com/in/purbon/
> > > >>
> > > >
> > >
> >
>
