chia7712 commented on code in PR #17813:
URL: https://github.com/apache/kafka/pull/17813#discussion_r1858567773
##########
docs/security.html:
##########
@@ -2373,157 +2326,4 @@ <h3 class="anchor-heading"><a
id="security_rolling_upgrade" class="anchor-link">
<pre><code
class="language-text">listeners=SSL://broker1:9092,SASL_SSL://broker1:9093
security.inter.broker.protocol=SSL</code></pre>
- ZooKeeper can be secured independently of the Kafka cluster. The steps for
doing this are covered in section <a href="#zk_authz_migration">7.7.2</a>.
-
-
- <h3 class="anchor-heading"><a id="zk_authz" class="anchor-link"></a><a
href="#zk_authz">7.7 ZooKeeper Authentication</a></h3>
- ZooKeeper supports mutual TLS (mTLS) authentication beginning with the
3.5.x versions.
- Kafka supports authenticating to ZooKeeper with SASL and mTLS -- either
individually or both together --
- beginning with version 2.5. See
- <a
href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication";>KIP-515:
Enable ZK client to use the new TLS supported authentication</a>
- for more details.
- <p>When using mTLS alone, every broker and any CLI tools (such as the <a
href="#zk_authz_migration">ZooKeeper Security Migration Tool</a>)
- should identify itself with the same Distinguished Name (DN) because
it is the DN that is ACL'ed.
- This can be changed as described below, but it involves writing and
deploying a custom ZooKeeper authentication provider.
- Generally each certificate should have the same DN but a different
Subject Alternative Name (SAN)
- so that hostname verification of the brokers and any CLI tools by
ZooKeeper will succeed.
- </p>
- <p>
- When using SASL authentication to ZooKeeper together with mTLS, both
the SASL identity and
- either the DN that created the znode (i.e. the creating broker's
certificate)
- or the DN of the Security Migration Tool (if migration was performed
after the znode was created)
- will be ACL'ed, and all brokers and CLI tools will be authorized even
if they all use different DNs
- because they will all use the same ACL'ed SASL identity.
- It is only when using mTLS authentication alone that all the DNs must
match (and SANs become critical --
- again, in the absence of writing and deploying a custom ZooKeeper
authentication provider as described below).
- </p>
- <p>
- Use the broker properties file to set TLS configs for brokers as
described below.
- </p>
- <p>
- Use the <code>--zk-tls-config-file <file></code> option to set
TLS configs in the Zookeeper Security Migration Tool.
- The <code>kafka-acls.sh</code> and <code>kafka-configs.sh</code> CLI
tools also support the <code>--zk-tls-config-file <file></code> option.
- </p>
- <p>
- Use the <code>-zk-tls-config-file <file></code> option (note the
single-dash rather than double-dash)
- to set TLS configs for the <code>zookeeper-shell.sh</code> CLI tool.
- </p>
- <h4 class="anchor-heading"><a id="zk_authz_new" class="anchor-link"></a><a
href="#zk_authz_new">7.7.1 New clusters</a></h4>
- <h5 class="anchor-heading"><a id="zk_authz_new_sasl"
class="anchor-link"></a><a href="#zk_authz_new_sasl">7.7.1.1 ZooKeeper SASL
Authentication</a></h5>
- To enable ZooKeeper SASL authentication on brokers, there are two
necessary steps:
- <ol>
- <li> Create a JAAS login file and set the appropriate system property
to point to it as described above</li>
- <li> Set the configuration property <code>zookeeper.set.acl</code> in
each broker to true</li>
- </ol>
-
- The metadata stored in ZooKeeper for the Kafka cluster is world-readable,
but can only be modified by the brokers. The rationale behind this decision is
that the data stored in ZooKeeper is not sensitive, but inappropriate
manipulation of that data can cause cluster disruption. We also recommend
limiting the access to ZooKeeper via network segmentation (only brokers and
some admin tools need access to ZooKeeper).
-
- <h5 class="anchor-heading"><a id="zk_authz_new_mtls"
class="anchor-link"></a><a href="#zk_authz_new_mtls">7.7.1.2 ZooKeeper Mutual
TLS Authentication</a></h5>
- ZooKeeper mTLS authentication can be enabled with or without SASL
authentication. As mentioned above,
- when using mTLS alone, every broker and any CLI tools (such as the <a
href="#zk_authz_migration">ZooKeeper Security Migration Tool</a>)
- must generally identify itself with the same Distinguished Name (DN)
because it is the DN that is ACL'ed, which means
- each certificate should have an appropriate Subject Alternative Name (SAN)
so that
- hostname verification of the brokers and any CLI tool by ZooKeeper will
succeed.
- <p>
- It is possible to use something other than the DN for the identity of
mTLS clients by writing a class that
- extends
<code>org.apache.zookeeper.server.auth.X509AuthenticationProvider</code> and
overrides the method
- <code>protected String getClientId(X509Certificate clientCert)</code>.
- Choose a scheme name and set <code>authProvider.[scheme]</code> in
ZooKeeper to be the fully-qualified class name
- of the custom implementation; then set
<code>ssl.authProvider=[scheme]</code> to use it.
- </p>
- Here is a sample (partial) ZooKeeper configuration for enabling TLS
authentication.
- These configurations are described in the
- <a
href="https://zookeeper.apache.org/doc/r3.5.7/zookeeperAdmin.html#sc_authOptions";>ZooKeeper
Admin Guide</a>.
- <pre><code class="language-text">secureClientPort=2182
-serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
-authProvider.x509=org.apache.zookeeper.server.auth.X509AuthenticationProvider
-ssl.keyStore.location=/path/to/zk/keystore.jks
-ssl.keyStore.password=zk-ks-passwd
-ssl.trustStore.location=/path/to/zk/truststore.jks
-ssl.trustStore.password=zk-ts-passwd</code></pre>
- <strong>IMPORTANT</strong>: ZooKeeper does not support setting the key
password in the ZooKeeper server keystore
- to a value different from the keystore password itself.
- Be sure to set the key password to be the same as the keystore password.
-
- <p>Here is a sample (partial) Kafka Broker configuration for connecting to
ZooKeeper with mTLS authentication.
- These configurations are described above in <a
href="#brokerconfigs">Broker Configs</a>.
- </p>
- <pre><code class="language-text"># connect to the ZooKeeper port
configured for TLS
-zookeeper.connect=zk1:2182,zk2:2182,zk3:2182
-# required to use TLS to ZooKeeper (default is false)
-zookeeper.ssl.client.enable=true
-# required to use TLS to ZooKeeper
-zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
-# define key/trust stores to use TLS to ZooKeeper; ignored unless
zookeeper.ssl.client.enable=true
-zookeeper.ssl.keystore.location=/path/to/kafka/keystore.jks
-zookeeper.ssl.keystore.password=kafka-ks-passwd
-zookeeper.ssl.truststore.location=/path/to/kafka/truststore.jks
-zookeeper.ssl.truststore.password=kafka-ts-passwd
-# tell broker to create ACLs on znodes
-zookeeper.set.acl=true</code></pre>
- <strong>IMPORTANT</strong>: ZooKeeper does not support setting the key
password in the ZooKeeper client (i.e. broker) keystore
- to a value different from the keystore password itself.
- Be sure to set the key password to be the same as the keystore password.
-
- <h4 class="anchor-heading"><a id="zk_authz_migration"
class="anchor-link"></a><a href="#zk_authz_migration">7.7.2 Migrating
clusters</a></h4>
- If you are running a version of Kafka that does not support security or
simply with security disabled, and you want to make the cluster secure, then
you need to execute the following steps to enable ZooKeeper authentication with
minimal disruption to your operations:
- <ol>
- <li>Enable SASL and/or mTLS authentication on ZooKeeper. If enabling
mTLS, you would now have both a non-TLS port and a TLS port, like this:
- <pre><code class="language-text">clientPort=2181
-secureClientPort=2182
-serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
-authProvider.x509=org.apache.zookeeper.server.auth.X509AuthenticationProvider
-ssl.keyStore.location=/path/to/zk/keystore.jks
-ssl.keyStore.password=zk-ks-passwd
-ssl.trustStore.location=/path/to/zk/truststore.jks
-ssl.trustStore.password=zk-ts-passwd</code></pre>
- </li>
- <li>Perform a rolling restart of brokers setting the JAAS login file
and/or defining ZooKeeper mutual TLS configurations (including connecting to
the TLS-enabled ZooKeeper port) as required, which enables brokers to
authenticate to ZooKeeper. At the end of the rolling restart, brokers are able
to manipulate znodes with strict ACLs, but they will not create znodes with
those ACLs</li>
- <li>If you enabled mTLS, disable the non-TLS port in ZooKeeper</li>
- <li>Perform a second rolling restart of brokers, this time setting the
configuration parameter <code>zookeeper.set.acl</code> to true, which enables
the use of secure ACLs when creating znodes</li>
- <li>Execute the ZkSecurityMigrator tool. To execute the tool, there is
this script: <code>bin/zookeeper-security-migration.sh</code> with
<code>zookeeper.acl</code> set to secure. This tool traverses the corresponding
sub-trees changing the ACLs of the znodes. Use the <code>--zk-tls-config-file
<file></code> option if you enable mTLS.</li>
- </ol>
- <p>It is also possible to turn off authentication in a secure cluster. To
do it, follow these steps:</p>
- <ol>
- <li>Perform a rolling restart of brokers setting the JAAS login file
and/or defining ZooKeeper mutual TLS configurations, which enables brokers to
authenticate, but setting <code>zookeeper.set.acl</code> to false. At the end
of the rolling restart, brokers stop creating znodes with secure ACLs, but are
still able to authenticate and manipulate all znodes</li>
- <li>Execute the ZkSecurityMigrator tool. To execute the tool, run this
script <code>bin/zookeeper-security-migration.sh</code> with
<code>zookeeper.acl</code> set to unsecure. This tool traverses the
corresponding sub-trees changing the ACLs of the znodes. Use the
<code>--zk-tls-config-file <file></code> option if you need to set TLS
configuration.</li>
- <li>If you are disabling mTLS, enable the non-TLS port in
ZooKeeper</li>
- <li>Perform a second rolling restart of brokers, this time omitting
the system property that sets the JAAS login file and/or removing ZooKeeper
mutual TLS configuration (including connecting to the non-TLS-enabled ZooKeeper
port) as required</li>
- <li>If you are disabling mTLS, disable the TLS port in ZooKeeper</li>
- </ol>
- Here is an example of how to run the migration tool:
- <pre><code class="language-bash">$ bin/zookeeper-security-migration.sh
--zookeeper.acl=secure --zookeeper.connect=localhost:2181</code></pre>
- <p>Run this to see the full list of parameters:</p>
- <pre><code class="language-bash">$ bin/zookeeper-security-migration.sh
--help</code></pre>
- <h4 class="anchor-heading"><a id="zk_authz_ensemble"
class="anchor-link"></a><a href="#zk_authz_ensemble">7.7.3 Migrating the
ZooKeeper ensemble</a></h4>
- It is also necessary to enable SASL and/or mTLS authentication on the
ZooKeeper ensemble. To do it, we need to perform a rolling restart of the
server and set a few properties. See above for mTLS information. Please refer
to the ZooKeeper documentation for more detail:
- <ol>
- <li><a
href="https://zookeeper.apache.org/doc/r3.5.7/zookeeperProgrammers.html#sc_ZooKeeperAccessControl";>Apache
ZooKeeper documentation</a></li>
- <li><a
href="https://cwiki.apache.org/confluence/display/ZOOKEEPER/Zookeeper+and+SASL";>Apache
ZooKeeper wiki</a></li>
- </ol>
- <h4 class="anchor-heading"><a id="zk_authz_quorum"
class="anchor-link"></a><a href="#zk_authz_quorum">7.7.4 ZooKeeper Quorum
Mutual TLS Authentication</a></h4>
- It is possible to enable mTLS authentication between the ZooKeeper servers
themselves.
- Please refer to the <a
href="https://zookeeper.apache.org/doc/r3.5.7/zookeeperAdmin.html#Quorum+TLS";>ZooKeeper
documentation</a> for more detail.
-
- <h3 class="anchor-heading"><a id="zk_encryption"
class="anchor-link"></a><a href="#zk_encryption">7.8 ZooKeeper
Encryption</a></h3>
- ZooKeeper connections that use mutual TLS are encrypted.
- Beginning with ZooKeeper version 3.5.7 (the version shipped with Kafka
version 2.5) ZooKeeper supports a sever-side config
- <code>ssl.clientAuth</code> (case-insensitively:
<code>want</code>/<code>need</code>/<code>none</code> are the valid options,
the default is <code>need</code>),
- and setting this value to <code>none</code> in ZooKeeper allows clients to
connect via a TLS-encrypted connection
- without presenting their own certificate. Here is a sample (partial)
Kafka Broker configuration for connecting to ZooKeeper with just TLS encryption.
- These configurations are described above in <a
href="#brokerconfigs">Broker Configs</a>.
- <pre><code class="language-text"># connect to the ZooKeeper port
configured for TLS
-zookeeper.connect=zk1:2182,zk2:2182,zk3:2182
-# required to use TLS to ZooKeeper (default is false)
-zookeeper.ssl.client.enable=true
-# required to use TLS to ZooKeeper
-zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
-# define trust stores to use TLS to ZooKeeper; ignored unless
zookeeper.ssl.client.enable=true
-# no need to set keystore information assuming ssl.clientAuth=none on ZooKeeper
-zookeeper.ssl.truststore.location=/path/to/kafka/truststore.jks
-zookeeper.ssl.truststore.password=kafka-ts-passwd
-# tell broker to create ACLs on znodes (if using SASL authentication,
otherwise do not set this)
-zookeeper.set.acl=true</code></pre>
-</script>
Review Comment:
@FrankYang0529 FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
