This is an automated email from the ASF dual-hosted git repository.
cmccabe pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/kafka.git
The following commit(s) were added to refs/heads/trunk by this push:
new 5b422613a66 KAFKA-16101: Fix KRaft migration documentation (#15193)
5b422613a66 is described below
commit 5b422613a664740282440a5c65c7589095149d35
Author: Colin Patrick McCabe <[email protected]>
AuthorDate: Mon Jan 29 14:34:07 2024 -0800
KAFKA-16101: Fix KRaft migration documentation (#15193)
This PR fixes some bugs in the KRaft migration documentation and
reorganizes it to be easier to read. (Specifically, there were some steps that
were previously out of order.)
In order to keep it all straight, the revert documentation is now in the
form of a table which maps the latest migration state to the actions which the
system administrator should perform.
Reviewers: Luke Chen <[email protected]>, David Arthur <[email protected]>,
Liu Zeyu <[email protected]>, Paolo Patierno <[email protected]>
---
docs/ops.html | 160 ++++++++++++++++++++++++++++++++++++++++++++++++++--------
1 file changed, 139 insertions(+), 21 deletions(-)
diff --git a/docs/ops.html b/docs/ops.html
index f943dbed630..8cb65549db6 100644
--- a/docs/ops.html
+++ b/docs/ops.html
@@ -3874,7 +3874,7 @@ zookeeper.connect=localhost:2181
<p><em>Note: The KRaft cluster <code>node.id</code> values must be different
from any existing ZK broker <code>broker.id</code>.
In KRaft-mode, the brokers and controllers share the same Node ID
namespace.</em></p>
- <h3>Enabling the migration on the brokers</h3>
+ <h3>Enter Migration Mode on the Brokers</h3>
<p>
Once the KRaft controller quorum has been started, the brokers will need
to be reconfigured and restarted. Brokers
may be restarted in a rolling fashion to avoid impacting cluster
availability. Each broker requires the
@@ -3919,9 +3919,10 @@ controller.listener.names=CONTROLLER</pre>
<h3>Migrating brokers to KRaft</h3>
<p>
- Once the KRaft controller completes the metadata migration, the brokers
will still be running in ZK mode. While the
- KRaft controller is in migration mode, it will continue sending controller
RPCs to the ZK mode brokers. This includes
- RPCs like UpdateMetadata and LeaderAndIsr.
+ Once the KRaft controller completes the metadata migration, the brokers
will still be running
+ in ZooKeeper mode. While the KRaft controller is in migration mode, it
will continue sending
+ controller RPCs to the ZooKeeper mode brokers. This includes RPCs like
UpdateMetadata and
+ LeaderAndIsr.
</p>
<p>
@@ -3956,29 +3957,16 @@ controller.listener.names=CONTROLLER</pre>
Each broker is restarted with a KRaft configuration until the entire
cluster is running in KRaft mode.
</p>
- <h3>Reverting to ZooKeeper mode During the Migration</h3>
- While the cluster is still in migration mode, it is possible to revert to
ZK mode. In order to do this:
- <ol>
- <li>
- For each KRaft broker:
- <ul>
- <li>Stop the broker.</li>
- <li>Remove the __cluster_metadata directory on the broker.</li>
- <li>Remove the <code>zookeeper.metadata.migration.enable</code>
configuration and the KRaft controllers related configurations like
<code>controller.quorum.voters</code>
- and <code>controller.listener.names</code> from the broker
configuration properties file.</li>
- <li>Restart the broker in ZooKeeper mode.</li>
- </ul>
- </li>
- <li>Take down the KRaft quorum.</li>
- <li>Using ZooKeeper shell, delete the controller node using <code>rmr
/controller</code>, so that a ZooKeeper-based broker can become the next
controller.</li>
- </ol>
-
<h3>Finalizing the migration</h3>
<p>
Once all brokers have been restarted in KRaft mode, the last step to
finalize the migration is to take the
KRaft controllers out of migration mode. This is done by removing the
"zookeeper.metadata.migration.enable"
property from each of their configs and restarting them one at a time.
</p>
+ <p>
+ Once the migration has been finalized, you can safely deprovision your
ZooKeeper cluster, assuming you are
+ not using it for anything else. After this point, it is no longer possible
to revert to ZooKeeper mode.
+ </p>
<pre>
# Sample KRaft cluster controller.properties listening on 9093
@@ -3996,6 +3984,136 @@ listeners=CONTROLLER://:9093
# Other configs ...</pre>
+ <h3>Reverting to ZooKeeper mode During the Migration</h3>
+ <p>
+ While the cluster is still in migration mode, it is possible to revert to
ZooKeeper mode. The process
+ to follow depends on how far the migration has progressed. In order to
find out how to revert,
+ select the <b>final</b> migration step that you have <b>completed</b> in
this table.
+ </p>
+ <p>
+ Note that the directions given here assume that each step was fully
completed, and they were
+ done in order. So, for example, we assume that if "Enter Migration Mode on
the Brokers" was
+ completed, "Provisioning the KRaft controller quorum" was also fully
completed previously.
+ </p>
+ <p>
+ If you did not fully complete any step, back out whatever you have done
and then follow revert
+ directions for the last fully completed step.
+ </p>
+
+ <table class="data-table">
+ <tbody>
+ <tr>
+ <th>Final Migration Section Completed</th>
+ <th>Directions for Reverting</th>
+ <th>Notes</th>
+ </tr>
+ <tr>
+ <td>Preparing for migration</td>
+ <td>
+ The preparation section does not involve leaving ZooKeeper mode. So
there is nothing to do in the
+ case of a revert.
+ </td>
+ <td>
+ </td>
+ </tr>
+ <tr>
+ <td>Provisioning the KRaft controller quorum</td>
+ <td>
+ <ul>
+ <li>
+ Deprovision the KRaft controller quorum.
+ </li>
+ <li>
+ Then you are done.
+ </li>
+ </ul>
+ </td>
+ <td>
+ </td>
+ </tr>
+ <tr>
+ <td>Enter Migration Mode on the brokers</td>
+ <td>
+ <ul>
+ <li>
+ Deprovision the KRaft controller quorum.
+ </li>
+ <li>
+ Using <code>zookeeper-shell.sh</code>, run <code>rmr
/controller</code> so that one
+ of the brokers can become the new old-style controller.
+ </li>
+ <li>
+ On each broker, remove the
<code>zookeeper.metadata.migration.enable</code>,
+ <code>controller.listener.names</code>, and
<code>controller.quorum.voters</code>
+ configurations, and replace <code>node.id</code> with
<code>broker.id</code>.
+ Then perform a rolling restart of all brokers.
+ </li>
+ <li>
+ Then you are done.
+ </li>
+ </ul>
+ </td>
+ <td>
+ It is important to perform the <code>zookeeper-shell.sh</code> step
quickly, to minimize the amount of
+ time that the cluster lacks a controller.
+ </td>
+ </tr>
+ <tr>
+ <td>Migrating brokers to KRaft</td>
+ <td>
+ <ul>
+ <li>
+ On each broker, remove the <code>process.roles</code>
configuration, and
+ restore the <code>zookeeper.connect</code> configuration to its
previous value.
+ If your cluster requires other ZooKeeper configurations for
brokers, such as
+ <code>zookeeper.ssl.protocol</code>, re-add those configurations
as well.
+ Then perform a rolling restart of all brokers.
+ </li>
+ <li>
+ Deprovision the KRaft controller quorum.
+ </li>
+ <li>
+ Using <code>zookeeper-shell.sh</code>, run <code>rmr
/controller</code> so that one
+ of the brokers can become the new old-style controller.
+ </li>
+ <li>
+ On each broker, remove the
<code>zookeeper.metadata.migration.enable</code>,
+ <code>controller.listener.names</code>, and
<code>controller.quorum.voters</code>
+ configurations. Replace <code>node.id</code> with
<code>broker.id</code>.
+ Then perform a second rolling restart of all brokers.
+ </li>
+ <li>
+ Then you are done.
+ </li>
+ </ul>
+ </td>
+ <td>
+ <ul>
+ <li>
+ It is important to perform the <code>zookeeper-shell.sh</code>
step <b>quickly</b>, to minimize the amount of
+ time that the cluster lacks a controller.
+ </li>
+ <li>
+ Make sure that on the first cluster roll,
<code>zookeeper.metadata.migration.enable</code> remains set to
+ </code>true</code>. <b>Do not set it to false until the second
cluster roll.</b>
+ </li>
+ </ul>
+ </td>
+ </tr>
+ <tr>
+ <td>Finalizing the migration</td>
+ <td>
+ If you have finalized the ZK migration, then you cannot revert.
+ </td>
+ <td>
+ Some users prefer to wait for a week or two before finalizing the
migration. While this
+ requires you to keep the ZooKeeper cluster running for a while
longer, it may be helpful
+ in validating KRaft mode in your cluster.
+ </td>
+ </tr>
+ </tbody>
+ </table>
+
<h3 class="anchor-heading"><a id="tiered_storage" class="anchor-link"></a><a
href="#kraft">6.11 Tiered Storage</a></h3>
