This is an automated email from the ASF dual-hosted git repository. jlprat pushed a commit to branch MINOR-fix-upgrade-instructions in repository https://gitbox.apache.org/repos/asf/kafka-site.git
commit 4b737d82485872c86ac3ca47ccbbb6399feb004b Author: Josep Prat <[email protected]> AuthorDate: Tue Oct 8 10:38:27 2024 +0200 MINOR Add upgrade instructiosn for 38 and 39 Signed-off-by: Josep Prat <[email protected]> --- 38/upgrade.html | 61 ++++++++++++++++++++++++++++ 39/upgrade.html | 124 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 185 insertions(+) diff --git a/38/upgrade.html b/38/upgrade.html index be12f0441..2dcdba7e8 100644 --- a/38/upgrade.html +++ b/38/upgrade.html @@ -21,6 +21,67 @@ <h4><a id="upgrade_3_8_0" href="#upgrade_3_8_0">Upgrading to 3.8.0 from any version 0.8.x through 3.7.x</a></h4> + <h5><a id="upgrade_380_zk" href="#upgrade_380_zk">Upgrading ZooKeeper-based clusters</a></h5> + <p><b>If you are upgrading from a version prior to 2.1.x, please see the note in step 5 below about the change to the schema used to store consumer offsets. + Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.</b></p> + + <p><b>For a rolling upgrade:</b></p> + + <ol> + <li>Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you + are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously + overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior + to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION. + <ul> + <li>inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. <code>3.7</code>, <code>3.6</code>, etc.)</li> + <li>log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See <a href="#upgrade_10_performance_impact">potential performance impact + following the upgrade</a> for the details on what this configuration does.)</li> + </ul> + If you are upgrading from version 0.11.0.x or above, and you have not overridden the message format, then you only need to override + the inter-broker protocol version. + <ul> + <li>inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. <code>3.7</code>, <code>3.6</code>, etc.)</li> + </ul> + </li> + <li>Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the + brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. + It is still possible to downgrade at this point if there are any problems. + </li> + <li>Once the cluster's behavior and performance has been verified, bump the protocol version by editing + <code>inter.broker.protocol.version</code> and setting it to <code>3.8</code>. + </li> + <li>Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest + protocol version, it will no longer be possible to downgrade the cluster to an older version. + </li> + <li>If you have overridden the message format version as instructed above, then you need to do one more rolling restart to + upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, + change log.message.format.version to 3.8 on each broker and restart them one by one. Note that the older Scala clients, + which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs + (or to take advantage of <a href="#upgrade_11_exactly_once_semantics">exactly once semantics</a>), + the newer Java clients must be used. + </li> + </ol> + + <h5><a id="upgrade_380_kraft" href="#upgrade_380_kraft">Upgrading KRaft-based clusters</a></h5> + <p><b>If you are upgrading from a version prior to 3.3.0, please see the note in step 3 below. Once you have changed the metadata.version to the latest version, it will not be possible to downgrade to a version prior to 3.3-IV0.</b></p> + + <p><b>For a rolling upgrade:</b></p> + + <ol> + <li>Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the + brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. + </li> + <li>Once the cluster's behavior and performance has been verified, bump the metadata.version by running + <code> + bin/kafka-features.sh upgrade --metadata 3.8 + </code> + </li> + <li>Note that cluster metadata downgrade is not supported in this version since it has metadata changes. + Every <a href="https://github.com/apache/kafka/blob/trunk/server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java";>MetadataVersion</a> + after 3.2.x has a boolean parameter that indicates if there are metadata changes (i.e. <code>IBP_3_3_IV3(7, "3.3", "IV3", true)</code> means this version has metadata changes). + Given your current and target versions, a downgrade is only possible if there are no metadata changes in the versions between.</li> + </ol> + <h5><a id="upgrade_380_notable" href="#upgrade_380_notable">Notable changes in 3.8.0</a></h5> <ul> <li>MirrorMaker 2 can now emit checkpoints for offsets mirrored before the start of the Checkpoint task for improved offset translation. diff --git a/39/upgrade.html b/39/upgrade.html index faa486215..d769cc70d 100644 --- a/39/upgrade.html +++ b/39/upgrade.html @@ -20,6 +20,69 @@ <script id="upgrade-template" type="text/x-handlebars-template"> <h4><a id="upgrade_3_9_0" href="#upgrade_3_9_0">Upgrading to 3.9.0 from any version 0.8.x through 3.8.x</a></h4> + + <h5><a id="upgrade_390_zk" href="#upgrade_390_zk">Upgrading ZooKeeper-based clusters</a></h5> + <p><b>If you are upgrading from a version prior to 2.1.x, please see the note in step 5 below about the change to the schema used to store consumer offsets. + Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.</b></p> + + <p><b>For a rolling upgrade:</b></p> + + <ol> + <li>Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you + are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously + overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior + to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION. + <ul> + <li>inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. <code>3.8</code>, <code>3.7</code>, etc.)</li> + <li>log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See <a href="#upgrade_10_performance_impact">potential performance impact + following the upgrade</a> for the details on what this configuration does.)</li> + </ul> + If you are upgrading from version 0.11.0.x or above, and you have not overridden the message format, then you only need to override + the inter-broker protocol version. + <ul> + <li>inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. <code>3.8</code>, <code>3.7</code>, etc.)</li> + </ul> + </li> + <li>Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the + brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. + It is still possible to downgrade at this point if there are any problems. + </li> + <li>Once the cluster's behavior and performance has been verified, bump the protocol version by editing + <code>inter.broker.protocol.version</code> and setting it to <code>3.9</code>. + </li> + <li>Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest + protocol version, it will no longer be possible to downgrade the cluster to an older version. + </li> + <li>If you have overridden the message format version as instructed above, then you need to do one more rolling restart to + upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, + change log.message.format.version to 3.8 on each broker and restart them one by one. Note that the older Scala clients, + which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs + (or to take advantage of <a href="#upgrade_11_exactly_once_semantics">exactly once semantics</a>), + the newer Java clients must be used. + </li> + </ol> + + <h5><a id="upgrade_390_kraft" href="#upgrade_390_kraft">Upgrading KRaft-based clusters</a></h5> + <p><b>If you are upgrading from a version prior to 3.3.0, please see the note in step 3 below. Once you have changed the metadata.version to the latest version, it will not be possible to downgrade to a version prior to 3.3-IV0.</b></p> + + <p><b>For a rolling upgrade:</b></p> + + <ol> + <li>Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the + brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. + </li> + <li>Once the cluster's behavior and performance has been verified, bump the metadata.version by running + <code> + bin/kafka-features.sh upgrade --metadata 3.9 + </code> + </li> + <li>Note that cluster metadata downgrade is not supported in this version since it has metadata changes. + Every <a href="https://github.com/apache/kafka/blob/trunk/server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java";>MetadataVersion</a> + after 3.2.x has a boolean parameter that indicates if there are metadata changes (i.e. <code>IBP_3_3_IV3(7, "3.3", "IV3", true)</code> means this version has metadata changes). + Given your current and target versions, a downgrade is only possible if there are no metadata changes in the versions between.</li> + </ol> + + <h5><a id="upgrade_390_notable" href="#upgrade_390_notable">Notable changes in 3.9.0</a></h5> <ul> <li>In case you run your Kafka clusters with no execution permission for the <code>/tmp</code> partition, Kafka will not work properly. It might either refuse to start or fail @@ -45,6 +108,67 @@ <h4><a id="upgrade_3_8_0" href="#upgrade_3_8_0">Upgrading to 3.8.0 from any version 0.8.x through 3.7.x</a></h4> + <h5><a id="upgrade_380_zk" href="#upgrade_380_zk">Upgrading ZooKeeper-based clusters</a></h5> + <p><b>If you are upgrading from a version prior to 2.1.x, please see the note in step 5 below about the change to the schema used to store consumer offsets. + Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.</b></p> + + <p><b>For a rolling upgrade:</b></p> + + <ol> + <li>Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you + are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously + overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior + to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION. + <ul> + <li>inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. <code>3.7</code>, <code>3.6</code>, etc.)</li> + <li>log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See <a href="#upgrade_10_performance_impact">potential performance impact + following the upgrade</a> for the details on what this configuration does.)</li> + </ul> + If you are upgrading from version 0.11.0.x or above, and you have not overridden the message format, then you only need to override + the inter-broker protocol version. + <ul> + <li>inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. <code>3.7</code>, <code>3.6</code>, etc.)</li> + </ul> + </li> + <li>Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the + brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. + It is still possible to downgrade at this point if there are any problems. + </li> + <li>Once the cluster's behavior and performance has been verified, bump the protocol version by editing + <code>inter.broker.protocol.version</code> and setting it to <code>3.8</code>. + </li> + <li>Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest + protocol version, it will no longer be possible to downgrade the cluster to an older version. + </li> + <li>If you have overridden the message format version as instructed above, then you need to do one more rolling restart to + upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, + change log.message.format.version to 3.8 on each broker and restart them one by one. Note that the older Scala clients, + which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs + (or to take advantage of <a href="#upgrade_11_exactly_once_semantics">exactly once semantics</a>), + the newer Java clients must be used. + </li> + </ol> + + <h5><a id="upgrade_380_kraft" href="#upgrade_380_kraft">Upgrading KRaft-based clusters</a></h5> + <p><b>If you are upgrading from a version prior to 3.3.0, please see the note in step 3 below. Once you have changed the metadata.version to the latest version, it will not be possible to downgrade to a version prior to 3.3-IV0.</b></p> + + <p><b>For a rolling upgrade:</b></p> + + <ol> + <li>Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the + brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations. + </li> + <li>Once the cluster's behavior and performance has been verified, bump the metadata.version by running + <code> + bin/kafka-features.sh upgrade --metadata 3.8 + </code> + </li> + <li>Note that cluster metadata downgrade is not supported in this version since it has metadata changes. + Every <a href="https://github.com/apache/kafka/blob/trunk/server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java";>MetadataVersion</a> + after 3.2.x has a boolean parameter that indicates if there are metadata changes (i.e. <code>IBP_3_3_IV3(7, "3.3", "IV3", true)</code> means this version has metadata changes). + Given your current and target versions, a downgrade is only possible if there are no metadata changes in the versions between.</li> + </ol> + <h5><a id="upgrade_380_notable" href="#upgrade_380_notable">Notable changes in 3.8.0</a></h5> <ul> <li>MirrorMaker 2 can now emit checkpoints for offsets mirrored before the start of the Checkpoint task for improved offset translation.
