This is an automated email from the ASF dual-hosted git repository. lianetm pushed a commit to branch lm-411-updates in repository https://gitbox.apache.org/repos/asf/kafka-site.git
commit 73957b001320cf815bf00cf78318a09e1c38d220 Author: Lianet Magrans <[email protected]> AuthorDate: Wed Nov 12 12:51:43 2025 -0500 updates for 4.1.1 --- 41/generated/connect_rest.yaml | 2 +- 41/generated/mirror_connector_config.html | 8 ++-- 41/generated/sink_connector_config.html | 8 ++-- 41/generated/source_connector_config.html | 8 ++-- 41/js/templateData.js | 2 +- 41/ops.html | 60 ++++++++++---------------- 41/streams/developer-guide/config-streams.html | 1 - 41/streams/upgrade-guide.html | 20 ++++++--- 41/upgrade.html | 22 ++++++++-- 9 files changed, 68 insertions(+), 63 deletions(-) diff --git a/41/generated/connect_rest.yaml b/41/generated/connect_rest.yaml index 79f59295c..46e944885 100644 --- a/41/generated/connect_rest.yaml +++ b/41/generated/connect_rest.yaml @@ -8,7 +8,7 @@ info: name: Apache 2.0 url: https://www.apache.org/licenses/LICENSE-2.0.html title: Kafka Connect REST API - version: 4.1.0 + version: 4.1.1 paths: /: get: diff --git a/41/generated/mirror_connector_config.html b/41/generated/mirror_connector_config.html index 25c8d171b..54c1ce1da 100644 --- a/41/generated/mirror_connector_config.html +++ b/41/generated/mirror_connector_config.html @@ -765,7 +765,7 @@ <table><tbody> <tr><th>Type:</th><td>string</td></tr> <tr><th>Default:</th><td>null</td></tr> -<tr><th>Valid Values:</th><td>org.apache.kafka.connect.runtime.ConnectorConfig$PluginVersionValidator@442d9b6e</td></tr> +<tr><th>Valid Values:</th><td>org.apache.kafka.connect.runtime.ConnectorConfig$PluginVersionValidator@12bb4df8</td></tr> <tr><th>Importance:</th><td>medium</td></tr> </tbody></table> </li> @@ -805,7 +805,7 @@ <table><tbody> <tr><th>Type:</th><td>string</td></tr> <tr><th>Default:</th><td>null</td></tr> -<tr><th>Valid Values:</th><td>org.apache.kafka.connect.runtime.ConnectorConfig$PluginVersionValidator@ee7d9f1</td></tr> +<tr><th>Valid Values:</th><td>org.apache.kafka.connect.runtime.ConnectorConfig$PluginVersionValidator@4cc77c2e</td></tr> <tr><th>Importance:</th><td>low</td></tr> </tbody></table> </li> @@ -825,7 +825,7 @@ <table><tbody> <tr><th>Type:</th><td>string</td></tr> <tr><th>Default:</th><td>null</td></tr> -<tr><th>Valid Values:</th><td>org.apache.kafka.connect.runtime.ConnectorConfig$PluginVersionValidator@15615099</td></tr> +<tr><th>Valid Values:</th><td>org.apache.kafka.connect.runtime.ConnectorConfig$PluginVersionValidator@7a7b0070</td></tr> <tr><th>Importance:</th><td>low</td></tr> </tbody></table> </li> @@ -845,7 +845,7 @@ <table><tbody> <tr><th>Type:</th><td>string</td></tr> <tr><th>Default:</th><td>null</td></tr> -<tr><th>Valid Values:</th><td>org.apache.kafka.connect.runtime.ConnectorConfig$PluginVersionValidator@1edf1c96</td></tr> +<tr><th>Valid Values:</th><td>org.apache.kafka.connect.runtime.ConnectorConfig$PluginVersionValidator@39a054a5</td></tr> <tr><th>Importance:</th><td>low</td></tr> </tbody></table> </li> diff --git a/41/generated/sink_connector_config.html b/41/generated/sink_connector_config.html index c8a00db60..f03bf6d61 100644 --- a/41/generated/sink_connector_config.html +++ b/41/generated/sink_connector_config.html @@ -25,7 +25,7 @@ <table><tbody> <tr><th>Type:</th><td>string</td></tr> <tr><th>Default:</th><td>null</td></tr> -<tr><th>Valid Values:</th><td>org.apache.kafka.connect.runtime.ConnectorConfig$PluginVersionValidator@799f7e29</td></tr> +<tr><th>Valid Values:</th><td>org.apache.kafka.connect.runtime.ConnectorConfig$PluginVersionValidator@573fd745</td></tr> <tr><th>Importance:</th><td>medium</td></tr> </tbody></table> </li> @@ -85,7 +85,7 @@ <table><tbody> <tr><th>Type:</th><td>string</td></tr> <tr><th>Default:</th><td>null</td></tr> -<tr><th>Valid Values:</th><td>org.apache.kafka.connect.runtime.ConnectorConfig$PluginVersionValidator@531d72ca</td></tr> +<tr><th>Valid Values:</th><td>org.apache.kafka.connect.runtime.ConnectorConfig$PluginVersionValidator@20fa23c1</td></tr> <tr><th>Importance:</th><td>low</td></tr> </tbody></table> </li> @@ -105,7 +105,7 @@ <table><tbody> <tr><th>Type:</th><td>string</td></tr> <tr><th>Default:</th><td>null</td></tr> -<tr><th>Valid Values:</th><td>org.apache.kafka.connect.runtime.ConnectorConfig$PluginVersionValidator@22d8cfe0</td></tr> +<tr><th>Valid Values:</th><td>org.apache.kafka.connect.runtime.ConnectorConfig$PluginVersionValidator@3581c5f3</td></tr> <tr><th>Importance:</th><td>low</td></tr> </tbody></table> </li> @@ -125,7 +125,7 @@ <table><tbody> <tr><th>Type:</th><td>string</td></tr> <tr><th>Default:</th><td>null</td></tr> -<tr><th>Valid Values:</th><td>org.apache.kafka.connect.runtime.ConnectorConfig$PluginVersionValidator@579bb367</td></tr> +<tr><th>Valid Values:</th><td>org.apache.kafka.connect.runtime.ConnectorConfig$PluginVersionValidator@6aa8ceb6</td></tr> <tr><th>Importance:</th><td>low</td></tr> </tbody></table> </li> diff --git a/41/generated/source_connector_config.html b/41/generated/source_connector_config.html index ecbc0bc5c..bea98361a 100644 --- a/41/generated/source_connector_config.html +++ b/41/generated/source_connector_config.html @@ -25,7 +25,7 @@ <table><tbody> <tr><th>Type:</th><td>string</td></tr> <tr><th>Default:</th><td>null</td></tr> -<tr><th>Valid Values:</th><td>org.apache.kafka.connect.runtime.ConnectorConfig$PluginVersionValidator@2d38eb89</td></tr> +<tr><th>Valid Values:</th><td>org.apache.kafka.connect.runtime.ConnectorConfig$PluginVersionValidator@73a8dfcc</td></tr> <tr><th>Importance:</th><td>medium</td></tr> </tbody></table> </li> @@ -65,7 +65,7 @@ <table><tbody> <tr><th>Type:</th><td>string</td></tr> <tr><th>Default:</th><td>null</td></tr> -<tr><th>Valid Values:</th><td>org.apache.kafka.connect.runtime.ConnectorConfig$PluginVersionValidator@5fa7e7ff</td></tr> +<tr><th>Valid Values:</th><td>org.apache.kafka.connect.runtime.ConnectorConfig$PluginVersionValidator@ea30797</td></tr> <tr><th>Importance:</th><td>low</td></tr> </tbody></table> </li> @@ -85,7 +85,7 @@ <table><tbody> <tr><th>Type:</th><td>string</td></tr> <tr><th>Default:</th><td>null</td></tr> -<tr><th>Valid Values:</th><td>org.apache.kafka.connect.runtime.ConnectorConfig$PluginVersionValidator@4629104a</td></tr> +<tr><th>Valid Values:</th><td>org.apache.kafka.connect.runtime.ConnectorConfig$PluginVersionValidator@7e774085</td></tr> <tr><th>Importance:</th><td>low</td></tr> </tbody></table> </li> @@ -105,7 +105,7 @@ <table><tbody> <tr><th>Type:</th><td>string</td></tr> <tr><th>Default:</th><td>null</td></tr> -<tr><th>Valid Values:</th><td>org.apache.kafka.connect.runtime.ConnectorConfig$PluginVersionValidator@27f8302d</td></tr> +<tr><th>Valid Values:</th><td>org.apache.kafka.connect.runtime.ConnectorConfig$PluginVersionValidator@3f8f9dd6</td></tr> <tr><th>Importance:</th><td>low</td></tr> </tbody></table> </li> diff --git a/41/js/templateData.js b/41/js/templateData.js index cb834035f..d63a81b50 100644 --- a/41/js/templateData.js +++ b/41/js/templateData.js @@ -19,6 +19,6 @@ limitations under the License. var context={ "version": "41", "dotVersion": "4.1", - "fullDotVersion": "4.1.0", + "fullDotVersion": "4.1.1", "scalaVersion": "2.13" }; diff --git a/41/ops.html b/41/ops.html index dbb914e60..2b87e07f5 100644 --- a/41/ops.html +++ b/41/ops.html @@ -4072,42 +4072,26 @@ In the replica description 0@controller-0:1234:3Db5QLSqSZieL3rJBUUegA, 0 is the If you are not sure whether you are using static or dynamic quorums, you can determine this by running something like the following:<p> -<pre><code class="language-bash"> - $ bin/kafka-features.sh --bootstrap-controller localhost:9093 describe -</code></pre><p> - - If the <code>kraft.version</code> field is level 0 or absent, you are using a static quorum. If - it is 1 or above, you are using a dynamic quorum. For example, here is an example of a static - quorum:<p/> -<pre><code class="language-bash"> -Feature: kraft.version SupportedMinVersion: 0 SupportedMaxVersion: 1 FinalizedVersionLevel: 0 Epoch: 5 -Feature: metadata.version SupportedMinVersion: 3.3-IV3 SupportedMaxVersion: 3.9-IV0 FinalizedVersionLevel: 3.9-IV0 Epoch: 5 -</code></pre><p/> - - Here is another example of a static quorum:<p/> -<pre><code class="language-bash"> -Feature: metadata.version SupportedMinVersion: 3.3-IV3 SupportedMaxVersion: 3.8-IV0 FinalizedVersionLevel: 3.8-IV0 Epoch: 5 -</code></pre><p/> - - Here is an example of a dynamic quorum:<p/> -<pre><code class="language-bash"> -Feature: kraft.version SupportedMinVersion: 0 SupportedMaxVersion: 1 FinalizedVersionLevel: 1 Epoch: 5 -Feature: metadata.version SupportedMinVersion: 3.3-IV3 SupportedMaxVersion: 3.9-IV0 FinalizedVersionLevel: 3.9-IV0 Epoch: 5 -</code></pre><p/> - - The static versus dynamic nature of the quorum is determined at the time of formatting. - Specifically, the quorum will be formatted as dynamic if <code>controller.quorum.voters</code> is - <b>not</b> present, and if the software version is Apache Kafka 3.9 or newer. If you have - followed the instructions earlier in this document, you will get a dynamic quorum.<p> - - If you would like the formatting process to fail if a dynamic quorum cannot be achieved, format your - controllers using the <code>--feature kraft.version=1</code>. (Note that you should not supply - this flag when formatting brokers -- only when formatting controllers.)<p> - -<pre><code class="language-bash"> - $ bin/kafka-storage.sh format -t KAFKA_CLUSTER_ID --feature kraft.version=1 -c controller.properties -</code></pre><p> - + <pre><code class="language-bash">$ bin/kafka-features.sh --bootstrap-controller localhost:9093 describe</code></pre> + <p> + If the <code>kraft.version</code> field is level 0 or absent, you are using a static quorum. If + it is 1 or above, you are using a dynamic quorum. For example, here is an example of a static + quorum:<p> + <pre><code class="language-bash">Feature: kraft.version SupportedMinVersion: 0 SupportedMaxVersion: 1 FinalizedVersionLevel: 0 Epoch: 5 +Feature: metadata.version SupportedMinVersion: 3.3-IV3 SupportedMaxVersion: 3.9-IV0 FinalizedVersionLevel: 3.9-IV0 Epoch: 5</code></pre> + <p> + Here is another example of a static quorum:<p> + <pre><code class="language-bash">Feature: metadata.version SupportedMinVersion: 3.3-IV3 SupportedMaxVersion: 3.8-IV0 FinalizedVersionLevel: 3.8-IV0 Epoch: 5</code></pre> + <p> + Here is an example of a dynamic quorum:<p> + <pre><code class="language-bash">Feature: kraft.version SupportedMinVersion: 0 SupportedMaxVersion: 1 FinalizedVersionLevel: 1 Epoch: 5 +Feature: metadata.version SupportedMinVersion: 3.3-IV3 SupportedMaxVersion: 3.9-IV0 FinalizedVersionLevel: 3.9-IV0 Epoch: 5</code></pre> + <p> + The static versus dynamic nature of the quorum is determined at the time of formatting. + Specifically, the quorum will be formatted as dynamic if <code>controller.quorum.voters</code> is + <b>not</b> present, and one of --standalone, --initial-controllers, or --no-initial-controllers is set. + If you have followed the instructions earlier in this document, you will get a dynamic quorum. + <p> Note: To migrate from static voter set to dynamic voter set, please refer to the <a href="#kraft_upgrade">Upgrade</a> section. <h5 class="anchor-heading"><a id="kraft_reconfig_add" class="anchor-link"></a><a href="#kraft_reconfig_add">Add New Controller</a></h5> @@ -4156,7 +4140,7 @@ CurrentObservers: [{"id": 0, "directoryId": "3Db5QLSqSZieL3rJBUUegA"}, <pre><code class="language-bash">$ bin/kafka-dump-log.sh --cluster-metadata-decoder --files metadata_log_dir/__cluster_metadata-0/00000000000000000000.log</code></pre> - <p>This command decodes and prints the records in the a cluster metadata snapshot:</p> + <p>This command decodes and prints the records in a cluster metadata snapshot:</p> <pre><code class="language-bash">$ bin/kafka-dump-log.sh --cluster-metadata-decoder --files metadata_log_dir/__cluster_metadata-0/00000000000000000100-0000000001.checkpoint</code></pre> @@ -4186,7 +4170,7 @@ foo <h4 class="anchor-heading"><a id="kraft_deployment" class="anchor-link"></a><a href="#kraft_deployment">Deploying Considerations</a></h4> <ul> - <li>Kafka server's <code>process.role</code> should be set to either <code>broker</code> or <code>controller</code> but not both. Combined mode can be used in development environments, but it should be avoided in critical deployment environments.</li> + <li>Kafka server's <code>process.roles</code> should be set to either <code>broker</code> or <code>controller</code> but not both. Combined mode can be used in development environments, but it should be avoided in critical deployment environments.</li> <li>For redundancy, a Kafka cluster should use 3 or more controllers, depending on factors like cost and the number of concurrent failures your system should withstand without availability impact. For the KRaft controller cluster to withstand <code>N</code> concurrent failures the controller cluster must include <code>2N + 1</code> controllers.</li> <li>The Kafka controllers store all the metadata for the cluster in memory and on disk. We believe that for a typical Kafka cluster 5GB of main memory and 5GB of disk space on the metadata log director is sufficient.</li> </ul> diff --git a/41/streams/developer-guide/config-streams.html b/41/streams/developer-guide/config-streams.html index 439c33098..8c45d0c79 100644 --- a/41/streams/developer-guide/config-streams.html +++ b/41/streams/developer-guide/config-streams.html @@ -1509,4 +1509,3 @@ streamsSettings.put(StreamsConfig.topicPrefix("PARAMETER_NAME"), "topic-value"); $('.b-nav__docs').parent().toggleClass('nav__item__with__subs--expanded'); }); </script> - \ No newline at end of file diff --git a/41/streams/upgrade-guide.html b/41/streams/upgrade-guide.html index 3831c2f1e..ac529e3eb 100644 --- a/41/streams/upgrade-guide.html +++ b/41/streams/upgrade-guide.html @@ -141,8 +141,6 @@ <h3><a id="streams_api_changes_410" href="#streams_api_changes_410">Streams API changes in 4.1.0</a></h3> - <p><b>Note:</b> Kafka Streams 4.1.0 contains a critical memory leak bug (<a href="https://issues.apache.org/jira/browse/KAFKA-19748";>KAFKA-19748</a>) that affects users of range scans and certain DSL operators (session windows, sliding windows, stream-stream joins, foreign-key joins). Users running Kafka Streams should consider upgrading directly to 4.1.1 when available.</p> - <h4>Early Access of the Streams Rebalance Protocol</h4> <p> @@ -215,9 +213,19 @@ Set <code>unstable.feature.versions.enable=true</code> for controllers and brokers, and set <code>unstable.api.versions.enable=true</code> on the brokers as well. In your Kafka Streams application configuration, set <code>group.protocol=streams</code>. - After the new feature is configured, check - <code>kafka-features.sh --bootstrap-server localhost:9092 describe</code> - and `streams.version` should now have FinalizedVersionLevel 1. + </p> + + <p> + When <code>unstable.api.versions.enable=true</code> is set when the kafka storage is first created, + and no explicit metadata version is set, the feature will be enabled by default. In other configurations + (e.g. if the cluster already existed, or a metadata version was hardcoded), you may have to enable it explicitly. + First, check the current feature level by running + <code>kafka-features.sh --bootstrap-server localhost:9092 describe</code>. + If <code>streams.version</code> shows <code>FinalizedVersionLevel</code> is 1, no action is needed. + Otherwise, upgrade by running + <code>kafka-features.sh --bootstrap-server localhost:9092 upgrade --feature streams.version=1</code>. + After the upgrade, verify the change by running + <code>kafka-features.sh --bootstrap-server localhost:9092 describe</code>. </p> <p> @@ -263,6 +271,7 @@ A few important ones are listed below. The full list can be found in <a href="https://issues.apache.org/jira/browse/KAFKA-12822";>KAFKA-12822</a>. <ul> <li><a href="https://issues.apache.org/jira/browse/KAFKA-12829";>Old processor APIs</a></li> + <li><a href="https://issues.apache.org/jira/browse/KAFKA-12823";>KStream#through() in both Java and Scala</a></li> <li> <a href="https://issues.apache.org/jira/browse/KAFKA-16339";>"transformer" methods and classes in both Java and Scala</a> <ul> @@ -272,7 +281,6 @@ </li> </ul> </li> - <li><a href="https://issues.apache.org/jira/browse/KAFKA-16339";>"transformer" methods and classes in both Java and Scala</a></li> <li><a href="https://issues.apache.org/jira/browse/KAFKA-12824";>kstream.KStream#branch in both Java and Scala</a></li> <li><a href="https://issues.apache.org/jira/browse/KAFKA-16332";>builder methods for Time/Session/Join/SlidingWindows</a></li> <li><a href="https://issues.apache.org/jira/browse/KAFKA-12827";>KafkaStreams#setUncaughtExceptionHandler()</a></li> diff --git a/41/upgrade.html b/41/upgrade.html index 0ef6e6376..77e6194b3 100644 --- a/41/upgrade.html +++ b/41/upgrade.html @@ -19,9 +19,18 @@ <script id="upgrade-template" type="text/x-handlebars-template"> -<h4><a id="upgrade_4_1_0" href="#upgrade_4_1_0">Upgrading to 4.1.0</a></h4> +<h4><a id="upgrade_4_1_1" href="#upgrade_4_1_0">Upgrading to 4.1.1</a></h4> +<h5><a id="upgrade_4_1_1_notable" href="#upgrade_4_1_1_notable">Notable changes in 4.1.1</a></h5> + <ul> + <li> + Includes a fix for the critical Kafka Streams bug (<a href="https://issues.apache.org/jira/browse/KAFKA-19748";>KAFKA-19748</a>), solving the memory leak issues that affected users of range scans and certain DSL operators (session windows, sliding windows, stream-stream joins, foreign-key joins). + </li> + <li> + Includes a fix for the critical Kafka Streams bug (<a href="https://issues.apache.org/jira/browse/KAFKA-19479";>KAFKA-19479</a>), related to potential data loss. + </li> + </ul> -<p><b>Note:</b> Kafka Streams 4.1.0 contains a critical memory leak bug (<a href="https://issues.apache.org/jira/browse/KAFKA-19748";>KAFKA-19748</a>) that affects users of range scans and certain DSL operators (session windows, sliding windows, stream-stream joins, foreign-key joins). Users running Kafka Streams should consider upgrading directly to 4.1.1 when available.</p> +<h4><a id="upgrade_4_1_0" href="#upgrade_4_1_0">Upgrading to 4.1.0</a></h4> <h5><a id="upgrade_4_1_0_from" href="#upgrade_4_1_0_from">Upgrading Servers to 4.1.0 from any version 3.3.x through 4.0.x</a></h5> <h5><a id="upgrade_410_notable" href="#upgrade_410_notable">Notable changes in 4.1.0</a></h5> @@ -132,6 +141,11 @@ The filename for rotated <code>state-change.log</code> files has been updated from <code>stage-change.log.[date]</code> to <code>state-change.log.[date]</code> in the log4j2.yaml configuration file. See <a href="https://issues.apache.org/jira/browse/KAFKA-19576";>KAFKA-19576</a> for details. </li> + <li> + Kafka Streams include a critical fix to upgrade from <code>KStreams#transformValues()</code> (remove with 4.0.0 release) + to <code>KStreams#processValues()</code>. + For more details, see the <a href="/{{version}}/documentation/streams/developer-guide/dsl-api.html#transformers-removal-and-migration-to-processors">migration guide</a>. + </li> </ul> <h5><a id="upgrade_servers_400_notable" href="#upgrade_servers_400_notable">Notable changes in 4.0.0</a></h5> <ul> @@ -411,8 +425,8 @@ <li><a id="upgrade_400_notable_kafka_streams" href="#upgrade_400_notable_kafka_streams"><b>Kafka Streams</b></a> <ul> <li> - All public APIs, deprecated in Apache Kafka 3.6 or an earlier release, have been removed, with the exception of <code>JoinWindows.of()</code> and <code>JoinWindows#grace()</code>. - See <a href="https://issues.apache.org/jira/browse/KAFKA-17531";>KAFKA-17531</a> for details. + All public APIs, deprecated in Apache Kafka 3.6 or an earlier release, have been removed, with the exception of <code>JoinWindows.of()</code> and <code>JoinWindows#grace()</code>. + See <a href="https://issues.apache.org/jira/browse/KAFKA-17531";>KAFKA-17531</a> for details. </li> <li> The most important changes are highlighted in the <a href="/{{version}}/documentation/streams/upgrade-guide.html#streams_api_changes_400">Kafka Streams upgrade guide</a>.
