[
https://issues.apache.org/jira/browse/KAFKA-7021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16514001#comment-16514001
]
ASF GitHub Bot commented on KAFKA-7021:
---------------------------------------
guozhangwang closed pull request #5195: KAFKA-7021: Update upgrade guide
section for reusing source topic
URL: https://github.com/apache/kafka/pull/5195
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/docs/streams/upgrade-guide.html b/docs/streams/upgrade-guide.html
index 07f85446424..cd9278262b7 100644
--- a/docs/streams/upgrade-guide.html
+++ b/docs/streams/upgrade-guide.html
@@ -34,7 +34,7 @@ <h1>Upgrade Guide and API Changes</h1>
</div>
<p>
- Upgrading from any older version to 2.0.0 is possible: (1) you need to
make sure to update you code accordingly, because there are some minor
non-compatible API changes since older
+ Upgrading from any older version to 2.0.0 is possible: (1) you need to
make sure to update you code and config accordingly, because there are some
minor non-compatible API changes since older
releases (the code changes are expected to be minimal, please see
below for the details),
(2) upgrading to 2.0.0 in the online mode requires two rolling bounces.
For (2), in the first rolling bounce phase users need to set config
<code>upgrade.from="older version"</code> (possible values are <code>"0.10.0",
"0.10.1", "0.10.2", "0.11.0", "1.0", and "1.1"</code>)
@@ -59,6 +59,16 @@ <h1>Upgrade Guide and API Changes</h1>
For Kafka Streams 0.10.0, broker version 0.10.0 or higher is required.
</p>
+ <p>
+ Another important thing to keep in mind: in deprecated
<code>KStreamBuilder</code> class, when a <code>KTable</code> is created from a
source topic via <code>KStreamBuilder.table()</code>, its materialized state
store
+ will reuse the source topic as its changelog topic for restoring, and
will disable logging to avoid appending new updates to the source topic; in the
<code>StreamsBuilder</code> class introduced in 1.0, this behavior was changed
+ accidentally: we still reuse the source topic as the changelog topic
for restoring, but will also create a separate changelog topic to append the
update records from source topic to. In the 2.0 release, we have fixed this
issue and now users
+ can choose whether or not to reuse the source topic based on the
<code>StreamsConfig#TOPOLOGY_OPTIMIZATION</code>: if you are upgrading from the
old <code>KStreamBuilder</code> class and hence you need to change your code to
use
+ the new <code>StreamsBuilder</code>, you should set this config value
to <code>StreamsConfig#OPTIMIZE</code> to continue reusing the source topic; if
you are upgrading from 1.0 or 1.1 where you are already using
<code>StreamsBuilder</code> and hence have already
+ created a separate changelog topic, you should set this config value
to <code>StreamsConfig#NO_OPTIMIZATION</code> when upgrading to 2.0.0 in order
to use that changelog topic for restoring the state store.
+ More details about the new config
<code>StreamsConfig#TOPOLOGY_OPTIMIZATION</code> can be found in <a
href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-295%3A+Add+Streams+Configuration+Allowing+for+Optional+Topology+Optimization";>KIP-295</a>.
+ </p>
+
<p>
In 2.0.0 we have added a few new APIs on the
<code>ReadOnlyWindowStore</code> interface (for details please read <a
href="#streams_api_changes_200">Streams API changes</a> below).
If you have customized window store implementations that extends the
<code>ReadOnlyWindowStore</code> interface you need to make code changes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Source KTable checkpoint is not correct
> ---------------------------------------
>
> Key: KAFKA-7021
> URL: https://issues.apache.org/jira/browse/KAFKA-7021
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 0.10.0.0
> Reporter: Matthias J. Sax
> Assignee: Guozhang Wang
> Priority: Major
> Fix For: 0.10.2.2, 2.0.0, 0.11.0.3, 1.0.2, 1.1.1
>
>
> Kafka Streams treats source KTables,ie, table created via `builder.table()`,
> differently. Instead of creating a changelog topic, the original source topic
> is use to avoid unnecessary data redundancy.
> However, Kafka Streams does not write a correct local state checkpoint file.
> This results in unnecessary state restore after a rebalance. Instead of the
> latest committed offset, the latest restored offset is written into the
> checkpoint file in `ProcessorStateManager#close()`
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
