[
https://issues.apache.org/jira/browse/KAFKA-13132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justine Olshan updated KAFKA-13132:
-----------------------------------
Description:
With the change in 3.0 to how topic IDs are assigned to logs, a bug was
inadvertently introduced. Now, topic IDs will only be assigned on the load of
the log to a partition in LISR requests. This means we will only assign topic
IDs for newly created topics/partitions, on broker startup, or potentially when
a partition is reassigned.
In the case of upgrading from an IBP before 2.8, we may have a scenario where
we upgrade the controller to IBP 3.0 last. (Ie, the controller is IBP < 2.8 and
all other brokers are on IBP 3.0) Upon the last broker upgrading, we will elect
a new controller but its LISR request may not result in topic IDs being
assigned to logs.
Keep in mind, in this scenario, topic IDs will be still be newly assigned to
all pre-existing topics and will show up in metadata. *It is just the
LISR/partition.metadata part of the code that is affected.* This means we are
not ensured the same guarantees we had in 2.8.
The problem is two-fold
1. We ignore LISR requests when the partition leader epoch has not increased
(previously we assigned the ID before this check)
2. We only assign the topic ID when we are associating the log with the
partition in replicamanager for the first time. Though in the scenario
described above, we have logs associated with partitions that need to be
upgraded.
We should check the if the LISR request is resulting in a topic ID addition and
add logic to logs already associated to partitions in replica manager.
was:
With the change in 3.0 to how topic IDs are assigned to logs, a bug was
inadvertently introduced. Now, topic IDs will only be assigned on the load of
the log to a partition in LISR requests. This means we will only assign topic
IDs for newly created topics/partitions, on broker startup, or potentially when
a partition is reassigned.
In the case of upgrading from an IBP before 2.8, we may have a scenario where
we upgrade the controller to IBP 3.0 last. (Ie, the controller is IBP < 2.8 and
all other brokers are on IBP 3.0) Upon the last broker upgrading, we will elect
a new controller but its LISR request may not result in topic IDs being
assigned to logs.
Keep in mind, in this scenario, topic IDs will be assigned to all pre-existing
topics and will show up in metadata. It is just the LISR/partition.metadata
part of the code that is affected. This means we are not ensured the same
guarantees we had in 2.8.
The problem is two-fold
1. We ignore LISR requests when the partition leader epoch has not increased
(previously we assigned the ID before this check)
2. We only assign the topic ID when we are associating the log with the
partition in replicamanager for the first time. Though in the scenario
described above, we have logs associated with partitions that need to be
upgraded.
We should check the if the LISR request is resulting in a topic ID addition and
add logic to logs already associated to partitions in replica manager.
> Upgrading to topic IDs in LISR requests has gaps introduced in 3.0
> ------------------------------------------------------------------
>
> Key: KAFKA-13132
> URL: https://issues.apache.org/jira/browse/KAFKA-13132
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 3.0.0
> Reporter: Justine Olshan
> Assignee: Justine Olshan
> Priority: Major
>
> With the change in 3.0 to how topic IDs are assigned to logs, a bug was
> inadvertently introduced. Now, topic IDs will only be assigned on the load of
> the log to a partition in LISR requests. This means we will only assign topic
> IDs for newly created topics/partitions, on broker startup, or potentially
> when a partition is reassigned.
>
> In the case of upgrading from an IBP before 2.8, we may have a scenario where
> we upgrade the controller to IBP 3.0 last. (Ie, the controller is IBP < 2.8
> and all other brokers are on IBP 3.0) Upon the last broker upgrading, we will
> elect a new controller but its LISR request may not result in topic IDs being
> assigned to logs.
> Keep in mind, in this scenario, topic IDs will be still be newly assigned to
> all pre-existing topics and will show up in metadata. *It is just the
> LISR/partition.metadata part of the code that is affected.* This means we are
> not ensured the same guarantees we had in 2.8.
>
> The problem is two-fold
> 1. We ignore LISR requests when the partition leader epoch has not increased
> (previously we assigned the ID before this check)
> 2. We only assign the topic ID when we are associating the log with the
> partition in replicamanager for the first time. Though in the scenario
> described above, we have logs associated with partitions that need to be
> upgraded.
>
> We should check the if the LISR request is resulting in a topic ID addition
> and add logic to logs already associated to partitions in replica manager.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
