[
https://issues.apache.org/jira/browse/KAFKA-13132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justine Olshan updated KAFKA-13132:
-----------------------------------
Description:
With the change in 3.0 to how topic IDs are assigned to logs, a bug was
inadvertently introduced. Now, topic IDs will only be assigned on the load of
the log to a partition in LISR requests. This means we will only assign topic
IDs for newly created topics/partitions, on broker startup, or potentially when
a partition is reassigned.
In the case of upgrading from an IBP before 2.8, we may have a scenario where
we upgrade the controller to IBP 3.0 (or even 2.8) last. (Ie, the controller is
IBP < 2.8 and all other brokers are on the newest IBP) Upon the last broker
upgrading, we will elect a new controller but its LISR request will not result
in topic IDs being assigned to logs of existing topics. They will only be
assigned in the cases mentioned above.
*Keep in mind, in this scenario, topic IDs will be still be assigned in the
controller/ZK to all new and pre-existing topics and will show up in metadata.*
This means we are not ensured the same guarantees we had in 2.8. *It is just
the LISR/partition.metadata part of the code that is affected.*
The problem is two-fold
1. We ignore LISR requests when the partition leader epoch has not increased
(previously we assigned the ID before this check)
2. We only assign the topic ID when we are associating the log with the
partition in replicamanager for the first time. Though in the scenario
described above, we have logs associated with partitions that need to be
upgraded.
We should check the if the LISR request is resulting in a topic ID addition and
add logic to logs already associated to partitions in replica manager.
was:
With the change in 3.0 to how topic IDs are assigned to logs, a bug was
inadvertently introduced. Now, topic IDs will only be assigned on the load of
the log to a partition in LISR requests. This means we will only assign topic
IDs for newly created topics/partitions, on broker startup, or potentially when
a partition is reassigned.
In the case of upgrading from an IBP before 2.8, we may have a scenario where
we upgrade the controller to IBP 3.0 last. (Ie, the controller is IBP < 2.8 and
all other brokers are on IBP 3.0) Upon the last broker upgrading, we will elect
a new controller but its LISR request will not result in topic IDs being
assigned to logs. They will only be assigned in the cases mentioned above.
Keep in mind, in this scenario, topic IDs will be still be newly assigned to
all pre-existing topics and will show up in metadata. This means we are not
ensured the same guarantees we had in 2.8. *It is just the
LISR/partition.metadata part of the code that is affected. The controller and
ZooKeeper will still correctly assign topic IDs to new topics upon upgrade. We
will also see this reflected in metadata responses.*
The problem is two-fold
1. We ignore LISR requests when the partition leader epoch has not increased
(previously we assigned the ID before this check)
2. We only assign the topic ID when we are associating the log with the
partition in replicamanager for the first time. Though in the scenario
described above, we have logs associated with partitions that need to be
upgraded.
We should check the if the LISR request is resulting in a topic ID addition and
add logic to logs already associated to partitions in replica manager.
> Upgrading to topic IDs in LISR requests has gaps introduced in 3.0
> ------------------------------------------------------------------
>
> Key: KAFKA-13132
> URL: https://issues.apache.org/jira/browse/KAFKA-13132
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 3.0.0
> Reporter: Justine Olshan
> Assignee: Justine Olshan
> Priority: Major
>
> With the change in 3.0 to how topic IDs are assigned to logs, a bug was
> inadvertently introduced. Now, topic IDs will only be assigned on the load of
> the log to a partition in LISR requests. This means we will only assign topic
> IDs for newly created topics/partitions, on broker startup, or potentially
> when a partition is reassigned.
>
> In the case of upgrading from an IBP before 2.8, we may have a scenario where
> we upgrade the controller to IBP 3.0 (or even 2.8) last. (Ie, the controller
> is IBP < 2.8 and all other brokers are on the newest IBP) Upon the last
> broker upgrading, we will elect a new controller but its LISR request will
> not result in topic IDs being assigned to logs of existing topics. They will
> only be assigned in the cases mentioned above.
> *Keep in mind, in this scenario, topic IDs will be still be assigned in the
> controller/ZK to all new and pre-existing topics and will show up in
> metadata.* This means we are not ensured the same guarantees we had in 2.8.
> *It is just the LISR/partition.metadata part of the code that is affected.*
>
> The problem is two-fold
> 1. We ignore LISR requests when the partition leader epoch has not increased
> (previously we assigned the ID before this check)
> 2. We only assign the topic ID when we are associating the log with the
> partition in replicamanager for the first time. Though in the scenario
> described above, we have logs associated with partitions that need to be
> upgraded.
>
> We should check the if the LISR request is resulting in a topic ID addition
> and add logic to logs already associated to partitions in replica manager.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
