I noticed that the following KoP tests need to be updated: OffsetResetTest.java, GroupCoordinatorTest.java, and TransactionStateManagerTest.java. The breakage is limited to test scenarios only. > It has been valid and allowed to create producers or readers directly on partitions if the partition count is known in advance
PR #24118 does not break this feature. If the partition count (i.e., partitioned metadata) is known, the broker still allows producers and consumers to create partitions as expected. > There is a trick that this partitioned topic can be deleted by force and then producers will reconnect the partitions to continue producing. However, with this change, the producers will fail to reconnect due to `NotAllowedException`, Sometimes, users just want to discard huge legacy data. Regarding the loss of partition metadata, it can be recreated by creating the same partitioned topic again. I admit it's a tricky solution. In cases where a partitioned topic is forcibly deleted while the producer or consumer is online, the client will retry and may end up creating a non-partitioned topic with `-partition-`. Since the partitioned metadata is deleted, the client will no longer recognize it as a partitioned topic. Upon restart, producers and consumers won’t be able to publish or consume messages as expected from a partitioned topic. PR #24118 can prevent this scenario. It enforces that users must explicitly create the partitioned metadata. I also discuss this issue on slack: https://apache-pulsar.slack.com/archives/C5ZSVEN4E/p1742803935952389 Once we've concluded this discussion, I'll review your PR. Thanks again for the fix! Thanks, Zixuan Yunze Xu <x...@apache.org> 于2025年4月23日周三 00:34写道： > Hi all, > > I prepared a quick fix: https://github.com/apache/pulsar/pull/24195 so > that we can fix the regression quickly in 3.0, 3.3 and 4.0 releases. I > devoted my time addressing this breaking change because it's my > responsibility that I missed the previous open discussion in the ML > list. But I didn't have more time to write a PIP and go through the > PIP process for that. > > Thanks, > Yunze > > On Tue, Apr 22, 2025 at 11:24 PM Yunze Xu <x...@apache.org> wrote: > > > > Hi Zixuan, > > > > If you're maintaining your own private KoP fork and upgrading the > > Pulsar dependency to include this PR, all tests in > > `GroupCoordinatorTest` fail without changes. I didn't have a chance to > > look into all the failed tests for now. But at least, there are 37 > > tests failing due to the same reason. I've verified that some can be > > fixed by creating a partitioned topic manually. > > > > Even out of the scope of KoP, Pulsar's partitioned topic can be used > > as a few partitions and a metadata that represents the partition > > count. It has been valid and allowed to create producers or readers > > directly on partitions if the partition count is known in advance > > (e.g. via the admin API or UI). > > - producer0 on `topic-partition-0` > > - producer1 on `topic-partition-1` > > - ... > > > > There is a trick that this partitioned topic can be deleted by force > > and then producers will reconnect the partitions to continue > > producing. However, with this change, the producers will fail to > > reconnect due to `NotAllowedException`, Sometimes, users just want to > > discard huge legacy data. Regarding the loss of partition metadata, it > > can be recreated by creating the same partitioned topic again. I admit > > it's a tricky solution. > > > > Thanks, > > Yunze > > > > > > > > > > On Tue, Apr 22, 2025 at 10:33 PM Zixuan Liu <zix...@apache.org> wrote: > > > > > > I'm aware that #24118 breaks some test cases, but I believe those test > > > cases were incorrect. I also flagged this PR as introducing breaking > > > changes in the mailing list, please see > > > https://lists.apache.org/thread/g1jq0vkq0wt28mdnbg63bpw8fpx9ml42 > > > > > > I'd like to better understand the scope of the breakage. Is it limited > to > > > internal KoP usage—such as test cases where developers use > > > partitioned-topic format without metadata? Or does it also affect > users who > > > might be doing this in practice? In our organization, we also maintain > KoP > > > and have updated our test cases accordingly. > > > > > > Personally, I believe that the partitioned topic should have associated > > > metadata. If someone uses the partitioned format without metadata, it > feels > > > more like a workaround or an unsupported edge case. > > > > > > I'm open to discussion. If this behavior is something users actually > depend > > > on, then we should talk about how to handle it more gracefully. > > > > > > Thanks, > > > Zixuan >
