Hi Zixuan,
The use case is that the client side does not care about how many
partitions the topic has. Creating a partitioned topic for this use
case just acts as a convenient approach to create and delete all these
partitions, as well as querying the total count and the whole
properties.
However, the partition metadata is not required for this use case.
Actually we don't have to respect the number of partitions. For
example, given a topic with 50 partitions, creating 50 producers and
50 consumers on that partitioned topic might bring too much pressure
on an application without much CPU. We can configure the number of
partitions manually, here is an example that explains the use case
```java
// Before the restart (numPartitions is configured statically)
final var numPartitions = 50;
admin.topics().createPartitionedTopic(topic, 50);
final var producers = new ArrayList<Producer<byte[]>>();
for (int i = 0; i < numPartitions; i++) {
producers.add(pulsarClient.newProducer().topic(topic +
TopicName.PARTITIONED_TOPIC_SUFFIX + i).create());
}
// Now there are 50 producers
for (var producer : producers) {
producer.close();
}
producers.clear();
// After the restart (we changed the numPartitions config from 50 to 4)
admin.topics().deletePartitionedTopic(topic, true);
final var newNumPartitions = 4;
for (int i = 0; i < newNumPartitions; i++) {
producers.add(pulsarClient.newProducer().topic(topic +
TopicName.PARTITIONED_TOPIC_SUFFIX + i).create());
}
// Now there are only 4 producers
```
In the code above, deleting the partitioned topic can clear old data
and avoid new partitions being deleted by deleting the partitioned
topic.
Thanks,
yunze
On Wed, Apr 23, 2025 at 11:47 AM Zixuan Liu <zix...@apache.org> wrote:
>
> I noticed that the following KoP tests need to be updated:
> OffsetResetTest.java, GroupCoordinatorTest.java, and
> TransactionStateManagerTest.java. The breakage is limited to test scenarios
> only.
> > It has been valid and allowed to create producers or readers
> directly on partitions if the partition count is known in advance
>
> PR #24118 does not break this feature. If the partition count (i.e.,
> partitioned metadata) is known, the broker still allows producers and
> consumers to create partitions as expected.
> > There is a trick that this partitioned topic can be deleted by force
> and then producers will reconnect the partitions to continue
> producing. However, with this change, the producers will fail to
> reconnect due to `NotAllowedException`, Sometimes, users just want to
> discard huge legacy data. Regarding the loss of partition metadata, it
> can be recreated by creating the same partitioned topic again. I admit
> it's a tricky solution.
>
> In cases where a partitioned topic is forcibly deleted while the producer
> or consumer is online, the client will retry and may end up creating a
> non-partitioned topic with `-partition-`. Since the partitioned metadata is
> deleted, the client will no longer recognize it as a partitioned topic.
> Upon restart, producers and consumers won’t be able to publish or consume
> messages as expected from a partitioned topic.
> PR #24118 can prevent this scenario. It enforces that users must explicitly
> create the partitioned metadata.
>
> I also discuss this issue on slack:
> https://apache-pulsar.slack.com/archives/C5ZSVEN4E/p1742803935952389
>
> Once we've concluded this discussion, I'll review your PR. Thanks again for
> the fix!
> Thanks,
> Zixuan
>
> Yunze Xu <x...@apache.org> 于2025年4月23日周三 00:34写道：
>
> > Hi all,
> >
> > I prepared a quick fix: https://github.com/apache/pulsar/pull/24195 so
> > that we can fix the regression quickly in 3.0, 3.3 and 4.0 releases. I
> > devoted my time addressing this breaking change because it's my
> > responsibility that I missed the previous open discussion in the ML
> > list. But I didn't have more time to write a PIP and go through the
> > PIP process for that.
> >
> > Thanks,
> > Yunze
> >
> > On Tue, Apr 22, 2025 at 11:24 PM Yunze Xu <x...@apache.org> wrote:
> > >
> > > Hi Zixuan,
> > >
> > > If you're maintaining your own private KoP fork and upgrading the
> > > Pulsar dependency to include this PR, all tests in
> > > `GroupCoordinatorTest` fail without changes. I didn't have a chance to
> > > look into all the failed tests for now. But at least, there are 37
> > > tests failing due to the same reason. I've verified that some can be
> > > fixed by creating a partitioned topic manually.
> > >
> > > Even out of the scope of KoP, Pulsar's partitioned topic can be used
> > > as a few partitions and a metadata that represents the partition
> > > count. It has been valid and allowed to create producers or readers
> > > directly on partitions if the partition count is known in advance
> > > (e.g. via the admin API or UI).
> > > - producer0 on `topic-partition-0`
> > > - producer1 on `topic-partition-1`
> > > - ...
> > >
> > > There is a trick that this partitioned topic can be deleted by force
> > > and then producers will reconnect the partitions to continue
> > > producing. However, with this change, the producers will fail to
> > > reconnect due to `NotAllowedException`, Sometimes, users just want to
> > > discard huge legacy data. Regarding the loss of partition metadata, it
> > > can be recreated by creating the same partitioned topic again. I admit
> > > it's a tricky solution.
> > >
> > > Thanks,
> > > Yunze
> > >
> > >
> > >
> > >
> > > On Tue, Apr 22, 2025 at 10:33 PM Zixuan Liu <zix...@apache.org> wrote:
> > > >
> > > > I'm aware that #24118 breaks some test cases, but I believe those test
> > > > cases were incorrect. I also flagged this PR as introducing breaking
> > > > changes in the mailing list, please see
> > > > https://lists.apache.org/thread/g1jq0vkq0wt28mdnbg63bpw8fpx9ml42
> > > >
> > > > I'd like to better understand the scope of the breakage. Is it limited
> > to
> > > > internal KoP usage—such as test cases where developers use
> > > > partitioned-topic format without metadata? Or does it also affect
> > users who
> > > > might be doing this in practice? In our organization, we also maintain
> > KoP
> > > > and have updated our test cases accordingly.
> > > >
> > > > Personally, I believe that the partitioned topic should have associated
> > > > metadata. If someone uses the partitioned format without metadata, it
> > feels
> > > > more like a workaround or an unsupported edge case.
> > > >
> > > > I'm open to discussion. If this behavior is something users actually
> > depend
> > > > on, then we should talk about how to handle it more gracefully.
> > > >
> > > > Thanks,
> > > > Zixuan
> >
