Hi Rajan > Therefore, this PR is important to make sure > tenants don't face topic > unavailability due to well known issues of > missing schema ledgers.
Firstly, could you submit an issue that you encountered a ledger lost issue, and we should solve the issue Secondly, the ledger containing user messages will also be lost if the schema ledger is lost. Thirdly, after the PR, users will encounter a issue that is more worse than before. They can not consume the original messages before the schema ledger is broken. After the PR they can not consume the messages continuously published. If users want to remove all schemas, they can call the API that is used to remove all schemas manually. Thanks Yubiao Feng On Wed, Oct 9, 2024 at 12:51 PM Rajan Dhabalia <dhabalia...@gmail.com> wrote: > >> When the schemas of a topic are lost, all of the messages in the topic > can not be consumed successfully, and producers can not publish messages > anymore. > > Well, there are already many issues created related to the lost schema > ledgers so, it is a very well known issue that the topic becomes > unavailable and producers are not able to publish messages anymore just > because of server issues which is extremely critical for mission critical > usecases and not acceptable as well. Because of that brokers must be fault > tolerant to recover in case of missing ledgers. Broker already handles such > issues during manged-ledger or managed-cursor legers but it should also > handle in case of schema ledger as well and make sure that broker won't > impact topic's unavailability. It is also very important to just not rely > on operational effort to fix such unavailability issues manually but broker > should have a mechanism to recover by itself, > Therefore, this PR is important to make sure tenants don't face topic > unavailability due to well known issues of missing schema ledgers. > > Thanks, > Rajan > > On Tue, Oct 8, 2024 at 8:29 PM Yubiao Feng > <yubiao.f...@streamnative.io.invalid> wrote: > > > Hi all > > > > Background: When the schemas of a topic are lost, all of the messages in > > the topic can not be consumed successfully, and producers can not publish > > messages anymore. This mechanism alerts users to try to recover their > > schemas or recreate their topics. > > > > https://github.com/apache/pulsar/pull/23395 added a patch: producers > will > > rebuild schemas if the original schemas were lost, which will mix the old > > schema and new schema as the same schema ID. For example: > > - send M1 with schema `Int32`, get schema id: `1` > > - send M2 with schema `String`, get schema id: `2` > > - schemas are lost > > - send M3 with schema `String`, get schema id `1` > > > > The messages `M1` and `M3` use different schemas, but have the same > schema > > id, but users assume all things are fine, which is dangerous. So I want > to > > revert this PR. > > > > Thanks > > Yubiao Feng > > >
