Now that PIP-43 is released in 2.5.0, I wanted to follow up on the messages below.
What is remaining to be done in Pulsar to support having multiple different types on one topic in Pulsar? Yi indicates below that PIP-43 sets the stage for this, but that the schema compatibility implementation still would need some work. Would this require another PIP, or just an issue to track the work? Regards, Raman On 2019/09/16 01:32:39, Yi Tang <ssnailt...@gmail.com> wrote: > Hi rarma, > > It's a great and important feature, I think. This PIP requires the > compatibility check from bottom registry only and doesn't touch the > implementation detail. I think we should address this feature in the > future, and this PIP provides the essential ability to implement it. > > Thanks, > Yi > > rocketra...@gmail.com <rocketra...@gmail.com> 于 2019年9月15日周日 22:36写道： > > > I see a mention of compatibility in the PIP but with no details. The docs > > about schema compatibility state this: > > > > > Consequently, those events need to go in the same Pulsar partition to > > maintain order. This application can use ALWAYS_COMPATIBLE to allow > > different kinds of events co-exist in the same topic. > > > > With this PIP, this limitation can be relaxed, and schema compatibility > > should be able to be strengthened, since each type of message on a topic > > can have its own schema, and compatibility can then be checked against only > > other schemas for the same type. Kafka does this via the concept of > > "subjects" in the schema registry, and subjects default to just the topic > > name (plus a "-key" or "-value" suffix since keys and values can both have > > their own schemas), but can also include (via an injectable strategy) the > > message type. Compatibility is managed at the subject level. > > > > Is this something that should be addressed in this PIP, or in future > > follow-on work? This is critical to supporting ordering across different > > message types, with schema compatibility verification by Pulsar. > > > > Regards, > > Raman > > > > > > > > On 2019/09/03 05:12:32, 唐谊 <ssnailt...@gmail.com> wrote: > > > Hi all; > > > > > > I am drafting a proposal to support the producer to send messages with > > > different schema. > > > > > > ## Motivation > > > For now, Pulsar producer can only produce messages of one type of schema > > > which is determined by user when it is created, or by fecthing the latest > > > version of schema from registry if AUTO_PRODUCE_BYTES type is specified. > > > Schema, however, can be updated by external system after producer > > started, > > > which would lead to inconsistency between messsage payload and schema > > > version metadata. Also some senarios like replicating from kafka require > > a > > > single producer for replicating messages of different schemas from one > > > Kafka partition to one Pulsar partition to guarantee the order and no > > > duplicates. > > > > > > Here proposing that messages can indicate the associated schema by > > itself, > > > for more detail, > > > > > https://gist.github.com/yittg/56c6dedf7509f634ec7effc4f6f3631d#file-pip-md > > > > > > Looking forward to any feedback. > > > > > > Thanks, > > > Yi > > > > > >
