Hi all, 🥳 I'd like to open the discussion on "*PIP-246: Improved PROTOBUF_NATIVE schema compatibility checks without using avro-protobuf"*.
This is a PIP from a long time ago, I want to restart it. ( https://github.com/apache/pulsar/issues/19565) *Detailed pip documentation at: *https://github.com/apache/pulsar/pull/26695 *The implemented code is at: *https://github.com/apache/pulsar/pull/19566 # Background knowledge Pulsar's schema registry stores schema versions for a topic and checks whether a proposed schema can coexist with previously registered versions. A compatibility strategy determines the direction of the check. `BACKWARD` asks whether a new reader can read data written with the previous schema; `FORWARD` asks whether the previous reader can read data written with the new schema; `FULL` requires both. Their transitive variants compare against all versions selected by the registry, rather than only the latest version. `PROTOBUF_NATIVE` stores a Protobuf schema as a serialized `FileDescriptorSet`, the root file name, and the root message's fully qualified name. The descriptor set contains the message definitions and their dependencies. The broker can reconstruct these descriptors without loading application classes or converting the schema to Avro. This representation is distinct from Pulsar's legacy `PROTOBUF` schema type. Protobuf binary messages identify fields by number and encoding, rather than by field name. Compatibility also depends on required fields, repeated fields, message types, enums, maps, and `oneof` groups. A `oneof` allows only one of its member fields to be set at a time. A closed enum exposes only its declared numeric values through the field; an open enum can also represent unrecognized numeric values. Protobuf supports `proto2`, `proto3`, and Editions. Editions express behaviors such as field presence, enum openness, and repeated-field encoding through inherited feature settings. Consequently, comparing syntax strings or raw field labels is insufficient to determine compatibility. Further background is available in the [Protobuf language guide]( https://protobuf.dev/programming-guides/proto3/#updating) and [Editions feature documentation](https://protobuf.dev/editions/features/). The existing `PROTOBUF_NATIVE` compatibility checker requires equal root message full names for the six directional strategies. It does not compare fields. Checkers are selected through the broker's `schemaRegistryCompatibilityCheckers` configuration, with one checker per schema type. # Motivation Two schemas can retain the same root message name while making incompatible changes to their fields. For example, changing `Order.id` from `int64` to `string` at field number `1` passes the root-name check. The schemas describe different encodings, and a reader may no longer obtain the expected field value. Even a successful parse does not establish safe evolution. Changing an integer field's width can truncate values, and combining independent fields into a `oneof` can discard a value. The registry does not know application value restrictions or the order in which every producer and consumer will be upgraded, so it cannot establish the additional conditions needed to permit such changes. This proposal introduces field-level compatibility checks using Protobuf descriptors directly, while retaining the existing checker as the default. It permits ordinary field additions without an explicit default and changes between packed and expanded representations of the same repeated scalar field. It rejects changes whose safety depends on assumptions the broker cannot verify. Thanks, sinan
