gortiz opened a new pull request, #11753: URL: https://github.com/apache/pinot/pull/11753
Recently we had some issues reading nulls from optional proto fields. After several tests, we found that the actual issue was using a very old confluent version (5.5.3). Before 7.1.x, `kafka-protobuf-serializer` was not able to read optional fields in proto3 and instead relay on the producer rewriting the proto descriptor in order to change all optional fields to oneof. That is a requirement in older versions of proto but it is not a requirement in modern ones. This transformation is not required in `kafka-protobuf-serializer` in 7.1.x and higher and therefore if the producer uses one of these modern versions, the rewrite is not applied. That means that if the consumer used `kafka-protobuf-serializer` >= 7.1.x, Pinot is not able to recognize optional fields, which had the lateral effect of either detecting default values (like 0 for ints) as null (which happened before https://github.com/apache/pinot/pull/11723) or transforming actual nulls to default values (which happened after https://github.com/apache/pinot/pull/11723) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
