Sanil15 commented on PR #26276: URL: https://github.com/apache/beam/pull/26276#issuecomment-1515095461
> General questions > > 1. Why use JSON as the serialization format when protobuf is the recommended way of serializing internal components within beam? > 2. Can we leverage existing protobuf representation of the pipeline and use that instead of JSON? i.e., populating the same configuration w/ toProto(pipeline) and deserializing from the config? > 3. Are there any potential pitfalls with varying versions of jackson across beam and samza which can cause issues? > > @xinyuiscool in case you have something to chime in on leveraging protobuf representations - IMO using protobuf might over-complicate things for a simple MapEntry<String, String> serializer - Protobuf is better since it has a schema defined but here we are just converting a list of string -> string, I wanted to to ideally use Pair<String, String>, but there is no serializer for it - PipelineJsonRenderer already uses JSON - Configs needs to be readable for debugging, if we serialize this with protobuf, the serialised string will not be readable via configs while debugging - it would need a deserializer -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
