Sanil15 commented on PR #26276: URL: https://github.com/apache/beam/pull/26276#issuecomment-1515465360
> > * Configs needs to be readable for debugging, if we serialize this with protobuf, the serialised string will not be readable via configs while debugging - it would need a deserializer > > This is an internal configuration not for external consumption for user. I'd rather keep it hidden than have it readable. If you want it for debug purpose, you can use the debug logs to infer as opposed to exposing this configuration externally. > > > PipelineJsonRenderer already uses JSON > > It is external facing so that we can render the DAG for observability. It is not necessarily used internally. > > > IMO using protobuf might over-complicate things for a simple MapEntry<String, String> serializer > > `Map<string, string>` is not an evolvable data model and if you need additional metadata evolving the string to have record delimitters and so-on complicates evolution and compatibility handling. > > All said, If we can't leverage the protobuf representation of the pipeline, can we use `BEAM_JSON_GRAPH` which also has information about the transforms. Can that be leveraged? @mynameborat Found a way to use BEAM_JSON_GRAPH to attach transformIoInfo in addition to graphLinks and the graph there - check the ExpectedDag as the sample. - Removed Jackson with custom MapEntrySerializer we do not need it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
