Hi,

I intend to use Protobuf options to trigger different transforms and use
metadata from storage proto options for sink partitioning  etc.. and also
allow different protobuf message types flowing via the same pipeline,
running as different instances of the pipeline.

I am able to parse descriptors, fields and options from file descriptors
compiled externally to the beam pipeline jar.


I am not able to use dynamicMessage.getDefaultInstanceForType() in the Sink
transforms PTransform<PCollection<T>, PDone> which need a defaultInstance
of the message type to persist the data since it throws
com.google.protobuf.DynamicMessage not Serializable.

I wanted to check if there is a way to use a generic proto in a beam
pipeline and if there are any examples of protobuf reflection which can be
used in this case or if there is any recommended way to achieve this
functionality.



Many Thanks,

Amit

Reply via email to