The write can be done as a normal ParDo / DoFn. The read needs to expose some watermark logic, which at the time of writing wasn't available, since no unbounded source API was available. We may be able to write the read / source as a SplittableDoFn since that API was introduced as an unbounded source API.
On Sat, Apr 28, 2018, 5:56 AM Udi Meiri <eh...@google.com> wrote: > Hi, > I'm having trouble understanding why there's an extra level of indirection > when doing pubsub reads via directrunner vs writes. > > For reads, we have these translations: > beam_pubsub.ReadFromPubSub -> > direct_runner._DirectReadFromPubSub -> > transform_evaluator._PubSubReadEvaluator > > For writes, this is abbreviated: > beam_pubsub.WriteStringsToPubSub -> > _DirectWriteToPubSub > > What is the role of transform_evaluator._TransformEvaluator? > Why do we need it for reads and not for writes? > >
