For doing something before starting the pipeline, can you do it in the main
program? The only disadvantage I can see is that it wouldn't be amenable to
using templates (ValueProvider's) - is that the blocker?
For doing something after a transform finishes processing a window of a
PCollection - we
My use case is that I have generic code to transfer for example tables
from mongo to bigquery. I iterate over all tables in mongo and create
a PCollection for each. But there are things that need to be checked
before running, and to run only if validated.
I tried the visitor but there is no way to
Hi,
I don't think it makes sense on a transform (as it expects a PCollection).
However, why not introducing a specific hook for that.
I think you can workaround using a Pipeline Visitor, but it would be runner
level.
Regards
JB
On 09/14/2017 08:21 AM, Chaim Turkel wrote:
Hi,
I have a
Hi,
I have a few scenarios where I would like to have code that is
before the PBegin and after the PDone.
This is usually for monitoring purposes.
It would be nice to be able to transform from PBegin to PBegin, and
PDone to PDone, so that code can be run before and after and not in
the driver
