aaltay commented on a change in pull request #11141: [BEAM-7923] Include side
effects in p.run
URL: https://github.com/apache/beam/pull/11141#discussion_r393351076
##########
File path: sdks/python/apache_beam/runners/interactive/pipeline_instrument.py
##########
@@ -418,10 +420,16 @@ def visit_transform(self, transform_node):
tuple(ie.current_env().options.capturable_sources)):
unbounded_source_pcolls.update(transform_node.outputs.values())
cacheable_inputs.update(self._pin._cacheable_inputs(transform_node))
+ ins, outs = self._pin._all_inputs_outputs(transform_node)
+ all_inputs.update(ins)
+ all_outputs.update(outs)
v = InstrumentVisitor(self)
self._pipeline.visit(v)
+ # Every output PCollection that is never used as an input PCollection is
+ # considered as a side effect of the pipeline run and should be included.
+ self._extended_targets.update(all_outputs.difference(all_inputs))
Review comment:
Do you want to track, mark side effects differently? Does users want to
specifically track these pcollections?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
