*If you are not an Interactive Beam user, you can ignore this email.* Hi Interactive Beam users,
We've recently made some changes to how Interactive Beam gets to understand the context of the pipelines/PCollections defined in your notebook/code. If you write Beam pipelines with the InteractiveRunner directly in notebook cells like the Interactive Beam Examples <https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/interactive/examples/Interactive%20Beam%20Example.ipynb> or define everything in "__main__", you will not be affected by the changes. If you define your pipelines in local scope such as functions (an example scenario, unit tests) and you rely on interactive features to introspect the data of a PCollection after a pipeline run, you might see such "raise ValueError('PCollection not available, please run the pipeline.')". It's because Interactive Beam now "watches" the "__main__" scope by default to provide features implicitly. To avoid the error, you only need to tell Interactive Beam to "watch <https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/interactive/interactive_beam.py#L36>" your local scopes too. An example to fix the issue, from apache_beam.runners.interactive import interactive_beam ... def some_func(...): p = beam.Pipeline(InteractiveRunner()) pcoll = p | 'SomeTransform' >> SomeTransform() ... interactive_beam.watch(locals()) result = p.run() ... ... Thanks for using Interactive Beam! Ning.
