IMO returning the input PCollection in a PTransform should be a valid albeit trivial use case. I have put a suggested fix for supporting these kinds of transforms in the interactive runner here: https://github.com/apache/beam/pull/9865 . I'm also new to beam dev so if there's something I'm missing please let me know :-)
On Mon, Oct 21, 2019 at 4:46 PM Robert Bradshaw <rober...@google.com> wrote: > > Thanks for trying this out. Yes, this is definitely something that > should be supported (and tested). > > On Mon, Oct 21, 2019 at 3:40 PM Igor Durovic <id.te...@gmail.com> wrote: > > > > Hi everyone, > > > > The interactive beam example using the DirectRunner fails after execution > > of the last cell. The recursion limit is exceeded during the calculation of > > the cache label because of a circular reference in the PipelineInfo object. > > > > The constructor for the PipelineInfo class creates a mapping from each > > pcollection to the transforms that produce and consume it. The issue arises > > when there exists a transform that is both a producer and a consumer for > > the same pcollection. This occurs when a transform's expand method returns > > the same pcoll object that's passed into it. The specific transform causing > > the failure of the example is MaybeReshuffle, which is used in the Create > > transform. Replacing "return pcoll" with "return pcoll | Map(lambda x: x)" > > seems to fix the problem. > > > > A workaround for this issue on the interactive beam side would be fairly > > simple, but it seems to me that there should be more validation of > > pipelines to prevent the use of transforms that return the same pcoll > > that's passed in, or at least a mention of this in the transform style > > guide. My understanding is that pcollections are produced by a single > > transform (they even have a field called "producer" that references only > > one transform). If that's the case then that property of pcollections > > should be enforced. > > > > I made ticket BEAM-8451 to track this issue. > > > > I'm still new to beam so I apologize if I'm fundamentally misunderstanding > > something. I'm not exactly sure what the next step should be and would > > appreciate some recommendations. I can submit a PR to solve the immediate > > problem of the failing example but the underlying problem should also be > > addressed at some point. I also apologize if people are already aware of > > this problem. > > > > Thank You! > > Igor Durovic -- *Confidentiality Note:* We care about protecting our proprietary information, confidential material, and trade secrets. This message may contain some or all of those things. Cruise will suffer material harm if anyone other than the intended recipient disseminates or takes any action based on this message. If you have received this message (including any attachments) in error, please delete it immediately and notify the sender promptly.
