This is a known issue and was fixed with https://github.com/apache/beam/commit/5d9bb4595c763025a369a959e18c6dd288e72314#diff-f149847d2c06f56ea591cab8d862c960
It is meant to be released as part of 2.16.0 On Tue, Aug 27, 2019 at 11:41 AM Zhiheng Huang <[email protected]> wrote: > Hi all, > > Looking for help to understand an internal error from beam dataflow runner. > > I have a streaming pipeline that is on google dataflow runner(beam version > 2.11, java). Recently I added a SplittableDoFn to my pipeline to > continuously generate a sequence. However, after the job runs for a few > hours, I start to see the following exception: > > java.util.NoSuchElementException > > org.apache.beam.vendor.guava.v20_0.com.google.common.collect.MultitransformedIterator.next(MultitransformedIterator.java:63) > > org.apache.beam.vendor.guava.v20_0.com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47) > > org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterators.getOnlyElement(Iterators.java:308) > > org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables.getOnlyElement(Iterables.java:294) > > org.apache.beam.runners.dataflow.worker.DataflowProcessFnRunner.getUnderlyingWindow(DataflowProcessFnRunner.java:98) > > org.apache.beam.runners.dataflow.worker.DataflowProcessFnRunner.placeIntoElementWindow(DataflowProcessFnRunner.java:72) > > org.apache.beam.runners.dataflow.worker.DataflowProcessFnRunner.processElement(DataflowProcessFnRunner.java:62) > > org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:325) > > org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44) > > org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49) > > org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201) > > org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159) > > org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77) > > org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1269) > > org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000(StreamingDataflowWorker.java:146) > > org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run(StreamingDataflowWorker.java:1008) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > java.lang.Thread.run(Thread.java:745) > > This is 100% reproducible and I am clueless about how this stacktrace can > be debugged since there's no pointer to user code nor the failing > PTransform. This happens only if I add the SplittableDoFn, whose rough > structure is like this > <https://gist.github.com/sylvon/cbcccdcb64aeb15002721977398dc308>. > > Appreciate any pointers on what might be going wrong or how this exception > can be debugged. > > Thanks a lot! > >
