BEAM-6855 is still open and I updated it linking to this thread that a user is still being impacted.
On Tue, Aug 4, 2020 at 10:20 AM Mohil Khare <mo...@prosimo.io> wrote: > yeah .. looks like a bug still exists. > > So how does this work ? Shall I open a new Jira ? > > Thanks and regards > Mohil > > On Thu, Jul 30, 2020 at 10:39 PM Reuven Lax <re...@google.com> wrote: > >> I believe that the person trying to fix BEAM-6855 was unable to reproduce >> it in test, and therefore assumed that the bug was fixed. However it >> appears that the bug still exists. >> >> On Wed, Jul 29, 2020 at 10:36 AM Kenneth Knowles <k...@apache.org> wrote: >> >>> Hi Mohil, >>> >>> It helps also to tell us what version of Beam you are using and some >>> more details. This looks related to >>> https://issues.apache.org/jira/browse/BEAM-6855 which claims to be >>> resolved in 2.17.0 >>> >>> Kenn >>> >>> On Mon, Jul 27, 2020 at 11:47 PM Mohil Khare <mo...@prosimo.io> wrote: >>> >>>> Hello all, >>>> >>>> I think I found the reason for the issue. Since the exception was >>>> thrown by StreamingSideInputDoFnRunner.java, I realize that I recently >>>> added side input to one of my ParDo that does stateful transformations. >>>> It looks like there is some issue when you add side input (My side >>>> input was coming via Global window to ParDo in a Fixed Window) to stateful >>>> DoFn. >>>> >>>> As a work around, instead of adding side input to stateful ParDo, I >>>> introduced another ParDo that enriches streaming data with side input >>>> before flowing into stateful DoFn. That seems to have fixed the problem. >>>> >>>> >>>> Thanks and regards >>>> Mohil >>>> >>>> >>>> >>>> On Mon, Jul 27, 2020 at 10:50 AM Mohil Khare <mo...@prosimo.io> wrote: >>>> >>>>> Hello All, >>>>> >>>>> Any idea how to debug this and find out which stage, which DoFn or >>>>> which side input is causing the problem? >>>>> Do I need to override OnTimer with every DoFn to avoid this problem? >>>>> I thought that some uncaught exceptions were causing this and added >>>>> various checks and exception handling in all DoFn and still seeing this >>>>> issue. >>>>> It has been driving me nuts. And now forget DRAIN, it happens during >>>>> normal functioning as well. Any help would be appreciated. >>>>> >>>>> java.lang.UnsupportedOperationException: Attempt to deliver a timer to >>>>> a DoFn, but timers are not supported in Dataflow. >>>>> >>>>> 1. >>>>> 1. at org.apache.beam.runners.dataflow.worker. >>>>> StreamingSideInputDoFnRunner.onTimer ( >>>>> StreamingSideInputDoFnRunner.java:86 >>>>> >>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingSideInputDoFnRunner.java&line=86&project=prosimo-test> >>>>> ) >>>>> 2. at org.apache.beam.runners.dataflow.worker. >>>>> SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360 >>>>> >>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=360&project=prosimo-test> >>>>> ) >>>>> 3. at org.apache.beam.runners.dataflow.worker. >>>>> SimpleParDoFn.access$600 (SimpleParDoFn.java:73 >>>>> >>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=73&project=prosimo-test> >>>>> ) >>>>> 4. at org.apache.beam.runners.dataflow.worker. >>>>> SimpleParDoFn$TimerType$1.processTimer (SimpleParDoFn.java:444 >>>>> >>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=444&project=prosimo-test> >>>>> ) >>>>> 5. at org.apache.beam.runners.dataflow.worker. >>>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:473 >>>>> >>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=473&project=prosimo-test> >>>>> ) >>>>> 6. at org.apache.beam.runners.dataflow.worker. >>>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:353 >>>>> >>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=353&project=prosimo-test> >>>>> ) >>>>> 7. at >>>>> org.apache.beam.runners.dataflow.worker.util.common.worker. >>>>> ParDoOperation.finish (ParDoOperation.java:52 >>>>> >>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FParDoOperation.java&line=52&project=prosimo-test> >>>>> ) >>>>> 8. at >>>>> org.apache.beam.runners.dataflow.worker.util.common.worker. >>>>> MapTaskExecutor.execute (MapTaskExecutor.java:85 >>>>> >>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FMapTaskExecutor.java&line=85&project=prosimo-test> >>>>> ) >>>>> 9. at org.apache.beam.runners.dataflow.worker. >>>>> StreamingDataflowWorker.process ( >>>>> StreamingDataflowWorker.java:1350 >>>>> >>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=1350&project=prosimo-test> >>>>> ) >>>>> 10. at org.apache.beam.runners.dataflow.worker. >>>>> StreamingDataflowWorker.access$1100 ( >>>>> StreamingDataflowWorker.java:152 >>>>> >>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=152&project=prosimo-test> >>>>> ) >>>>> 2. >>>>> 1. at org.apache.beam.runners.dataflow.worker. >>>>> StreamingDataflowWorker$7.run (StreamingDataflowWorker.java:1073 >>>>> ) >>>>> 2. at java.util.concurrent.ThreadPoolExecutor.runWorker ( >>>>> ThreadPoolExecutor.java:1149) >>>>> 3. at java.util.concurrent.ThreadPoolExecutor$Worker.run ( >>>>> ThreadPoolExecutor.java:624) >>>>> 4. at java.lang.Thread.run (Thread.java:748) >>>>> >>>>> Thanks >>>>> Mohil >>>>> >>>>> >>>>> On Sun, Jul 26, 2020 at 1:50 PM Mohil Khare <mo...@prosimo.io> wrote: >>>>> >>>>>> and it seems be due to TimerType User >>>>>> >>>>>> Thanks >>>>>> Mohil >>>>>> >>>>>> On Sun, Jul 26, 2020 at 1:42 PM Mohil Khare <mo...@prosimo.io> wrote: >>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> I was looking at source code of >>>>>>> https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingSideInputDoFnRunner.java >>>>>>> , >>>>>>> >>>>>>> It seems the default implementation of OnTimer is to throw (Attempt >>>>>>> to deliver a timer to a DoFn, but timers are not supported in Dataflow). >>>>>>> >>>>>>> Do you know under what circumstances, My code might be throwing this >>>>>>> ? Not sure if its some issue in 2.19 which might have been fixed now >>>>>>> with >>>>>>> 2.22 >>>>>>> >>>>>>> Thanks and Regards >>>>>>> Mohil >>>>>>> >>>>>>> >>>>>>> On Fri, Jul 24, 2020 at 5:21 PM Mohil Khare <mo...@prosimo.io> >>>>>>> wrote: >>>>>>> >>>>>>>> Actually NOT JUST DRAIN, seeing it during regular RUN as well. >>>>>>>> It's getting flooded with this exception >>>>>>>> >>>>>>>> Attempt to deliver a timer to a DoFn, but timers are not supported >>>>>>>> in Dataflow. >>>>>>>> The changes that I did are the following: >>>>>>>> 1. Read one set of logs from one of the kafka topics and create >>>>>>>> KV(ID, log1) >>>>>>>> 2. Read 2nd set of logs from another kafka topic and create KV(ID, >>>>>>>> log2) >>>>>>>> 3. Put above in session window >>>>>>>> 4. CoGroupByKey both logs >>>>>>>> >>>>>>>> Apart from aforementioned exceptions, I am also seeing the >>>>>>>> following in worker logs. What I have noticed is that it happens when >>>>>>>> the >>>>>>>> pipeline is silent for a while. i.e. no new logs to be read from Kafka >>>>>>>> (Not >>>>>>>> sure if it is the actual reason). >>>>>>>> >>>>>>>> 2020-07-24 17:06:43.532 PDT >>>>>>>> Execution of work for P188 for key >>>>>>>> cloud@prosimo.ioHe8fc8079-c844-11ea-a6d5-dabe1eb9c630 failed. Will >>>>>>>> retry locally. >>>>>>>> >>>>>>>> <https://console.cloud.google.com/logs/viewer?advancedFilter=insertId%3D%224729149996009968456:27267:0:42419165%22%20resource.type%3D%22dataflow_step%22%20resource.labels.job_id%3D%222020-07-24_16_23_20-4320159532106629306%22%20(logName%3D%22projects%2Fprosimo-test%2Flogs%2Fdataflow.googleapis.com%252Fworker%22)%20severity%3E%3DERROR%0A(timestamp%3E%222020-07-25T00:06:33.224Z%22%20OR%20(insertId%3E%224729149996009968456:27267:0:42264272%22%20AND%20timestamp%3D%222020-07-25T00:06:33.224Z%22))&expandAll=true&interval=JUMP_TO_TIME&dateRangeStart=2020-07-25T00:06:43.532Z&dateRangeEnd=2020-07-25T00:06:43.532Z&project=prosimo-test> >>>>>>>> 2020-07-24 17:06:53.863 PDT >>>>>>>> Uncaught exception: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Jul 24, 2020 at 2:59 PM Mohil Khare <mo...@prosimo.io> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> I am on java sdk 2.19 and using dataflow for beam job. I use >>>>>>>>> Timers for my stateful transformations, but recently I started seeing >>>>>>>>> the >>>>>>>>> following exception on DRAINING a job. It used to work fine and not >>>>>>>>> sure >>>>>>>>> what changed. >>>>>>>>> >>>>>>>>> *java.lang.UnsupportedOperationException:* >>>>>>>>> >>>>>>>> *Attempt to deliver a timer to a DoFn, but timers are not supported >>>>>>>> in Dataflow. * >>>>>>>> >>>>>>>>> >>>>>>>>> 1. >>>>>>>>> 1. atorg.apache.beam.runners.dataflow.worker. >>>>>>>>> StreamingSideInputDoFnRunner.onTimer ( >>>>>>>>> StreamingSideInputDoFnRunner.java:86 >>>>>>>>> >>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingSideInputDoFnRunner.java&line=86&project=prosimo-test> >>>>>>>>> ) >>>>>>>>> 2. atorg.apache.beam.runners.dataflow.worker. >>>>>>>>> SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360 >>>>>>>>> >>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=360&project=prosimo-test> >>>>>>>>> ) >>>>>>>>> 3. atorg.apache.beam.runners.dataflow.worker. >>>>>>>>> SimpleParDoFn.access$600 (SimpleParDoFn.java:73 >>>>>>>>> >>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=73&project=prosimo-test> >>>>>>>>> ) >>>>>>>>> 4. atorg.apache.beam.runners.dataflow.worker. >>>>>>>>> SimpleParDoFn$TimerType$1.processTimer ( >>>>>>>>> SimpleParDoFn.java:444 >>>>>>>>> >>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=444&project=prosimo-test> >>>>>>>>> ) >>>>>>>>> 5. atorg.apache.beam.runners.dataflow.worker. >>>>>>>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:473 >>>>>>>>> >>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=473&project=prosimo-test> >>>>>>>>> ) >>>>>>>>> 6. atorg.apache.beam.runners.dataflow.worker. >>>>>>>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:353 >>>>>>>>> >>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=353&project=prosimo-test> >>>>>>>>> ) >>>>>>>>> 7. at >>>>>>>>> org.apache.beam.runners.dataflow.worker.util.common.worker. >>>>>>>>> ParDoOperation.finish (ParDoOperation.java:52 >>>>>>>>> >>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FParDoOperation.java&line=52&project=prosimo-test> >>>>>>>>> ) >>>>>>>>> 8. at >>>>>>>>> org.apache.beam.runners.dataflow.worker.util.common.worker. >>>>>>>>> MapTaskExecutor.execute (MapTaskExecutor.java:85 >>>>>>>>> >>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FMapTaskExecutor.java&line=85&project=prosimo-test> >>>>>>>>> ) >>>>>>>>> 9. atorg.apache.beam.runners.dataflow.worker. >>>>>>>>> StreamingDataflowWorker.process ( >>>>>>>>> StreamingDataflowWorker.java:1350 >>>>>>>>> >>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=1350&project=prosimo-test> >>>>>>>>> ) >>>>>>>>> 10. atorg.apache.beam.runners.dataflow.worker. >>>>>>>>> StreamingDataflowWorker.access$1100 ( >>>>>>>>> StreamingDataflowWorker.java:152 >>>>>>>>> >>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=152&project=prosimo-test> >>>>>>>>> ) >>>>>>>>> 2. Sometime back I opened jira for an issue which is related >>>>>>>>> to this while doing DRAIN: >>>>>>>>> https://issues.apache.org/jira/browse/BEAM-10053 (Looks like >>>>>>>>> no one has taken a stab on this Jira) >>>>>>>>> 3. Not sure if the reason is same and due to multiple side >>>>>>>>> inputs that i use in my PTransforms. >>>>>>>>> >>>>>>>>> >>>>>>>>> Any help would be appreciated. >>>>>>>>> >>>>>>>>> Thanks and Regards >>>>>>>>> Mohil >>>>>>>>> >>>>>>>>>
