Re: Exceptions: Attempt to deliver a timer to a DoFn, but timers are not supported in Dataflow.

[Mohil Khare]

Thanks a lot Luke..

Regards
Mohil

On Tue, Aug 4, 2020 at 12:01 PM Luke Cwik  wrote:

> BEAM-6855 is still open and I updated it linking to this thread that a
> user is still being impacted.
>
> On Tue, Aug 4, 2020 at 10:20 AM Mohil Khare  wrote:
>
>> yeah .. looks like a bug still exists.
>>
>> So how does this work ? Shall I open a new Jira ?
>>
>> Thanks and regards
>> Mohil
>>
>> On Thu, Jul 30, 2020 at 10:39 PM Reuven Lax  wrote:
>>
>>> I believe that the person trying to fix BEAM-6855 was unable to
>>> reproduce it in test, and therefore assumed that the bug was fixed. However
>>> it appears that the bug still exists.
>>>
>>> On Wed, Jul 29, 2020 at 10:36 AM Kenneth Knowles 
>>> wrote:
>>>
 Hi Mohil,

 It helps also to tell us what version of Beam you are using and some
 more details. This looks related to
 https://issues.apache.org/jira/browse/BEAM-6855 which claims to be
 resolved in 2.17.0

 Kenn

 On Mon, Jul 27, 2020 at 11:47 PM Mohil Khare  wrote:

> Hello all,
>
> I think I found the reason for the issue.  Since the exception was
> thrown by StreamingSideInputDoFnRunner.java, I realize that I recently
> added side input to one of my ParDo that does stateful transformations.
> It looks like there is some issue when you add side input (My side
> input was coming via Global window to ParDo in a Fixed Window) to stateful
> DoFn.
>
> As a work around, instead of adding side input to stateful ParDo, I
> introduced another ParDo  that enriches streaming data with side input
> before flowing into stateful DoFn. That seems to have fixed the problem.
>
>
> Thanks and regards
> Mohil
>
>
>
> On Mon, Jul 27, 2020 at 10:50 AM Mohil Khare  wrote:
>
>> Hello All,
>>
>> Any idea how to debug this and find out which stage, which DoFn or
>> which side input is causing the problem?
>> Do I need to override OnTimer with every DoFn to avoid this problem?
>> I thought that some uncaught exceptions were causing this and added
>> various checks and exception handling in all DoFn and still seeing this
>> issue.
>> It has been driving me nuts. And now forget DRAIN, it happens during
>> normal functioning as well. Any help would be appreciated.
>>
>> java.lang.UnsupportedOperationException: Attempt to deliver a timer
>> to a DoFn, but timers are not supported in Dataflow.
>>
>>1.
>>   1. at org.apache.beam.runners.dataflow.worker.
>>   StreamingSideInputDoFnRunner.onTimer (
>>   StreamingSideInputDoFnRunner.java:86
>>   
>> 
>>   )
>>   2. at org.apache.beam.runners.dataflow.worker.
>>   SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360
>>   
>> 
>>   )
>>   3. at org.apache.beam.runners.dataflow.worker.
>>   SimpleParDoFn.access$600 (SimpleParDoFn.java:73
>>   
>> 
>>   )
>>   4. at org.apache.beam.runners.dataflow.worker.
>>   SimpleParDoFn$TimerType$1.processTimer (SimpleParDoFn.java:444
>>   
>> 
>>   )
>>   5. at org.apache.beam.runners.dataflow.worker.
>>   SimpleParDoFn.processTimers (SimpleParDoFn.java:473
>>   
>> 
>>   )
>>   6. at org.apache.beam.runners.dataflow.worker.
>>   SimpleParDoFn.processTimers (SimpleParDoFn.java:353
>>   
>> 
>>   )
>>   7. at
>>   org.apache.beam.runners.dataflow.worker.util.common.worker.
>>   ParDoOperation.finish (ParDoOperation.java:52
>>   
>> 

Re: Exceptions: Attempt to deliver a timer to a DoFn, but timers are not supported in Dataflow.

[Luke Cwik]

BEAM-6855 is still open and I updated it linking to this thread that a user
is still being impacted.

On Tue, Aug 4, 2020 at 10:20 AM Mohil Khare  wrote:

> yeah .. looks like a bug still exists.
>
> So how does this work ? Shall I open a new Jira ?
>
> Thanks and regards
> Mohil
>
> On Thu, Jul 30, 2020 at 10:39 PM Reuven Lax  wrote:
>
>> I believe that the person trying to fix BEAM-6855 was unable to reproduce
>> it in test, and therefore assumed that the bug was fixed. However it
>> appears that the bug still exists.
>>
>> On Wed, Jul 29, 2020 at 10:36 AM Kenneth Knowles  wrote:
>>
>>> Hi Mohil,
>>>
>>> It helps also to tell us what version of Beam you are using and some
>>> more details. This looks related to
>>> https://issues.apache.org/jira/browse/BEAM-6855 which claims to be
>>> resolved in 2.17.0
>>>
>>> Kenn
>>>
>>> On Mon, Jul 27, 2020 at 11:47 PM Mohil Khare  wrote:
>>>
 Hello all,

 I think I found the reason for the issue.  Since the exception was
 thrown by StreamingSideInputDoFnRunner.java, I realize that I recently
 added side input to one of my ParDo that does stateful transformations.
 It looks like there is some issue when you add side input (My side
 input was coming via Global window to ParDo in a Fixed Window) to stateful
 DoFn.

 As a work around, instead of adding side input to stateful ParDo, I
 introduced another ParDo  that enriches streaming data with side input
 before flowing into stateful DoFn. That seems to have fixed the problem.


 Thanks and regards
 Mohil



 On Mon, Jul 27, 2020 at 10:50 AM Mohil Khare  wrote:

> Hello All,
>
> Any idea how to debug this and find out which stage, which DoFn or
> which side input is causing the problem?
> Do I need to override OnTimer with every DoFn to avoid this problem?
> I thought that some uncaught exceptions were causing this and added
> various checks and exception handling in all DoFn and still seeing this
> issue.
> It has been driving me nuts. And now forget DRAIN, it happens during
> normal functioning as well. Any help would be appreciated.
>
> java.lang.UnsupportedOperationException: Attempt to deliver a timer to
> a DoFn, but timers are not supported in Dataflow.
>
>1.
>   1. at org.apache.beam.runners.dataflow.worker.
>   StreamingSideInputDoFnRunner.onTimer (
>   StreamingSideInputDoFnRunner.java:86
>   
> 
>   )
>   2. at org.apache.beam.runners.dataflow.worker.
>   SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360
>   
> 
>   )
>   3. at org.apache.beam.runners.dataflow.worker.
>   SimpleParDoFn.access$600 (SimpleParDoFn.java:73
>   
> 
>   )
>   4. at org.apache.beam.runners.dataflow.worker.
>   SimpleParDoFn$TimerType$1.processTimer (SimpleParDoFn.java:444
>   
> 
>   )
>   5. at org.apache.beam.runners.dataflow.worker.
>   SimpleParDoFn.processTimers (SimpleParDoFn.java:473
>   
> 
>   )
>   6. at org.apache.beam.runners.dataflow.worker.
>   SimpleParDoFn.processTimers (SimpleParDoFn.java:353
>   
> 
>   )
>   7. at
>   org.apache.beam.runners.dataflow.worker.util.common.worker.
>   ParDoOperation.finish (ParDoOperation.java:52
>   
> 
>   )
>   8. at
>   org.apache.beam.runners.dataflow.worker.util.common.worker.
> 

Re: Exceptions: Attempt to deliver a timer to a DoFn, but timers are not supported in Dataflow.

[Mohil Khare]

yeah .. looks like a bug still exists.

So how does this work ? Shall I open a new Jira ?

Thanks and regards
Mohil

On Thu, Jul 30, 2020 at 10:39 PM Reuven Lax  wrote:

> I believe that the person trying to fix BEAM-6855 was unable to reproduce
> it in test, and therefore assumed that the bug was fixed. However it
> appears that the bug still exists.
>
> On Wed, Jul 29, 2020 at 10:36 AM Kenneth Knowles  wrote:
>
>> Hi Mohil,
>>
>> It helps also to tell us what version of Beam you are using and some more
>> details. This looks related to
>> https://issues.apache.org/jira/browse/BEAM-6855 which claims to be
>> resolved in 2.17.0
>>
>> Kenn
>>
>> On Mon, Jul 27, 2020 at 11:47 PM Mohil Khare  wrote:
>>
>>> Hello all,
>>>
>>> I think I found the reason for the issue.  Since the exception was
>>> thrown by StreamingSideInputDoFnRunner.java, I realize that I recently
>>> added side input to one of my ParDo that does stateful transformations.
>>> It looks like there is some issue when you add side input (My side input
>>> was coming via Global window to ParDo in a Fixed Window) to stateful DoFn.
>>>
>>> As a work around, instead of adding side input to stateful ParDo, I
>>> introduced another ParDo  that enriches streaming data with side input
>>> before flowing into stateful DoFn. That seems to have fixed the problem.
>>>
>>>
>>> Thanks and regards
>>> Mohil
>>>
>>>
>>>
>>> On Mon, Jul 27, 2020 at 10:50 AM Mohil Khare  wrote:
>>>
 Hello All,

 Any idea how to debug this and find out which stage, which DoFn or
 which side input is causing the problem?
 Do I need to override OnTimer with every DoFn to avoid this problem?
 I thought that some uncaught exceptions were causing this and added
 various checks and exception handling in all DoFn and still seeing this
 issue.
 It has been driving me nuts. And now forget DRAIN, it happens during
 normal functioning as well. Any help would be appreciated.

 java.lang.UnsupportedOperationException: Attempt to deliver a timer to
 a DoFn, but timers are not supported in Dataflow.

1.
   1. at org.apache.beam.runners.dataflow.worker.
   StreamingSideInputDoFnRunner.onTimer (
   StreamingSideInputDoFnRunner.java:86
   
 
   )
   2. at org.apache.beam.runners.dataflow.worker.
   SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360
   
 
   )
   3. at org.apache.beam.runners.dataflow.worker.
   SimpleParDoFn.access$600 (SimpleParDoFn.java:73
   
 
   )
   4. at org.apache.beam.runners.dataflow.worker.
   SimpleParDoFn$TimerType$1.processTimer (SimpleParDoFn.java:444
   
 
   )
   5. at org.apache.beam.runners.dataflow.worker.
   SimpleParDoFn.processTimers (SimpleParDoFn.java:473
   
 
   )
   6. at org.apache.beam.runners.dataflow.worker.
   SimpleParDoFn.processTimers (SimpleParDoFn.java:353
   
 
   )
   7. at org.apache.beam.runners.dataflow.worker.util.common.worker.
   ParDoOperation.finish (ParDoOperation.java:52
   
 
   )
   8. at org.apache.beam.runners.dataflow.worker.util.common.worker.
   MapTaskExecutor.execute (MapTaskExecutor.java:85
   
 
 

Re: Exceptions: Attempt to deliver a timer to a DoFn, but timers are not supported in Dataflow.

[Mohil Khare]

Hi Kenneth,
I am on beam java sdk 2.19 With enableStreamingEngine set to true and using
default machine type (n1-standard-2).

Thanks and regards
Mohil



On Wed, Jul 29, 2020 at 10:36 AM Kenneth Knowles  wrote:

> Hi Mohil,
>
> It helps also to tell us what version of Beam you are using and some more
> details. This looks related to
> https://issues.apache.org/jira/browse/BEAM-6855 which claims to be
> resolved in 2.17.0
>
> Kenn
>
> On Mon, Jul 27, 2020 at 11:47 PM Mohil Khare  wrote:
>
>> Hello all,
>>
>> I think I found the reason for the issue.  Since the exception was thrown
>> by StreamingSideInputDoFnRunner.java, I realize that I recently added side
>> input to one of my ParDo that does stateful transformations.
>> It looks like there is some issue when you add side input (My side input
>> was coming via Global window to ParDo in a Fixed Window) to stateful DoFn.
>>
>> As a work around, instead of adding side input to stateful ParDo, I
>> introduced another ParDo  that enriches streaming data with side input
>> before flowing into stateful DoFn. That seems to have fixed the problem.
>>
>>
>> Thanks and regards
>> Mohil
>>
>>
>>
>> On Mon, Jul 27, 2020 at 10:50 AM Mohil Khare  wrote:
>>
>>> Hello All,
>>>
>>> Any idea how to debug this and find out which stage, which DoFn or which
>>> side input is causing the problem?
>>> Do I need to override OnTimer with every DoFn to avoid this problem?
>>> I thought that some uncaught exceptions were causing this and added
>>> various checks and exception handling in all DoFn and still seeing this
>>> issue.
>>> It has been driving me nuts. And now forget DRAIN, it happens during
>>> normal functioning as well. Any help would be appreciated.
>>>
>>> java.lang.UnsupportedOperationException: Attempt to deliver a timer to a
>>> DoFn, but timers are not supported in Dataflow.
>>>
>>>1.
>>>   1. at org.apache.beam.runners.dataflow.worker.
>>>   StreamingSideInputDoFnRunner.onTimer (
>>>   StreamingSideInputDoFnRunner.java:86
>>>   
>>> 
>>>   )
>>>   2. at org.apache.beam.runners.dataflow.worker.
>>>   SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360
>>>   
>>> 
>>>   )
>>>   3. at org.apache.beam.runners.dataflow.worker.
>>>   SimpleParDoFn.access$600 (SimpleParDoFn.java:73
>>>   
>>> 
>>>   )
>>>   4. at org.apache.beam.runners.dataflow.worker.
>>>   SimpleParDoFn$TimerType$1.processTimer (SimpleParDoFn.java:444
>>>   
>>> 
>>>   )
>>>   5. at org.apache.beam.runners.dataflow.worker.
>>>   SimpleParDoFn.processTimers (SimpleParDoFn.java:473
>>>   
>>> 
>>>   )
>>>   6. at org.apache.beam.runners.dataflow.worker.
>>>   SimpleParDoFn.processTimers (SimpleParDoFn.java:353
>>>   
>>> 
>>>   )
>>>   7. at org.apache.beam.runners.dataflow.worker.util.common.worker.
>>>   ParDoOperation.finish (ParDoOperation.java:52
>>>   
>>> 
>>>   )
>>>   8. at org.apache.beam.runners.dataflow.worker.util.common.worker.
>>>   MapTaskExecutor.execute (MapTaskExecutor.java:85
>>>   
>>> 
>>>   )
>>>   9. at org.apache.beam.runners.dataflow.worker.
>>>   StreamingDataflowWorker.process (StreamingDataflowWorker.java:1350
>>>   
>>> 

Re: Exceptions: Attempt to deliver a timer to a DoFn, but timers are not supported in Dataflow.

[Kenneth Knowles]

Hi Mohil,

It helps also to tell us what version of Beam you are using and some more
details. This looks related to
https://issues.apache.org/jira/browse/BEAM-6855 which claims to be resolved
in 2.17.0

Kenn

On Mon, Jul 27, 2020 at 11:47 PM Mohil Khare  wrote:

> Hello all,
>
> I think I found the reason for the issue.  Since the exception was thrown
> by StreamingSideInputDoFnRunner.java, I realize that I recently added side
> input to one of my ParDo that does stateful transformations.
> It looks like there is some issue when you add side input (My side input
> was coming via Global window to ParDo in a Fixed Window) to stateful DoFn.
>
> As a work around, instead of adding side input to stateful ParDo, I
> introduced another ParDo  that enriches streaming data with side input
> before flowing into stateful DoFn. That seems to have fixed the problem.
>
>
> Thanks and regards
> Mohil
>
>
>
> On Mon, Jul 27, 2020 at 10:50 AM Mohil Khare  wrote:
>
>> Hello All,
>>
>> Any idea how to debug this and find out which stage, which DoFn or which
>> side input is causing the problem?
>> Do I need to override OnTimer with every DoFn to avoid this problem?
>> I thought that some uncaught exceptions were causing this and added
>> various checks and exception handling in all DoFn and still seeing this
>> issue.
>> It has been driving me nuts. And now forget DRAIN, it happens during
>> normal functioning as well. Any help would be appreciated.
>>
>> java.lang.UnsupportedOperationException: Attempt to deliver a timer to a
>> DoFn, but timers are not supported in Dataflow.
>>
>>1.
>>   1. at org.apache.beam.runners.dataflow.worker.
>>   StreamingSideInputDoFnRunner.onTimer (
>>   StreamingSideInputDoFnRunner.java:86
>>   
>> 
>>   )
>>   2. at org.apache.beam.runners.dataflow.worker.
>>   SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360
>>   
>> 
>>   )
>>   3. at org.apache.beam.runners.dataflow.worker.
>>   SimpleParDoFn.access$600 (SimpleParDoFn.java:73
>>   
>> 
>>   )
>>   4. at org.apache.beam.runners.dataflow.worker.
>>   SimpleParDoFn$TimerType$1.processTimer (SimpleParDoFn.java:444
>>   
>> 
>>   )
>>   5. at org.apache.beam.runners.dataflow.worker.
>>   SimpleParDoFn.processTimers (SimpleParDoFn.java:473
>>   
>> 
>>   )
>>   6. at org.apache.beam.runners.dataflow.worker.
>>   SimpleParDoFn.processTimers (SimpleParDoFn.java:353
>>   
>> 
>>   )
>>   7. at org.apache.beam.runners.dataflow.worker.util.common.worker.
>>   ParDoOperation.finish (ParDoOperation.java:52
>>   
>> 
>>   )
>>   8. at org.apache.beam.runners.dataflow.worker.util.common.worker.
>>   MapTaskExecutor.execute (MapTaskExecutor.java:85
>>   
>> 
>>   )
>>   9. at org.apache.beam.runners.dataflow.worker.
>>   StreamingDataflowWorker.process (StreamingDataflowWorker.java:1350
>>   
>> 
>>   )
>>   10. at org.apache.beam.runners.dataflow.worker.
>>   StreamingDataflowWorker.access$1100 (
>>   StreamingDataflowWorker.java:152
>>   
>> 

Re: Exceptions: Attempt to deliver a timer to a DoFn, but timers are not supported in Dataflow.

[Mohil Khare]

Hello all,

I think I found the reason for the issue.  Since the exception was thrown
by StreamingSideInputDoFnRunner.java, I realize that I recently added side
input to one of my ParDo that does stateful transformations.
It looks like there is some issue when you add side input (My side input
was coming via Global window to ParDo in a Fixed Window) to stateful DoFn.

As a work around, instead of adding side input to stateful ParDo, I
introduced another ParDo  that enriches streaming data with side input
before flowing into stateful DoFn. That seems to have fixed the problem.


Thanks and regards
Mohil



On Mon, Jul 27, 2020 at 10:50 AM Mohil Khare  wrote:

> Hello All,
>
> Any idea how to debug this and find out which stage, which DoFn or which
> side input is causing the problem?
> Do I need to override OnTimer with every DoFn to avoid this problem?
> I thought that some uncaught exceptions were causing this and added
> various checks and exception handling in all DoFn and still seeing this
> issue.
> It has been driving me nuts. And now forget DRAIN, it happens during
> normal functioning as well. Any help would be appreciated.
>
> java.lang.UnsupportedOperationException: Attempt to deliver a timer to a
> DoFn, but timers are not supported in Dataflow.
>
>1.
>   1. at org.apache.beam.runners.dataflow.worker.
>   StreamingSideInputDoFnRunner.onTimer (
>   StreamingSideInputDoFnRunner.java:86
>   
> 
>   )
>   2. at org.apache.beam.runners.dataflow.worker.
>   SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360
>   
> 
>   )
>   3. at org.apache.beam.runners.dataflow.worker.
>   SimpleParDoFn.access$600 (SimpleParDoFn.java:73
>   
> 
>   )
>   4. at org.apache.beam.runners.dataflow.worker.
>   SimpleParDoFn$TimerType$1.processTimer (SimpleParDoFn.java:444
>   
> 
>   )
>   5. at org.apache.beam.runners.dataflow.worker.
>   SimpleParDoFn.processTimers (SimpleParDoFn.java:473
>   
> 
>   )
>   6. at org.apache.beam.runners.dataflow.worker.
>   SimpleParDoFn.processTimers (SimpleParDoFn.java:353
>   
> 
>   )
>   7. at org.apache.beam.runners.dataflow.worker.util.common.worker.
>   ParDoOperation.finish (ParDoOperation.java:52
>   
> 
>   )
>   8. at org.apache.beam.runners.dataflow.worker.util.common.worker.
>   MapTaskExecutor.execute (MapTaskExecutor.java:85
>   
> 
>   )
>   9. at org.apache.beam.runners.dataflow.worker.
>   StreamingDataflowWorker.process (StreamingDataflowWorker.java:1350
>   
> 
>   )
>   10. at org.apache.beam.runners.dataflow.worker.
>   StreamingDataflowWorker.access$1100 (
>   StreamingDataflowWorker.java:152
>   
> 
>   )
>2.
>   1. at org.apache.beam.runners.dataflow.worker.
>   StreamingDataflowWorker$7.run (StreamingDataflowWorker.java:1073)
>   2. at java.util.concurrent.ThreadPoolExecutor.runWorker (
>   

Exceptions: Attempt to deliver a timer to a DoFn, but timers are not supported in Dataflow.

[Mohil Khare]

Hello All,

Any idea how to debug this and find out which stage, which DoFn or which
side input is causing the problem?
Do I need to override OnTimer with every DoFn to avoid this problem?
I thought that some uncaught exceptions were causing this and added various
checks and exception handling in all DoFn and still seeing this issue.
It has been driving me nuts. And now forget DRAIN, it happens during normal
functioning as well. Any help would be appreciated.

java.lang.UnsupportedOperationException: Attempt to deliver a timer to a
DoFn, but timers are not supported in Dataflow.

   1.
  1. at org.apache.beam.runners.dataflow.worker.
  StreamingSideInputDoFnRunner.onTimer (
  StreamingSideInputDoFnRunner.java:86
  

  )
  2. at org.apache.beam.runners.dataflow.worker.
  SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360
  

  )
  3. at org.apache.beam.runners.dataflow.worker.SimpleParDoFn.access$600
   (SimpleParDoFn.java:73
  

  )
  4. at org.apache.beam.runners.dataflow.worker.
  SimpleParDoFn$TimerType$1.processTimer (SimpleParDoFn.java:444
  

  )
  5. at org.apache.beam.runners.dataflow.worker.
  SimpleParDoFn.processTimers (SimpleParDoFn.java:473
  

  )
  6. at org.apache.beam.runners.dataflow.worker.
  SimpleParDoFn.processTimers (SimpleParDoFn.java:353
  

  )
  7. at org.apache.beam.runners.dataflow.worker.util.common.worker.
  ParDoOperation.finish (ParDoOperation.java:52
  

  )
  8. at org.apache.beam.runners.dataflow.worker.util.common.worker.
  MapTaskExecutor.execute (MapTaskExecutor.java:85
  

  )
  9. at org.apache.beam.runners.dataflow.worker.
  StreamingDataflowWorker.process (StreamingDataflowWorker.java:1350
  

  )
  10. at org.apache.beam.runners.dataflow.worker.
  StreamingDataflowWorker.access$1100 (StreamingDataflowWorker.java:152
  

  )
   2.
  1. at org.apache.beam.runners.dataflow.worker.
  StreamingDataflowWorker$7.run (StreamingDataflowWorker.java:1073)
  2. at java.util.concurrent.ThreadPoolExecutor.runWorker (
  ThreadPoolExecutor.java:1149)
  3. at java.util.concurrent.ThreadPoolExecutor$Worker.run (
  ThreadPoolExecutor.java:624)
  4. at java.lang.Thread.run (Thread.java:748)

Thanks
Mohil


On Sun, Jul 26, 2020 at 1:50 PM Mohil Khare  wrote:

> and it seems be due  to TimerType User
>
> Thanks
> Mohil
>
> On Sun, Jul 26, 2020 at 1:42 PM Mohil Khare  wrote:
>
>> Hello,
>>
>> I was looking at source code of
>> https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingSideInputDoFnRunner.java
>> ,
>>
>> It seems the default implementation of OnTimer is to throw (Attempt to
>> deliver a timer to a DoFn, but timers are not supported in Dataflow).
>>
>> Do you know under what circumstances, My code might be throwing