Re: Pipeline AttributeError on Python3

2019-11-22 Thread Thomas Weise
We have not seen the issue with Python 3.6 on 2.16+ after applying this patch. 🎉 Thanks! On Thu, Nov 21, 2019 at 4:41 PM Thomas Weise wrote: > We are currently verifying the patch. Will report back tomorrow. > > On Thu, Nov 21, 2019 at 8:40 AM Valentyn Tymofieiev > wrote: > >> That would be he

Re: Pipeline AttributeError on Python3

2019-11-21 Thread Thomas Weise
We are currently verifying the patch. Will report back tomorrow. On Thu, Nov 21, 2019 at 8:40 AM Valentyn Tymofieiev wrote: > That would be helpful, thanks a lot! It should be a straightforward patch. > Also, thanks Guenther, for sharing your investigation on > https://bugs.python.org/issue34572

Re: Pipeline AttributeError on Python3

2019-11-21 Thread Valentyn Tymofieiev
That would be helpful, thanks a lot! It should be a straightforward patch. Also, thanks Guenther, for sharing your investigation on https://bugs.python.org/issue34572, it was very helpful. On Thu, Nov 21, 2019 at 8:25 AM Thomas Weise wrote: > Valentyn, thanks a lot for following up on this. > >

Re: Pipeline AttributeError on Python3

2019-11-21 Thread Thomas Weise
Valentyn, thanks a lot for following up on this. If the change can be cherry picked in isolation, we should be able to verify this soon (with 2.16). On Thu, Nov 21, 2019 at 8:12 AM Valentyn Tymofieiev wrote: > To close the loop here: To my knowledge this issue affects all Python 3 > users of P

Re: Pipeline AttributeError on Python3

2019-11-21 Thread Valentyn Tymofieiev
To close the loop here: To my knowledge this issue affects all Python 3 users of Portable Flink/Spark runners, and Dataflow Python Streaming users, including users on Python 3.7.3 and newer versions. The issue is addressed on Beam master, and we have a cherry-pick out for Beam 2.17.0. Workaround

Re: Pipeline AttributeError on Python3

2019-11-13 Thread Valentyn Tymofieiev
I also opened https://issues.apache.org/jira/browse/BEAM-8651 to track this issue and any recommendation for the users that will come out of it. On Thu, Nov 7, 2019 at 6:25 PM Valentyn Tymofieiev wrote: > I think we have heard of this issue from the same source: > > This looks exactly like a ra

Re: Pipeline AttributeError on Python3

2019-11-07 Thread Valentyn Tymofieiev
I think we have heard of this issue from the same source: This looks exactly like a race condition that we've encountered on Python > 3.7.1: There's a bug in some older 3.7.x releases that breaks the > thread-safety of the unpickler, as concurrent unpickle threads can access a > module before it

Re: Pipeline AttributeError on Python3

2019-11-06 Thread Rakesh Kumar
Thanks Valentyn, Aggregation_transform.py doesn't have any transformation method which extends beam.DoFn. We are using plain python method which we passed in beam.Map(). I am not sure how to get the dump of serialized_fn. Can you please let me the process? I also heard that some people ran into

Re: Pipeline AttributeError on Python3

2019-10-28 Thread Valentyn Tymofieiev
+user@, bcc: dev@ https://issues.apache.org/jira/browse/BEAM-6158 may be contributing to this issue, although we saw instances of this bug in exactly opposite scenarios - when pipeline was defined *in one file*, but not in multiple files. Could you try replacing instances of super() in aggregation

Pipeline AttributeError on Python3

2019-10-28 Thread Rakesh Kumar
Hi All, We have noticed a weird intermittent issue on Python3 but we don't run into this issue on python2. Sometimes when we are trying to submit the pipeline, we get AttributeError (Check the stack trace below). we have double-checked and we do find the attribute/methods are present in the right