This was discovered because Repeatedly(AfterProcessingTime(n)) does require the workaround, because AfterProcessingTime(n) does terminate and that is propagated by the Repeatedly implementation.
Kenn On Mon, Sep 27, 2021 at 2:49 PM Robert Bradshaw <rober...@google.com> wrote: > On Mon, Sep 27, 2021 at 2:42 PM Kenneth Knowles <k...@apache.org> wrote: > > > > True. It is the use case of Repeatedly(ProcessingTime(n)) that I would > guess to be the primary case of concern. > > This shouldn't require the workaround, right? > > If you're not comfortable with this, I'd say let's patch and re-cut. > > > > On Mon, Sep 27, 2021 at 2:38 PM Robert Bradshaw <rober...@google.com> > wrote: > >> > >> As I read the code, it's only pipelines that use a Repeatedly trigger > >> that wrap an already lossy trigger that are declared to be themselves > >> lossy. If I'm mistaken, I'll certainly reconsider my vote (and thanks > >> for bringing this up). > >> > >> On Mon, Sep 27, 2021 at 2:21 PM Kenneth Knowles <k...@apache.org> > wrote: > >> > > >> > My concern is that the error message is incorrect and every user of > 2.33.0 may be educated wrong, or be worried about data loss in Beam, or > fail to find the blog post or CHANGES, etc. > >> > > >> > Kenn > >> > > >> > On Mon, Sep 27, 2021 at 2:16 PM Udi Meiri <eh...@google.com> wrote: > >> >> > >> >> I don't know how rare it is, but there is a flag documented in > CHANGES and blog post that reverts to the old behavior. > >> >> > >> >> On Mon, Sep 27, 2021 at 2:12 PM Kenneth Knowles <k...@apache.org> > wrote: > >> >>> > >> >>> I guess my vote is -0 since I don't have enough context on this > issue. A number of people with more awareness of how severe this is have > voted +1 so I will not try to block the release. > >> >>> > >> >>> Kenn > >> >>> > >> >>> On Mon, Sep 27, 2021 at 2:11 PM Kenneth Knowles <k...@apache.org> > wrote: > >> >>>> > >> >>>> I have to disagree with the other PMC members here, or at least > dig in to the question: every pipeline that uses a Repeatedly trigger at > the top level will be rejected. Is this so rare in Python that it is OK? > >> >>>> > >> >>>> Kenn > >> >>>> > >> >>>> On Mon, Sep 27, 2021 at 1:56 PM Robert Burke <r...@google.com> > wrote: > >> >>>>> > >> >>>>> On it. Thanks! > >> >>>>> > >> >>>>> On Mon, Sep 27, 2021 at 1:18 PM Udi Meiri <eh...@google.com> > wrote: > >> >>>>>> > >> >>>>>> Daniel/Robert, feel free to make changes to this PR: > https://github.com/apache/beam/pull/15543 > >> >>>>>> > >> >>>>>> On Mon, Sep 27, 2021 at 12:08 PM Daniel Oliveira < > danolive...@google.com> wrote: > >> >>>>>>> > >> >>>>>>> +1 > >> >>>>>>> > >> >>>>>>> I hadn't realized the pipelines were still finishing > successfully. I retried wordcount with that in mind and confirmed it > finishes successfully, so this isn't a blocker. > >> >>>>>>> > >> >>>>>>> Although maybe we should add this to the "Known Issues" because > I can easily see those messages being interpreted as a pipeline failure. > >> >>>>>>> > >> >>>>>>> On Sun, Sep 26, 2021 at 7:26 PM Robert Burke < > rob...@frantil.com> wrote: > >> >>>>>>>> > >> >>>>>>>> That's on the SDK side, and it just means the PCollection > metrics are being returned, buy not handled by the SDK. At present the > pipeline results only handle PTransform metrics. > >> >>>>>>>> > >> >>>>>>>> As such it's not a regression, as adding those is still under > development. > >> >>>>>>>> > >> >>>>>>>> On Sun, Sep 26, 2021, 7:02 PM Daniel Oliveira < > danolive...@google.com> wrote: > >> >>>>>>>>> > >> >>>>>>>>> I tried validating wordcount with the Go SDK on Flink. The > pipeline failed with a wall of errors like the following. I tried this on > Flink 1.11, 1.12, and 1.13 job servers built from source at the RC1 commit, > same errors on all of them. > >> >>>>>>>>> > >> >>>>>>>>>> 2021/09/26 18:21:49 Failed to deduce Step from > MonitoringInfo: urn:"beam:metric:element_count:v1" > type:"beam:metrics:sum_int64:v1" payload:"\x8d%" > labels:{key:"PCOLLECTION" value:"n9"} > >> >>>>>>>>>> 2021/09/26 18:21:49 Failed to deduce Step from > MonitoringInfo: urn:"beam:metric:element_count:v1" > type:"beam:metrics:sum_int64:v1" payload:"\x8d%" > labels:{key:"PCOLLECTION" value:"n10"} > >> >>>>>>>>>> 2021/09/26 18:21:49 Failed to deduce Step from > MonitoringInfo: urn:"beam:metric:element_count:v1" > type:"beam:metrics:sum_int64:v1" payload:"\x8d%" > labels:{key:"PCOLLECTION" value:"n8"} > >> >>>>>>>>>> 2021/09/26 18:21:49 Failed to deduce Step from > MonitoringInfo: urn:"beam:metric:element_count:v1" > type:"beam:metrics:sum_int64:v1" payload:"\x8d%" > labels:{key:"PCOLLECTION" value:"n7"} > >> >>>>>>>>>> 2021/09/26 18:21:49 Failed to deduce Step from > MonitoringInfo: urn:"beam:metric:sampled_byte_size:v1" > type:"beam:metrics:distribution_int64:v1" > payload:"\x8d%\xfa\xbf\x05\x04\xa8\x0c" labels:{key:"PCOLLECTION" > value:"n7"} > >> >>>>>>>>>> 2021/09/26 18:21:49 Failed to deduce Step from > MonitoringInfo: urn:"beam:metric:sampled_byte_size:v1" > type:"beam:metrics:distribution_int64:v1" payload:"\xb9\x05\xcf6\x05\x11" > labels:{key:"PCOLLECTION" value:"n9"} > >> >>>>>>>>>> 2021/09/26 18:21:49 Failed to deduce Step from > MonitoringInfo: urn:"beam:metric:sampled_byte_size:v1" > type:"beam:metrics:distribution_int64:v1" payload:"\xc5\x05\x8a1\x04\x12" > labels:{key:"PCOLLECTION" value:"n8"} > >> >>>>>>>>>> 2021/09/26 18:21:49 Failed to deduce Step from > MonitoringInfo: urn:"beam:metric:sampled_byte_size:v1" > type:"beam:metrics:distribution_int64:v1" > payload:"\x8d%\xb3\xa7\x07\x14\"" labels:{key:"PCOLLECTION" value:"n10"} > >> >>>>>>>>>> {...} > >> >>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>>> @Robert Burke I think you might know what's going on here. > Is this solvable with a cherry-pick and a new RC? > >> >>>>>>>>> > >> >>>>>>>>> On Fri, Sep 24, 2021 at 4:25 PM Ahmet Altay <al...@google.com> > wrote: > >> >>>>>>>>>> > >> >>>>>>>>>> > >> >>>>>>>>>> > >> >>>>>>>>>> On Fri, Sep 24, 2021 at 4:21 PM Udi Meiri <eh...@google.com> > wrote: > >> >>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>>>> On Fri, Sep 24, 2021 at 4:18 PM Ahmet Altay < > al...@google.com> wrote: > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> Thank you for the update related to allow_unsafe_triggers. > My vote is still a +1. > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> If we decide to move forward with this RC, could you > please include this bug in the known issues list under the changes.md for > this release? > >> >>>>>>>>>>> > >> >>>>>>>>>>> Yes, that's included in > https://github.com/apache/beam/pull/15543/files. > >> >>>>>>>>>> > >> >>>>>>>>>> > >> >>>>>>>>>> Thank you! > >> >>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> On Fri, Sep 24, 2021 at 3:56 PM Chamikara Jayalath < > chamik...@google.com> wrote: > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> +1 (binding) > >> >>>>>>>>>>>>> Validated a few scenarios from the spreadsheet. > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> Thanks, > >> >>>>>>>>>>>>> Cham > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> On Fri, Sep 24, 2021 at 3:07 PM Robert Bradshaw < > rober...@google.com> wrote: > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> +1 (binding) > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> All the artifacts and signatures look good. > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> I don't think the unsafe trigger check is severe enough > to block the release. > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> On Fri, Sep 24, 2021 at 2:36 PM Udi Meiri < > eh...@google.com> wrote: > >> >>>>>>>>>>>>>> > > >> >>>>>>>>>>>>>> > Hi everyone, we found a bug during testing. It has to > do with Python SDK's allow_unsafe_triggers check. > >> >>>>>>>>>>>>>> > There is a preliminary fix that will go to master. > >> >>>>>>>>>>>>>> > > >> >>>>>>>>>>>>>> > For the 2.33.0 release, I'm leaning towards not making > a new RC since there is a workaround: pass the flag --allow_unsafe_triggers. > >> >>>>>>>>>>>>>> > > >> >>>>>>>>>>>>>> > Please reevaluate your votes accordingly and recast if > you've changed your vote. > >> >>>>>>>>>>>>>> > > >> >>>>>>>>>>>>>> > Thanks > >> >>>>>>>>>>>>>> > > >> >>>>>>>>>>>>>> > On Fri, Sep 24, 2021 at 12:48 PM Alexey Romanenko < > aromanenko....@gmail.com> wrote: > >> >>>>>>>>>>>>>> >> > >> >>>>>>>>>>>>>> >> > >> >>>>>>>>>>>>>> >> On 24 Sep 2021, at 20:45, Udi Meiri <eh...@google.com> > wrote: > >> >>>>>>>>>>>>>> >> Alexey is this something that we should put in the > release notes, or some other change? > >> >>>>>>>>>>>>>> >> > >> >>>>>>>>>>>>>> >> > >> >>>>>>>>>>>>>> >> Yes, I think it could be helpful to mention that Beam > Jackson’s deps was bumped and it may require an update of Jackson’s runtime > deps for Spark 2 users as well. > >> >>>>>>>>>>>>>> >> > >> >>>>>>>>>>>>>> >> — > >> >>>>>>>>>>>>>> >> Alexey > >> >>>>>>>>>>>>>> >> > >> >>>>>>>>>>>>>> >> > >> >>>>>>>>>>>>>> >> > >> >>>>>>>>>>>>>> >>> > >> >>>>>>>>>>>>>> >>> > >> >>>>>>>>>>>>>> >>> — > >> >>>>>>>>>>>>>> >>> Alexey > >> >>>>>>>>>>>>>> >>> > >> >>>>>>>>>>>>>> >>> On 24 Sep 2021, at 16:17, Alexey Romanenko < > aromanenko....@gmail.com> wrote: > >> >>>>>>>>>>>>>> >>> > >> >>>>>>>>>>>>>> >>> I checked with beam-samples [1] and noticed an issue > to run some pipelines with Spark 2 runner (Spark 3 seems is ok). > >> >>>>>>>>>>>>>> >>> It looks that it’s caused by new Jackson's version > updated recently [2], even if it’s a minor update but it works fine with a > previous one. > >> >>>>>>>>>>>>>> >>> I’ll try to find a workaround and get back with a > results of this. > >> >>>>>>>>>>>>>> >>> > >> >>>>>>>>>>>>>> >>> [1] https://github.com/Talend/beam-samples/ > >> >>>>>>>>>>>>>> >>> [2] > https://github.com/apache/beam/commit/9694f70df1447e96684b665279679edafec13a0c > >> >>>>>>>>>>>>>> >>> > >> >>>>>>>>>>>>>> >>> — > >> >>>>>>>>>>>>>> >>> Alexey > >> >>>>>>>>>>>>>> >>> > >> >>>>>>>>>>>>>> >>> On 24 Sep 2021, at 11:17, Jan Lukavský < > je...@seznam.cz> wrote: > >> >>>>>>>>>>>>>> >>> > >> >>>>>>>>>>>>>> >>> +1 (non-binding) > >> >>>>>>>>>>>>>> >>> > >> >>>>>>>>>>>>>> >>> Validated several use-cases using non-portable Flink > with Java SDK. > >> >>>>>>>>>>>>>> >>> > >> >>>>>>>>>>>>>> >>> On 9/24/21 4:55 AM, Valentyn Tymofieiev wrote: > >> >>>>>>>>>>>>>> >>> > >> >>>>>>>>>>>>>> >>> +1. Ran several Python batch and streaming pipelines > on Dataflow and checked that Dataflow containers have required dependencies > of Beam. > >> >>>>>>>>>>>>>> >>> > >> >>>>>>>>>>>>>> >>> On Thu, Sep 23, 2021 at 7:03 PM Robert Burke < > lostl...@apache.org> wrote: > >> >>>>>>>>>>>>>> >>>> > >> >>>>>>>>>>>>>> >>>> +1 (non-binding) > >> >>>>>>>>>>>>>> >>>> I validated the Go Quickstart (wordcount), and my > ray tracer against the Go Direct runner, Dataflow, and Spark (ensuring the > rc1 tagged container was used) and they executed successfully. > >> >>>>>>>>>>>>>> >>>> > >> >>>>>>>>>>>>>> >>>> I needed to manually synthesize a pseudo-version to > ensure I was using the tagged branch version > (v2.0.0-20210914211513-b358127f9859) instead of simply using v2.33.0-RC1. > >> >>>>>>>>>>>>>> >>>> > >> >>>>>>>>>>>>>> >>>> It either can't find the package with the right > tagged version, or it can't find the version. It's not clear to me what the > issue is, but it's not notionally a release blocker. I'll investigate > further once we have a full release, as it's probably some unspecified > behavior due to how we transitioned to Go Modules (which strongly > recommended doing a major version bump for such transitions, which seems a > bit excessive for Beam as a whole...). > >> >>>>>>>>>>>>>> >>>> > >> >>>>>>>>>>>>>> >>>> On 2021/09/23 03:59:18, Ahmet Altay < > al...@google.com> wrote: > >> >>>>>>>>>>>>>> >>>> > +1 on the RC. I validated python quick start > examples on direct runners. > >> >>>>>>>>>>>>>> >>>> > Thank you Udi. > >> >>>>>>>>>>>>>> >>>> > > >> >>>>>>>>>>>>>> >>>> > On Wed, Sep 22, 2021 at 2:20 PM Robert Burke < > rob...@frantil.com> wrote: > >> >>>>>>>>>>>>>> >>>> > > >> >>>>>>>>>>>>>> >>>> > > Just an FYI that intend to validate the Go SDK > for this release but can't > >> >>>>>>>>>>>>>> >>>> > > get to it until tomorrow (Thursday). I'm > catching up from a week of > >> >>>>>>>>>>>>>> >>>> > > vacation. Apologies for the inconvenience. > >> >>>>>>>>>>>>>> >>>> > > > >> >>>>>>>>>>>>>> >>>> > > On Tue, Sep 21, 2021, 10:59 AM Udi Meiri < > eh...@google.com> wrote: > >> >>>>>>>>>>>>>> >>>> > > > >> >>>>>>>>>>>>>> >>>> > >> I updated the affected and fixed version > fields for > >> >>>>>>>>>>>>>> >>>> > >> > https://issues.apache.org/jira/browse/BEAM-12356. > >> >>>>>>>>>>>>>> >>>> > >> > >> >>>>>>>>>>>>>> >>>> > >> > >> >>>>>>>>>>>>>> >>>> > >> On Tue, Sep 21, 2021 at 10:48 AM Reuven Lax < > re...@google.com> wrote: > >> >>>>>>>>>>>>>> >>>> > >> > >> >>>>>>>>>>>>>> >>>> > >>> Unfortunate - I didn't realize that > >> >>>>>>>>>>>>>> >>>> > >>> https://github.com/apache/beam/pull/15480 > didn't make the cut. > >> >>>>>>>>>>>>>> >>>> > >>> > >> >>>>>>>>>>>>>> >>>> > >>> This bug was a regression in Beam 2.32.0, and > is blocking multiple users > >> >>>>>>>>>>>>>> >>>> > >>> from updating. > >> >>>>>>>>>>>>>> >>>> > >>> > >> >>>>>>>>>>>>>> >>>> > >>> On Tue, Sep 21, 2021 at 10:33 AM Udi Meiri < > u...@apache.org> wrote: > >> >>>>>>>>>>>>>> >>>> > >>> > >> >>>>>>>>>>>>>> >>>> > >>>> Hi everyone, > >> >>>>>>>>>>>>>> >>>> > >>>> Please review and vote on the release > candidate #1 for the version > >> >>>>>>>>>>>>>> >>>> > >>>> 2.33.0, as follows: > >> >>>>>>>>>>>>>> >>>> > >>>> [ ] +1, Approve the release > >> >>>>>>>>>>>>>> >>>> > >>>> [ ] -1, Do not approve the release (please > provide specific comments) > >> >>>>>>>>>>>>>> >>>> > >>>> > >> >>>>>>>>>>>>>> >>>> > >>>> > >> >>>>>>>>>>>>>> >>>> > >>>> Reviewers are encouraged to test their own > use cases with the release > >> >>>>>>>>>>>>>> >>>> > >>>> candidate, and vote +1 if > >> >>>>>>>>>>>>>> >>>> > >>>> no issues are found. > >> >>>>>>>>>>>>>> >>>> > >>>> > >> >>>>>>>>>>>>>> >>>> > >>>> The complete staging area is available for > your review, which includes: > >> >>>>>>>>>>>>>> >>>> > >>>> * JIRA release notes [1], > >> >>>>>>>>>>>>>> >>>> > >>>> * the official Apache source release to be > deployed to dist.apache.org > >> >>>>>>>>>>>>>> >>>> > >>>> [2], which is signed with the key with > fingerprint 587B049C36DAAFE6 [3], > >> >>>>>>>>>>>>>> >>>> > >>>> * all artifacts to be deployed to the Maven > Central Repository [4], > >> >>>>>>>>>>>>>> >>>> > >>>> * source code tag "v2.33.0-RC1" [5], > >> >>>>>>>>>>>>>> >>>> > >>>> * website pull request listing the release > [6], the blog post [6], and > >> >>>>>>>>>>>>>> >>>> > >>>> publishing the API reference manual [7]. > >> >>>>>>>>>>>>>> >>>> > >>>> * Java artifacts were built with Maven 3.6.3 > and OpenJDK 1.8.0_181. > >> >>>>>>>>>>>>>> >>>> > >>>> * Python artifacts are deployed along with > the source release to the > >> >>>>>>>>>>>>>> >>>> > >>>> dist.apache.org [2] and pypy[8]. > >> >>>>>>>>>>>>>> >>>> > >>>> * Validation sheet with a tab for 2.33.0 > release to help with > >> >>>>>>>>>>>>>> >>>> > >>>> validation [9]. > >> >>>>>>>>>>>>>> >>>> > >>>> * Docker images published to Docker Hub [10]. > >> >>>>>>>>>>>>>> >>>> > >>>> > >> >>>>>>>>>>>>>> >>>> > >>>> The vote will be open for at least 72 hours. > It is adopted by majority > >> >>>>>>>>>>>>>> >>>> > >>>> approval, with at least 3 PMC affirmative > votes. > >> >>>>>>>>>>>>>> >>>> > >>>> > >> >>>>>>>>>>>>>> >>>> > >>>> For guidelines on how to try the release in > your projects, check out > >> >>>>>>>>>>>>>> >>>> > >>>> our blog post at > https://beam.apache.org/blog/validate-beam-release/. > >> >>>>>>>>>>>>>> >>>> > >>>> > >> >>>>>>>>>>>>>> >>>> > >>>> Thanks, > >> >>>>>>>>>>>>>> >>>> > >>>> Release Manager > >> >>>>>>>>>>>>>> >>>> > >>>> > >> >>>>>>>>>>>>>> >>>> > >>>> [1] > >> >>>>>>>>>>>>>> >>>> > >>>> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12350404 > >> >>>>>>>>>>>>>> >>>> > >>>> [2] > https://dist.apache.org/repos/dist/dev/beam/2.33.0/ > >> >>>>>>>>>>>>>> >>>> > >>>> [3] > https://dist.apache.org/repos/dist/release/beam/KEYS > >> >>>>>>>>>>>>>> >>>> > >>>> [4] > >> >>>>>>>>>>>>>> >>>> > >>>> > https://repository.apache.org/content/repositories/orgapachebeam-1234/ > >> >>>>>>>>>>>>>> >>>> > >>>> [5] > https://github.com/apache/beam/tree/v2.33.0-RC1 > >> >>>>>>>>>>>>>> >>>> > >>>> [6] > https://github.com/apache/beam/pull/15543 > >> >>>>>>>>>>>>>> >>>> > >>>> [7] > https://github.com/apache/beam-site/pull/619 > >> >>>>>>>>>>>>>> >>>> > >>>> [8] > https://pypi.org/project/apache-beam/2.33.0rc1/ > >> >>>>>>>>>>>>>> >>>> > >>>> [9] > >> >>>>>>>>>>>>>> >>>> > >>>> > https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1705275493 > >> >>>>>>>>>>>>>> >>>> > >>>> [10] > https://hub.docker.com/search?q=apache%2Fbeam&type=image > >> >>>>>>>>>>>>>> >>>> > >>>> > >> >>>>>>>>>>>>>> >>>> > >>>> > >> >>>>>>>>>>>>>> >>>> > > >> >>>>>>>>>>>>>> >>> > >> >>>>>>>>>>>>>> >>> > >> >>>>>>>>>>>>>> >>> > >> >>>>>>>>>>>>>> >> >
