My concern is that the error message is incorrect and every user of 2.33.0 may be educated wrong, or be worried about data loss in Beam, or fail to find the blog post or CHANGES, etc.
Kenn On Mon, Sep 27, 2021 at 2:16 PM Udi Meiri <eh...@google.com> wrote: > I don't know how rare it is, but there is a flag documented in CHANGES and > blog post that reverts to the old behavior. > > On Mon, Sep 27, 2021 at 2:12 PM Kenneth Knowles <k...@apache.org> wrote: > >> I guess my vote is -0 since I don't have enough context on this issue. A >> number of people with more awareness of how severe this is have voted +1 so >> I will not try to block the release. >> >> Kenn >> >> On Mon, Sep 27, 2021 at 2:11 PM Kenneth Knowles <k...@apache.org> wrote: >> >>> I have to disagree with the other PMC members here, or at least dig in >>> to the question: every pipeline that uses a Repeatedly trigger at the top >>> level will be rejected. Is this so rare in Python that it is OK? >>> >>> Kenn >>> >>> On Mon, Sep 27, 2021 at 1:56 PM Robert Burke <r...@google.com> wrote: >>> >>>> On it. Thanks! >>>> >>>> On Mon, Sep 27, 2021 at 1:18 PM Udi Meiri <eh...@google.com> wrote: >>>> >>>>> Daniel/Robert, feel free to make changes to this PR: >>>>> https://github.com/apache/beam/pull/15543 >>>>> >>>>> On Mon, Sep 27, 2021 at 12:08 PM Daniel Oliveira < >>>>> danolive...@google.com> wrote: >>>>> >>>>>> +1 >>>>>> >>>>>> I hadn't realized the pipelines were still finishing successfully. I >>>>>> retried wordcount with that in mind and confirmed it finishes >>>>>> successfully, >>>>>> so this isn't a blocker. >>>>>> >>>>>> Although maybe we should add this to the "Known Issues" because I can >>>>>> easily see those messages being interpreted as a pipeline failure. >>>>>> >>>>>> On Sun, Sep 26, 2021 at 7:26 PM Robert Burke <rob...@frantil.com> >>>>>> wrote: >>>>>> >>>>>>> That's on the SDK side, and it just means the PCollection metrics >>>>>>> are being returned, buy not handled by the SDK. At present the pipeline >>>>>>> results only handle PTransform metrics. >>>>>>> >>>>>>> As such it's not a regression, as adding those is still under >>>>>>> development. >>>>>>> >>>>>>> On Sun, Sep 26, 2021, 7:02 PM Daniel Oliveira < >>>>>>> danolive...@google.com> wrote: >>>>>>> >>>>>>>> I tried validating wordcount with the Go SDK on Flink. The pipeline >>>>>>>> failed with a wall of errors like the following. I tried this on Flink >>>>>>>> 1.11, 1.12, and 1.13 job servers built from source at the RC1 commit, >>>>>>>> same >>>>>>>> errors on all of them. >>>>>>>> >>>>>>>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo: >>>>>>>>> urn:"beam:metric:element_count:v1" type:"beam:metrics:sum_int64:v1" >>>>>>>>> payload:"\x8d%" labels:{key:"PCOLLECTION" value:"n9"} >>>>>>>>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo: >>>>>>>>> urn:"beam:metric:element_count:v1" type:"beam:metrics:sum_int64:v1" >>>>>>>>> payload:"\x8d%" labels:{key:"PCOLLECTION" value:"n10"} >>>>>>>>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo: >>>>>>>>> urn:"beam:metric:element_count:v1" type:"beam:metrics:sum_int64:v1" >>>>>>>>> payload:"\x8d%" labels:{key:"PCOLLECTION" value:"n8"} >>>>>>>>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo: >>>>>>>>> urn:"beam:metric:element_count:v1" type:"beam:metrics:sum_int64:v1" >>>>>>>>> payload:"\x8d%" labels:{key:"PCOLLECTION" value:"n7"} >>>>>>>>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo: >>>>>>>>> urn:"beam:metric:sampled_byte_size:v1" >>>>>>>>> type:"beam:metrics:distribution_int64:v1" >>>>>>>>> payload:"\x8d%\xfa\xbf\x05\x04\xa8\x0c" labels:{key:"PCOLLECTION" >>>>>>>>> value:"n7"} >>>>>>>>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo: >>>>>>>>> urn:"beam:metric:sampled_byte_size:v1" >>>>>>>>> type:"beam:metrics:distribution_int64:v1" >>>>>>>>> payload:"\xb9\x05\xcf6\x05\x11" >>>>>>>>> labels:{key:"PCOLLECTION" value:"n9"} >>>>>>>>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo: >>>>>>>>> urn:"beam:metric:sampled_byte_size:v1" >>>>>>>>> type:"beam:metrics:distribution_int64:v1" >>>>>>>>> payload:"\xc5\x05\x8a1\x04\x12" >>>>>>>>> labels:{key:"PCOLLECTION" value:"n8"} >>>>>>>>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo: >>>>>>>>> urn:"beam:metric:sampled_byte_size:v1" >>>>>>>>> type:"beam:metrics:distribution_int64:v1" >>>>>>>>> payload:"\x8d%\xb3\xa7\x07\x14\"" labels:{key:"PCOLLECTION" >>>>>>>>> value:"n10"} >>>>>>>>> {...} >>>>>>>> >>>>>>>> >>>>>>>> @Robert Burke <r...@google.com> I think you might know what's >>>>>>>> going on here. Is this solvable with a cherry-pick and a new RC? >>>>>>>> >>>>>>>> On Fri, Sep 24, 2021 at 4:25 PM Ahmet Altay <al...@google.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Sep 24, 2021 at 4:21 PM Udi Meiri <eh...@google.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Sep 24, 2021 at 4:18 PM Ahmet Altay <al...@google.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Thank you for the update related to allow_unsafe_triggers. My >>>>>>>>>>> vote is still a +1. >>>>>>>>>>> >>>>>>>>>>> If we decide to move forward with this RC, could you please >>>>>>>>>>> include this bug in the known issues list under the changes.md for >>>>>>>>>>> this >>>>>>>>>>> release? >>>>>>>>>>> >>>>>>>>>> Yes, that's included in >>>>>>>>>> https://github.com/apache/beam/pull/15543/files. >>>>>>>>>> >>>>>>>>> >>>>>>>>> Thank you! >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Fri, Sep 24, 2021 at 3:56 PM Chamikara Jayalath < >>>>>>>>>>> chamik...@google.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> +1 (binding) >>>>>>>>>>>> Validated a few scenarios from the spreadsheet. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Cham >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Sep 24, 2021 at 3:07 PM Robert Bradshaw < >>>>>>>>>>>> rober...@google.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> +1 (binding) >>>>>>>>>>>>> >>>>>>>>>>>>> All the artifacts and signatures look good. >>>>>>>>>>>>> >>>>>>>>>>>>> I don't think the unsafe trigger check is severe enough to >>>>>>>>>>>>> block the release. >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Sep 24, 2021 at 2:36 PM Udi Meiri <eh...@google.com> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> > >>>>>>>>>>>>> > Hi everyone, we found a bug during testing. It has to do >>>>>>>>>>>>> with Python SDK's allow_unsafe_triggers check. >>>>>>>>>>>>> > There is a preliminary fix that will go to master. >>>>>>>>>>>>> > >>>>>>>>>>>>> > For the 2.33.0 release, I'm leaning towards not making a new >>>>>>>>>>>>> RC since there is a workaround: pass the flag >>>>>>>>>>>>> --allow_unsafe_triggers. >>>>>>>>>>>>> > >>>>>>>>>>>>> > Please reevaluate your votes accordingly and recast if >>>>>>>>>>>>> you've changed your vote. >>>>>>>>>>>>> > >>>>>>>>>>>>> > Thanks >>>>>>>>>>>>> > >>>>>>>>>>>>> > On Fri, Sep 24, 2021 at 12:48 PM Alexey Romanenko < >>>>>>>>>>>>> aromanenko....@gmail.com> wrote: >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> On 24 Sep 2021, at 20:45, Udi Meiri <eh...@google.com> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >> Alexey is this something that we should put in the release >>>>>>>>>>>>> notes, or some other change? >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> Yes, I think it could be helpful to mention that Beam >>>>>>>>>>>>> Jackson’s deps was bumped and it may require an update of >>>>>>>>>>>>> Jackson’s runtime >>>>>>>>>>>>> deps for Spark 2 users as well. >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> — >>>>>>>>>>>>> >> Alexey >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >>> >>>>>>>>>>>>> >>> >>>>>>>>>>>>> >>> — >>>>>>>>>>>>> >>> Alexey >>>>>>>>>>>>> >>> >>>>>>>>>>>>> >>> On 24 Sep 2021, at 16:17, Alexey Romanenko < >>>>>>>>>>>>> aromanenko....@gmail.com> wrote: >>>>>>>>>>>>> >>> >>>>>>>>>>>>> >>> I checked with beam-samples [1] and noticed an issue to >>>>>>>>>>>>> run some pipelines with Spark 2 runner (Spark 3 seems is ok). >>>>>>>>>>>>> >>> It looks that it’s caused by new Jackson's version updated >>>>>>>>>>>>> recently [2], even if it’s a minor update but it works fine with >>>>>>>>>>>>> a previous >>>>>>>>>>>>> one. >>>>>>>>>>>>> >>> I’ll try to find a workaround and get back with a results >>>>>>>>>>>>> of this. >>>>>>>>>>>>> >>> >>>>>>>>>>>>> >>> [1] https://github.com/Talend/beam-samples/ >>>>>>>>>>>>> >>> [2] >>>>>>>>>>>>> https://github.com/apache/beam/commit/9694f70df1447e96684b665279679edafec13a0c >>>>>>>>>>>>> >>> >>>>>>>>>>>>> >>> — >>>>>>>>>>>>> >>> Alexey >>>>>>>>>>>>> >>> >>>>>>>>>>>>> >>> On 24 Sep 2021, at 11:17, Jan Lukavský <je...@seznam.cz> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>> >>>>>>>>>>>>> >>> +1 (non-binding) >>>>>>>>>>>>> >>> >>>>>>>>>>>>> >>> Validated several use-cases using non-portable Flink with >>>>>>>>>>>>> Java SDK. >>>>>>>>>>>>> >>> >>>>>>>>>>>>> >>> On 9/24/21 4:55 AM, Valentyn Tymofieiev wrote: >>>>>>>>>>>>> >>> >>>>>>>>>>>>> >>> +1. Ran several Python batch and streaming pipelines on >>>>>>>>>>>>> Dataflow and checked that Dataflow containers have required >>>>>>>>>>>>> dependencies of >>>>>>>>>>>>> Beam. >>>>>>>>>>>>> >>> >>>>>>>>>>>>> >>> On Thu, Sep 23, 2021 at 7:03 PM Robert Burke < >>>>>>>>>>>>> lostl...@apache.org> wrote: >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> >>>> +1 (non-binding) >>>>>>>>>>>>> >>>> I validated the Go Quickstart (wordcount), and my ray >>>>>>>>>>>>> tracer against the Go Direct runner, Dataflow, and Spark >>>>>>>>>>>>> (ensuring the rc1 >>>>>>>>>>>>> tagged container was used) and they executed successfully. >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> >>>> I needed to manually synthesize a pseudo-version to >>>>>>>>>>>>> ensure I was using the tagged branch version >>>>>>>>>>>>> (v2.0.0-20210914211513-b358127f9859) instead of simply using >>>>>>>>>>>>> v2.33.0-RC1. >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> >>>> It either can't find the package with the right tagged >>>>>>>>>>>>> version, or it can't find the version. It's not clear to me what >>>>>>>>>>>>> the issue >>>>>>>>>>>>> is, but it's not notionally a release blocker. I'll investigate >>>>>>>>>>>>> further >>>>>>>>>>>>> once we have a full release, as it's probably some unspecified >>>>>>>>>>>>> behavior due >>>>>>>>>>>>> to how we transitioned to Go Modules (which strongly recommended >>>>>>>>>>>>> doing a >>>>>>>>>>>>> major version bump for such transitions, which seems a bit >>>>>>>>>>>>> excessive for >>>>>>>>>>>>> Beam as a whole...). >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> >>>> On 2021/09/23 03:59:18, Ahmet Altay <al...@google.com> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>> > +1 on the RC. I validated python quick start examples >>>>>>>>>>>>> on direct runners. >>>>>>>>>>>>> >>>> > Thank you Udi. >>>>>>>>>>>>> >>>> > >>>>>>>>>>>>> >>>> > On Wed, Sep 22, 2021 at 2:20 PM Robert Burke < >>>>>>>>>>>>> rob...@frantil.com> wrote: >>>>>>>>>>>>> >>>> > >>>>>>>>>>>>> >>>> > > Just an FYI that intend to validate the Go SDK for >>>>>>>>>>>>> this release but can't >>>>>>>>>>>>> >>>> > > get to it until tomorrow (Thursday). I'm catching up >>>>>>>>>>>>> from a week of >>>>>>>>>>>>> >>>> > > vacation. Apologies for the inconvenience. >>>>>>>>>>>>> >>>> > > >>>>>>>>>>>>> >>>> > > On Tue, Sep 21, 2021, 10:59 AM Udi Meiri < >>>>>>>>>>>>> eh...@google.com> wrote: >>>>>>>>>>>>> >>>> > > >>>>>>>>>>>>> >>>> > >> I updated the affected and fixed version fields for >>>>>>>>>>>>> >>>> > >> https://issues.apache.org/jira/browse/BEAM-12356. >>>>>>>>>>>>> >>>> > >> >>>>>>>>>>>>> >>>> > >> >>>>>>>>>>>>> >>>> > >> On Tue, Sep 21, 2021 at 10:48 AM Reuven Lax < >>>>>>>>>>>>> re...@google.com> wrote: >>>>>>>>>>>>> >>>> > >> >>>>>>>>>>>>> >>>> > >>> Unfortunate - I didn't realize that >>>>>>>>>>>>> >>>> > >>> https://github.com/apache/beam/pull/15480 didn't >>>>>>>>>>>>> make the cut. >>>>>>>>>>>>> >>>> > >>> >>>>>>>>>>>>> >>>> > >>> This bug was a regression in Beam 2.32.0, and is >>>>>>>>>>>>> blocking multiple users >>>>>>>>>>>>> >>>> > >>> from updating. >>>>>>>>>>>>> >>>> > >>> >>>>>>>>>>>>> >>>> > >>> On Tue, Sep 21, 2021 at 10:33 AM Udi Meiri < >>>>>>>>>>>>> u...@apache.org> wrote: >>>>>>>>>>>>> >>>> > >>> >>>>>>>>>>>>> >>>> > >>>> Hi everyone, >>>>>>>>>>>>> >>>> > >>>> Please review and vote on the release candidate #1 >>>>>>>>>>>>> for the version >>>>>>>>>>>>> >>>> > >>>> 2.33.0, as follows: >>>>>>>>>>>>> >>>> > >>>> [ ] +1, Approve the release >>>>>>>>>>>>> >>>> > >>>> [ ] -1, Do not approve the release (please provide >>>>>>>>>>>>> specific comments) >>>>>>>>>>>>> >>>> > >>>> >>>>>>>>>>>>> >>>> > >>>> >>>>>>>>>>>>> >>>> > >>>> Reviewers are encouraged to test their own use >>>>>>>>>>>>> cases with the release >>>>>>>>>>>>> >>>> > >>>> candidate, and vote +1 if >>>>>>>>>>>>> >>>> > >>>> no issues are found. >>>>>>>>>>>>> >>>> > >>>> >>>>>>>>>>>>> >>>> > >>>> The complete staging area is available for your >>>>>>>>>>>>> review, which includes: >>>>>>>>>>>>> >>>> > >>>> * JIRA release notes [1], >>>>>>>>>>>>> >>>> > >>>> * the official Apache source release to be >>>>>>>>>>>>> deployed to dist.apache.org >>>>>>>>>>>>> >>>> > >>>> [2], which is signed with the key with fingerprint >>>>>>>>>>>>> 587B049C36DAAFE6 [3], >>>>>>>>>>>>> >>>> > >>>> * all artifacts to be deployed to the Maven >>>>>>>>>>>>> Central Repository [4], >>>>>>>>>>>>> >>>> > >>>> * source code tag "v2.33.0-RC1" [5], >>>>>>>>>>>>> >>>> > >>>> * website pull request listing the release [6], >>>>>>>>>>>>> the blog post [6], and >>>>>>>>>>>>> >>>> > >>>> publishing the API reference manual [7]. >>>>>>>>>>>>> >>>> > >>>> * Java artifacts were built with Maven 3.6.3 and >>>>>>>>>>>>> OpenJDK 1.8.0_181. >>>>>>>>>>>>> >>>> > >>>> * Python artifacts are deployed along with the >>>>>>>>>>>>> source release to the >>>>>>>>>>>>> >>>> > >>>> dist.apache.org [2] and pypy[8]. >>>>>>>>>>>>> >>>> > >>>> * Validation sheet with a tab for 2.33.0 release >>>>>>>>>>>>> to help with >>>>>>>>>>>>> >>>> > >>>> validation [9]. >>>>>>>>>>>>> >>>> > >>>> * Docker images published to Docker Hub [10]. >>>>>>>>>>>>> >>>> > >>>> >>>>>>>>>>>>> >>>> > >>>> The vote will be open for at least 72 hours. It is >>>>>>>>>>>>> adopted by majority >>>>>>>>>>>>> >>>> > >>>> approval, with at least 3 PMC affirmative votes. >>>>>>>>>>>>> >>>> > >>>> >>>>>>>>>>>>> >>>> > >>>> For guidelines on how to try the release in your >>>>>>>>>>>>> projects, check out >>>>>>>>>>>>> >>>> > >>>> our blog post at >>>>>>>>>>>>> https://beam.apache.org/blog/validate-beam-release/. >>>>>>>>>>>>> >>>> > >>>> >>>>>>>>>>>>> >>>> > >>>> Thanks, >>>>>>>>>>>>> >>>> > >>>> Release Manager >>>>>>>>>>>>> >>>> > >>>> >>>>>>>>>>>>> >>>> > >>>> [1] >>>>>>>>>>>>> >>>> > >>>> >>>>>>>>>>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12350404 >>>>>>>>>>>>> >>>> > >>>> [2] >>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/beam/2.33.0/ >>>>>>>>>>>>> >>>> > >>>> [3] >>>>>>>>>>>>> https://dist.apache.org/repos/dist/release/beam/KEYS >>>>>>>>>>>>> >>>> > >>>> [4] >>>>>>>>>>>>> >>>> > >>>> >>>>>>>>>>>>> https://repository.apache.org/content/repositories/orgapachebeam-1234/ >>>>>>>>>>>>> >>>> > >>>> [5] >>>>>>>>>>>>> https://github.com/apache/beam/tree/v2.33.0-RC1 >>>>>>>>>>>>> >>>> > >>>> [6] https://github.com/apache/beam/pull/15543 >>>>>>>>>>>>> >>>> > >>>> [7] https://github.com/apache/beam-site/pull/619 >>>>>>>>>>>>> >>>> > >>>> [8] >>>>>>>>>>>>> https://pypi.org/project/apache-beam/2.33.0rc1/ >>>>>>>>>>>>> >>>> > >>>> [9] >>>>>>>>>>>>> >>>> > >>>> >>>>>>>>>>>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1705275493 >>>>>>>>>>>>> >>>> > >>>> [10] >>>>>>>>>>>>> https://hub.docker.com/search?q=apache%2Fbeam&type=image >>>>>>>>>>>>> >>>> > >>>> >>>>>>>>>>>>> >>>> > >>>> >>>>>>>>>>>>> >>>> > >>>>>>>>>>>>> >>> >>>>>>>>>>>>> >>> >>>>>>>>>>>>> >>> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >>>>>>>>>>>>
