That's on the SDK side, and it just means the PCollection metrics are being
returned, buy not handled by the SDK. At present the pipeline results only
handle PTransform metrics.
As such it's not a regression, as adding those is still under development.
On Sun, Sep 26, 2021, 7:02 PM Daniel Oliveira <danolive...@google.com>
wrote:
> I tried validating wordcount with the Go SDK on Flink. The pipeline failed
> with a wall of errors like the following. I tried this on Flink 1.11, 1.12,
> and 1.13 job servers built from source at the RC1 commit, same errors on
> all of them.
>
> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
>> urn:"beam:metric:element_count:v1" type:"beam:metrics:sum_int64:v1"
>> payload:"\x8d%" labels:{key:"PCOLLECTION" value:"n9"}
>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
>> urn:"beam:metric:element_count:v1" type:"beam:metrics:sum_int64:v1"
>> payload:"\x8d%" labels:{key:"PCOLLECTION" value:"n10"}
>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
>> urn:"beam:metric:element_count:v1" type:"beam:metrics:sum_int64:v1"
>> payload:"\x8d%" labels:{key:"PCOLLECTION" value:"n8"}
>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
>> urn:"beam:metric:element_count:v1" type:"beam:metrics:sum_int64:v1"
>> payload:"\x8d%" labels:{key:"PCOLLECTION" value:"n7"}
>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
>> urn:"beam:metric:sampled_byte_size:v1"
>> type:"beam:metrics:distribution_int64:v1"
>> payload:"\x8d%\xfa\xbf\x05\x04\xa8\x0c" labels:{key:"PCOLLECTION"
>> value:"n7"}
>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
>> urn:"beam:metric:sampled_byte_size:v1"
>> type:"beam:metrics:distribution_int64:v1" payload:"\xb9\x05\xcf6\x05\x11"
>> labels:{key:"PCOLLECTION" value:"n9"}
>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
>> urn:"beam:metric:sampled_byte_size:v1"
>> type:"beam:metrics:distribution_int64:v1" payload:"\xc5\x05\x8a1\x04\x12"
>> labels:{key:"PCOLLECTION" value:"n8"}
>> 2021/09/26 18:21:49 Failed to deduce Step from MonitoringInfo:
>> urn:"beam:metric:sampled_byte_size:v1"
>> type:"beam:metrics:distribution_int64:v1"
>> payload:"\x8d%\xb3\xa7\x07\x14\"" labels:{key:"PCOLLECTION" value:"n10"}
>> {...}
>
>
> @Robert Burke <r...@google.com> I think you might know what's going on
> here. Is this solvable with a cherry-pick and a new RC?
>
> On Fri, Sep 24, 2021 at 4:25 PM Ahmet Altay <al...@google.com> wrote:
>
>>
>>
>> On Fri, Sep 24, 2021 at 4:21 PM Udi Meiri <eh...@google.com> wrote:
>>
>>>
>>>
>>> On Fri, Sep 24, 2021 at 4:18 PM Ahmet Altay <al...@google.com> wrote:
>>>
>>>> Thank you for the update related to allow_unsafe_triggers. My vote is
>>>> still a +1.
>>>>
>>>> If we decide to move forward with this RC, could you please include
>>>> this bug in the known issues list under the changes.md for this release?
>>>>
>>> Yes, that's included in https://github.com/apache/beam/pull/15543/files.
>>>
>>
>> Thank you!
>>
>>
>>>
>>>
>>>>
>>>> On Fri, Sep 24, 2021 at 3:56 PM Chamikara Jayalath <
>>>> chamik...@google.com> wrote:
>>>>
>>>>> +1 (binding)
>>>>> Validated a few scenarios from the spreadsheet.
>>>>>
>>>>> Thanks,
>>>>> Cham
>>>>>
>>>>> On Fri, Sep 24, 2021 at 3:07 PM Robert Bradshaw <rober...@google.com>
>>>>> wrote:
>>>>>
>>>>>> +1 (binding)
>>>>>>
>>>>>> All the artifacts and signatures look good.
>>>>>>
>>>>>> I don't think the unsafe trigger check is severe enough to block the
>>>>>> release.
>>>>>>
>>>>>> On Fri, Sep 24, 2021 at 2:36 PM Udi Meiri <eh...@google.com> wrote:
>>>>>> >
>>>>>> > Hi everyone, we found a bug during testing. It has to do with
>>>>>> Python SDK's allow_unsafe_triggers check.
>>>>>> > There is a preliminary fix that will go to master.
>>>>>> >
>>>>>> > For the 2.33.0 release, I'm leaning towards not making a new RC
>>>>>> since there is a workaround: pass the flag --allow_unsafe_triggers.
>>>>>> >
>>>>>> > Please reevaluate your votes accordingly and recast if you've
>>>>>> changed your vote.
>>>>>> >
>>>>>> > Thanks
>>>>>> >
>>>>>> > On Fri, Sep 24, 2021 at 12:48 PM Alexey Romanenko <
>>>>>> aromanenko....@gmail.com> wrote:
>>>>>> >>
>>>>>> >>
>>>>>> >> On 24 Sep 2021, at 20:45, Udi Meiri <eh...@google.com> wrote:
>>>>>> >> Alexey is this something that we should put in the release notes,
>>>>>> or some other change?
>>>>>> >>
>>>>>> >>
>>>>>> >> Yes, I think it could be helpful to mention that Beam Jackson’s
>>>>>> deps was bumped and it may require an update of Jackson’s runtime deps
>>>>>> for
>>>>>> Spark 2 users as well.
>>>>>> >>
>>>>>> >> —
>>>>>> >> Alexey
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> —
>>>>>> >>> Alexey
>>>>>> >>>
>>>>>> >>> On 24 Sep 2021, at 16:17, Alexey Romanenko <
>>>>>> aromanenko....@gmail.com> wrote:
>>>>>> >>>
>>>>>> >>> I checked with beam-samples [1] and noticed an issue to run some
>>>>>> pipelines with Spark 2 runner (Spark 3 seems is ok).
>>>>>> >>> It looks that it’s caused by new Jackson's version updated
>>>>>> recently [2], even if it’s a minor update but it works fine with a
>>>>>> previous
>>>>>> one.
>>>>>> >>> I’ll try to find a workaround and get back with a results of this.
>>>>>> >>>
>>>>>> >>> [1] https://github.com/Talend/beam-samples/
>>>>>> >>> [2]
>>>>>> https://github.com/apache/beam/commit/9694f70df1447e96684b665279679edafec13a0c
>>>>>> >>>
>>>>>> >>> —
>>>>>> >>> Alexey
>>>>>> >>>
>>>>>> >>> On 24 Sep 2021, at 11:17, Jan Lukavský <je...@seznam.cz> wrote:
>>>>>> >>>
>>>>>> >>> +1 (non-binding)
>>>>>> >>>
>>>>>> >>> Validated several use-cases using non-portable Flink with Java
>>>>>> SDK.
>>>>>> >>>
>>>>>> >>> On 9/24/21 4:55 AM, Valentyn Tymofieiev wrote:
>>>>>> >>>
>>>>>> >>> +1. Ran several Python batch and streaming pipelines on Dataflow
>>>>>> and checked that Dataflow containers have required dependencies of Beam.
>>>>>> >>>
>>>>>> >>> On Thu, Sep 23, 2021 at 7:03 PM Robert Burke <lostl...@apache.org>
>>>>>> wrote:
>>>>>> >>>>
>>>>>> >>>> +1 (non-binding)
>>>>>> >>>> I validated the Go Quickstart (wordcount), and my ray tracer
>>>>>> against the Go Direct runner, Dataflow, and Spark (ensuring the rc1
>>>>>> tagged
>>>>>> container was used) and they executed successfully.
>>>>>> >>>>
>>>>>> >>>> I needed to manually synthesize a pseudo-version to ensure I was
>>>>>> using the tagged branch version (v2.0.0-20210914211513-b358127f9859)
>>>>>> instead of simply using v2.33.0-RC1.
>>>>>> >>>>
>>>>>> >>>> It either can't find the package with the right tagged version,
>>>>>> or it can't find the version. It's not clear to me what the issue is, but
>>>>>> it's not notionally a release blocker. I'll investigate further once we
>>>>>> have a full release, as it's probably some unspecified behavior due to
>>>>>> how
>>>>>> we transitioned to Go Modules (which strongly recommended doing a major
>>>>>> version bump for such transitions, which seems a bit excessive for Beam
>>>>>> as
>>>>>> a whole...).
>>>>>> >>>>
>>>>>> >>>> On 2021/09/23 03:59:18, Ahmet Altay <al...@google.com> wrote:
>>>>>> >>>> > +1 on the RC. I validated python quick start examples on
>>>>>> direct runners.
>>>>>> >>>> > Thank you Udi.
>>>>>> >>>> >
>>>>>> >>>> > On Wed, Sep 22, 2021 at 2:20 PM Robert Burke <
>>>>>> rob...@frantil.com> wrote:
>>>>>> >>>> >
>>>>>> >>>> > > Just an FYI that intend to validate the Go SDK for this
>>>>>> release but can't
>>>>>> >>>> > > get to it until tomorrow (Thursday). I'm catching up from a
>>>>>> week of
>>>>>> >>>> > > vacation. Apologies for the inconvenience.
>>>>>> >>>> > >
>>>>>> >>>> > > On Tue, Sep 21, 2021, 10:59 AM Udi Meiri <eh...@google.com>
>>>>>> wrote:
>>>>>> >>>> > >
>>>>>> >>>> > >> I updated the affected and fixed version fields for
>>>>>> >>>> > >> https://issues.apache.org/jira/browse/BEAM-12356.
>>>>>> >>>> > >>
>>>>>> >>>> > >>
>>>>>> >>>> > >> On Tue, Sep 21, 2021 at 10:48 AM Reuven Lax <
>>>>>> re...@google.com> wrote:
>>>>>> >>>> > >>
>>>>>> >>>> > >>> Unfortunate - I didn't realize that
>>>>>> >>>> > >>> https://github.com/apache/beam/pull/15480 didn't make the
>>>>>> cut.
>>>>>> >>>> > >>>
>>>>>> >>>> > >>> This bug was a regression in Beam 2.32.0, and is blocking
>>>>>> multiple users
>>>>>> >>>> > >>> from updating.
>>>>>> >>>> > >>>
>>>>>> >>>> > >>> On Tue, Sep 21, 2021 at 10:33 AM Udi Meiri <
>>>>>> u...@apache.org> wrote:
>>>>>> >>>> > >>>
>>>>>> >>>> > >>>> Hi everyone,
>>>>>> >>>> > >>>> Please review and vote on the release candidate #1 for
>>>>>> the version
>>>>>> >>>> > >>>> 2.33.0, as follows:
>>>>>> >>>> > >>>> [ ] +1, Approve the release
>>>>>> >>>> > >>>> [ ] -1, Do not approve the release (please provide
>>>>>> specific comments)
>>>>>> >>>> > >>>>
>>>>>> >>>> > >>>>
>>>>>> >>>> > >>>> Reviewers are encouraged to test their own use cases with
>>>>>> the release
>>>>>> >>>> > >>>> candidate, and vote +1 if
>>>>>> >>>> > >>>> no issues are found.
>>>>>> >>>> > >>>>
>>>>>> >>>> > >>>> The complete staging area is available for your review,
>>>>>> which includes:
>>>>>> >>>> > >>>> * JIRA release notes [1],
>>>>>> >>>> > >>>> * the official Apache source release to be deployed to
>>>>>> dist.apache.org
>>>>>> >>>> > >>>> [2], which is signed with the key with fingerprint
>>>>>> 587B049C36DAAFE6 [3],
>>>>>> >>>> > >>>> * all artifacts to be deployed to the Maven Central
>>>>>> Repository [4],
>>>>>> >>>> > >>>> * source code tag "v2.33.0-RC1" [5],
>>>>>> >>>> > >>>> * website pull request listing the release [6], the blog
>>>>>> post [6], and
>>>>>> >>>> > >>>> publishing the API reference manual [7].
>>>>>> >>>> > >>>> * Java artifacts were built with Maven 3.6.3 and OpenJDK
>>>>>> 1.8.0_181.
>>>>>> >>>> > >>>> * Python artifacts are deployed along with the source
>>>>>> release to the
>>>>>> >>>> > >>>> dist.apache.org [2] and pypy[8].
>>>>>> >>>> > >>>> * Validation sheet with a tab for 2.33.0 release to help
>>>>>> with
>>>>>> >>>> > >>>> validation [9].
>>>>>> >>>> > >>>> * Docker images published to Docker Hub [10].
>>>>>> >>>> > >>>>
>>>>>> >>>> > >>>> The vote will be open for at least 72 hours. It is
>>>>>> adopted by majority
>>>>>> >>>> > >>>> approval, with at least 3 PMC affirmative votes.
>>>>>> >>>> > >>>>
>>>>>> >>>> > >>>> For guidelines on how to try the release in your
>>>>>> projects, check out
>>>>>> >>>> > >>>> our blog post at
>>>>>> https://beam.apache.org/blog/validate-beam-release/.
>>>>>> >>>> > >>>>
>>>>>> >>>> > >>>> Thanks,
>>>>>> >>>> > >>>> Release Manager
>>>>>> >>>> > >>>>
>>>>>> >>>> > >>>> [1]
>>>>>> >>>> > >>>>
>>>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12350404
>>>>>> >>>> > >>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.33.0/
>>>>>> >>>> > >>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>>>>>> >>>> > >>>> [4]
>>>>>> >>>> > >>>>
>>>>>> https://repository.apache.org/content/repositories/orgapachebeam-1234/
>>>>>> >>>> > >>>> [5] https://github.com/apache/beam/tree/v2.33.0-RC1
>>>>>> >>>> > >>>> [6] https://github.com/apache/beam/pull/15543
>>>>>> >>>> > >>>> [7] https://github.com/apache/beam-site/pull/619
>>>>>> >>>> > >>>> [8] https://pypi.org/project/apache-beam/2.33.0rc1/
>>>>>> >>>> > >>>> [9]
>>>>>> >>>> > >>>>
>>>>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1705275493
>>>>>> >>>> > >>>> [10]
>>>>>> https://hub.docker.com/search?q=apache%2Fbeam&type=image
>>>>>> >>>> > >>>>
>>>>>> >>>> > >>>>
>>>>>> >>>> >
>>>>>> >>>
>>>>>> >>>
>>>>>> >>>
>>>>>> >>
>>>>>>
>>>>>
