Thank you Valentyn. Being a release manager is difficult. It requires
balancing between stability, following the process, regressions,
timelines. Thank you for following the process, thank you for asking the
right questions, thank you for doing the release.
On Fri, Jul 17, 2020 at 3:59 PM Robert Bradshaw <[email protected]
<mailto:[email protected]>> wrote:
Thank you, Valentyn!
On Fri, Jul 17, 2020 at 3:25 PM Chamikara Jayalath
<[email protected] <mailto:[email protected]>> wrote:
>
>
>
> On Fri, Jul 17, 2020 at 3:01 PM Valentyn Tymofieiev
<[email protected] <mailto:[email protected]>> wrote:
>>
>> As a general rule, fixes pertaining to new functionality are not
a good candidate for a cherry-pick.
>>
>> A case for an exception can be made for polishing features
related to major wide announcements with a hard deadline, which
appears to be the case for xlang on Dataflow.
>>
>> I will prepare an RC2 with xlang fixes and consider other
low-risk additions from issues that were brought to my attention.
>
>
> Thanks Valentyn.
>
>>
>>
>> Thanks
>>
>>
>> On Fri, Jul 17, 2020 at 10:36 AM Chamikara Jayalath
<[email protected] <mailto:[email protected]>> wrote:
>>>
>>>
>>>
>>> On Fri, Jul 17, 2020 at 10:01 AM Robert Bradshaw
<[email protected] <mailto:[email protected]>> wrote:
>>>>
>>>> Taking a step back, the goal of avoiding cherry-picks is to reduce
>>>> risk and increase the velocity of our releases, as otherwise the
>>>> release manager gets inundated by a never ending list of features
>>>> people want to get in that puts the releases further and further
>>>> behind (increasing the desire to get features in in a vicious
cycle).
>>>> On the flip side, the reason we have a release process with
candidates
>>>> and voting (as opposed to just declaring a commit id every N
weeks to
>>>> be "the release") is to give us the flexibility to achieve a
level of
>>>> quality and polish that may not ever occur in HEAD itself.
>>>>
>>>> With regards to this specific cross-langauge fix, the
motivation is
>>>> that those working on it at Google want to widely publish this
feature
>>>> as newly available on Dataflow. The question to answer here
(Cham) is
>>>> whether this bug is debilitating enough that were it not to be
in the
>>>> release we would want to hold off advertising this (and related)
>>>> features until the next release. (In my understanding, it
would result
>>>> in a poor enough user experience that it is.)
>>>
>>>
>>> Yes, I think we will have to either hold off on widely
publishing the feature or list this as a potential issue that will
be fixed in the next release for anybody who tries cross-language
pipelines and runs into this.
>>> Note that we are getting in a Python Kafka example [1]. So
users will potentially try this out anyways.
>>>
>>> [1] https://github.com/apache/beam/pull/12188
>>>
>>>
>>>>
>>>>
>>>> On the other hand, there's the question of the cost of getting
this
>>>> fix into the release. The change is simple and well contained,
so I
>>>> think the risk is low (and, in particular, the cost to include
it here
>>>> is low enough that it's worth the value provided above).
>>>>
>>>> Looking at the other proposals,
>>>> https://github.com/apache/beam/pull/12196 also seems to meet
this bar
>>>> (there are possible xlang correctness issues at play here), as
does
>>>> https://github.com/apache/beam/pull/12175 (mostly due to its
>>>> simplicity and the fact that doing it later would be a backwards
>>>> compatible change). I'm on the fence about
>>>> https://github.com/apache/beam/pull/12171 (if an RC2 is in the
works
>>>> anyway), and IMHO the others are less compelling as having to
be done
>>>> now.
>>>
>>>
>>> +1
>>>
>>>>
>>>>
>>>> (On the question of a point release, IMHO anything worth
considering
>>>> for an x.y.1 release definitely meets the bar for inclusion
into an RC
>>>> of an ongoing release.)
>>>>
>>>> - Robert
>>>>
>>>>
>>>> On Thu, Jul 16, 2020 at 8:00 PM Chamikara Jayalath
<[email protected] <mailto:[email protected]>> wrote:
>>>> >
>>>> >
>>>> >
>>>> > On Thu, Jul 16, 2020 at 7:46 PM Chamikara Jayalath
<[email protected] <mailto:[email protected]>> wrote:
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Thu, Jul 16, 2020 at 7:28 PM Valentyn Tymofieiev
<[email protected] <mailto:[email protected]>> wrote:
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> On Thu, Jul 16, 2020, 19:07 Chamikara Jayalath
<[email protected] <mailto:[email protected]>> wrote:
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> On Thu, Jul 16, 2020 at 6:16 PM Valentyn Tymofieiev
<[email protected] <mailto:[email protected]>> wrote:
>>>> >>>>>
>>>> >>>>> Thanks for the feedback, help with release validation,
and for reaching out on dev@ regarding a cherry-pick request.
>>>> >>>>>
>>>> >>>>> BEAM-10397 pertains to new functionality (xlang support
on Dataflow). Are there any reasons that this fix cannot wait until
2.24.0 (release cut date 4 weeks from now)?
>>>> >>>>>
>>>> >>>>> For transparency, I would like to list other cherry-pick
requests that I received off-the list (stakeholders bcc'ed):
>>>> >>>>> - https://github.com/apache/beam/pull/12175
>>>> >>>>> - https://github.com/apache/beam/pull/12196
>>>> >>>>> - https://github.com/apache/beam/pull/12171
>>>> >>>>> - https://issues.apache.org/jira/browse/BEAM-10492
(recently added)
>>>> >>>>> - https://issues.apache.org/jira/browse/BEAM-10385
>>>> >>>>> - https://github.com/apache/beam/pull/12187 (was
available before any of RC1 artifacts were created and integrated)
>>>> >>>>
>>>> >>>>
>>>> >>>> My main concern is Python changes in
https://github.com/apache/beam/pull/12164. Other changes (at least
related to x-lang) can wait.
>>>> >>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> My response to such requests is guided by the release
guide [1]:
>>>> >>>>>
>>>> >>>>> - None of the issues were a regression from a previous
release.
>>>> >>>>> - Most are related to new or recently introduced
functionality.
>>>> >>>>> - 3 of the requests are related to xlang io, which is
very exciting and important functionality, but arguably does not
impact a large percentage of [existing] users.
>>>> >>>>
>>>> >>>>
>>>> >>>> Agree that this is not a regression from the previous
release but it may result in inconsistent behavior when users
execute x-lang pipelines. Actually I think this is a pretty serious
issue for portability (we are not setting the environment in
WindowingStrategy) but for some reason we are not hitting this in
other tests.
>>>> >>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> So they do not seem to be release-blocking according to
the guide.
>>>> >>>>>
>>>> >>>>> At this point creating a new RC would delay 2.23.0
availability by at least a week. While a new RC will improve the
stability of xlang IO, it will also delay the release of features
and bug fixes available in 2.23.0. It will also create a precedent
of inconsistency with release policy. Should we delay the release if
we discover another xlang issue during validation next week?
>>>> >>>>
>>>> >>>>
>>>> >>>> To be honest, I don't think re-validating after the
cherry-pick mentioned above will take a week (unless we find other
issues). We just need to rebuild and re-validate the Python
distribution and may be rebuild Dataflow containers. I'm
volunteering to help you with this :)
>>>> >>>
>>>> >>>
>>>> >>> I was taking 72hrs of voting Window into account that must
happen outside of the weekend and the fact that I will be OOO for
one day.
>>>> >>
>>>> >>
>>>> >> Got it.
>>>> >>
>>>> >>>
>>>> >>>
>>>> >>> If the issue you mention seriously impacts (can cause data
loss, pipeline failures) all of users on portable stack or other
large user base (not just cross-language support in Dataflow (new
user-base) ), this is definitely a candidate for an ASAP fix.
>>>> >>>
>>>> >>> What is your assessment of the size of the user base that
is affected by the issue (large, medium, small, does not affect
production for any of existing users)?
>>>> >>
>>>> >>
>>>> >> Impact today I think is low but potential for impact in the
future is high. For example, if we update Dataflow service or
portable runners to require environment in WindowingStrategy, we'll
have to either fork for this or require users to upgrade to a Beam
version with the fix.
>>>> >
>>>> >
>>>> > Actually, ignore the "portable runners" part. Seems like we
already set "context.default_environment_id()" in the
WindowingStrategy so impact is likely only for Dataflow where we do
not set an environment_id in serialized WindowingStrategy that is
set in GBK.
>>>> >
>>>> >>
>>>> >>
>>>> >> Thanks,
>>>> >> Cham
>>>> >>
>>>> >>>
>>>> >>>
>>>> >>> Thanks!
>>>> >>>
>>>> >>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> My preferred course of action is to continue with RC0,
since release velocity is important for product health.
>>>> >>>>>
>>>> >>>>> Given that we are having this conversation, we can
revise the cherry-pick policy if we think it does not adequately
cover this situation.
>>>> >>>>
>>>> >>>>
>>>> >>>> Agree. We have a very strong policy currently regarding
cherry-picks but it's up to the release manager to look into
requests on a case-by-case basis.
>>>> >>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> We can also propose a patch-version release with urgent
cherry-picks (release 2.23.1), or consider a faster release cadence
if 6 weeks is too slow.
>>>> >>>>
>>>> >>>>
>>>> >>>> Honestly I don't think this is practical. Making a new
patch release, validation, vote etc will take 2 weeks or so. We
either should cherry-pick this into current release or wait till the
next one. I think patch releases should be reserved for critical
updates to LTS releases.
>>>> >>>>
>>>> >>>> Thanks,
>>>> >>>> Cham
>>>> >>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> Thanks,
>>>> >>>>> Valentyn
>>>> >>>>>
>>>> >>>>> [1]
https://beam.apache.org/contribute/release-guide/#review-cherry-picks
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> On Wed, Jul 15, 2020 at 5:41 PM Chamikara Jayalath
<[email protected] <mailto:[email protected]>> wrote:
>>>> >>>>>>
>>>> >>>>>> I agree. I think Dataflow x-lang users could run into
flaky pipelines due to this. Valentyn, are you OK with creating a
new RC that includes the fix (already merged -
https://github.com/apache/beam/pull/12164) and preferably
https://github.com/apache/beam/pull/12196 ?
>>>> >>>>>>
>>>> >>>>>> Thanks,
>>>> >>>>>> Cham
>>>> >>>>>>
>>>> >>>>>> On Wed, Jul 15, 2020 at 5:27 PM Heejong Lee
<[email protected] <mailto:[email protected]>> wrote:
>>>> >>>>>>>
>>>> >>>>>>> I think we need to cherry-pick
https://issues.apache.org/jira/browse/BEAM-10397 which fixes missing
environment errors for Dataflow xlang pipelines. Internally, we have
a flaky xlang kafkaio test because of missing environment errors and
any xlang pipelines using GroupByKey could encounter this.
>>>> >>>>>>>
>>>> >>>>>>> On Wed, Jul 15, 2020 at 5:08 PM Ahmet Altay
<[email protected] <mailto:[email protected]>> wrote:
>>>> >>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>>> On Wed, Jul 15, 2020 at 4:55 PM Robert Bradshaw
<[email protected] <mailto:[email protected]>> wrote:
>>>> >>>>>>>>>
>>>> >>>>>>>>> All the artifacts, signatures, and hashes look good.
>>>> >>>>>>>>>
>>>> >>>>>>>>> I would like to understand the severity of
>>>> >>>>>>>>> https://issues.apache.org/jira/browse/BEAM-10397
before giving my
>>>> >>>>>>>>> vote.
>>>> >>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>>> +Heejong Lee to comment on this.
>>>> >>>>>>>>
>>>> >>>>>>>>>
>>>> >>>>>>>>>
>>>> >>>>>>>>> On Wed, Jul 15, 2020 at 10:51 AM Pablo Estrada
<[email protected] <mailto:[email protected]>> wrote:
>>>> >>>>>>>>> >
>>>> >>>>>>>>> > +1
>>>> >>>>>>>>> > I was able to run the python 3.8 quickstart from
wheels on DirectRunner.
>>>> >>>>>>>>> > I verified hashes for Python files.
>>>> >>>>>>>>> > -P.
>>>> >>>>>>>>> >
>>>> >>>>>>>>> > On Fri, Jul 10, 2020 at 4:34 PM Ahmet Altay
<[email protected] <mailto:[email protected]>> wrote:
>>>> >>>>>>>>> >>
>>>> >>>>>>>>> >> I validated the python 3 quickstarts. I had
issues with running with python 3.8 wheel files, but did not have
issues with source distributions, or other python wheel files. I
have not tested python 2 quickstarts.
>>>> >>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>>> Did someone validate python 3.8 wheels on Dataflow? I
was not able to run that.
>>>> >>>>>>>>
>>>> >>>>>>>>>
>>>> >>>>>>>>> >>
>>>> >>>>>>>>> >> On Thu, Jul 9, 2020 at 10:53 PM Valentyn
Tymofieiev <[email protected] <mailto:[email protected]>> wrote:
>>>> >>>>>>>>> >>>
>>>> >>>>>>>>> >>> Hi everyone,
>>>> >>>>>>>>> >>>
>>>> >>>>>>>>> >>> Please review and vote on the release candidate
#1 for the version 2.23.0, as follows:
>>>> >>>>>>>>> >>> [ ] +1, Approve the release
>>>> >>>>>>>>> >>> [ ] -1, Do not approve the release (please
provide specific comments)
>>>> >>>>>>>>> >>>
>>>> >>>>>>>>> >>>
>>>> >>>>>>>>> >>> The complete staging area is available for your
review, which includes:
>>>> >>>>>>>>> >>> * JIRA release notes [1],
>>>> >>>>>>>>> >>> * the official Apache source release to be
deployed to dist.apache.org <http://dist.apache.org> [2], which is
signed with the key with fingerprint 1DF50603225D29A4 [3],
>>>> >>>>>>>>> >>> * all artifacts to be deployed to the Maven
Central Repository [4],
>>>> >>>>>>>>> >>> * source code tag "v2.23.0-RС1" [5],
>>>> >>>>>>>>> >>> * website pull request listing the release [6],
publishing the API reference manual [7], and the blog post [8].
>>>> >>>>>>>>> >>> * Java artifacts were built with Maven 3.6.0 and
Oracle JDK 1.8.0_201-b09 .
>>>> >>>>>>>>> >>> * Python artifacts are deployed along with the
source release to the dist.apache.org <http://dist.apache.org> [2].
>>>> >>>>>>>>> >>> * Validation sheet with a tab for 2.23.0 release
to help with validation [9].
>>>> >>>>>>>>> >>> * Docker images published to Docker Hub [10].
>>>> >>>>>>>>> >>>
>>>> >>>>>>>>> >>> The vote will be open for at least 72 hours. It
is adopted by majority approval, with at least 3 PMC affirmative votes.
>>>> >>>>>>>>> >>>
>>>> >>>>>>>>> >>> Thanks,
>>>> >>>>>>>>> >>> Release Manager
>>>> >>>>>>>>> >>>
>>>> >>>>>>>>> >>> [1]
https://jira.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12347145
>>>> >>>>>>>>> >>> [2]
https://dist.apache.org/repos/dist/dev/beam/2.23.0/
>>>> >>>>>>>>> >>> [3]
https://dist.apache.org/repos/dist/release/beam/KEYS
>>>> >>>>>>>>> >>> [4]
https://repository.apache.org/content/repositories/orgapachebeam-1105/
>>>> >>>>>>>>> >>> [5] https://github.com/apache/beam/tree/v2.23.0-RC1
>>>> >>>>>>>>> >>> [6] https://github.com/apache/beam/pull/12212
>>>> >>>>>>>>> >>> [7] https://github.com/apache/beam-site/pull/605
>>>> >>>>>>>>> >>> [8] https://github.com/apache/beam/pull/12213
>>>> >>>>>>>>> >>> [9]
https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=596347973
>>>> >>>>>>>>> >>> [10]
https://hub.docker.com/search?q=apache%2Fbeam&type=image
