DataflowRunner | Cross-language

[Paweł Urbanowicz]

Hello, community,

I found information that Google is working on supporting Dataflow runner for 
cross-language
(https://beam.apache.org/roadmap/connectors-multi-sdk/)

Is there any more information about the expected release of this feature?

Thanks 

Re: Try Beam Katas Today

[Rion Williams]

Hi Henry,

I submitted a pull request related to the Beam Katas that can be found here 
(https://github.com/apache/beam/pull/11761) and included you as a reviewer. I 
updated all of the related metadata, generated the course, and tested through 
it to ensure it worked as expected (and the placeholders all worked as expected 
as well).

The generated course can be found here on Stepik 
(https://stepik.org/course/72488) and I’ve reached out to a few folks to put it 
through its paces in the wild. 

Let me know if there’s anything else I can do or changes that need to be made 
in the PR or elsewhere.

Thanks again,

Rion

> On May 20, 2020, at 2:12 AM, Henry Suryawirawan  
> wrote:
> 
> 
> Yeah there was a recent pull request merged for the md file format change.
> I checked your repo and it still contains the task.html, so need your help to 
> merge with the latest master.
> 
> For the answer placeholder, you may refer to this doc first to understand how 
> it works.
> It will auto update the placeholder position in the task-info.yaml.
> 
> If you encounter any issue, just let me know.
> Thanks Rion.
> 
> 
> Regards,
> Henry
> 
> 
> 
>> On Wed, May 20, 2020 at 12:43 PM Rion Williams  wrote:
>> Hi Henry,
>> 
>> Thanks for the quick response, I appreciate it. I believe that I pulled the 
>> latest from master a day or so ago, so I’ll make sure to pull the most 
>> recent changes in.
>> 
>> As far as the placeholders, they aren’t currently present (as I don’t 
>> believe they were present in the Java ones within the learning/katas 
>> directory), however I can easily add those in to align with the content of 
>> the existing course. I wasn’t entirely sure based on the existing 
>> directories if the files should contain the placeholders or the actual 
>> implementations, either way, it’s a pretty trivial series of changes.
>> 
>> I’ll try to put these together tomorrow and push up a PR. I’ll make sure to 
>> include you as a reviewer.
>> 
>> Thanks for the initial feedback,
>> 
>> Rion
>> 
 On May 19, 2020, at 11:15 PM, Henry Suryawirawan 
  wrote:
 
>>> 
>>> Thanks Rion for adding the Kotlin version.
>>> This is great to show other people that Beam can be done in Kotlin too!
>>> 
>>> I can help to review your work.
>>> Please help to incorporate the Java Katas latest changes from master.
>>> There are recent changes to the task description file format from html to 
>>> md.
>>> Please also help to remove all the *-remote-info.yaml files.
>>> I assume that you've adjusted the answer placeholders in all tasks as well.
>>> Afterwards, you can create a pull request and assign me as reviewer.
>>> 
>>> Please reach out to me if you have any questions.
>>> 
>>> 
>>> Regards,
>>> Henry
>>> 
>>> 
>>> 
>>> 
 On Wed, May 20, 2020 at 3:33 AM Rion Williams  
 wrote:
 Sure! I ran through all of the tests locally on my branch (as tests) and 
 then performed a check against all of the known tasks (via Course Creator 
 > Check All Tasks) and 35/36 tasks passed successfully with the only one 
 that didn't being a Built-in IO one that doesn't currently have any 
 implementation. Although, I'd love for someone else to try the same thing 
 since as far as I can tell it "works on my machine".
 
 Thanks!
 
 Rion
 
 On 2020/05/19 19:12:57, Pablo Estrada  wrote: 
 > This is really cool Rion!
 > 
 > I believe it's possible to start trying out the katas from your branch? 
 > If
 > so, I can give them a try, and use that as a review...
 > Henry, any other ideas?
 > 
 > On Tue, May 19, 2020 at 12:04 PM Rion Williams 
 > wrote:
 > 
 > > Hi all,
 > >
 > > I was recently added as a contributor and created a JIRA ticket 
 > > related to
 > > the existing Katas (https://issues.apache.org/jira/browse/BEAM-10027),
 > > specifically creating one that targets Kotlin specific as there are 
 > > quite a
 > > few existing examples out there for Kotlin, so I thought a Kata course 
 > > that
 > > would parallel the existing Java, Go, and Python ones.
 > >
 > > I basically ported over the existing Java Katas, added the appropriate
 > > dependencies, and converted all of the Java files over to Kotlin, and
 > > ensured that all of the tests pass as expected. I'd love outside of 
 > > this to
 > > see if we can shift it to a Stepik course as well if that seems 
 > > reasonable
 > > similar to those mentioned in this thread.
 > >
 > > My current branch awaiting a PR can be found here (
 > > https://github.com/rionmonster/beam/tree/BEAM-10027), however I'm 
 > > unsure
 > > who would be the best to review such a PR and what other steps might 
 > > need
 > > to be taken before trying to get it merged in.
 > >
 > > Any feedback would be welcome!
 > >
 > > Thanks,
 > >
 > > Rion
 > >
 > > On 2020/05/14 23:40:45, Rion 

deploy_travis.sh ?

[Austin Bennett]

Hi All,

Was digging into https://github.com/apache/beam-wheels after being reminded
of https://issues.apache.org/jira/browse/BEAM-9388 (I recall the original
conversation -->
https://lists.apache.org/thread.html/r4a7d34e64a34e9fe589d06aec74d9b464d252c516fe96c35b2d6c9ae%40%3Cdev.beam.apache.org%3E
Not
sure whether taking on that issue is too ambitious, given would likely be
lacking much context.

Eliminating manual steps is the thing, so esp. happy to help with such
things to make our lives easier!

Trying to track down what all is involved here.  Seems I am missing
something,  deploy_travis.sh   is mentioned in the README
, but I don't
see it in this helper repo, nor in the main beam repo.  It actually looks
like step now relies upon:
https://github.com/apache/beam/blob/master/release/src/main/scripts/sign_hash_python_wheels.sh
to take from GCS to SVN?

If my understanding is correct, I will at least update the
documentation/README in beam-wheels.

Cheers,
Austin

Re: [RESULT][VOTE] Accept the Firefly design donation as Beam Mascot - Deadline Mon April 6

[Aizhamal Nurmamat kyzy]

Hello everyone,
Julian and I have created this pr[1] to create a page with the model sheet,
and links to the mascot image files. Can someone please review?
Thanks!
Aizhamal

[1] https://github.com/apache/beam/pull/11780

On Mon, May 11, 2020 at 10:14 AM Aizhamal Nurmamat kyzy 
wrote:

> @Ismael, this is in my work items for as soon as we complete the migration
> of the website.
> @Kyle, thanks for filing Jira, I assigned it to myself.
>
> On Mon, May 11, 2020 at 10:03 AM Kyle Weaver  wrote:
>
>> > Now that the vote has passed maybe we should add the images somewhere
>> > in the website so people can easily find the Firefly to use it
>>
>> +1 Maybe something to revisit after the website overhaul is complete. I
>> filed https://jira.apache.org/jira/browse/BEAM-9948 if anyone wants to
>> take it.
>>
>> On Mon, May 11, 2020 at 12:57 PM Ismaël Mejía  wrote:
>>
>>> Now that the vote has passed maybe we should add the images somewhere
>>> in the website so people can easily find the Firefly to use it.
>>> Something like what we do with our logos
>>> https://beam.apache.org/community/logos/
>>>
>>> WDYT? any taker?
>>>
>>> On Tue, Apr 28, 2020 at 7:43 PM Pablo Estrada 
>>> wrote:
>>> >
>>> > I'll be happy to as well!
>>> >
>>> > On Sun, Apr 26, 2020 at 4:18 AM Maximilian Michels 
>>> wrote:
>>> >>
>>> >> Hey Maria,
>>> >>
>>> >> I can testify :)
>>> >>
>>> >> Cheers,
>>> >> Max
>>> >>
>>> >> On 23.04.20 20:49, María Cruz wrote:
>>> >> > Hi everyone!
>>> >> > It is amazing to see how this process developed to collaboratively
>>> >> > create Apache Beam's mascot. Thank you to everyone who got involved!
>>> >> > I would like to write a blogpost for the Beam website, and I wanted
>>> to
>>> >> > ask you: would anyone like to offer their testimony about the
>>> process of
>>> >> > creating the Beam mascot, and what this means to you? Everyone's
>>> >> > testimony is welcome! If you witnessed the development of a mascot
>>> for
>>> >> > another open source project, even better =)
>>> >> >
>>> >> > Please feel free to express interest on this thread, and I'll reach
>>> out
>>> >> > to you off-list.
>>> >> >
>>> >> > Thanks,
>>> >> >
>>> >> > María
>>> >> >
>>> >> > On Fri, Apr 17, 2020 at 6:19 AM Jeff Klukas >> >> > > wrote:
>>> >> >
>>> >> > I personally like the sound of "Datum" as a name. I also like
>>> the
>>> >> > idea of not assigning them a gender.
>>> >> >
>>> >> > As a counterpoint on the naming side, one of the slide decks
>>> >> > provided while iterating on the design mentions:
>>> >> >
>>> >> > > Mascot can change colors when it is “full of data” or has a
>>> “batch
>>> >> > of data” to process.  Yellow is supercharged and ready to
>>> process!
>>> >> >
>>> >> > Based on that, I'd argue that the mascot maps to the concept of
>>> a
>>> >> > bundle in the beam execution model and we should consider a name
>>> >> > that's a play on "bundle" or perhaps a play on "checkpoint".
>>> >> >
>>> >> > On Thu, Apr 16, 2020 at 3:44 PM Julian Bruno <
>>> juliangbr...@gmail.com
>>> >> > > wrote:
>>> >> >
>>> >> > Hi all,
>>> >> >
>>> >> > While working on the design of our Mascot
>>> >> > Some ideas showed up and I wish to share them.
>>> >> > In regard to Alex Van Boxel's question about the name of
>>> our Mascot.
>>> >> >
>>> >> > I was thinking about this yesterday night and feel it could
>>> be a
>>> >> > great idea to name the Mascot "*Data*" or "*Datum*". Both
>>> names
>>> >> > sound cute and make sense to me. I prefer the later. Datum
>>> means
>>> >> > a single piece of information. The Mascot is the first
>>> piece of
>>> >> > information and its job is to collect batches of data and
>>> >> > process it. Datum is in charge of linking information
>>> together.
>>> >> >
>>> >> > In addition, our Mascot should have no gender. Rendering it
>>> >> > accessible to all users.
>>> >> >
>>> >> > Beam as a name for the mascot is pretty straight forward
>>> but I
>>> >> > think there are many things carrying that same name already.
>>> >> >
>>> >> > What do you think?
>>> >> >
>>> >> > Looking forward to hearing your feedback. Names are
>>> important
>>> >> > and I feel it can expand the personality and create a cool
>>> >> > background for our Mascot.
>>> >> >
>>> >> > Cheers!
>>> >> >
>>> >> > Julian
>>> >> >
>>> >> > On Mon, Apr 13, 2020, 3:40 PM Kyle Weaver <
>>> kcwea...@google.com
>>> >> > > wrote:
>>> >> >
>>> >> > Beam Firefly is fine with me (I guess people tend to
>>> forget
>>> >> > mascot names anyway). But if anyone comes up with
>>> something
>>> >> > particularly cute/clever we can consider it.
>>> >> >
>>> >> > On Mon, Apr 13, 2020 at 6:33 PM Aizhamal 

Re: [VOTE] Release 2.21.0, release candidate #1

[Thomas Weise]

Please note https://github.com/apache/beam/pull/11777 - another bug we
found while trying to upgrade to 2.21

Other than that things look good.

On Thu, May 21, 2020 at 4:14 PM Ahmet Altay  wrote:

> +1 (again, validated with new whl files.)
>
> What about https://issues.apache.org/jira/browse/BEAM-10015? Is this a
> blocker?
>
> +Kenneth Knowles  +Reuven Lax  -- since
> you are both tagged on the JIRA.
>
> On Thu, May 21, 2020 at 4:09 PM Kyle Weaver  wrote:
>
>> > We *really* need to automate the building and deploying of artifacts,
>> rather than have so many manual steps...
>>
>> Agreed. Luckily building wheels is one of only a couple steps that aren't
>> automated yet. Partially related:
>> https://issues.apache.org/jira/browse/BEAM-9388.
>>
>> On Thu, May 21, 2020 at 6:55 PM Luke Cwik  wrote:
>>
>>> The 2.22 release is also being worked on. Rather allow that release pick
>>> up anything that isn't critical.
>>>
>>> On Thu, May 21, 2020 at 3:51 PM Robert Bradshaw 
>>> wrote:
>>>
 We *really* need to automate the building and deploying of artifacts,
 rather than have so many manual steps...

 The new set of wheels look good now. Verified all the hashes and
 signatures and source tarball contents as well. Ran a couple of test
 pipelines.

 I noticed https://github.com/apache/beam/pull/11722 was just merged.
 Are we OK excluding that? Other than that looks good.

 +1 (binding) pending the one question above.

 On Thu, May 21, 2020 at 10:49 AM Kyle Weaver 
 wrote:

> Nevermind, uploading the wheels to dist.apache.org is part
> of ./sign_hash_python_wheels.sh, which I forgot to run. Wheels should be 
> up
> to date now, PTAL.
>
> On Thu, May 21, 2020 at 1:27 PM Kyle Weaver 
> wrote:
>
>> > -1, the wheel files seem to be built against the wrong commit.
>>
>> Thanks for catching that Robert. I had to rebuild the wheels after
>> some cherry picks. I validated that the wheels in 
>> gs://beam-wheels-staging
>> are up to date. They then must not have overwritten the wheels on
>> dist.apache.org properly, which I assume we expect the Travis build
>> to do. I might have to copy over the new wheels myself.
>>
>> > Since the current RC has been -1ed maybe we can include BEAM-9887 as
>> > part of the next RC, no?
>>
>> At this point, there is no need to go to a full second RC. If there
>> turn out to be blocking issues with RC #1 that necessitate RC #2, we can
>> consider including BEAM-9887 then.
>>
>> On Thu, May 21, 2020 at 3:41 AM Ismaël Mejía 
>> wrote:
>>
>>> Since the current RC has been -1ed maybe we can include BEAM-9887 as
>>> part of the next RC, no?
>>> It is definitely not a blocker but a nice to have.
>>>
>>> On Thu, May 21, 2020 at 2:26 AM Robert Bradshaw 
>>> wrote:
>>> >
>>> > -1, the wheel files seem to be built against the wrong commit. E.g.
>>> >
>>> > unzip -p
>>> apache_beam-2.21.0-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
>>> apache_beam/runners/worker/bundle_processor.py | head -n 40
>>> >
>>> > notice the missing "import bisect" (among other things) missing
>>> from
>>> https://github.com/apache/beam/blob/release-2.21.0/sdks/python/apache_beam/runners/worker/bundle_processor.py
>>> .
>>> >
>>> > (I do agree that BEAM-9887 isn't severe enough to hold up the
>>> release at this point.)
>>> >
>>> >
>>> > On Tue, May 19, 2020 at 8:48 PM rahul patwari <
>>> rahulpatwari8...@gmail.com> wrote:
>>> >>
>>> >> Hi Luke,
>>> >>
>>> >> The release is not severely broken without PR #11609.
>>> >> The PR ensures that, while building a Row with Logical Type, the
>>> input value provided is proper. If we take FixedBytes logical type with
>>> length 10, for example, the proper input value will be a byte array of
>>> length 10. But, without this PR, for FixedBytes logical type, the Row 
>>> will
>>> be built with input value with length less than the expected length.
>>> >> But, as long as the input value provided is correct, there
>>> shouldn't be any problems.
>>> >> I will change the fix version as 2.22.0 for BEAM-9887.
>>> >>
>>> >> Regards,
>>> >> Rahul
>>> >>
>>> >> On Wed, May 20, 2020 at 8:51 AM Luke Cwik 
>>> wrote:
>>> >>>
>>> >>> Rahul, do you believe that the release is severely broken
>>> without PR/11609 enough to require another release candidate or would
>>> waiting till 2.22 (which is due to be cut tomorrow)?
>>> >>>
>>> >>> On Tue, May 19, 2020 at 8:13 PM rahul patwari <
>>> rahulpatwari8...@gmail.com> wrote:
>>> 
>>>  Hi,
>>> 
>>>  Can the PR: https://github.com/apache/beam/pull/11609 be
>>> cherry-picked for 2.21.0 release?

Re: [VOTE] Release 2.21.0, release candidate #1

[Thomas Weise]

On Thu, May 21, 2020 at 4:08 PM Robert Bradshaw  wrote:

> On Thu, May 21, 2020 at 3:55 PM Luke Cwik  wrote:
>
>> The 2.22 release is also being worked on. Rather allow that release pick
>> up anything that isn't critical.
>>
>
> +1. The question is whether this will result in broken installs (e.g. if
> we don't exclude Flink 1.10.1 and the user tries to use it with this
> release).
>

That's fine. The RC, which is based on Flink 1.10.0, will also work with a
Flink 1.10.1 cluster.

Re: Apache Beam Fixit 2020-06-08

[Valentyn Tymofieiev]

Thanks, Tyson.

I noticed a few links could be corrected, fixes inline.

On Thu, May 21, 2020 at 4:13 PM Tyson Hamilton  wrote:

> Why
>
> Google has a Fixit week to improve Engineering Excellence and we want to
> use it to improve Apache Beam!
> What
>
> For the week of 2020-06-08 you will see an increased focus from Google
> engineers on improving Engineering Excellence. We’re asking the community
> to join in and commit as much, or as little, as you’re able to -- anything
> helps!
>
> Goals:
>
>-
>
>Reduce flakiness
>
> 
>[1] in tests across unit/integration tests in Jenkins.
>-
>
>Reduce overall Jenkins usage.
>-
>
>Improve project health signals: add signal, fix signal, split up jobs
>for clearer signals.
>-
>
>   See post-commit greenness
>   
> 
>   [2], coverage
>   
> 
>   [3], precommit latency
>   
> 
>   [4].
>
>
> Examples
>
>-
>
>triaging @Ignored tests and possibly removing decrepit ones
>
> 
>[5]
>-
>
>[BEAM-] Remove support for EOLed runners (Apex, etc.)
> [6]
>-
>
>[BEAM-2762] Coverage report for Python code
> [7]
>-
>
>[BEAM-1399] Coverage numbers are not accurate
> [8]
>-
>
>Jira issues for test failures
>
> 
>[9]
>
> Jira issues for test failures link

.

>
> How
>
>-
>
>Check the ‘beam-fixit’
>
> 
>[10] Jira label for open issues or see the suggestions above.
>
> Jira issues with beam-fixit label link
.


>-
>
>   Ensure the issue you’re working on, or completed, is on the
>   beam-fixit
>   
> 
>   Jira label.
>   -
>
>Assign an issue to yourself and FIXIT!
>
>
> Text Links:
>
> [1]:
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20(labels%20%3D%20flake%20OR%20text%20~%20%22flake%22)%20AND%20statusCategory%20%3D%20%22To%20Do%22%20
>
> [2]:
> http://metrics.beam.apache.org/d/8N6LVeCmk/post-commits-status-dashboard?orgId=1=30s
>
> [3]:
> https://issues.apache.org/jira/browse/BEAM-2762?jql=project%20%3D%20BEAM%20AND%20text%20~%20%22coverage%22
>
> [4]:
> http://metrics.beam.apache.org/d/_TNndF2iz/pre-commit-test-latency?orgId=1
>
> [5]:
> https://lists.apache.org/thread.html/r5d62b55a6c2792c0a560dc1dcad71ef4e70668cd62ef44da76672264%40%3Cdev.beam.apache.org%3E
>
> [6]: https://issues.apache.org/jira/browse/BEAM-
>
> [7]: https://issues.apache.org/jira/browse/BEAM-2762
>
> [8]: https://issues.apache.org/jira/browse/BEAM-1399
>
> [9]:
> https://issues.apache.org/jira/browse/BEAM-9745?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20test-
> 
>
>
failures%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> 
>

Re: [VOTE] Release 2.21.0, release candidate #1

[Ahmet Altay]

+1 (again, validated with new whl files.)

What about https://issues.apache.org/jira/browse/BEAM-10015? Is this a
blocker?

+Kenneth Knowles  +Reuven Lax  -- since
you are both tagged on the JIRA.

On Thu, May 21, 2020 at 4:09 PM Kyle Weaver  wrote:

> > We *really* need to automate the building and deploying of artifacts,
> rather than have so many manual steps...
>
> Agreed. Luckily building wheels is one of only a couple steps that aren't
> automated yet. Partially related:
> https://issues.apache.org/jira/browse/BEAM-9388.
>
> On Thu, May 21, 2020 at 6:55 PM Luke Cwik  wrote:
>
>> The 2.22 release is also being worked on. Rather allow that release pick
>> up anything that isn't critical.
>>
>> On Thu, May 21, 2020 at 3:51 PM Robert Bradshaw 
>> wrote:
>>
>>> We *really* need to automate the building and deploying of artifacts,
>>> rather than have so many manual steps...
>>>
>>> The new set of wheels look good now. Verified all the hashes and
>>> signatures and source tarball contents as well. Ran a couple of test
>>> pipelines.
>>>
>>> I noticed https://github.com/apache/beam/pull/11722 was just merged.
>>> Are we OK excluding that? Other than that looks good.
>>>
>>> +1 (binding) pending the one question above.
>>>
>>> On Thu, May 21, 2020 at 10:49 AM Kyle Weaver 
>>> wrote:
>>>
 Nevermind, uploading the wheels to dist.apache.org is part
 of ./sign_hash_python_wheels.sh, which I forgot to run. Wheels should be up
 to date now, PTAL.

 On Thu, May 21, 2020 at 1:27 PM Kyle Weaver 
 wrote:

> > -1, the wheel files seem to be built against the wrong commit.
>
> Thanks for catching that Robert. I had to rebuild the wheels after
> some cherry picks. I validated that the wheels in gs://beam-wheels-staging
> are up to date. They then must not have overwritten the wheels on
> dist.apache.org properly, which I assume we expect the Travis build
> to do. I might have to copy over the new wheels myself.
>
> > Since the current RC has been -1ed maybe we can include BEAM-9887 as
> > part of the next RC, no?
>
> At this point, there is no need to go to a full second RC. If there
> turn out to be blocking issues with RC #1 that necessitate RC #2, we can
> consider including BEAM-9887 then.
>
> On Thu, May 21, 2020 at 3:41 AM Ismaël Mejía 
> wrote:
>
>> Since the current RC has been -1ed maybe we can include BEAM-9887 as
>> part of the next RC, no?
>> It is definitely not a blocker but a nice to have.
>>
>> On Thu, May 21, 2020 at 2:26 AM Robert Bradshaw 
>> wrote:
>> >
>> > -1, the wheel files seem to be built against the wrong commit. E.g.
>> >
>> > unzip -p
>> apache_beam-2.21.0-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
>> apache_beam/runners/worker/bundle_processor.py | head -n 40
>> >
>> > notice the missing "import bisect" (among other things) missing
>> from
>> https://github.com/apache/beam/blob/release-2.21.0/sdks/python/apache_beam/runners/worker/bundle_processor.py
>> .
>> >
>> > (I do agree that BEAM-9887 isn't severe enough to hold up the
>> release at this point.)
>> >
>> >
>> > On Tue, May 19, 2020 at 8:48 PM rahul patwari <
>> rahulpatwari8...@gmail.com> wrote:
>> >>
>> >> Hi Luke,
>> >>
>> >> The release is not severely broken without PR #11609.
>> >> The PR ensures that, while building a Row with Logical Type, the
>> input value provided is proper. If we take FixedBytes logical type with
>> length 10, for example, the proper input value will be a byte array of
>> length 10. But, without this PR, for FixedBytes logical type, the Row 
>> will
>> be built with input value with length less than the expected length.
>> >> But, as long as the input value provided is correct, there
>> shouldn't be any problems.
>> >> I will change the fix version as 2.22.0 for BEAM-9887.
>> >>
>> >> Regards,
>> >> Rahul
>> >>
>> >> On Wed, May 20, 2020 at 8:51 AM Luke Cwik 
>> wrote:
>> >>>
>> >>> Rahul, do you believe that the release is severely broken without
>> PR/11609 enough to require another release candidate or would waiting 
>> till
>> 2.22 (which is due to be cut tomorrow)?
>> >>>
>> >>> On Tue, May 19, 2020 at 8:13 PM rahul patwari <
>> rahulpatwari8...@gmail.com> wrote:
>> 
>>  Hi,
>> 
>>  Can the PR: https://github.com/apache/beam/pull/11609 be
>> cherry-picked for 2.21.0 release?
>>  If not, the fix version has to be changed for BEAM-9887.
>> 
>>  Regards,
>>  Rahul
>> 
>>  On Wed, May 20, 2020 at 6:05 AM Ahmet Altay 
>> wrote:
>> >
>> > +1, I validated python 2 and 3 quickstarts.
>> >
>> > On Tue, May 19, 2020 at 4:57 PM 

Apache Beam Fixit 2020-06-08

[Tyson Hamilton]

Why

Google has a Fixit week to improve Engineering Excellence and we want to
use it to improve Apache Beam!
What

For the week of 2020-06-08 you will see an increased focus from Google
engineers on improving Engineering Excellence. We’re asking the community
to join in and commit as much, or as little, as you’re able to -- anything
helps!

Goals:

   -

   Reduce flakiness
   

   [1] in tests across unit/integration tests in Jenkins.
   -

   Reduce overall Jenkins usage.
   -

   Improve project health signals: add signal, fix signal, split up jobs
   for clearer signals.
   -

  See post-commit greenness
  

  [2], coverage
  

  [3], precommit latency
  

  [4].


Examples

   -

   triaging @Ignored tests and possibly removing decrepit ones
   

   [5]
   -

   [BEAM-] Remove support for EOLed runners (Apex, etc.)
    [6]
   -

   [BEAM-2762] Coverage report for Python code
    [7]
   -

   [BEAM-1399] Coverage numbers are not accurate
    [8]
   -

   Jira issues for test failures
   

   [9]


How

   -

   Check the ‘beam-fixit’
   

   [10] Jira label for open issues or see the suggestions above.
   -

  Ensure the issue you’re working on, or completed, is on the beam-fixit
  

  Jira label.
  -

   Assign an issue to yourself and FIXIT!


Text Links:

[1]:
https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20(labels%20%3D%20flake%20OR%20text%20~%20%22flake%22)%20AND%20statusCategory%20%3D%20%22To%20Do%22%20

[2]:
http://metrics.beam.apache.org/d/8N6LVeCmk/post-commits-status-dashboard?orgId=1=30s

[3]:
https://issues.apache.org/jira/browse/BEAM-2762?jql=project%20%3D%20BEAM%20AND%20text%20~%20%22coverage%22

[4]:
http://metrics.beam.apache.org/d/_TNndF2iz/pre-commit-test-latency?orgId=1

[5]:
https://lists.apache.org/thread.html/r5d62b55a6c2792c0a560dc1dcad71ef4e70668cd62ef44da76672264%40%3Cdev.beam.apache.org%3E

[6]: https://issues.apache.org/jira/browse/BEAM-

[7]: https://issues.apache.org/jira/browse/BEAM-2762

[8]: https://issues.apache.org/jira/browse/BEAM-1399

[9]:
https://issues.apache.org/jira/browse/BEAM-9745?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20test-failures%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC

[10]:
https://issues.apache.org/jira/browse/BEAM-?jql=labels%20%3D%20beam-fixit

Best,
-Tyson

Re: [VOTE] Release 2.21.0, release candidate #1

[Kyle Weaver]

> We *really* need to automate the building and deploying of artifacts,
rather than have so many manual steps...

Agreed. Luckily building wheels is one of only a couple steps that aren't
automated yet. Partially related:
https://issues.apache.org/jira/browse/BEAM-9388.

On Thu, May 21, 2020 at 6:55 PM Luke Cwik  wrote:

> The 2.22 release is also being worked on. Rather allow that release pick
> up anything that isn't critical.
>
> On Thu, May 21, 2020 at 3:51 PM Robert Bradshaw 
> wrote:
>
>> We *really* need to automate the building and deploying of artifacts,
>> rather than have so many manual steps...
>>
>> The new set of wheels look good now. Verified all the hashes and
>> signatures and source tarball contents as well. Ran a couple of test
>> pipelines.
>>
>> I noticed https://github.com/apache/beam/pull/11722 was just merged. Are
>> we OK excluding that? Other than that looks good.
>>
>> +1 (binding) pending the one question above.
>>
>> On Thu, May 21, 2020 at 10:49 AM Kyle Weaver  wrote:
>>
>>> Nevermind, uploading the wheels to dist.apache.org is part
>>> of ./sign_hash_python_wheels.sh, which I forgot to run. Wheels should be up
>>> to date now, PTAL.
>>>
>>> On Thu, May 21, 2020 at 1:27 PM Kyle Weaver  wrote:
>>>
 > -1, the wheel files seem to be built against the wrong commit.

 Thanks for catching that Robert. I had to rebuild the wheels after some
 cherry picks. I validated that the wheels in gs://beam-wheels-staging are
 up to date. They then must not have overwritten the wheels on
 dist.apache.org properly, which I assume we expect the Travis build to
 do. I might have to copy over the new wheels myself.

 > Since the current RC has been -1ed maybe we can include BEAM-9887 as
 > part of the next RC, no?

 At this point, there is no need to go to a full second RC. If there
 turn out to be blocking issues with RC #1 that necessitate RC #2, we can
 consider including BEAM-9887 then.

 On Thu, May 21, 2020 at 3:41 AM Ismaël Mejía  wrote:

> Since the current RC has been -1ed maybe we can include BEAM-9887 as
> part of the next RC, no?
> It is definitely not a blocker but a nice to have.
>
> On Thu, May 21, 2020 at 2:26 AM Robert Bradshaw 
> wrote:
> >
> > -1, the wheel files seem to be built against the wrong commit. E.g.
> >
> > unzip -p
> apache_beam-2.21.0-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
> apache_beam/runners/worker/bundle_processor.py | head -n 40
> >
> > notice the missing "import bisect" (among other things) missing from
> https://github.com/apache/beam/blob/release-2.21.0/sdks/python/apache_beam/runners/worker/bundle_processor.py
> .
> >
> > (I do agree that BEAM-9887 isn't severe enough to hold up the
> release at this point.)
> >
> >
> > On Tue, May 19, 2020 at 8:48 PM rahul patwari <
> rahulpatwari8...@gmail.com> wrote:
> >>
> >> Hi Luke,
> >>
> >> The release is not severely broken without PR #11609.
> >> The PR ensures that, while building a Row with Logical Type, the
> input value provided is proper. If we take FixedBytes logical type with
> length 10, for example, the proper input value will be a byte array of
> length 10. But, without this PR, for FixedBytes logical type, the Row will
> be built with input value with length less than the expected length.
> >> But, as long as the input value provided is correct, there
> shouldn't be any problems.
> >> I will change the fix version as 2.22.0 for BEAM-9887.
> >>
> >> Regards,
> >> Rahul
> >>
> >> On Wed, May 20, 2020 at 8:51 AM Luke Cwik  wrote:
> >>>
> >>> Rahul, do you believe that the release is severely broken without
> PR/11609 enough to require another release candidate or would waiting till
> 2.22 (which is due to be cut tomorrow)?
> >>>
> >>> On Tue, May 19, 2020 at 8:13 PM rahul patwari <
> rahulpatwari8...@gmail.com> wrote:
> 
>  Hi,
> 
>  Can the PR: https://github.com/apache/beam/pull/11609 be
> cherry-picked for 2.21.0 release?
>  If not, the fix version has to be changed for BEAM-9887.
> 
>  Regards,
>  Rahul
> 
>  On Wed, May 20, 2020 at 6:05 AM Ahmet Altay 
> wrote:
> >
> > +1, I validated python 2 and 3 quickstarts.
> >
> > On Tue, May 19, 2020 at 4:57 PM Hannah Jiang <
> hannahji...@google.com> wrote:
> >>
> >> I confirmed that licenses/notices/source code are added to Java
> and Python docker images as expected.
> >>
> >>
> >> On Tue, May 19, 2020 at 2:36 PM Kyle Weaver <
> kcwea...@google.com> wrote:
> >>>
> >>> Thanks for bringing that up Steve. I'll leave it to others to
> vote on whether that 

Re: [VOTE] Release 2.21.0, release candidate #1

[Robert Bradshaw]

On Thu, May 21, 2020 at 3:55 PM Luke Cwik  wrote:

> The 2.22 release is also being worked on. Rather allow that release pick
> up anything that isn't critical.
>

+1. The question is whether this will result in broken installs (e.g. if we
don't exclude Flink 1.10.1 and the user tries to use it with this release).


> On Thu, May 21, 2020 at 3:51 PM Robert Bradshaw 
> wrote:
>
>> We *really* need to automate the building and deploying of artifacts,
>> rather than have so many manual steps...
>>
>> The new set of wheels look good now. Verified all the hashes and
>> signatures and source tarball contents as well. Ran a couple of test
>> pipelines.
>>
>> I noticed https://github.com/apache/beam/pull/11722 was just merged. Are
>> we OK excluding that? Other than that looks good.
>>
>> +1 (binding) pending the one question above.
>>
>> On Thu, May 21, 2020 at 10:49 AM Kyle Weaver  wrote:
>>
>>> Nevermind, uploading the wheels to dist.apache.org is part
>>> of ./sign_hash_python_wheels.sh, which I forgot to run. Wheels should be up
>>> to date now, PTAL.
>>>
>>> On Thu, May 21, 2020 at 1:27 PM Kyle Weaver  wrote:
>>>
 > -1, the wheel files seem to be built against the wrong commit.

 Thanks for catching that Robert. I had to rebuild the wheels after some
 cherry picks. I validated that the wheels in gs://beam-wheels-staging are
 up to date. They then must not have overwritten the wheels on
 dist.apache.org properly, which I assume we expect the Travis build to
 do. I might have to copy over the new wheels myself.

 > Since the current RC has been -1ed maybe we can include BEAM-9887 as
 > part of the next RC, no?

 At this point, there is no need to go to a full second RC. If there
 turn out to be blocking issues with RC #1 that necessitate RC #2, we can
 consider including BEAM-9887 then.

 On Thu, May 21, 2020 at 3:41 AM Ismaël Mejía  wrote:

> Since the current RC has been -1ed maybe we can include BEAM-9887 as
> part of the next RC, no?
> It is definitely not a blocker but a nice to have.
>
> On Thu, May 21, 2020 at 2:26 AM Robert Bradshaw 
> wrote:
> >
> > -1, the wheel files seem to be built against the wrong commit. E.g.
> >
> > unzip -p
> apache_beam-2.21.0-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
> apache_beam/runners/worker/bundle_processor.py | head -n 40
> >
> > notice the missing "import bisect" (among other things) missing from
> https://github.com/apache/beam/blob/release-2.21.0/sdks/python/apache_beam/runners/worker/bundle_processor.py
> .
> >
> > (I do agree that BEAM-9887 isn't severe enough to hold up the
> release at this point.)
> >
> >
> > On Tue, May 19, 2020 at 8:48 PM rahul patwari <
> rahulpatwari8...@gmail.com> wrote:
> >>
> >> Hi Luke,
> >>
> >> The release is not severely broken without PR #11609.
> >> The PR ensures that, while building a Row with Logical Type, the
> input value provided is proper. If we take FixedBytes logical type with
> length 10, for example, the proper input value will be a byte array of
> length 10. But, without this PR, for FixedBytes logical type, the Row will
> be built with input value with length less than the expected length.
> >> But, as long as the input value provided is correct, there
> shouldn't be any problems.
> >> I will change the fix version as 2.22.0 for BEAM-9887.
> >>
> >> Regards,
> >> Rahul
> >>
> >> On Wed, May 20, 2020 at 8:51 AM Luke Cwik  wrote:
> >>>
> >>> Rahul, do you believe that the release is severely broken without
> PR/11609 enough to require another release candidate or would waiting till
> 2.22 (which is due to be cut tomorrow)?
> >>>
> >>> On Tue, May 19, 2020 at 8:13 PM rahul patwari <
> rahulpatwari8...@gmail.com> wrote:
> 
>  Hi,
> 
>  Can the PR: https://github.com/apache/beam/pull/11609 be
> cherry-picked for 2.21.0 release?
>  If not, the fix version has to be changed for BEAM-9887.
> 
>  Regards,
>  Rahul
> 
>  On Wed, May 20, 2020 at 6:05 AM Ahmet Altay 
> wrote:
> >
> > +1, I validated python 2 and 3 quickstarts.
> >
> > On Tue, May 19, 2020 at 4:57 PM Hannah Jiang <
> hannahji...@google.com> wrote:
> >>
> >> I confirmed that licenses/notices/source code are added to Java
> and Python docker images as expected.
> >>
> >>
> >> On Tue, May 19, 2020 at 2:36 PM Kyle Weaver <
> kcwea...@google.com> wrote:
> >>>
> >>> Thanks for bringing that up Steve. I'll leave it to others to
> vote on whether that necessitates an RC #2.
> >>>
> >>> On Tue, May 19, 2020 at 5:22 PM Steve Niemitz <
> 

Re: [VOTE] Release 2.21.0, release candidate #1

[Luke Cwik]

The 2.22 release is also being worked on. Rather allow that release pick up
anything that isn't critical.

On Thu, May 21, 2020 at 3:51 PM Robert Bradshaw  wrote:

> We *really* need to automate the building and deploying of artifacts,
> rather than have so many manual steps...
>
> The new set of wheels look good now. Verified all the hashes and
> signatures and source tarball contents as well. Ran a couple of test
> pipelines.
>
> I noticed https://github.com/apache/beam/pull/11722 was just merged. Are
> we OK excluding that? Other than that looks good.
>
> +1 (binding) pending the one question above.
>
> On Thu, May 21, 2020 at 10:49 AM Kyle Weaver  wrote:
>
>> Nevermind, uploading the wheels to dist.apache.org is part
>> of ./sign_hash_python_wheels.sh, which I forgot to run. Wheels should be up
>> to date now, PTAL.
>>
>> On Thu, May 21, 2020 at 1:27 PM Kyle Weaver  wrote:
>>
>>> > -1, the wheel files seem to be built against the wrong commit.
>>>
>>> Thanks for catching that Robert. I had to rebuild the wheels after some
>>> cherry picks. I validated that the wheels in gs://beam-wheels-staging are
>>> up to date. They then must not have overwritten the wheels on
>>> dist.apache.org properly, which I assume we expect the Travis build to
>>> do. I might have to copy over the new wheels myself.
>>>
>>> > Since the current RC has been -1ed maybe we can include BEAM-9887 as
>>> > part of the next RC, no?
>>>
>>> At this point, there is no need to go to a full second RC. If there turn
>>> out to be blocking issues with RC #1 that necessitate RC #2, we can
>>> consider including BEAM-9887 then.
>>>
>>> On Thu, May 21, 2020 at 3:41 AM Ismaël Mejía  wrote:
>>>
 Since the current RC has been -1ed maybe we can include BEAM-9887 as
 part of the next RC, no?
 It is definitely not a blocker but a nice to have.

 On Thu, May 21, 2020 at 2:26 AM Robert Bradshaw 
 wrote:
 >
 > -1, the wheel files seem to be built against the wrong commit. E.g.
 >
 > unzip -p
 apache_beam-2.21.0-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
 apache_beam/runners/worker/bundle_processor.py | head -n 40
 >
 > notice the missing "import bisect" (among other things) missing from
 https://github.com/apache/beam/blob/release-2.21.0/sdks/python/apache_beam/runners/worker/bundle_processor.py
 .
 >
 > (I do agree that BEAM-9887 isn't severe enough to hold up the release
 at this point.)
 >
 >
 > On Tue, May 19, 2020 at 8:48 PM rahul patwari <
 rahulpatwari8...@gmail.com> wrote:
 >>
 >> Hi Luke,
 >>
 >> The release is not severely broken without PR #11609.
 >> The PR ensures that, while building a Row with Logical Type, the
 input value provided is proper. If we take FixedBytes logical type with
 length 10, for example, the proper input value will be a byte array of
 length 10. But, without this PR, for FixedBytes logical type, the Row will
 be built with input value with length less than the expected length.
 >> But, as long as the input value provided is correct, there shouldn't
 be any problems.
 >> I will change the fix version as 2.22.0 for BEAM-9887.
 >>
 >> Regards,
 >> Rahul
 >>
 >> On Wed, May 20, 2020 at 8:51 AM Luke Cwik  wrote:
 >>>
 >>> Rahul, do you believe that the release is severely broken without
 PR/11609 enough to require another release candidate or would waiting till
 2.22 (which is due to be cut tomorrow)?
 >>>
 >>> On Tue, May 19, 2020 at 8:13 PM rahul patwari <
 rahulpatwari8...@gmail.com> wrote:
 
  Hi,
 
  Can the PR: https://github.com/apache/beam/pull/11609 be
 cherry-picked for 2.21.0 release?
  If not, the fix version has to be changed for BEAM-9887.
 
  Regards,
  Rahul
 
  On Wed, May 20, 2020 at 6:05 AM Ahmet Altay 
 wrote:
 >
 > +1, I validated python 2 and 3 quickstarts.
 >
 > On Tue, May 19, 2020 at 4:57 PM Hannah Jiang <
 hannahji...@google.com> wrote:
 >>
 >> I confirmed that licenses/notices/source code are added to Java
 and Python docker images as expected.
 >>
 >>
 >> On Tue, May 19, 2020 at 2:36 PM Kyle Weaver 
 wrote:
 >>>
 >>> Thanks for bringing that up Steve. I'll leave it to others to
 vote on whether that necessitates an RC #2.
 >>>
 >>> On Tue, May 19, 2020 at 5:22 PM Steve Niemitz <
 sniem...@apache.org> wrote:
 
  https://issues.apache.org/jira/browse/BEAM-10015 was marked
 as 2.21 but isn't in the RC1 tag.  It's marked as P1, and seems like the
 implication is that without the fix, pipelines can produce incorrect data.
 Is this a blocker?
 >
 >
 > +Reuven Lax, would this be a 

Re: [VOTE] Release 2.21.0, release candidate #1

[Robert Bradshaw]

We *really* need to automate the building and deploying of artifacts,
rather than have so many manual steps...

The new set of wheels look good now. Verified all the hashes and signatures
and source tarball contents as well. Ran a couple of test pipelines.

I noticed https://github.com/apache/beam/pull/11722 was just merged. Are we
OK excluding that? Other than that looks good.

+1 (binding) pending the one question above.

On Thu, May 21, 2020 at 10:49 AM Kyle Weaver  wrote:

> Nevermind, uploading the wheels to dist.apache.org is part
> of ./sign_hash_python_wheels.sh, which I forgot to run. Wheels should be up
> to date now, PTAL.
>
> On Thu, May 21, 2020 at 1:27 PM Kyle Weaver  wrote:
>
>> > -1, the wheel files seem to be built against the wrong commit.
>>
>> Thanks for catching that Robert. I had to rebuild the wheels after some
>> cherry picks. I validated that the wheels in gs://beam-wheels-staging are
>> up to date. They then must not have overwritten the wheels on
>> dist.apache.org properly, which I assume we expect the Travis build to
>> do. I might have to copy over the new wheels myself.
>>
>> > Since the current RC has been -1ed maybe we can include BEAM-9887 as
>> > part of the next RC, no?
>>
>> At this point, there is no need to go to a full second RC. If there turn
>> out to be blocking issues with RC #1 that necessitate RC #2, we can
>> consider including BEAM-9887 then.
>>
>> On Thu, May 21, 2020 at 3:41 AM Ismaël Mejía  wrote:
>>
>>> Since the current RC has been -1ed maybe we can include BEAM-9887 as
>>> part of the next RC, no?
>>> It is definitely not a blocker but a nice to have.
>>>
>>> On Thu, May 21, 2020 at 2:26 AM Robert Bradshaw 
>>> wrote:
>>> >
>>> > -1, the wheel files seem to be built against the wrong commit. E.g.
>>> >
>>> > unzip -p
>>> apache_beam-2.21.0-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
>>> apache_beam/runners/worker/bundle_processor.py | head -n 40
>>> >
>>> > notice the missing "import bisect" (among other things) missing from
>>> https://github.com/apache/beam/blob/release-2.21.0/sdks/python/apache_beam/runners/worker/bundle_processor.py
>>> .
>>> >
>>> > (I do agree that BEAM-9887 isn't severe enough to hold up the release
>>> at this point.)
>>> >
>>> >
>>> > On Tue, May 19, 2020 at 8:48 PM rahul patwari <
>>> rahulpatwari8...@gmail.com> wrote:
>>> >>
>>> >> Hi Luke,
>>> >>
>>> >> The release is not severely broken without PR #11609.
>>> >> The PR ensures that, while building a Row with Logical Type, the
>>> input value provided is proper. If we take FixedBytes logical type with
>>> length 10, for example, the proper input value will be a byte array of
>>> length 10. But, without this PR, for FixedBytes logical type, the Row will
>>> be built with input value with length less than the expected length.
>>> >> But, as long as the input value provided is correct, there shouldn't
>>> be any problems.
>>> >> I will change the fix version as 2.22.0 for BEAM-9887.
>>> >>
>>> >> Regards,
>>> >> Rahul
>>> >>
>>> >> On Wed, May 20, 2020 at 8:51 AM Luke Cwik  wrote:
>>> >>>
>>> >>> Rahul, do you believe that the release is severely broken without
>>> PR/11609 enough to require another release candidate or would waiting till
>>> 2.22 (which is due to be cut tomorrow)?
>>> >>>
>>> >>> On Tue, May 19, 2020 at 8:13 PM rahul patwari <
>>> rahulpatwari8...@gmail.com> wrote:
>>> 
>>>  Hi,
>>> 
>>>  Can the PR: https://github.com/apache/beam/pull/11609 be
>>> cherry-picked for 2.21.0 release?
>>>  If not, the fix version has to be changed for BEAM-9887.
>>> 
>>>  Regards,
>>>  Rahul
>>> 
>>>  On Wed, May 20, 2020 at 6:05 AM Ahmet Altay 
>>> wrote:
>>> >
>>> > +1, I validated python 2 and 3 quickstarts.
>>> >
>>> > On Tue, May 19, 2020 at 4:57 PM Hannah Jiang <
>>> hannahji...@google.com> wrote:
>>> >>
>>> >> I confirmed that licenses/notices/source code are added to Java
>>> and Python docker images as expected.
>>> >>
>>> >>
>>> >> On Tue, May 19, 2020 at 2:36 PM Kyle Weaver 
>>> wrote:
>>> >>>
>>> >>> Thanks for bringing that up Steve. I'll leave it to others to
>>> vote on whether that necessitates an RC #2.
>>> >>>
>>> >>> On Tue, May 19, 2020 at 5:22 PM Steve Niemitz <
>>> sniem...@apache.org> wrote:
>>> 
>>>  https://issues.apache.org/jira/browse/BEAM-10015 was marked as
>>> 2.21 but isn't in the RC1 tag.  It's marked as P1, and seems like the
>>> implication is that without the fix, pipelines can produce incorrect data.
>>> Is this a blocker?
>>> >
>>> >
>>> > +Reuven Lax, would this be a release blocker?
>>> >
>>> 
>>> 
>>>  On Tue, May 19, 2020 at 4:51 PM Kyle Weaver <
>>> kcwea...@google.com> wrote:
>>> >
>>> > Hi everyone,
>>> > Please review and vote on the release candidate #1 for the
>>> version 2.21.0, as follows:
>>> 

Re: joining Beam JIRA

[Tyson Hamilton]

Hey Ted! Welcome =)

On Thu, May 21, 2020 at 10:34 AM Ahmet Altay  wrote:

> Done. And welcome!
>
> On Thu, May 21, 2020 at 10:32 AM Ted Romer  wrote:
>
>> Hi, this is Ted Romer at Google ... I'd like to get permission to join
>> the JIRA project so that I can assign
>> https://issues.apache.org/jira/browse/BEAM-10055 to myself.
>>
>> My JIRA username is "tedromer".
>>
>> Thanks!
>> Ted
>>
>

Re: [VOTE] Release 2.21.0, release candidate #1

[Kyle Weaver]

Nevermind, uploading the wheels to dist.apache.org is part
of ./sign_hash_python_wheels.sh, which I forgot to run. Wheels should be up
to date now, PTAL.

On Thu, May 21, 2020 at 1:27 PM Kyle Weaver  wrote:

> > -1, the wheel files seem to be built against the wrong commit.
>
> Thanks for catching that Robert. I had to rebuild the wheels after some
> cherry picks. I validated that the wheels in gs://beam-wheels-staging are
> up to date. They then must not have overwritten the wheels on
> dist.apache.org properly, which I assume we expect the Travis build to
> do. I might have to copy over the new wheels myself.
>
> > Since the current RC has been -1ed maybe we can include BEAM-9887 as
> > part of the next RC, no?
>
> At this point, there is no need to go to a full second RC. If there turn
> out to be blocking issues with RC #1 that necessitate RC #2, we can
> consider including BEAM-9887 then.
>
> On Thu, May 21, 2020 at 3:41 AM Ismaël Mejía  wrote:
>
>> Since the current RC has been -1ed maybe we can include BEAM-9887 as
>> part of the next RC, no?
>> It is definitely not a blocker but a nice to have.
>>
>> On Thu, May 21, 2020 at 2:26 AM Robert Bradshaw 
>> wrote:
>> >
>> > -1, the wheel files seem to be built against the wrong commit. E.g.
>> >
>> > unzip -p
>> apache_beam-2.21.0-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
>> apache_beam/runners/worker/bundle_processor.py | head -n 40
>> >
>> > notice the missing "import bisect" (among other things) missing from
>> https://github.com/apache/beam/blob/release-2.21.0/sdks/python/apache_beam/runners/worker/bundle_processor.py
>> .
>> >
>> > (I do agree that BEAM-9887 isn't severe enough to hold up the release
>> at this point.)
>> >
>> >
>> > On Tue, May 19, 2020 at 8:48 PM rahul patwari <
>> rahulpatwari8...@gmail.com> wrote:
>> >>
>> >> Hi Luke,
>> >>
>> >> The release is not severely broken without PR #11609.
>> >> The PR ensures that, while building a Row with Logical Type, the input
>> value provided is proper. If we take FixedBytes logical type with length
>> 10, for example, the proper input value will be a byte array of length 10.
>> But, without this PR, for FixedBytes logical type, the Row will be built
>> with input value with length less than the expected length.
>> >> But, as long as the input value provided is correct, there shouldn't
>> be any problems.
>> >> I will change the fix version as 2.22.0 for BEAM-9887.
>> >>
>> >> Regards,
>> >> Rahul
>> >>
>> >> On Wed, May 20, 2020 at 8:51 AM Luke Cwik  wrote:
>> >>>
>> >>> Rahul, do you believe that the release is severely broken without
>> PR/11609 enough to require another release candidate or would waiting till
>> 2.22 (which is due to be cut tomorrow)?
>> >>>
>> >>> On Tue, May 19, 2020 at 8:13 PM rahul patwari <
>> rahulpatwari8...@gmail.com> wrote:
>> 
>>  Hi,
>> 
>>  Can the PR: https://github.com/apache/beam/pull/11609 be
>> cherry-picked for 2.21.0 release?
>>  If not, the fix version has to be changed for BEAM-9887.
>> 
>>  Regards,
>>  Rahul
>> 
>>  On Wed, May 20, 2020 at 6:05 AM Ahmet Altay 
>> wrote:
>> >
>> > +1, I validated python 2 and 3 quickstarts.
>> >
>> > On Tue, May 19, 2020 at 4:57 PM Hannah Jiang <
>> hannahji...@google.com> wrote:
>> >>
>> >> I confirmed that licenses/notices/source code are added to Java
>> and Python docker images as expected.
>> >>
>> >>
>> >> On Tue, May 19, 2020 at 2:36 PM Kyle Weaver 
>> wrote:
>> >>>
>> >>> Thanks for bringing that up Steve. I'll leave it to others to
>> vote on whether that necessitates an RC #2.
>> >>>
>> >>> On Tue, May 19, 2020 at 5:22 PM Steve Niemitz <
>> sniem...@apache.org> wrote:
>> 
>>  https://issues.apache.org/jira/browse/BEAM-10015 was marked as
>> 2.21 but isn't in the RC1 tag.  It's marked as P1, and seems like the
>> implication is that without the fix, pipelines can produce incorrect data.
>> Is this a blocker?
>> >
>> >
>> > +Reuven Lax, would this be a release blocker?
>> >
>> 
>> 
>>  On Tue, May 19, 2020 at 4:51 PM Kyle Weaver 
>> wrote:
>> >
>> > Hi everyone,
>> > Please review and vote on the release candidate #1 for the
>> version 2.21.0, as follows:
>> > [ ] +1, Approve the release
>> > [ ] -1, Do not approve the release (please provide specific
>> comments)
>> >
>> >
>> > The complete staging area is available for your review, which
>> includes:
>> > * JIRA release notes [1],
>> > * the official Apache source release to be deployed to
>> dist.apache.org [2], which is signed with the key with fingerprint
>> F11E37D7F006D086232876797B6D6673C79AEA72 [3],
>> > * all artifacts to be deployed to the Maven Central Repository
>> [4],
>> > * source code tag "v2.21.0-RC1" [5],
>> > * website pull 

Re: GSoD participation

[Chandan Prakash]

Thanks

On Thu, 21 May 2020, 23:04 Kyle Weaver,  wrote:

> Hi Chandan, sorry about that. Does this link work?
>
>
> https://join.slack.com/t/seasonofdocs/shared_invite/enQtNTc0NDgyOTQ5Nzc4LTliZjVlZjRmNmU5ZmFiNmViNmNiMTM5NTdlNTJiYzIzZTk3MDlhMjM3NzE2MzIxNzIxNmZiMmMzNzRmZTI4NmU
>
> On Thu, May 21, 2020 at 1:32 PM Chandan Prakash <
> b18...@students.iitmandi.ac.in> wrote:
>
>> Thanks,
>>
>> The link to join slack channel is expired .I am new to slack ,so , it
>> will be helpful for me if I get the slack URL .
>>
>>
>> On Thu, 21 May 2020, 22:13 Kyle Weaver,  wrote:
>>
>>> Hi Chandan,
>>>
>>>
>>> Thank you for the introduction and your interest to work on Apache Beam
>>> documentation with Season of Docs. To participate in the program you
>>> need to follow the guides here [1] [2]. If you are new to the program, we
>>> suggest:
>>>
>>>1.
>>>
>>>Start by studying our proposed project ideas and expected
>>>deliverables for each of them [3].
>>>2.
>>>
>>>Explore more in depth the existing related Beam documentation for
>>>each project idea. We provided links to the background material, known
>>>issues and current documentation for both project ideas [4] [5]. Choose 
>>> one
>>>project you like the most.
>>>3.
>>>
>>>Start drafting a proposal with the gaps you have found and ideas for
>>>improvement, and how you would present the new/updated/full 
>>> documentation.
>>>Here are more tips on how to make your proposal stronger [6]. Please,
>>>follow the guides and make sure you cover all points.
>>>4.
>>>
>>>Submit the project proposal to the Google program administrators
>>>during the technical writer application phase. It opens on June 9, 2020. 
>>> If
>>>you want any  feedback for your initial draft, consider using Google
>>>Docs and share on dev@beam.apache.org, so the community members can
>>>leave their comments and suggestions (please check the access to the doc
>>>before sending to the mailing list).
>>>
>>> If you have any ideas that you want to brainstorm about, don’t hesitate
>>> to start a discussion in the community list or reach out on Slack channel
>>> to discuss issues related to GSoD documentation [7]. Once you create an
>>> account join #beam-gsod channel.
>>>
>>> Project administrators will assess proposals based on these guidelines
>>> [8]
>>>
>>> Hope it helps. Let us know if you have more questions.
>>>
>>> Thanks,
>>>
>>> Beam GSoD team
>>>
>>> [1] https://developers.google.com/season-of-docs/docs/tech-writer-guide
>>>
>>> [2] https://developers.google.com/season-of-docs/terms/tech-writer-terms
>>>
>>> [3] https://cwiki.apache.org/confluence/display/BEAM/Google+Season+of+
>>> Docs
>>>
>>> [4] https://cwiki.apache.org/confluence/display/BEAM/Google+Season+of+
>>> Docs
>>> #GoogleSeasonofDocs-1.DeploymentofaFlinkandSparkClusterswithPortableBeam
>>>
>>> [5] https://cwiki.apache.org/confluence/display/BEAM/Google+Season+of+
>>> Docs
>>> #GoogleSeasonofDocs-2.Updateoftherunnercomparisonpage/capabilitymatrix
>>>
>>> [6] https://developers.google.com/season-of-docs/docs
>>> /tech-writer-application-hints
>>>
>>> [7] https://join.slack.com/share/zt-eaml657m-KMamnNZfRF2BB7eQpmvveg
>>>
>>> [8] https://developers.google.com/season-of-docs/docs
>>> /project-selection#assess-proposal
>>>
>>>
>>> On Thu, May 21, 2020 at 12:18 AM Chandan Prakash <
>>> b18...@students.iitmandi.ac.in> wrote:
>>>
 Hello community,
 I am Chandan Prakash , sophomore year student in CS from India. I
 really want to contribute to the documentation . I have gone through the
 project ideas . I am new to Apache beam as I have never used it before.
 Right now, I am exploring Apache beam , runner , SDKs ,etc. But I am
 quite familiar with open source .
 I also want to join slack channel for GSoD idea discussion . But , I
 don't know the workspace URL.
 Suggestions for where should I start are highly appreciated.


 Thanks

>>>

Re: joining Beam JIRA

[Ahmet Altay]

Done. And welcome!

On Thu, May 21, 2020 at 10:32 AM Ted Romer  wrote:

> Hi, this is Ted Romer at Google ... I'd like to get permission to join the
> JIRA project so that I can assign
> https://issues.apache.org/jira/browse/BEAM-10055 to myself.
>
> My JIRA username is "tedromer".
>
> Thanks!
> Ted
>

Re: GSoD participation

[Kyle Weaver]

Hi Chandan, sorry about that. Does this link work?

https://join.slack.com/t/seasonofdocs/shared_invite/enQtNTc0NDgyOTQ5Nzc4LTliZjVlZjRmNmU5ZmFiNmViNmNiMTM5NTdlNTJiYzIzZTk3MDlhMjM3NzE2MzIxNzIxNmZiMmMzNzRmZTI4NmU

On Thu, May 21, 2020 at 1:32 PM Chandan Prakash <
b18...@students.iitmandi.ac.in> wrote:

> Thanks,
>
> The link to join slack channel is expired .I am new to slack ,so , it will
> be helpful for me if I get the slack URL .
>
>
> On Thu, 21 May 2020, 22:13 Kyle Weaver,  wrote:
>
>> Hi Chandan,
>>
>>
>> Thank you for the introduction and your interest to work on Apache Beam
>> documentation with Season of Docs. To participate in the program you
>> need to follow the guides here [1] [2]. If you are new to the program, we
>> suggest:
>>
>>1.
>>
>>Start by studying our proposed project ideas and expected
>>deliverables for each of them [3].
>>2.
>>
>>Explore more in depth the existing related Beam documentation for
>>each project idea. We provided links to the background material, known
>>issues and current documentation for both project ideas [4] [5]. Choose 
>> one
>>project you like the most.
>>3.
>>
>>Start drafting a proposal with the gaps you have found and ideas for
>>improvement, and how you would present the new/updated/full documentation.
>>Here are more tips on how to make your proposal stronger [6]. Please,
>>follow the guides and make sure you cover all points.
>>4.
>>
>>Submit the project proposal to the Google program administrators
>>during the technical writer application phase. It opens on June 9, 2020. 
>> If
>>you want any  feedback for your initial draft, consider using Google
>>Docs and share on dev@beam.apache.org, so the community members can
>>leave their comments and suggestions (please check the access to the doc
>>before sending to the mailing list).
>>
>> If you have any ideas that you want to brainstorm about, don’t hesitate
>> to start a discussion in the community list or reach out on Slack channel
>> to discuss issues related to GSoD documentation [7]. Once you create an
>> account join #beam-gsod channel.
>>
>> Project administrators will assess proposals based on these guidelines [8]
>>
>> Hope it helps. Let us know if you have more questions.
>>
>> Thanks,
>>
>> Beam GSoD team
>>
>> [1] https://developers.google.com/season-of-docs/docs/tech-writer-guide
>>
>> [2] https://developers.google.com/season-of-docs/terms/tech-writer-terms
>>
>> [3] https://cwiki.apache.org/confluence/display/BEAM/Google+Season+of+
>> Docs
>>
>> [4] https://cwiki.apache.org/confluence/display/BEAM/Google+Season+of+
>> Docs
>> #GoogleSeasonofDocs-1.DeploymentofaFlinkandSparkClusterswithPortableBeam
>>
>> [5] https://cwiki.apache.org/confluence/display/BEAM/Google+Season+of+
>> Docs
>> #GoogleSeasonofDocs-2.Updateoftherunnercomparisonpage/capabilitymatrix
>>
>> [6] https://developers.google.com/season-of-docs/docs
>> /tech-writer-application-hints
>>
>> [7] https://join.slack.com/share/zt-eaml657m-KMamnNZfRF2BB7eQpmvveg
>>
>> [8] https://developers.google.com/season-of-docs/docs
>> /project-selection#assess-proposal
>>
>>
>> On Thu, May 21, 2020 at 12:18 AM Chandan Prakash <
>> b18...@students.iitmandi.ac.in> wrote:
>>
>>> Hello community,
>>> I am Chandan Prakash , sophomore year student in CS from India. I really
>>> want to contribute to the documentation . I have gone through the project
>>> ideas . I am new to Apache beam as I have never used it before.
>>> Right now, I am exploring Apache beam , runner , SDKs ,etc. But I am
>>> quite familiar with open source .
>>> I also want to join slack channel for GSoD idea discussion . But , I
>>> don't know the workspace URL.
>>> Suggestions for where should I start are highly appreciated.
>>>
>>>
>>> Thanks
>>>
>>

joining Beam JIRA

[Ted Romer]

Hi, this is Ted Romer at Google ... I'd like to get permission to join the
JIRA project so that I can assign
https://issues.apache.org/jira/browse/BEAM-10055 to myself.

My JIRA username is "tedromer".

Thanks!
Ted

Re: GSoD participation

[Chandan Prakash]

Thanks,

The link to join slack channel is expired .I am new to slack ,so , it will
be helpful for me if I get the slack URL .


On Thu, 21 May 2020, 22:13 Kyle Weaver,  wrote:

> Hi Chandan,
>
>
> Thank you for the introduction and your interest to work on Apache Beam
> documentation with Season of Docs. To participate in the program you need
> to follow the guides here [1] [2]. If you are new to the program, we
> suggest:
>
>1.
>
>Start by studying our proposed project ideas and expected deliverables
>for each of them [3].
>2.
>
>Explore more in depth the existing related Beam documentation for each
>project idea. We provided links to the background material, known issues
>and current documentation for both project ideas [4] [5]. Choose one
>project you like the most.
>3.
>
>Start drafting a proposal with the gaps you have found and ideas for
>improvement, and how you would present the new/updated/full documentation.
>Here are more tips on how to make your proposal stronger [6]. Please,
>follow the guides and make sure you cover all points.
>4.
>
>Submit the project proposal to the Google program administrators
>during the technical writer application phase. It opens on June 9, 2020. If
>you want any  feedback for your initial draft, consider using Google
>Docs and share on dev@beam.apache.org, so the community members can
>leave their comments and suggestions (please check the access to the doc
>before sending to the mailing list).
>
> If you have any ideas that you want to brainstorm about, don’t hesitate to
> start a discussion in the community list or reach out on Slack channel to
> discuss issues related to GSoD documentation [7]. Once you create an
> account join #beam-gsod channel.
>
> Project administrators will assess proposals based on these guidelines [8]
>
> Hope it helps. Let us know if you have more questions.
>
> Thanks,
>
> Beam GSoD team
>
> [1] https://developers.google.com/season-of-docs/docs/tech-writer-guide
>
> [2] https://developers.google.com/season-of-docs/terms/tech-writer-terms
>
> [3] https://cwiki.apache.org/confluence/display/BEAM/Google+Season+of+Docs
>
>
> [4] https://cwiki.apache.org/confluence/display/BEAM/Google+Season+of+Docs
> #GoogleSeasonofDocs-1.DeploymentofaFlinkandSparkClusterswithPortableBeam
>
> [5] https://cwiki.apache.org/confluence/display/BEAM/Google+Season+of+Docs
> #GoogleSeasonofDocs-2.Updateoftherunnercomparisonpage/capabilitymatrix
>
> [6] https://developers.google.com/season-of-docs/docs
> /tech-writer-application-hints
>
> [7] https://join.slack.com/share/zt-eaml657m-KMamnNZfRF2BB7eQpmvveg
>
> [8] https://developers.google.com/season-of-docs/docs
> /project-selection#assess-proposal
>
>
> On Thu, May 21, 2020 at 12:18 AM Chandan Prakash <
> b18...@students.iitmandi.ac.in> wrote:
>
>> Hello community,
>> I am Chandan Prakash , sophomore year student in CS from India. I really
>> want to contribute to the documentation . I have gone through the project
>> ideas . I am new to Apache beam as I have never used it before.
>> Right now, I am exploring Apache beam , runner , SDKs ,etc. But I am
>> quite familiar with open source .
>> I also want to join slack channel for GSoD idea discussion . But , I
>> don't know the workspace URL.
>> Suggestions for where should I start are highly appreciated.
>>
>>
>> Thanks
>>
>

Re: [VOTE] Release 2.21.0, release candidate #1

[Kyle Weaver]

> -1, the wheel files seem to be built against the wrong commit.

Thanks for catching that Robert. I had to rebuild the wheels after some
cherry picks. I validated that the wheels in gs://beam-wheels-staging are
up to date. They then must not have overwritten the wheels on
dist.apache.org properly, which I assume we expect the Travis build to do.
I might have to copy over the new wheels myself.

> Since the current RC has been -1ed maybe we can include BEAM-9887 as
> part of the next RC, no?

At this point, there is no need to go to a full second RC. If there turn
out to be blocking issues with RC #1 that necessitate RC #2, we can
consider including BEAM-9887 then.

On Thu, May 21, 2020 at 3:41 AM Ismaël Mejía  wrote:

> Since the current RC has been -1ed maybe we can include BEAM-9887 as
> part of the next RC, no?
> It is definitely not a blocker but a nice to have.
>
> On Thu, May 21, 2020 at 2:26 AM Robert Bradshaw 
> wrote:
> >
> > -1, the wheel files seem to be built against the wrong commit. E.g.
> >
> > unzip -p
> apache_beam-2.21.0-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
> apache_beam/runners/worker/bundle_processor.py | head -n 40
> >
> > notice the missing "import bisect" (among other things) missing from
> https://github.com/apache/beam/blob/release-2.21.0/sdks/python/apache_beam/runners/worker/bundle_processor.py
> .
> >
> > (I do agree that BEAM-9887 isn't severe enough to hold up the release at
> this point.)
> >
> >
> > On Tue, May 19, 2020 at 8:48 PM rahul patwari <
> rahulpatwari8...@gmail.com> wrote:
> >>
> >> Hi Luke,
> >>
> >> The release is not severely broken without PR #11609.
> >> The PR ensures that, while building a Row with Logical Type, the input
> value provided is proper. If we take FixedBytes logical type with length
> 10, for example, the proper input value will be a byte array of length 10.
> But, without this PR, for FixedBytes logical type, the Row will be built
> with input value with length less than the expected length.
> >> But, as long as the input value provided is correct, there shouldn't be
> any problems.
> >> I will change the fix version as 2.22.0 for BEAM-9887.
> >>
> >> Regards,
> >> Rahul
> >>
> >> On Wed, May 20, 2020 at 8:51 AM Luke Cwik  wrote:
> >>>
> >>> Rahul, do you believe that the release is severely broken without
> PR/11609 enough to require another release candidate or would waiting till
> 2.22 (which is due to be cut tomorrow)?
> >>>
> >>> On Tue, May 19, 2020 at 8:13 PM rahul patwari <
> rahulpatwari8...@gmail.com> wrote:
> 
>  Hi,
> 
>  Can the PR: https://github.com/apache/beam/pull/11609 be
> cherry-picked for 2.21.0 release?
>  If not, the fix version has to be changed for BEAM-9887.
> 
>  Regards,
>  Rahul
> 
>  On Wed, May 20, 2020 at 6:05 AM Ahmet Altay  wrote:
> >
> > +1, I validated python 2 and 3 quickstarts.
> >
> > On Tue, May 19, 2020 at 4:57 PM Hannah Jiang 
> wrote:
> >>
> >> I confirmed that licenses/notices/source code are added to Java and
> Python docker images as expected.
> >>
> >>
> >> On Tue, May 19, 2020 at 2:36 PM Kyle Weaver 
> wrote:
> >>>
> >>> Thanks for bringing that up Steve. I'll leave it to others to vote
> on whether that necessitates an RC #2.
> >>>
> >>> On Tue, May 19, 2020 at 5:22 PM Steve Niemitz 
> wrote:
> 
>  https://issues.apache.org/jira/browse/BEAM-10015 was marked as
> 2.21 but isn't in the RC1 tag.  It's marked as P1, and seems like the
> implication is that without the fix, pipelines can produce incorrect data.
> Is this a blocker?
> >
> >
> > +Reuven Lax, would this be a release blocker?
> >
> 
> 
>  On Tue, May 19, 2020 at 4:51 PM Kyle Weaver 
> wrote:
> >
> > Hi everyone,
> > Please review and vote on the release candidate #1 for the
> version 2.21.0, as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific
> comments)
> >
> >
> > The complete staging area is available for your review, which
> includes:
> > * JIRA release notes [1],
> > * the official Apache source release to be deployed to
> dist.apache.org [2], which is signed with the key with fingerprint
> F11E37D7F006D086232876797B6D6673C79AEA72 [3],
> > * all artifacts to be deployed to the Maven Central Repository
> [4],
> > * source code tag "v2.21.0-RC1" [5],
> > * website pull request listing the release [6], publishing the
> API reference manual [7], and the blog post [8].
> > * Java artifacts were built with Maven 3.6.3 and OpenJDK/Oracle
> JDK 1.8.0.
> > * Python artifacts are deployed along with the source release to
> the dist.apache.org [2].
> > * Validation sheet with a tab for 2.21.0 release to help with
> validation [9].
> > * 

Re: GSoD participation

[Kyle Weaver]

Hi Chandan,


Thank you for the introduction and your interest to work on Apache Beam
documentation with Season of Docs. To participate in the program you need
to follow the guides here [1] [2]. If you are new to the program, we
suggest:

   1.

   Start by studying our proposed project ideas and expected deliverables
   for each of them [3].
   2.

   Explore more in depth the existing related Beam documentation for each
   project idea. We provided links to the background material, known issues
   and current documentation for both project ideas [4] [5]. Choose one
   project you like the most.
   3.

   Start drafting a proposal with the gaps you have found and ideas for
   improvement, and how you would present the new/updated/full documentation.
   Here are more tips on how to make your proposal stronger [6]. Please,
   follow the guides and make sure you cover all points.
   4.

   Submit the project proposal to the Google program administrators during
   the technical writer application phase. It opens on June 9, 2020. If you
   want any  feedback for your initial draft, consider using Google Docs
   and share on dev@beam.apache.org, so the community members can leave
   their comments and suggestions (please check the access to the doc before
   sending to the mailing list).

If you have any ideas that you want to brainstorm about, don’t hesitate to
start a discussion in the community list or reach out on Slack channel to
discuss issues related to GSoD documentation [7]. Once you create an
account join #beam-gsod channel.

Project administrators will assess proposals based on these guidelines [8]

Hope it helps. Let us know if you have more questions.

Thanks,

Beam GSoD team

[1] https://developers.google.com/season-of-docs/docs/tech-writer-guide

[2] https://developers.google.com/season-of-docs/terms/tech-writer-terms

[3] https://cwiki.apache.org/confluence/display/BEAM/Google+Season+of+Docs

[4] https://cwiki.apache.org/confluence/display/BEAM/Google+Season+of+Docs
#GoogleSeasonofDocs-1.DeploymentofaFlinkandSparkClusterswithPortableBeam

[5] https://cwiki.apache.org/confluence/display/BEAM/Google+Season+of+Docs
#GoogleSeasonofDocs-2.Updateoftherunnercomparisonpage/capabilitymatrix

[6] https://developers.google.com/season-of-docs/docs
/tech-writer-application-hints

[7] https://join.slack.com/share/zt-eaml657m-KMamnNZfRF2BB7eQpmvveg

[8] https://developers.google.com/season-of-docs/docs
/project-selection#assess-proposal


On Thu, May 21, 2020 at 12:18 AM Chandan Prakash <
b18...@students.iitmandi.ac.in> wrote:

> Hello community,
> I am Chandan Prakash , sophomore year student in CS from India. I really
> want to contribute to the documentation . I have gone through the project
> ideas . I am new to Apache beam as I have never used it before.
> Right now, I am exploring Apache beam , runner , SDKs ,etc. But I am quite
> familiar with open source .
> I also want to join slack channel for GSoD idea discussion . But , I don't
> know the workspace URL.
> Suggestions for where should I start are highly appreciated.
>
>
> Thanks
>

Re: TextIO. Writing late files

[Reuven Lax]

On Tue, May 19, 2020 at 2:02 AM Maximilian Michels  wrote:

> > This is still confusing to me - why would the messages be dropped as
> late in this case?
>
> Since you previously mentioned that the bug is due to the pane info
> missing, I just pointed out that the WriteFiles logic is expected to
> drop the pane info.
>

I mentioned the other issue, where it appeared that Reshuffle in flink
drops pane info (which seems wrong to me). However the original issue here
is elements dropped due to lateness.


>
> @Jose Would it make sense to file a JIRA and summarize all the findings
> here?
>
> @Jozef What you describe in
> https://www.mail-archive.com/dev@beam.apache.org/msg20186.html is
> expected because Flink does not do a GroupByKey on Reshuffle but just
> redistributes the elements.
>
> Thanks,
> Max
>
> On 18.05.20 21:59, Jose Manuel wrote:
> > Hi Reuven,
> >
> > I can try to explaining what I guess.
> >
> > - There is a source which is reading data entries and updating the
> > watermark.
> > - Then, data entries are grouped and stored in files.
> > - The window information of these data entries are used to emit
> > filenames. Data entries's window and timestamp. PaneInfo is empty.
> > - When a second window is applied to filenames, if allowlateness is zero
> > of lower than the spent time in the previous reading/writing, the
> > filenames are discarded as late.
> >
> > I guess, the key is in
> >
> https://github.com/apache/beam/blob/master/runners/core-java/src/main/java/org/apache/beam/runners/core/LateDataDroppingDoFnRunner.java#L168
> >
> > My assumption is global watermark (or source watermark, I am not sure
> > about the name) is used to evaluate the filenames, what are in an
> > already emitted window.
> >
> > Thanks
> > Jose
> >
> >
> > El lun., 18 may. 2020 a las 18:37, Reuven Lax ( > >) escribió:
> >
> > This is still confusing to me - why would the messages be dropped as
> > late in this case?
> >
> > On Mon, May 18, 2020 at 6:14 AM Maximilian Michels  > > wrote:
> >
> > All runners which use the Beam reference implementation drop the
> > PaneInfo for
> > WriteFilesResult#getPerDestinationOutputFilenames(). That's
> > why we can observe this behavior not only in Flink but also
> Spark.
> >
> > The WriteFilesResult is returned here:
> >
> https://github.com/apache/beam/blob/d773f8ca7a4d63d01472b5eaef8b67157d60f40e/sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java#L363
> >
> > GatherBundlesPerWindow will discard the pane information because
> all
> > buffered elements are emitted in the FinishBundle method which
> > always
> > has a NO_FIRING (unknown) pane info:
> >
> https://github.com/apache/beam/blob/d773f8ca7a4d63d01472b5eaef8b67157d60f40e/sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java#L895
> >
> > So this seems expected behavior. We would need to preserve the
> > panes in
> > the Multimap buffer.
> >
> > -Max
> >
> > On 15.05.20 18:34, Reuven Lax wrote:
> > > Lateness should never be introduced inside a pipeline -
> > generally late
> > > data can only come from a source.  If data was not dropped as
> late
> > > earlier in the pipeline, it should not be dropped after the
> > file write.
> > > I suspect that this is a bug in how the Flink runner handles
> the
> > > Reshuffle transform, but I'm not sure what the exact bug is.
> > >
> > > Reuven
> > >
> > > On Fri, May 15, 2020 at 2:23 AM Jozef Vilcek
> > mailto:jozo.vil...@gmail.com>
> > > >>
> > wrote:
> > >
> > > Hi Jose,
> > >
> > > thank you for putting the effort to get example which
> > > demonstrate your problem.
> > >
> > > You are using a streaming pipeline and it seems that
> > watermark in
> > > downstream already advanced further, so when your File
> > pane arrives,
> > > it is already late. Since you define that lateness is not
> > tolerated,
> > > it is dropped.
> > > I myself never had requirement to specify zero allowed
> > lateness for
> > > streaming. It feels dangerous. Do you have a specific use
> > case?
> > > Also, in may cases, after windowed files are written, I
> > usually
> > > collect them into global window and specify a different
> > triggering
> > > policy for collecting them. Both cases are why I never
> > came across
> > > this situation.
> > >
> > > I do not have an explanation if it is a bug or not. I
> > would guess
> > > that watermark can advance 

Re: Transparency to Beam Digital Summit Planning

[Brittany Hermann]

Hi folks,

Sorry about that. This link should give public access now.

https://docs.google.com/document/d/11PXOBUbeldgPqz6OlTswCal6SxyX76Bb_ZVKBdwsd7o/edit?usp=sharing

Have a great weekend!

On Thu, May 21, 2020 at 5:01 AM Maximilian Michels  wrote:

> +1 for making the notes publicly available. This list is free to join by
> anyone.
>
> On 21.05.20 00:17, Austin Bennett wrote:
> > Should the link/meeting notes be publicly available?  Not just available
> > to individuals plus all of @google?
> >
> >
> >
> > On Wed, May 20, 2020 at 2:06 PM Brittany Hermann  > > wrote:
> >
> > Hi folks,
> >
> > I wanted to provide a few different ways of transparency to you
> > during the planning of the Beam Digital Summit.
> >
> > 1) *Beam Summit Status Reports:* I will be sending out weekly Beam
> > Summit Status Reports which will include the goals, attendees,
> > topics discussed, and decisions made every Wednesday.
> >
> > 2) *Community Guests on Committee Planning Calls:* We would like to
> > invite you to join as a guest to these planning calls. This would
> > allow for observation of the planning process and to see if there
> > are ways for future collaboration on promotions, etc. for the event.
> > If you are interested in joining the first bi-weekly meeting
> > starting next week, please reach out to me and I will send the
> > invite with call-in information directly to you.
> >
> > In the meantime, I have attached this week's Beam Summit Status
> > report below.
> >
> >
> https://docs.google.com/document/d/1_jLhKvW5MTtkHOZDJyzCTSLUDiD4RjlJmU35rXV-3n0/edit?usp=sharing
> >
> > Have a great rest of your week!
> >
> > --
> >
> >
> >
> > Brittany Hermann
> >
> > Open Source Program Manager (Provided by Adecco Staffing)
> >
> > 1190 Bordeaux Drive , Building 4, Sunnyvale, CA 94089
> > <
> https://www.google.com/maps/search/1190+Bordeaux+Drive+,+Building+4,+Sunnyvale,+CA+94089?entry=gmail=g
> >
> >
> >
>


-- 

Brittany Hermann

Open Source Program Manager (Provided by Adecco Staffing)

1190 Bordeaux Drive , Building 4, Sunnyvale, CA 94089

Re: Event Calendar?

[Maximilian Michels]

Would it make sense to combine it with the Apache Beam release calendar?

https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com=America%2FLos_Angeles

On 20.05.20 19:27, Tyson Hamilton wrote:
> +1 a calendar would be nice. 
> 
> On Tue, May 19, 2020 at 3:51 PM Austin Bennett
> mailto:whatwouldausti...@gmail.com>> wrote:
> 
> Hi All,
> 
> As we have events more often that are more accessible (digital),
> wondering whether others see a value of adding a calendar to the
> website?  
> 
> Perhaps related, is it worth
> updating https://beam.apache.org/community/in-person/ <- to
> something that isn't 'in-person' since doing things in-person is
> perhaps (hopefully not completely) a vestige of the past.  
> 
> Cheers,
> Austin
> 

Re: New Dates For Beam Summit Digital 2020

[Maximilian Michels]

Thanks Matthias!

We realized we want this to be a much more community-driven process.
That's why we are planning to be more transparent to give everyone a
chance to get involved in the summit. Given that we now have more time,
this will be much more feasible.

Cheers,
Max

On 18.05.20 20:00, Matthias Baetens wrote:
> Dear Beam community,
> 
> A few weeks ago, we announced the dates for the Beam Digital Summit and
> we know the community received this news with excitement. This is a
> great opportunity to create and share content about streaming analytics
> and the solutions that teams around the world have created using Apache
> Beam and its ecosystem. 
> 
> We have chosen August 24-28th as the new dates. While this has been a
> difficult decision, we think it’s the right decision to ensure we
> produce the best possible event. We encourage you to send your talk
> proposals, anything from use cases, lightning talks, or workshop ideas.
> 
> Based on this change, the CFP will remain open until June 15th. We would
> love to hear about what you are doing with Beam, how to improve it, and
> how to strengthen our community.
> 
> We thank you for your understanding! See you soon!
> 
> -Griselda Cuevas, Brittany Hermann, Maximilian Michels, Austin Bennett,
> Matthias Baetens, Alex Van Boxel
> 

Re: Transparency to Beam Digital Summit Planning

[Maximilian Michels]

+1 for making the notes publicly available. This list is free to join by
anyone.

On 21.05.20 00:17, Austin Bennett wrote:
> Should the link/meeting notes be publicly available?  Not just available
> to individuals plus all of @google?  
> 
> 
> 
> On Wed, May 20, 2020 at 2:06 PM Brittany Hermann  > wrote:
> 
> Hi folks,
> 
> I wanted to provide a few different ways of transparency to you
> during the planning of the Beam Digital Summit. 
> 
> 1) *Beam Summit Status Reports:* I will be sending out weekly Beam
> Summit Status Reports which will include the goals, attendees,
> topics discussed, and decisions made every Wednesday. 
> 
> 2) *Community Guests on Committee Planning Calls:* We would like to
> invite you to join as a guest to these planning calls. This would
> allow for observation of the planning process and to see if there
> are ways for future collaboration on promotions, etc. for the event.
> If you are interested in joining the first bi-weekly meeting
> starting next week, please reach out to me and I will send the
> invite with call-in information directly to you. 
> 
> In the meantime, I have attached this week's Beam Summit Status
> report below. 
> 
> 
> https://docs.google.com/document/d/1_jLhKvW5MTtkHOZDJyzCTSLUDiD4RjlJmU35rXV-3n0/edit?usp=sharing
> 
> Have a great rest of your week! 
> 
> -- 
> 
>   
> 
> Brittany Hermann
> 
> Open Source Program Manager (Provided by Adecco Staffing)
> 
> 1190 Bordeaux Drive , Building 4, Sunnyvale, CA 94089
> 
> 
> 
> 

Re: [VOTE] Release 2.21.0, release candidate #1

[Ismaël Mejía]

Since the current RC has been -1ed maybe we can include BEAM-9887 as
part of the next RC, no?
It is definitely not a blocker but a nice to have.

On Thu, May 21, 2020 at 2:26 AM Robert Bradshaw  wrote:
>
> -1, the wheel files seem to be built against the wrong commit. E.g.
>
> unzip -p 
> apache_beam-2.21.0-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
>  apache_beam/runners/worker/bundle_processor.py | head -n 40
>
> notice the missing "import bisect" (among other things) missing from 
> https://github.com/apache/beam/blob/release-2.21.0/sdks/python/apache_beam/runners/worker/bundle_processor.py.
>
> (I do agree that BEAM-9887 isn't severe enough to hold up the release at this 
> point.)
>
>
> On Tue, May 19, 2020 at 8:48 PM rahul patwari  
> wrote:
>>
>> Hi Luke,
>>
>> The release is not severely broken without PR #11609.
>> The PR ensures that, while building a Row with Logical Type, the input value 
>> provided is proper. If we take FixedBytes logical type with length 10, for 
>> example, the proper input value will be a byte array of length 10. But, 
>> without this PR, for FixedBytes logical type, the Row will be built with 
>> input value with length less than the expected length.
>> But, as long as the input value provided is correct, there shouldn't be any 
>> problems.
>> I will change the fix version as 2.22.0 for BEAM-9887.
>>
>> Regards,
>> Rahul
>>
>> On Wed, May 20, 2020 at 8:51 AM Luke Cwik  wrote:
>>>
>>> Rahul, do you believe that the release is severely broken without PR/11609 
>>> enough to require another release candidate or would waiting till 2.22 
>>> (which is due to be cut tomorrow)?
>>>
>>> On Tue, May 19, 2020 at 8:13 PM rahul patwari  
>>> wrote:

 Hi,

 Can the PR: https://github.com/apache/beam/pull/11609 be cherry-picked for 
 2.21.0 release?
 If not, the fix version has to be changed for BEAM-9887.

 Regards,
 Rahul

 On Wed, May 20, 2020 at 6:05 AM Ahmet Altay  wrote:
>
> +1, I validated python 2 and 3 quickstarts.
>
> On Tue, May 19, 2020 at 4:57 PM Hannah Jiang  
> wrote:
>>
>> I confirmed that licenses/notices/source code are added to Java and 
>> Python docker images as expected.
>>
>>
>> On Tue, May 19, 2020 at 2:36 PM Kyle Weaver  wrote:
>>>
>>> Thanks for bringing that up Steve. I'll leave it to others to vote on 
>>> whether that necessitates an RC #2.
>>>
>>> On Tue, May 19, 2020 at 5:22 PM Steve Niemitz  
>>> wrote:

 https://issues.apache.org/jira/browse/BEAM-10015 was marked as 2.21 
 but isn't in the RC1 tag.  It's marked as P1, and seems like the 
 implication is that without the fix, pipelines can produce incorrect 
 data.  Is this a blocker?
>
>
> +Reuven Lax, would this be a release blocker?
>


 On Tue, May 19, 2020 at 4:51 PM Kyle Weaver  
 wrote:
>
> Hi everyone,
> Please review and vote on the release candidate #1 for the version 
> 2.21.0, as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
>
> The complete staging area is available for your review, which 
> includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to 
> dist.apache.org [2], which is signed with the key with fingerprint 
> F11E37D7F006D086232876797B6D6673C79AEA72 [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.21.0-RC1" [5],
> * website pull request listing the release [6], publishing the API 
> reference manual [7], and the blog post [8].
> * Java artifacts were built with Maven 3.6.3 and OpenJDK/Oracle JDK 
> 1.8.0.
> * Python artifacts are deployed along with the source release to the 
> dist.apache.org [2].
> * Validation sheet with a tab for 2.21.0 release to help with 
> validation [9].
> * Docker images published to Docker Hub [10].
>
> The vote will be open for at least 72 hours. It is adopted by 
> majority approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Kyle
>
> [1] 
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12347143
> [2] https://dist.apache.org/repos/dist/dev/beam/2.21.0/
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> [4] 
> https://repository.apache.org/content/repositories/orgapachebeam-1103/
> [5] https://github.com/apache/beam/releases/tag/v2.21.0-RC1
> [6] https://github.com/apache/beam/pull/11727
> [7] https://github.com/apache/beam-site/pull/603
> [8]