Re: NOTICE: New Python PreCommit jobs

[Chad Dombrova]

There's a lot of value to switching to pytest even without xdist.  Could we
prune back the goals of this first PR to just achieving feature parity with
nose, and make a followup PR for xdist?

-chad

On Mon, Oct 7, 2019 at 12:04 PM Udi Meiri  wrote:

>
>
> On Fri, Oct 4, 2019 at 10:35 AM Chad Dombrova  wrote:
>
>>
>> I have a WiP PR to convert Beam to use pytest, but it's been stalled.
>>>
>>
>> What would it take to get it back on track?
>>
>
> Besides needing to convert ITs (removing save_main_session), which can be
> split out to a later PR, there's verifying that the same set of tests are
> collected for each suite.
>
>
>>
>>
>>> Another nice thing about pytest is that you'll be able to tell which
>>> suite a test belongs to.
>>>
>>
>> pytest has a lot of quality of life improvements over nose.  The biggest
>> and simplest one is that the test name that it prints is in the same format
>> as the runner expects for specifying individual tests to run, so you can
>> just copy and paste on the command line to run that one test.  Genius.
>> Also, since it uses directory names for tests and not module names, you can
>> tab complete.   The whole fixture
>>
> LOL at the copy-paste issue.
>
>
>> concept is also great, since it gives you a new axis for test
>> composability and reuse, instead of just complex sub-classing or
>> copy-and-paste.   After switching to pytest we went through our tests and
>> replaced all of our horrible test mixins with fixtures and the end result
>> is much more legible and maintainable.  There's honestly nothing I miss
>> about nose.
>>
>> -chad
>>
>>
>>
>>
>>

[ANNOUNCE] Beam 2.16.0 Released!

[Mark Liu]

The Apache Beam team is pleased to announce the release of version 2.16.0.

Apache Beam is an open source unified programming model to define and
execute data processing pipelines, including ETL, batch and stream
(continuous) processing. See https://beam.apache.org

You can download the release here:

https://beam.apache.org/get-started/downloads/

This release includes bug fixes, features, and improvements detailed on
the Beam blog: https://beam.apache.org/blog/2019/10/07/beam-2.16.0.html

Thanks to everyone who contributed to this release, and we hope you enjoy
using Beam 2.16.0.

Re: Long Jenkins queue

[Cyrus Maden]

Thanks Pablo! Good to know. The rerunning worked great and the PR is merged
now.

On Mon, Oct 7, 2019 at 3:37 PM Pablo Estrada  wrote:

> Hi Cyrus!
> Being stuck for the whole weekend is not normal. Sorry that you had to
> deal with that. I don't know why that happened - and yes, rerunning is a
> good thing to do.
> In my experience, most jobs should take <2,3 hours to run. If it takes
> longer, something is likely off with the testing infra.
> Best
> -P.
>
> On Mon, Oct 7, 2019 at 12:33 PM Cyrus Maden  wrote:
>
>> I'm re-running all precommits. Is this a good approach for this type of
>> issue?
>>
>> On Mon, Oct 7, 2019 at 3:06 PM Cyrus Maden  wrote:
>>
>>> I'm working on a docs PR[1] for the 2.16 release and the RAT precommit
>>> check is stuck in the Jenkins queue (since Friday). Is it possible to move
>>> this up, or is there a way to get an estimate for how long it will take
>>> the precommit check to run?
>>>
>>> [1] https://github.com/apache/beam/pull/9607
>>>
>>> Best,
>>> Cyrus
>>>
>>

Re: Long Jenkins queue

[Pablo Estrada]

Hi Cyrus!
Being stuck for the whole weekend is not normal. Sorry that you had to deal
with that. I don't know why that happened - and yes, rerunning is a good
thing to do.
In my experience, most jobs should take <2,3 hours to run. If it takes
longer, something is likely off with the testing infra.
Best
-P.

On Mon, Oct 7, 2019 at 12:33 PM Cyrus Maden  wrote:

> I'm re-running all precommits. Is this a good approach for this type of
> issue?
>
> On Mon, Oct 7, 2019 at 3:06 PM Cyrus Maden  wrote:
>
>> I'm working on a docs PR[1] for the 2.16 release and the RAT precommit
>> check is stuck in the Jenkins queue (since Friday). Is it possible to move
>> this up, or is there a way to get an estimate for how long it will take
>> the precommit check to run?
>>
>> [1] https://github.com/apache/beam/pull/9607
>>
>> Best,
>> Cyrus
>>
>

Re: Contributor permission

[Pablo Estrada]

We have some labels: easyfix, newbie, beginner, starter. Check them, and
let us know if they help you find an issue. If not, don't hesitate to ask -
I can try to look for a couple easier beginner contributions.

Best
-P.

On Mon, Oct 7, 2019 at 11:54 AM Leonardo Miguel <
leonardo.mig...@arquivei.com.br> wrote:

> I was going to start with katas.
> I have interest in the sdk-java and sdk-py, so maybe working on
> examples-java and examples-py would be a start.
>
> Em seg, 7 de out de 2019 às 15:30, Rui Wang  escreveu:
>
>> I am not aware of starter tags. Which component/project you are
>> interested in?
>>
>>
>> -Rui
>>
>> On Mon, Oct 7, 2019 at 11:02 AM Leonardo Miguel <
>> leonardo.mig...@arquivei.com.br> wrote:
>>
>>> Hi,
>>>
>>> I've been working with Beam for three years now and I would like to
>>> start contributing.
>>> Could someone please give me permission to assign issues to myself?
>>> My jira username is leonardo.miguel
>>>
>>> Do you have any Labels for begginers that I could start working on?
>>>
>>> Thanks!
>>>
>>> --
>>> []s
>>>
>>> Leonardo Alves Miguel
>>> Data Engineer
>>> (16) 3509-5515 | www.arquivei.com.br
>>> 
>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>> 
>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>> Silício]
>>> 
>>> 
>>> 
>>> 
>>>
>>
>
> --
> []s
>
> Leonardo Alves Miguel
> Data Engineer
> (16) 3509-5515 | www.arquivei.com.br
> 
> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
> 
> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
> Silício]
> 
> 
> 
> 
>

Long Jenkins queue

[Cyrus Maden]

I'm working on a docs PR[1] for the 2.16 release and the RAT precommit
check is stuck in the Jenkins queue (since Friday). Is it possible to move
this up, or is there a way to get an estimate for how long it will take
the precommit check to run?

[1] https://github.com/apache/beam/pull/9607

Best,
Cyrus

Re: Contributor permission

[Leonardo Miguel]

I was going to start with katas.
I have interest in the sdk-java and sdk-py, so maybe working on
examples-java and examples-py would be a start.

Em seg, 7 de out de 2019 às 15:30, Rui Wang  escreveu:

> I am not aware of starter tags. Which component/project you are
> interested in?
>
>
> -Rui
>
> On Mon, Oct 7, 2019 at 11:02 AM Leonardo Miguel <
> leonardo.mig...@arquivei.com.br> wrote:
>
>> Hi,
>>
>> I've been working with Beam for three years now and I would like to start
>> contributing.
>> Could someone please give me permission to assign issues to myself?
>> My jira username is leonardo.miguel
>>
>> Do you have any Labels for begginers that I could start working on?
>>
>> Thanks!
>>
>> --
>> []s
>>
>> Leonardo Alves Miguel
>> Data Engineer
>> (16) 3509-5515 | www.arquivei.com.br
>> 
>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>> 
>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>> Silício]
>> 
>> 
>> 
>> 
>>
>

-- 
[]s

Leonardo Alves Miguel
Data Engineer
(16) 3509-5515 | www.arquivei.com.br

[image: Arquivei.com.br – Inteligência em Notas Fiscais]

[image: Google seleciona Arquivei para imersão e mentoria no Vale do
Silício]

[RESULT] [VOTE] Release 2.16.0, release candidate #1

[Mark Liu]

I'm happy to announce that we have unanimously approved this release.

There are 8 approving votes, 5 of which are binding:
* Ahmet (al...@google.com)
* Pablo (pabl...@google.com)
* Robert (rober...@google.com)
* Kenneth (k...@apache.org)
* Jean-Baptiste (j...@nanthrax.net)

There are no disapproving votes.

Next step is to finalize the release (merge the docs/website/blog PRs,
publish artifacts). Please let me know if you have any questions.

Thanks everyone!

Regards,
Mark

Re: Contributor permission

[Rui Wang]

I am not aware of starter tags. Which component/project you are
interested in?


-Rui

On Mon, Oct 7, 2019 at 11:02 AM Leonardo Miguel <
leonardo.mig...@arquivei.com.br> wrote:

> Hi,
>
> I've been working with Beam for three years now and I would like to start
> contributing.
> Could someone please give me permission to assign issues to myself?
> My jira username is leonardo.miguel
>
> Do you have any Labels for begginers that I could start working on?
>
> Thanks!
>
> --
> []s
>
> Leonardo Alves Miguel
> Data Engineer
> (16) 3509-5515 | www.arquivei.com.br
> 
> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
> 
> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
> Silício]
> 
> 
> 
> 
>

Contributor permission

[Leonardo Miguel]

Hi,

I've been working with Beam for three years now and I would like to start
contributing.
Could someone please give me permission to assign issues to myself?
My jira username is leonardo.miguel

Do you have any Labels for begginers that I could start working on?

Thanks!

-- 
[]s

Leonardo Alves Miguel
Data Engineer
(16) 3509-5515 | www.arquivei.com.br

[image: Arquivei.com.br – Inteligência em Notas Fiscais]

[image: Google seleciona Arquivei para imersão e mentoria no Vale do
Silício]

Re: [VOTE] Release 2.16.0, release candidate #1

[Kyle Weaver]

@Valentyn Tymofieiev  You can use the
`--retain_docker_containers` pipeline option to avoid garbage collection.

On Mon, Oct 7, 2019 at 1:57 AM Maximilian Michels  wrote:

> @Pablo Thanks for asking. Since the user is satisfied with the
> workaround for now, I feel better about the release. It's also good to
> stay on track with the release schedule.
>
> @Ahmet/@Thomas Makes sense to go through the release manager, as it
> prevents confusion around what is currently in the release branch.
>
> On 04.10.19 10:17, Pablo Estrada wrote:
> > Hi all,
> > I looked at https://issues.apache.org/jira/browse/BEAM-8303, and it
> > seems like the user has a workaround - is that correct?
> > If that's the case, then I vote +1.
> >
> > @Max - lmk if you'd like to discuss further, but for now my vote is
> on +1.
> > Best
> > -P.
> >
> > On Fri, Oct 4, 2019 at 9:29 AM Mark Liu  > > wrote:
> >
> > +1 (forgot to vote)
> >
> > I also triggered Java Nexmark on direct, dataflow, spark and flink
> > runner. Didn't saw performance regression from the dashboard
> > (https://apache-beam-testing.appspot.com/dashboard-admin)
> >
> > On Fri, Oct 4, 2019 at 8:23 AM Mark Liu  > > wrote:
> >
> > Thanks for the validation work! I validated following:
> >
> > - Java Quickstart on direct, dataflow,spark local, flink local
> > runner
> > - Java mobile gaming on direct and dataflow runner
> > - Python Quickstart in batch and streaming in py2/3.5/3.6/3.7
> > using wheals/zip
> > - Python Mobile Game in batch/streaming in py2/3.5/3.6/3.7 using
> > wheals/zip on direct and dataflow runner
> >
> > Mark
> >
> > On Thu, Oct 3, 2019 at 6:57 PM Ahmet Altay  > > wrote:
> >
> > I see most of the release validations have been completed
> > and marked in the spreadsheet. Thank you all for doing that.
> > If you have not validated/voted yet please take a look at
> > the release candidate.
> >
> > On Thu, Oct 3, 2019 at 7:59 AM Thomas Weise  > > wrote:
> >
> > I think there is a different reason why the release
> > manager should probably merge/approve all PRs that go
> > into the release branch while the release is in progress:
> >
> > If/when the need arises for another RC, then only those
> > changes should be included that are deemed blockers or
> > explicitly agreed. Otherwise the release can potentially
> > be delayed by modifications that invalidate prior
> > verification or introduce new instability.
> >
> >
> > I agree with this reasoning. It expresses my concern in a
> > more clear way.
> >
> >
> > Thomas
> >
> > On Thu, Oct 3, 2019 at 3:12 AM Maximilian Michels
> > mailto:m...@apache.org>> wrote:
> >
> >   > For the next time, may I suggest asking release
> > manager to do the
> >   > merging to the release branch. We do not know
> > whether there will be an
> >   > RC2 or not. And if there will not be an RC2
> > release branch as of now
> >   > does not directly correspond to what will be
> > released.
> >
> > The ground truth for releases are the release tags,
> > not the release
> > branches. Downstream projects should not depend on
> > the release branches.
> > Release branches are merely important for the
> > process of creating a
> > release, but they lose validity after the RC has
> > been created and released.
> >
> > On 02.10.19 11:45, Ahmet Altay wrote:
> >  > +1 (validated python quickstarts). Thank you Mark.
> >  >
> >  > On Wed, Oct 2, 2019 at 10:49 AM Maximilian
> > Michels mailto:m...@apache.org>
> >  > >>
> > wrote:
> >  >
> >  > Thanks for preparing the release, Mark! I
> > would like to address
> >  > https://issues.apache.org/jira/browse/BEAM-8303
> > in the release. I've
> >  > already merged the fix to the release-2.16.0
> > branch. If we do another
> >  > RC, we could include it. As a user is blocked
> > on this, I would not vote
> >  > 

Re: [portability] Removing the old portable metrics API...

[Robert Bradshaw]

Yes, Dataflow still uses the old API, for both counters and for its
progress/autoscaling mechanisms. We'd need to convert that over as
well (which is on the TODO list but lower than finishing up support
for portability in general).

On Mon, Oct 7, 2019 at 9:56 AM Robert Burke  wrote:
>
> The Go SDK uses the old API [1], but it shouldn't be too hard to migrate it.
>
> The main thing I'd want to do at the same time is move the dependencies on 
> the protos out of that package and have those live only in the harness 
> package [2]. I wasn't aware of that particular separation of concerns until 
> much later, but allows for alternative harness implementations.
>
> I have some other work to get the Per-DoFn profiling metrics (eleemnt count, 
> size, time) into the Go SDK this quarter, so I can handle this then.
>
> [1] 
> https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/metrics/metrics.go#L474
> [2] 
> https://github.com/apache/beam/tree/master/sdks/go/pkg/beam/core/runtime/harness
>
> On Fri, Oct 4, 2019 at 6:14 PM Pablo Estrada  wrote:
>>
>> Hello devs,
>> I recently took a look at how Dataflow is retrieving metrics from the Beam 
>> SDK harnesses, and noticed something. As you may (or may not) remember, the 
>> portability API currently has two ways of reporting metrics. Namely, the 
>> newer MonitoringInfo API[1], and the older Metrics one[2].
>>
>> This is somewhat troublesome because now we have two things that do the same 
>> thing. The SDKs report double the amount of metrics[3][4], and I bet it's 
>> confusing for runner implementers.
>>
>> Luckily, it seems like the Flink and Spark runners do use the new API [5][6] 
>> - yay! : ) - so I guess then the only runner that uses the old API is 
>> Dataflow? (internally)
>>
>> Which way does the Samza runner use? +Hai Lu?
>> How about the Go SDK +Robert Burke ? - Ah I bet this uses the old API?
>>
>> If they all use the MonitoringInfos, we may be able to clean up the old api, 
>> and move to the new one (somewhat)soon : )
>>
>> [1] 
>> https://github.com/apache/beam/blob/v2.15.0/model/fn-execution/src/main/proto/beam_fn_api.proto#L395
>> [2] 
>> https://github.com/apache/beam/blob/v2.15.0/model/fn-execution/src/main/proto/beam_fn_api.proto#L391
>> [3] 
>> https://github.com/apache/beam/blob/c1007b678a00ea85671872236edef940a8e56adc/sdks/python/apache_beam/runners/worker/sdk_worker.py#L406-L414
>> [4] 
>> https://github.com/apache/beam/blob/c1007b678a00ea85671872236edef940a8e56adc/sdks/python/apache_beam/runners/worker/sdk_worker.py#L378-L384
>>
>> [5] 
>> https://github.com/apache/beam/blob/44fa33e6518574cb9561f47774e218e0910093fe/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java#L94-L97
>> [6] 
>> https://github.com/apache/beam/blob/932bd80a17171bd2d8157820ffe09e8389a52b9b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L219-L226

Re: [portability] Removing the old portable metrics API...

[Robert Burke]

The Go SDK uses the old API [1], but it shouldn't be too hard to migrate it.

The main thing I'd want to do at the same time is move the dependencies on
the protos out of that package and have those live only in the harness
package [2]. I wasn't aware of that particular separation of concerns until
much later, but allows for alternative harness implementations.

I have some other work to get the Per-DoFn profiling metrics (eleemnt
count, size, time) into the Go SDK this quarter, so I can handle this then.

[1]
https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/metrics/metrics.go#L474
[2]
https://github.com/apache/beam/tree/master/sdks/go/pkg/beam/core/runtime/harness

On Fri, Oct 4, 2019 at 6:14 PM Pablo Estrada  wrote:

> Hello devs,
> I recently took a look at how Dataflow is retrieving metrics from the Beam
> SDK harnesses, and noticed something. As you may (or may not) remember, the
> portability API currently has two ways of reporting metrics. Namely, the
> newer MonitoringInfo API[1], and the older Metrics one[2].
>
> This is somewhat troublesome because now we have two things that do the
> same thing. The SDKs report double the amount of metrics[3][4], and I bet
> it's confusing for runner implementers.
>
> Luckily, it seems like the Flink and Spark runners do use the new API
> [5][6] - yay! : ) - so I guess then the only runner that uses the old API
> is Dataflow? (internally)
>
> Which way does the Samza runner use? +Hai Lu?
> How about the Go SDK +Robert Burke  ? - Ah I bet this
> uses the old API?
>
> If they all use the MonitoringInfos, we may be able to clean up the old
> api, and move to the new one (somewhat)soon : )
>
> [1]
> https://github.com/apache/beam/blob/v2.15.0/model/fn-execution/src/main/proto/beam_fn_api.proto#L395
> [2]
> https://github.com/apache/beam/blob/v2.15.0/model/fn-execution/src/main/proto/beam_fn_api.proto#L391
> [3]
> https://github.com/apache/beam/blob/c1007b678a00ea85671872236edef940a8e56adc/sdks/python/apache_beam/runners/worker/sdk_worker.py#L406-L414
> [4]
> https://github.com/apache/beam/blob/c1007b678a00ea85671872236edef940a8e56adc/sdks/python/apache_beam/runners/worker/sdk_worker.py#L378-L384
>
> [5]
> https://github.com/apache/beam/blob/44fa33e6518574cb9561f47774e218e0910093fe/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java#L94-L97
> [6]
> https://github.com/apache/beam/blob/932bd80a17171bd2d8157820ffe09e8389a52b9b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L219-L226
>

Re: NOTICE: New Python PreCommit jobs

[Udi Meiri]

On Fri, Oct 4, 2019 at 10:35 AM Chad Dombrova  wrote:

>
> I have a WiP PR to convert Beam to use pytest, but it's been stalled.
>>
>
> What would it take to get it back on track?
>

Besides needing to convert ITs (removing save_main_session), which can be
split out to a later PR, there's verifying that the same set of tests are
collected for each suite.


>
>
>> Another nice thing about pytest is that you'll be able to tell which
>> suite a test belongs to.
>>
>
> pytest has a lot of quality of life improvements over nose.  The biggest
> and simplest one is that the test name that it prints is in the same format
> as the runner expects for specifying individual tests to run, so you can
> just copy and paste on the command line to run that one test.  Genius.
> Also, since it uses directory names for tests and not module names, you can
> tab complete.   The whole fixture
>
LOL at the copy-paste issue.


> concept is also great, since it gives you a new axis for test
> composability and reuse, instead of just complex sub-classing or
> copy-and-paste.   After switching to pytest we went through our tests and
> replaced all of our horrible test mixins with fixtures and the end result
> is much more legible and maintainable.  There's honestly nothing I miss
> about nose.
>
> -chad
>
>
>
>
>


smime.p7s
Description: S/MIME Cryptographic Signature

Re: Contributor permissions

[Paweł Kordek]

Works, thanks a lot!

P.

From: Kenneth Knowles 
Sent: Monday, October 7, 2019 6:02 PM
To: dev 
Subject: Re: Contributor permissions

Great. I have added you now. Try assigning the JIRA to yourself to confirm.

Kenn

On Mon, Oct 7, 2019 at 8:59 AM Paweł Kordek 
mailto:pawel.kor...@outlook.com>> wrote:
Hi Kenneth

Sorry, I forgot to include it in the email, the username is "pawel.kordek".

Thanks
Paweł

From: Kenneth Knowles mailto:k...@apache.org>>
Sent: Monday, October 7, 2019 5:53 PM
To: dev mailto:dev@beam.apache.org>>
Subject: Re: Contributor permissions

Hi Paweł,

Welcome & thanks you for contributing!

Do you have a JIRA account? I searched for your name and did not find anything. 
If you tell me your Jira username I will add you to the "Contributors" role so 
you can assign the JIRA to yourself.

Kenn

On Mon, Oct 7, 2019 at 8:43 AM Paweł Kordek 
mailto:pawel.kor...@outlook.com>> wrote:
Hello

My name is Paweł and I since some time I'm working on PoC of Beam pipelines in 
my company. I have been recently blocked by 
https://issues.apache.org/jira/browse/BEAM-6860 and started digging a little 
into it, finding some possible solutions, therefore I'd like to assign JIRA to 
myself, and then open a PR for discussion.

Best regards
Paweł Kordek

Re: Contributor permissions

[Kenneth Knowles]

Great. I have added you now. Try assigning the JIRA to yourself to confirm.

Kenn

On Mon, Oct 7, 2019 at 8:59 AM Paweł Kordek 
wrote:

> Hi Kenneth
>
> Sorry, I forgot to include it in the email, the username is "pawel.kordek".
>
> Thanks
> Paweł
> --
> *From:* Kenneth Knowles 
> *Sent:* Monday, October 7, 2019 5:53 PM
> *To:* dev 
> *Subject:* Re: Contributor permissions
>
> Hi Paweł,
>
> Welcome & thanks you for contributing!
>
> Do you have a JIRA account? I searched for your name and did not find
> anything. If you tell me your Jira username I will add you to the
> "Contributors" role so you can assign the JIRA to yourself.
>
> Kenn
>
> On Mon, Oct 7, 2019 at 8:43 AM Paweł Kordek 
> wrote:
>
> Hello
>
> My name is Paweł and I since some time I'm working on PoC of Beam
> pipelines in my company. I have been recently blocked by
> https://issues.apache.org/jira/browse/BEAM-6860 and started digging a
> little into it, finding some possible solutions, therefore I'd like to
> assign JIRA to myself, and then open a PR for discussion.
>
> Best regards
> Paweł Kordek
>
>

Re: Contributor permissions

[Paweł Kordek]

Hi Kenneth

Sorry, I forgot to include it in the email, the username is "pawel.kordek".

Thanks
Paweł

From: Kenneth Knowles 
Sent: Monday, October 7, 2019 5:53 PM
To: dev 
Subject: Re: Contributor permissions

Hi Paweł,

Welcome & thanks you for contributing!

Do you have a JIRA account? I searched for your name and did not find anything. 
If you tell me your Jira username I will add you to the "Contributors" role so 
you can assign the JIRA to yourself.

Kenn

On Mon, Oct 7, 2019 at 8:43 AM Paweł Kordek 
mailto:pawel.kor...@outlook.com>> wrote:
Hello

My name is Paweł and I since some time I'm working on PoC of Beam pipelines in 
my company. I have been recently blocked by 
https://issues.apache.org/jira/browse/BEAM-6860 and started digging a little 
into it, finding some possible solutions, therefore I'd like to assign JIRA to 
myself, and then open a PR for discussion.

Best regards
Paweł Kordek

Re: Contributor permissions

[Kenneth Knowles]

Hi Paweł,

Welcome & thanks you for contributing!

Do you have a JIRA account? I searched for your name and did not find
anything. If you tell me your Jira username I will add you to the
"Contributors" role so you can assign the JIRA to yourself.

Kenn

On Mon, Oct 7, 2019 at 8:43 AM Paweł Kordek 
wrote:

> Hello
>
> My name is Paweł and I since some time I'm working on PoC of Beam
> pipelines in my company. I have been recently blocked by
> https://issues.apache.org/jira/browse/BEAM-6860 and started digging a
> little into it, finding some possible solutions, therefore I'd like to
> assign JIRA to myself, and then open a PR for discussion.
>
> Best regards
> Paweł Kordek
>

Contributor permissions

[Paweł Kordek]

Hello

My name is Paweł and I since some time I'm working on PoC of Beam pipelines in 
my company. I have been recently blocked by 
https://issues.apache.org/jira/browse/BEAM-6860 and started digging a little 
into it, finding some possible solutions, therefore I'd like to assign JIRA to 
myself, and then open a PR for discussion.

Best regards
Paweł Kordek

Re: Intro & request to add me as a contributor

[Kenneth Knowles]

Welcome! That kind of feedback would be a huge contribution.

Kenn

On Mon, Oct 7, 2019 at 12:25 AM Ismaël Mejía  wrote:

> Hello and welcome !
>
> You can now self assign Jira tickets. You don't need any additional
> permissions to open PRs.
> User/customer feedback is really appreciated so feel free to bring more of
> that.
>
> Cheers,
> Ismaël
>
> On Mon, Oct 7, 2019 at 7:58 AM Zoltán Kauker 
> wrote:
> >
> > Dear All,
> >
> > I'm new to this community. I'm a cloud architect at Aliz Technologies,
> working with Beam at a lot of our customers and I think from time-to-time I
> can bring in some useful feedback which can be beneficial for everyone.
> >
> > In urgent cases, I'd like to contribute to the code as well so it would
> be great if you could add me.
> >
> > Jira username: Kauker
> > Github account: zkauker
> >
> > Cheers,
> > Zoltan
>

Re: Introduction to the mailing list

[Ismaël Mejía]

Welcome Manuela!

The documentation on the Beam website [1] can help you too, in
particular to get some familiarity with the runtime part of Nexmark
(and the Java version of the queries). Maybe worth to try to run some
queries to see if your work environment is ok, and see how things work
and if you see something that can be 'easily' improved.

Since you are more familiar with Python a good exercise is to look at
the existing implementation of some of the queries [2] and try to run
them first with python's direct runner (currently some things are
broken and the expected input format is not the same as in the java
version), so feel free to remove/hack it until it works locally :). A
simple json per line version of the Nexmark objects generated from
java (compatible) is available here [3], so reading it into the
existing python objects in nexmark_model.py could be a first step.

Don't hesitate to contact us if you have any questions or we can
discuss further in private if you prefer.

Regards,
Ismaël

[1] https://beam.apache.org/documentation/sdks/java/testing/nexmark/
[2] 
https://github.com/apache/beam/tree/master/sdks/python/apache_beam/testing/benchmarks/nexmark
[3] https://gist.github.com/iemejia/f4a8d7e78e13265dd4a20243e0c10d87


On Sat, Oct 5, 2019 at 8:31 AM Manuela Chamda Tchakoute
 wrote:
>
> Hello,
>
> Thank you for reaching out. I will take a look at all the links sent and get 
> back to you in case of any queries.
>
> Regards,
> Manuela.
>
> On Sat, Oct 5, 2019, 2:51 AM Kenneth Knowles  wrote:
>>
>> Welcome!
>>
>> On Fri, Oct 4, 2019 at 1:29 PM Ahmet Altay  wrote:
>>>
>>> Welcome Manuela.
>>>
>>> You can look at (https://issues.apache.org/jira/browse/BEAM-2855) as the 
>>> starting point. It has links to 2 previously merged PRs that can serve as 
>>> starting points for writing new nexmark queries. There are nexmark queries 
>>> described in (https://cwiki.apache.org/confluence/display/BEAM/Nexmark). 
>>> Second step would be picking a few more of those and starting with the 
>>> implementation.
>>>
>>> +Ismaël Mejía and I can help with PR reviews and specific questions as well.
>>>
>>> Hope this helps.
>>>
>>> Ahmet
>>>
>>> On Fri, Oct 4, 2019 at 9:43 AM Thomas Weise  wrote:

 Welcome, Manuela!

 For getting familiar with the Beam development environment in general, I 
 would recommend to take a look at:

 https://beam.apache.org/get-started/quickstart-py/

 https://cwiki.apache.org/confluence/display/BEAM/Nexmark

 For contributing and collaborating in general, please take a look at:

 https://beam.apache.org/contribute/

 There is a lot more useful information for contributors on our cwiki:

 https://cwiki.apache.org/confluence/display/BEAM/Apache+Beam

 And don't hesitate to reach out anytime here on this list with questions.

 Thanks,
 Thomas


 On Fri, Oct 4, 2019 at 4:53 AM Manuela Chamda Tchakoute 
  wrote:
>
> Hello.
>
>  My name is Chamda Manuela from the University of Buea, Cameroon. I am 
> new to open source and comfortable with Python programming language. I 
> will like to contribute to the outreachy project "Extend the Nextmark 
> Benchmarking suite in Apache Beam to include python and portable runners".
>
> I will be glad if someone could help me with a step by step guide to get 
> started.
>
>
> Thank you.

Beam Dependency Check Report (2019-10-07)

[Apache Jenkins Server]

High Priority Dependency Updates Of Beam Python SDK:


  Dependency Name
  Current Version
  Latest Version
  Release Date Of the Current Used Version
  Release Date Of The Latest Release
  JIRA Issue
  
mock
2.0.0
3.0.5
2019-05-20
2019-05-20BEAM-7369
oauth2client
3.0.0
4.1.3
2018-12-10
2018-12-10BEAM-6089
Sphinx
1.8.5
2.2.0
2019-05-20
2019-08-19BEAM-7370
High Priority Dependency Updates Of Beam Java SDK:


  Dependency Name
  Current Version
  Latest Version
  Release Date Of the Current Used Version
  Release Date Of The Latest Release
  JIRA Issue
  
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin
0.20.0
0.25.0
2019-02-11
2019-09-16BEAM-6645
com.github.spotbugs:spotbugs
3.1.12
4.0.0-beta4
2019-03-01
2019-09-18BEAM-7792
com.github.spotbugs:spotbugs-annotations
3.1.12
4.0.0-beta4
2019-03-01
2019-09-18BEAM-6951
javax.servlet:javax.servlet-api
3.1.0
4.0.1
2013-04-25
2018-04-20BEAM-5750
org.conscrypt:conscrypt-openjdk
1.1.3
2.2.1
2018-06-04
2019-08-08BEAM-5748
org.eclipse.jetty:jetty-server
9.2.10.v20150310
10.0.0-alpha0
2015-03-10
2019-07-11BEAM-5752
org.eclipse.jetty:jetty-servlet
9.2.10.v20150310
10.0.0-alpha0
2015-03-10
2019-07-11BEAM-5753
Gradle:
5.2.1
5.6.2
2019-08-19
2019-09-09BEAM-8002

 A dependency update is high priority if it satisfies one of following criteria: 

 It has major versions update available, e.g. org.assertj:assertj-core 2.5.0 -> 3.10.0; 


 It is over 3 minor versions behind the latest version, e.g. org.tukaani:xz 1.5 -> 1.8; 


 The current version is behind the later version for over 180 days, e.g. com.google.auto.service:auto-service 2014-10-24 -> 2017-12-11. 

 In Beam, we make a best-effort attempt at keeping all dependencies up-to-date.
 In the future, issues will be filed and tracked for these automatically,
 but in the meantime you can search for existing issues or open a new one.

 For more information:  Beam Dependency Guide  

Re: Beam Summit Videos in youtube

[Maximilian Michels]

Hi Chad,

We're working on this. Sorry this is taking so long. Last time we hired 
a company and we had the edited recordings the day after. This time, we 
are going through a more manual process.


I'll check back about the progress.

Cheers,
Max

On 05.10.19 07:51, Chad Dombrova wrote:

Hi Max,
Any progress on this?  There are a few talks I really want to review!

-chad


On Wed, Sep 18, 2019 at 12:56 PM Maximilian Michels > wrote:


Hi Rahul,

The Beam Summit committee is working on this at the moment. Stay tuned.

Thanks,
Max

On 18.09.19 11:39, rahul patwari wrote:
 > Hi,
 >
 > The videos of Beam Summit that has happened recently have
disappeared
 > from YouTube Apache Beam channel.
 >
 > Is uploading the videos a WIP?
 >
 > Thanks,
 > Rahul
 >
 >

Re: [VOTE] Release 2.16.0, release candidate #1

[Maximilian Michels]

@Pablo Thanks for asking. Since the user is satisfied with the 
workaround for now, I feel better about the release. It's also good to 
stay on track with the release schedule.


@Ahmet/@Thomas Makes sense to go through the release manager, as it 
prevents confusion around what is currently in the release branch.


On 04.10.19 10:17, Pablo Estrada wrote:

Hi all,
I looked at https://issues.apache.org/jira/browse/BEAM-8303, and it 
seems like the user has a workaround - is that correct?

If that's the case, then I vote +1.

@Max - lmk if you'd like to discuss further, but for now my vote is on +1.
Best
-P.

On Fri, Oct 4, 2019 at 9:29 AM Mark Liu > wrote:


+1 (forgot to vote)

I also triggered Java Nexmark on direct, dataflow, spark and flink
runner. Didn't saw performance regression from the dashboard
(https://apache-beam-testing.appspot.com/dashboard-admin)

On Fri, Oct 4, 2019 at 8:23 AM Mark Liu mailto:mark...@google.com>> wrote:

Thanks for the validation work! I validated following:

- Java Quickstart on direct, dataflow,spark local, flink local
runner
- Java mobile gaming on direct and dataflow runner
- Python Quickstart in batch and streaming in py2/3.5/3.6/3.7
using wheals/zip
- Python Mobile Game in batch/streaming in py2/3.5/3.6/3.7 using
wheals/zip on direct and dataflow runner

Mark

On Thu, Oct 3, 2019 at 6:57 PM Ahmet Altay mailto:al...@google.com>> wrote:

I see most of the release validations have been completed
and marked in the spreadsheet. Thank you all for doing that.
If you have not validated/voted yet please take a look at
the release candidate.

On Thu, Oct 3, 2019 at 7:59 AM Thomas Weise mailto:t...@apache.org>> wrote:

I think there is a different reason why the release
manager should probably merge/approve all PRs that go
into the release branch while the release is in progress:

If/when the need arises for another RC, then only those
changes should be included that are deemed blockers or
explicitly agreed. Otherwise the release can potentially
be delayed by modifications that invalidate prior
verification or introduce new instability.


I agree with this reasoning. It expresses my concern in a
more clear way.


Thomas

On Thu, Oct 3, 2019 at 3:12 AM Maximilian Michels
mailto:m...@apache.org>> wrote:

  > For the next time, may I suggest asking release
manager to do the
  > merging to the release branch. We do not know
whether there will be an
  > RC2 or not. And if there will not be an RC2
release branch as of now
  > does not directly correspond to what will be
released.

The ground truth for releases are the release tags,
not the release
branches. Downstream projects should not depend on
the release branches.
Release branches are merely important for the
process of creating a
release, but they lose validity after the RC has
been created and released.

On 02.10.19 11:45, Ahmet Altay wrote:
 > +1 (validated python quickstarts). Thank you Mark.
 >
 > On Wed, Oct 2, 2019 at 10:49 AM Maximilian
Michels mailto:m...@apache.org>
 > >>
wrote:
 >
 >     Thanks for preparing the release, Mark! I
would like to address
 > https://issues.apache.org/jira/browse/BEAM-8303
in the release. I've
 >     already merged the fix to the release-2.16.0
branch. If we do another
 >     RC, we could include it. As a user is blocked
on this, I would not vote
 >     +1 for this RC, but I also do not want to
block the release process.
 >
 >
 > Max, thank you for the clear communication for
the importance and at the
 > same time non-blocking status of the issue.
 >
 > For the next time, may I suggest asking release
manager to do the
 > merging to the release branch. We do not know
whether there will be an
 

Re: Intro & request to add me as a contributor

[Ismaël Mejía]

Hello and welcome !

You can now self assign Jira tickets. You don't need any additional
permissions to open PRs.
User/customer feedback is really appreciated so feel free to bring more of that.

Cheers,
Ismaël

On Mon, Oct 7, 2019 at 7:58 AM Zoltán Kauker  wrote:
>
> Dear All,
>
> I'm new to this community. I'm a cloud architect at Aliz Technologies, 
> working with Beam at a lot of our customers and I think from time-to-time I 
> can bring in some useful feedback which can be beneficial for everyone.
>
> In urgent cases, I'd like to contribute to the code as well so it would be 
> great if you could add me.
>
> Jira username: Kauker
> Github account: zkauker
>
> Cheers,
> Zoltan