Re: Python PVR Reference post-commit tests failing

[Kenneth Knowles]

OK. I don't have a strong feeling about preserving the Java ULR on
`master`. I think since deleting such a large chunk of code is a big deal,
this is a good thing for an explicit vote. That way, anyone who is
following dev@ and wants to preserve the Java ULR on `master` can issue
their -1 vote and we can respect it (but ask to get the testing green). I
would suggest opening a PR with the change, opening a "lazy consensus" vote
on dev@ and then waiting a decent amount of time before merging the
deletion PR.

Kenn

On Fri, Mar 15, 2019 at 4:43 PM Daniel Oliveira 
wrote:

> The ULR used a bunch of code forked from the DirectRunner but I don't
> think it currently shares anything. And if it does share any code that I
> don't know about I expect that the dependency is one-way, i.e. removing the
> ULR shouldn't affect the DirectRunner. The only shared code I know of is
> between the ULR and other portable runners, particularly Flink, but I don't
> think that would be difficult to isolate.
>
> I'm in support of disabling the ULR tests and ok with removing the ULR as
> long as we make sure it can be revived if we want, like with Mikhail's
> suggestion of tagging the commit. I can help with the removal of the ULR
> code since I know specifics about the codebase.
>
> On Thu, Mar 14, 2019 at 2:25 PM Kenneth Knowles  wrote:
>
>> I know the Java DirectRunner shares a lot of code with the ULR. I'm a bit
>> unclear on the delta and how independent they are.
>>
>> Kenn
>>
>> On Thu, Mar 14, 2019 at 2:10 PM Mikhail Gryzykhin 
>> wrote:
>>
>>> @Kenneth
>>> If we disable tests, I'd call Java ULR a dead code.
>>>
>>> One of the better compromises:
>>> 1. disable tests.
>>> 2. Add tag to the last commit where Java ULR existed.
>>> 3. Remove Java ULR from head.
>>>
>>> Keeping history, no extra dead code at head.
>>>
>>> --Mikhail
>>>
>>> Have feedback ?
>>>
>>>
>>> On Thu, Mar 14, 2019 at 1:02 PM Ankur Goenka  wrote:
>>>
 On that note, we should also think about adding PVR for python
 reference runners. Jira:
 https://issues.apache.org/jira/browse/BEAM-6837


 On Thu, Mar 14, 2019 at 12:57 PM Kenneth Knowles 
 wrote:

> How about this compromise:
>
> 1. disable the test since clearly no one is relying on the
> functionality that is broken
> 2. leave the Java ULR as-is for now, and a volunteer can pick it up
> and make it work if they want
>
> Kenn
>
> On Thu, Mar 14, 2019 at 11:41 AM Mikhail Gryzykhin 
> wrote:
>
>> Hi everyone,
>>
>> We have Python PVR Reference post-commit tests failing for quite some
>> time now. These are tests for java reference runner.
>>
>> According to this thread
>> ,
>> we are deciding what to do with java reference runner and might want to
>> remove it from code base.
>>
>> My question is: do we want to a) invest time in fixing python PVR
>> tests, or b) disable this test and start cleaning up code?
>>
>> a) Is worth it if we want to invest into java reference runner in the
>> future.
>> b) Is worth if we want to invest into Python and forfeit java
>> reference runner.
>>
>> Option b) seem more reasonable to me atm, since most people lean
>> towards going forward with Python reference runner.
>>
>> Please, share your thoughts.
>>
>> Regards,
>> --Mikhail
>>
>> Have feedback ?
>>
>

Investigating Jenkins Agents OOM

[Yifan Zou]

Hi,

Some of you are aware of our Jenkins nodes fell in trouble very often.
We're now getting the beam4, 7 and 13 disconnected and jobs failed due to
the similar reason: OOM. I am asking ASF Infra to dump the usage of
resources (memory, disk, cpu...) from those machines before we reboot. In
the meantime, you may see longer waiting time on jobs due to the agent
reduction. We are sorry about it.

I've recorded some Jenkins console logs from those agents into a doc. Please
let us know if you have insights on these problems. Any helps
are appreciated.
https://docs.google.com/document/d/1OBmWumaJCuHPNMHM4-V6JWhYY2ZV_ETqUdFQG5A7R4E/edit?usp=sharing

Regards.
Yifan

Re: Python PVR Reference post-commit tests failing

[Daniel Oliveira]

The ULR used a bunch of code forked from the DirectRunner but I don't think
it currently shares anything. And if it does share any code that I don't
know about I expect that the dependency is one-way, i.e. removing the ULR
shouldn't affect the DirectRunner. The only shared code I know of is
between the ULR and other portable runners, particularly Flink, but I don't
think that would be difficult to isolate.

I'm in support of disabling the ULR tests and ok with removing the ULR as
long as we make sure it can be revived if we want, like with Mikhail's
suggestion of tagging the commit. I can help with the removal of the ULR
code since I know specifics about the codebase.

On Thu, Mar 14, 2019 at 2:25 PM Kenneth Knowles  wrote:

> I know the Java DirectRunner shares a lot of code with the ULR. I'm a bit
> unclear on the delta and how independent they are.
>
> Kenn
>
> On Thu, Mar 14, 2019 at 2:10 PM Mikhail Gryzykhin 
> wrote:
>
>> @Kenneth
>> If we disable tests, I'd call Java ULR a dead code.
>>
>> One of the better compromises:
>> 1. disable tests.
>> 2. Add tag to the last commit where Java ULR existed.
>> 3. Remove Java ULR from head.
>>
>> Keeping history, no extra dead code at head.
>>
>> --Mikhail
>>
>> Have feedback ?
>>
>>
>> On Thu, Mar 14, 2019 at 1:02 PM Ankur Goenka  wrote:
>>
>>> On that note, we should also think about adding PVR for python reference
>>> runners. Jira: https://issues.apache.org/jira/browse/BEAM-6837
>>>
>>>
>>> On Thu, Mar 14, 2019 at 12:57 PM Kenneth Knowles 
>>> wrote:
>>>
 How about this compromise:

 1. disable the test since clearly no one is relying on the
 functionality that is broken
 2. leave the Java ULR as-is for now, and a volunteer can pick it up and
 make it work if they want

 Kenn

 On Thu, Mar 14, 2019 at 11:41 AM Mikhail Gryzykhin 
 wrote:

> Hi everyone,
>
> We have Python PVR Reference post-commit tests failing for quite some
> time now. These are tests for java reference runner.
>
> According to this thread
> ,
> we are deciding what to do with java reference runner and might want to
> remove it from code base.
>
> My question is: do we want to a) invest time in fixing python PVR
> tests, or b) disable this test and start cleaning up code?
>
> a) Is worth it if we want to invest into java reference runner in the
> future.
> b) Is worth if we want to invest into Python and forfeit java
> reference runner.
>
> Option b) seem more reasonable to me atm, since most people lean
> towards going forward with Python reference runner.
>
> Please, share your thoughts.
>
> Regards,
> --Mikhail
>
> Have feedback ?
>

Re: [PROPOSAL] Test performance of basic Apache Beam operations

[Pablo Estrada]

I really like these. Happy to have them.
Best
-P.

On Fri, Mar 15, 2019 at 11:16 AM Łukasz Gajowy  wrote:

> Hi Beamers,
>
> an update on this. Together With Kasia, Michał and cooperating closely
> with Pablo we have created and scheduled a Cron Job running daily 7 tests
> for GroupByKey batch scenarios. Description of the tests is in the proposal
> [1] and will be documented later. The dashboards for the tests:
>  - showing run times [2]
>  - showing total load size (bytes) [3]
>
> All the metrics are collected using Beam's Metrics API.
>
> Things we have on our horizon:
>  - the same set of tests for Java but in streaming mode
>  - similar jobs for Python SDK
>  - running similar suites on Flink runner
>
> We have also created a set of Dataproc bash scripts that can be used to
> set up a Flink cluster that supports portability [4]. It is ready to use
> and I've already successfully run the word count example using Python SDK
> on it. Hoping + aiming to run load tests on it soon. :)
>
> BTW/Last but not least: we also reused some code to collect metrics using
> Metrics API in TextIOIT too and are willing to do a similar change for
> other IOITs. Dashboards for TextIOIT: [5].
>
> Thanks,
> Łukasz
>
> [1] https://s.apache.org/load-test-basic-operations
> [2]
> https://apache-beam-testing.appspot.com/explore?dashboard=5643144871804928
> [3]
> https://apache-beam-testing.appspot.com/explore?dashboard=5701325169885184
> [4]
> https://github.com/apache/beam/blob/b1ed061fd0c1ed1da562089c939d55715907769d/.test-infra/dataproc/create_flink_cluster.sh
> [5]
> https://apache-beam-testing.appspot.com/explore?dashboard=5629522644828160
>
>
>
> śr., 12 wrz 2018 o 14:23 Etienne Chauchot 
> napisał(a):
>
>> Let me elaborate a bit my last sentence
>> Le mardi 11 septembre 2018 à 11:29 +0200, Etienne Chauchot a écrit :
>>
>> Hi Lukasz,
>>
>> Well, having low level byte[] based pure performance tests makes sense.
>> And having high level realistic model (Nexmark auction system) makes sense
>> also to avoid testing unrealistic pipelines as you describe.
>>
>> Have common code between the 2 seems difficult as both the architecture
>> and the model are different.
>>
>> I'm more concerned about having two CI mechanisms to detect
>> functionnal/performance regressions.
>>
>>
>> Even if parts of NexMark and performance tests are the same they could
>> target different objectives: raw performance tests (the new framework) and
>> user oriented tests (nexmark). So they might be complementary.
>>
>> We must just chose how to run them. I think we need to have only one
>> automatic regression detection tool. IMHO, the most relevant for func/perf
>> regression is Nexmark because it represents what a real user could do (it
>> simulates an auction system). So let's  keep it as post commits. Post
>> commits allow to target a particular commit that introduced a regression.
>>
>> We could schedule the new performance tests.
>>
>> Best
>> Etienne
>>
>>
>> Best
>> Etienne
>>
>> Le lundi 10 septembre 2018 à 18:33 +0200, Łukasz Gajowy a écrit :
>>
>> In my opinion and as far as I understand Nexmark, there are some benefits
>> to having both types of tests. The load tests we propose can be very
>> straightforward and clearly show what is being tested thanks to the fact
>> that there's no fixed model but very "low level" KV
>> collections only. They are more flexible in shapes of the pipelines they
>> can express e.g. fanout_64, without having to think about specific use
>> cases.
>>
>> Having both types would allow developers to decide whether they want to
>> create a new Nexmark query for their specific case or develop a new Load
>> test (whatever is easier and more fits their case). However, there is a
>> risk - with KV developer can overemphasize cases that can
>> never happen in practice, so we need to be careful about the exact
>> configurations we run.
>>
>> Still, I can imagine that there surely will be code that should be common
>> to both types of tests and we seek ways to not duplicate code.
>>
>> WDYT?
>>
>> Regards,
>> Łukasz
>>
>>
>>
>> pon., 10 wrz 2018 o 16:36 Etienne Chauchot 
>> napisał(a):
>>
>> Hi,
>> It seems that there is a notable overlap with what Nexmark already does:
>> Nexmark mesures performance and regression by exercising all the Beam
>> model in both batch and streaming modes with several runners. It also
>> computes on synthetic data. Also nexmark is already included as PostCommits
>> in the CI and dashboards.
>>
>> Shall we merge the two?
>>
>> Best
>>
>> Etienne
>>
>> Le lundi 10 septembre 2018 à 12:56 +0200, Łukasz Gajowy a écrit :
>>
>> Hello everyone,
>>
>> thank you for all your comments to the proposal. To sum up:
>>
>> A set of performance tests exercising Core Beam Transforms (ParDo,
>> GroupByKey, CoGroupByKey, Combine) will be implemented for Java and Python
>> SDKs. Those tests will allow to:
>>
>>- measure performance of the transforms on various runners
>>- exercise the transforms by creating 

Re: hi from DevRel land

[Aizhamal Nurmamat kyzy]

Welcome Reza!!!
On Fri, Mar 15, 2019 at 11:33 Chamikara Jayalath 
wrote:

> Welcome Reza.
>
> On Thu, Mar 14, 2019 at 4:47 PM Matthias Baetens <
> baetensmatth...@gmail.com> wrote:
>
>> It's great to have you on board, Rez! To join in on what Kenn was saying:
>> great to have worked on challenging problems with you in the past, it's
>> going to be very helpful to have your experience added to the project!
>>
>> -Matthias
>>
>> On Thu, Mar 14, 2019, 04:06 Connell O'Callaghan 
>> wrote:
>>
>>> Welcome on board Reza!!!
>>>
>>> On Wed, Mar 13, 2019 at 8:41 PM Joana Filipa Bernardo Carrasqueira <
>>> joanafil...@google.com> wrote:
>>>
 Welcome Reza!

 On Wed, Mar 13, 2019 at 3:12 PM David Cavazos 
 wrote:

> Welcome Reza!
>
> On Wed, Mar 13, 2019 at 9:48 AM Mark Liu  wrote:
>
>> Welcome Reza!
>>
>> On Wed, Mar 13, 2019 at 8:09 AM Etienne Chauchot <
>> echauc...@apache.org> wrote:
>>
>>> Welcome onboard Reza
>>>
>>> Etienne
>>>
>>> Le mardi 12 mars 2019 à 09:10 -0700, Rose Nguyen a écrit :
>>>
>>> Welcome, Reza! Really excited to have you!
>>>
>>> On Tue, Mar 12, 2019 at 9:00 AM Reza Ardeshir Rokni <
>>> raro...@gmail.com> wrote:
>>>
>>> Thanx folks!
>>>
>>> Oppsy on the link, here it is again:
>>>
>>> https://stackoverflow.com/questions/54422510/how-to-solve-duplicate-values-exception-when-i-create-pcollectionviewmapstring/54623618#54623618
>>>
>>> On Tue, 12 Mar 2019 at 23:32, Teja MVSR 
>>> wrote:
>>>
>>> Hi Reza,
>>>
>>> I am also interested to contribute towards documentation. Please let
>>> me know if I can be of any help.
>>>
>>> Thanks and Regards,
>>> Teja.
>>>
>>> On Tue, Mar 12, 2019, 11:30 AM Kenneth Knowles 
>>> wrote:
>>>
>>> This is great news.
>>>
>>> For the benefit of the list, I want to say how nice it has been when
>>> I have had a chance to work with you. I've learned a great deal real and
>>> complex use cases through those opportunities. I'm really excited that
>>> you'll be helping out Beam in this new role.
>>>
>>> Kenn
>>>
>>> On Tue, Mar 12, 2019 at 7:21 AM Valentyn Tymofieiev <
>>> valen...@google.com> wrote:
>>>
>>> Hi Reza!
>>>
>>> Welcome to Beam. Very nice to have you onboard. Btw, the link seems
>>> broken.
>>>
>>> Thanks,
>>> Valentyn
>>>
>>> On Tue, Mar 12, 2019 at 6:04 AM Reza Ardeshir Rokni <
>>> raro...@gmail.com> wrote:
>>>
>>> Hi Folks,
>>>
>>> Just wanted to say hi to the good folks in the Beam community in my
>>> new capacity as a Developer advocate for Beam/Dataflow @ Google. :-)
>>>
>>> At the moment I am working on a couple of blogs around the Timer and
>>> State API as well as some work on general patterns that I hope to
>>> contribute as documentation to the Beam site. An example of the patterns
>>> can be seen here:  LINK
>>>
>>> Hope to be adding many more in 2019 and really looking forward to
>>> being able to contribute to Beam in anyway that I can!
>>>
>>> Cheers
>>> Reza
>>>
>>>
>>>
>>>

 --

 *Joana Carrasqueira*

 Cloud Developer Relations Events Manager

 415-602-2507 <(415)%20602-2507> Mobile

 1160 N Mathilda Ave, Sunnyvale, CA 94089
 

Re: hi from DevRel land

[Chamikara Jayalath]

Welcome Reza.

On Thu, Mar 14, 2019 at 4:47 PM Matthias Baetens 
wrote:

> It's great to have you on board, Rez! To join in on what Kenn was saying:
> great to have worked on challenging problems with you in the past, it's
> going to be very helpful to have your experience added to the project!
>
> -Matthias
>
> On Thu, Mar 14, 2019, 04:06 Connell O'Callaghan 
> wrote:
>
>> Welcome on board Reza!!!
>>
>> On Wed, Mar 13, 2019 at 8:41 PM Joana Filipa Bernardo Carrasqueira <
>> joanafil...@google.com> wrote:
>>
>>> Welcome Reza!
>>>
>>> On Wed, Mar 13, 2019 at 3:12 PM David Cavazos 
>>> wrote:
>>>
 Welcome Reza!

 On Wed, Mar 13, 2019 at 9:48 AM Mark Liu  wrote:

> Welcome Reza!
>
> On Wed, Mar 13, 2019 at 8:09 AM Etienne Chauchot 
> wrote:
>
>> Welcome onboard Reza
>>
>> Etienne
>>
>> Le mardi 12 mars 2019 à 09:10 -0700, Rose Nguyen a écrit :
>>
>> Welcome, Reza! Really excited to have you!
>>
>> On Tue, Mar 12, 2019 at 9:00 AM Reza Ardeshir Rokni <
>> raro...@gmail.com> wrote:
>>
>> Thanx folks!
>>
>> Oppsy on the link, here it is again:
>>
>> https://stackoverflow.com/questions/54422510/how-to-solve-duplicate-values-exception-when-i-create-pcollectionviewmapstring/54623618#54623618
>>
>> On Tue, 12 Mar 2019 at 23:32, Teja MVSR 
>> wrote:
>>
>> Hi Reza,
>>
>> I am also interested to contribute towards documentation. Please let
>> me know if I can be of any help.
>>
>> Thanks and Regards,
>> Teja.
>>
>> On Tue, Mar 12, 2019, 11:30 AM Kenneth Knowles 
>> wrote:
>>
>> This is great news.
>>
>> For the benefit of the list, I want to say how nice it has been when
>> I have had a chance to work with you. I've learned a great deal real and
>> complex use cases through those opportunities. I'm really excited that
>> you'll be helping out Beam in this new role.
>>
>> Kenn
>>
>> On Tue, Mar 12, 2019 at 7:21 AM Valentyn Tymofieiev <
>> valen...@google.com> wrote:
>>
>> Hi Reza!
>>
>> Welcome to Beam. Very nice to have you onboard. Btw, the link seems
>> broken.
>>
>> Thanks,
>> Valentyn
>>
>> On Tue, Mar 12, 2019 at 6:04 AM Reza Ardeshir Rokni <
>> raro...@gmail.com> wrote:
>>
>> Hi Folks,
>>
>> Just wanted to say hi to the good folks in the Beam community in my
>> new capacity as a Developer advocate for Beam/Dataflow @ Google. :-)
>>
>> At the moment I am working on a couple of blogs around the Timer and
>> State API as well as some work on general patterns that I hope to
>> contribute as documentation to the Beam site. An example of the patterns
>> can be seen here:  LINK
>>
>> Hope to be adding many more in 2019 and really looking forward to
>> being able to contribute to Beam in anyway that I can!
>>
>> Cheers
>> Reza
>>
>>
>>
>>
>>>
>>> --
>>>
>>> *Joana Carrasqueira*
>>>
>>> Cloud Developer Relations Events Manager
>>>
>>> 415-602-2507 Mobile
>>>
>>> 1160 N Mathilda Ave, Sunnyvale, CA 94089
>>>
>>>
>>>

Re: [PROPOSAL] Test performance of basic Apache Beam operations

[Łukasz Gajowy]

Hi Beamers,

an update on this. Together With Kasia, Michał and cooperating closely with
Pablo we have created and scheduled a Cron Job running daily 7 tests for
GroupByKey batch scenarios. Description of the tests is in the proposal [1]
and will be documented later. The dashboards for the tests:
 - showing run times [2]
 - showing total load size (bytes) [3]

All the metrics are collected using Beam's Metrics API.

Things we have on our horizon:
 - the same set of tests for Java but in streaming mode
 - similar jobs for Python SDK
 - running similar suites on Flink runner

We have also created a set of Dataproc bash scripts that can be used to set
up a Flink cluster that supports portability [4]. It is ready to use and
I've already successfully run the word count example using Python SDK on
it. Hoping + aiming to run load tests on it soon. :)

BTW/Last but not least: we also reused some code to collect metrics using
Metrics API in TextIOIT too and are willing to do a similar change for
other IOITs. Dashboards for TextIOIT: [5].

Thanks,
Łukasz

[1] https://s.apache.org/load-test-basic-operations
[2]
https://apache-beam-testing.appspot.com/explore?dashboard=5643144871804928
[3]
https://apache-beam-testing.appspot.com/explore?dashboard=5701325169885184
[4]
https://github.com/apache/beam/blob/b1ed061fd0c1ed1da562089c939d55715907769d/.test-infra/dataproc/create_flink_cluster.sh
[5]
https://apache-beam-testing.appspot.com/explore?dashboard=5629522644828160


śr., 12 wrz 2018 o 14:23 Etienne Chauchot  napisał(a):

> Let me elaborate a bit my last sentence
> Le mardi 11 septembre 2018 à 11:29 +0200, Etienne Chauchot a écrit :
>
> Hi Lukasz,
>
> Well, having low level byte[] based pure performance tests makes sense.
> And having high level realistic model (Nexmark auction system) makes sense
> also to avoid testing unrealistic pipelines as you describe.
>
> Have common code between the 2 seems difficult as both the architecture
> and the model are different.
>
> I'm more concerned about having two CI mechanisms to detect
> functionnal/performance regressions.
>
>
> Even if parts of NexMark and performance tests are the same they could
> target different objectives: raw performance tests (the new framework) and
> user oriented tests (nexmark). So they might be complementary.
>
> We must just chose how to run them. I think we need to have only one
> automatic regression detection tool. IMHO, the most relevant for func/perf
> regression is Nexmark because it represents what a real user could do (it
> simulates an auction system). So let's  keep it as post commits. Post
> commits allow to target a particular commit that introduced a regression.
>
> We could schedule the new performance tests.
>
> Best
> Etienne
>
>
> Best
> Etienne
>
> Le lundi 10 septembre 2018 à 18:33 +0200, Łukasz Gajowy a écrit :
>
> In my opinion and as far as I understand Nexmark, there are some benefits
> to having both types of tests. The load tests we propose can be very
> straightforward and clearly show what is being tested thanks to the fact
> that there's no fixed model but very "low level" KV
> collections only. They are more flexible in shapes of the pipelines they
> can express e.g. fanout_64, without having to think about specific use
> cases.
>
> Having both types would allow developers to decide whether they want to
> create a new Nexmark query for their specific case or develop a new Load
> test (whatever is easier and more fits their case). However, there is a
> risk - with KV developer can overemphasize cases that can
> never happen in practice, so we need to be careful about the exact
> configurations we run.
>
> Still, I can imagine that there surely will be code that should be common
> to both types of tests and we seek ways to not duplicate code.
>
> WDYT?
>
> Regards,
> Łukasz
>
>
>
> pon., 10 wrz 2018 o 16:36 Etienne Chauchot 
> napisał(a):
>
> Hi,
> It seems that there is a notable overlap with what Nexmark already does:
> Nexmark mesures performance and regression by exercising all the Beam
> model in both batch and streaming modes with several runners. It also
> computes on synthetic data. Also nexmark is already included as PostCommits
> in the CI and dashboards.
>
> Shall we merge the two?
>
> Best
>
> Etienne
>
> Le lundi 10 septembre 2018 à 12:56 +0200, Łukasz Gajowy a écrit :
>
> Hello everyone,
>
> thank you for all your comments to the proposal. To sum up:
>
> A set of performance tests exercising Core Beam Transforms (ParDo,
> GroupByKey, CoGroupByKey, Combine) will be implemented for Java and Python
> SDKs. Those tests will allow to:
>
>- measure performance of the transforms on various runners
>- exercise the transforms by creating stressful conditions and big
>loads using Synthetic Source and Synthetic Step API (delays, keeping cpu
>busy or asleep, processing large keys and values, performing fanout or
>reiteration of inputs)
>- run both in batch and streaming context
>- gather 

Re: [BEAM-6835] beam-sdks-python-test-suites-tox-py35:sdist FAILED on clean branch

[Mark Liu]

A fix to your problem is discussed in this thread
 and is
included in my roll-forward PR: https://github.com/apache/beam/pull/8067.

On Thu, Mar 14, 2019 at 11:27 AM Ahmet Altay  wrote:

> PR #8059 reverted a change from yesterday that was causing the sdist
> issue. Could you try again and see if your problem is resolved?
>
> On Thu, Mar 14, 2019 at 11:23 AM Alex Amato  wrote:
>
>> https://issues.apache.org/jira/browse/BEAM-6835
>> Repro:
>> ./gradlew lint :beam-sdks-python:docs
>>
>> I tested this on a branch I just rebased from master, and the same issue
>> is occurring. I also recreated the virtual env.
>> https://gradle.com/s/lnlmhu2cggreg
>>
>> I deleted the build directories, but it still generates multiple files
>> and fails.
>>
>>
>> ---
>> > Task :beam-sdks-python-test-suites-tox-py35:sdist FAILED
>>
>> FAILURE: Build completed with 2 failures.
>>
>> 1: Task failed with an exception.
>> ---
>> * What went wrong:
>> Execution failed for task ':beam-sdks-python:sdist'.
>> > Expected directory '/usr/local/google/home/ajamato/go/src/
>> github.com/apache/beam/sdks/python/build' to contain exactly one file,
>> however, it contains more than one file.
>>
>> * Try:
>> Run with --stacktrace option to get the stack trace. Run with --info or
>> --debug option to get more log output. Run with --scan to get full insights.
>>
>> ==
>>
>> 2: Task failed with an exception.
>> ---
>> * What went wrong:
>> Execution failed for task ':beam-sdks-python-test-suites-tox-py35:sdist'.
>> > Expected directory '/usr/local/google/home/ajamato/go/src/
>> github.com/apache/beam/sdks/python/test-suites/tox/py35/build' to
>> contain exactly one file, however, it contains more than one file.
>>
>> * Try:
>> Run with --stacktrace option to get the stack trace. Run with --info or
>> --debug option to get more log output. Run with --scan to get full insights.
>>
>> ==
>>
>> * Get more help at https://help.gradle.org
>>
>> Deprecated Gradle features were used in this build, making it
>> incompatible with Gradle 6.0.
>> Use '--warning-mode all' to show the individual deprecation warnings.
>> See
>> https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
>>
>> BUILD FAILED in 17s
>> 4 actionable tasks: 4 executed
>>
>> Publishing build scan...
>> https://gradle.com/s/lnlmhu2cggreg
>>
>>

Re: KafkaIO Exactly-Once & Flink Runner

[Kenneth Knowles]

Yes, the ParDoPayload has to contain most of the information that is on
DoFnSignature. Everything except the details for feeding the bits to the
Java DoFn.

Kenn

On Fri, Mar 15, 2019 at 9:59 AM Reuven Lax  wrote:

> I think this attribute needs to be added to the portability protos.
>
> On Fri, Mar 15, 2019 at 9:49 AM Thomas Weise  wrote:
>
>>
>>
>> On Tue, Mar 12, 2019 at 10:13 AM Maximilian Michels 
>> wrote:
>>
>>> > I think that is what Max's PR does. KafkaIO writes entire list of
>>> values associated with a key in one transaction. So it depends on how Flink
>>> runner bundles > after a GBK. I would think all of the buffered
>>> records would be queued. Here, the key is the shard id.
>>>
>>> We do not change the execution logic in case of stable input. Elements
>>> will still be processed key-wise.
>>>
>>
>> Wouldn't that require the KafkaEOS to support a different processing mode
>> where the elements are committed with @FinishBundle? The runner could then
>> align bundles and checkpointing as needed.
>>
>> I'm now also curious how @RequiresStableInput is supposed to work with
>> portable pipelines? The runner is not able to inspect the ParDo, so this
>> would need to be provided explicitly as part of the executable stage?
>>
>>
>>
>>>
>>>

Re: KafkaIO Exactly-Once & Flink Runner

[Reuven Lax]

I think this attribute needs to be added to the portability protos.

On Fri, Mar 15, 2019 at 9:49 AM Thomas Weise  wrote:

>
>
> On Tue, Mar 12, 2019 at 10:13 AM Maximilian Michels 
> wrote:
>
>> > I think that is what Max's PR does. KafkaIO writes entire list of
>> values associated with a key in one transaction. So it depends on how Flink
>> runner bundles > after a GBK. I would think all of the buffered
>> records would be queued. Here, the key is the shard id.
>>
>> We do not change the execution logic in case of stable input. Elements
>> will still be processed key-wise.
>>
>
> Wouldn't that require the KafkaEOS to support a different processing mode
> where the elements are committed with @FinishBundle? The runner could then
> align bundles and checkpointing as needed.
>
> I'm now also curious how @RequiresStableInput is supposed to work with
> portable pipelines? The runner is not able to inspect the ParDo, so this
> would need to be provided explicitly as part of the executable stage?
>
>
>
>>
>>

Re: KafkaIO Exactly-Once & Flink Runner

[Thomas Weise]

On Tue, Mar 12, 2019 at 10:13 AM Maximilian Michels  wrote:

> > I think that is what Max's PR does. KafkaIO writes entire list of values
> associated with a key in one transaction. So it depends on how Flink runner
> bundles > after a GBK. I would think all of the buffered records
> would be queued. Here, the key is the shard id.
>
> We do not change the execution logic in case of stable input. Elements
> will still be processed key-wise.
>

Wouldn't that require the KafkaEOS to support a different processing mode
where the elements are committed with @FinishBundle? The runner could then
align bundles and checkpointing as needed.

I'm now also curious how @RequiresStableInput is supposed to work with
portable pipelines? The runner is not able to inspect the ParDo, so this
would need to be provided explicitly as part of the executable stage?



>
>

Re: [PROPOSAL] Preparing for Beam 2.12.0 release

[Ahmet Altay]

+1 to extending 2.7.x LTS lifetime for a little longer and simultaneously
making a 2.7.1 release.

On Fri, Mar 15, 2019 at 9:32 AM Kenneth Knowles  wrote:

> We actually have some issues queued up for 2.7.1, and IMO it makes sense
> to extend 2.7 since the 6 month period was just a pilot and like you say we
> haven't really exercised LTS.
>
> Re 2.12.0 I strongly feel LTS should be designated after a release has
> seen some use. If we extend 2.7 for another while then we will have some
> candidate by the time it expires. (2.8, 2.9, 2.10 all have major issues,
> while 2.11 and 2.12 are untried)
>
> Kenn
>
> On Fri, Mar 15, 2019 at 7:50 AM Thomas Weise  wrote:
>
>> Given no LTS activity for 2.7.x - do we really need it?
>>
>>
>> On Fri, Mar 15, 2019 at 6:54 AM Ismaël Mejía  wrote:
>>
>>> After looking at the dates it seems that 2.12 should be the next LTS
>>> since it will be exactly 6 months after the release of 2.7.0. Anyone
>>> has comments, or prefer to do the LTS better for the next version
>>> (2.13) ?
>>>
>>> On Thu, Mar 14, 2019 at 12:13 PM Michael Luckey 
>>> wrote:
>>> >
>>> > @mxm
>>> >
>>> > Sure we should. Unfortunately the scripts to not have any '--dry-run'
>>> toggle. Implementing this seemed not too easy on first sight, as those
>>> release scripts do assume committed outputs of their predecessors and are
>>> not yet in the shape to be parameterised.
>>> >
>>> > So here is what I did:
>>> > 1. As I did not wanted the scripts to do 'sudo' installs on my
>>> machine, I first created a docker image with required prerequisites.
>>> > 2. Cloned beam to that machine (to get the release.scripts)
>>> > 3. Edited the places which seemed to call to the outside
>>> > - disabled any git push
>>> > - changed git url to point to some copy on local filesystem to
>>> pull required changes from there
>>> > - changed './gradlew' build to './gradlew assemble' as build will
>>> not work on docker anyway
>>> > - changed publish to publishToMavenLocal
>>> > - probably some more changes to ensure I do not write to remote
>>> > 4. run the scripts
>>> >
>>> > What I missed out:
>>> > 1. There is some communication with svn (signing artefacts downloaded
>>> from svn and committing). I just skipped those steps, as I was just too
>>> scared to miss some commit and doing an accidental push to some remote
>>> system (where I am hopefully not authorised anyway without doing proper
>>> authentication)
>>> >
>>> > If you believe I missed something which could be tested in advance, I
>>> d happily do more testing to ensure a smooth release process.
>>> >
>>> > michel
>>> >
>>> > On Thu, Mar 14, 2019 at 11:23 AM Maximilian Michels 
>>> wrote:
>>> >>
>>> >> Hi Andrew,
>>> >>
>>> >> Sounds good. Thank you for being the release manager.
>>> >>
>>> >> @Michael Shall we perform some dry-run release testing for ensuring
>>> >> Gradle 5 compatibility?
>>> >>
>>> >> Thanks,
>>> >> Max
>>> >>
>>> >> On 14.03.19 00:28, Michael Luckey wrote:
>>> >> > Sounds good. Thanks for volunteering.
>>> >> >
>>> >> > Just as a side note: @aaltay had trouble releasing caused by the
>>> switch
>>> >> > to gradle5. Although that should be fixed now, you will be the first
>>> >> > using those changes in production. So if you encounter any issues.
>>> do
>>> >> > not hesitate to blame and contact me. Also I am currently looking
>>> into
>>> >> > some improvements to the process suggested by @kenn. So your
>>> feedback on
>>> >> > the current state would be greatly appreciated. I hope to get at
>>> least
>>> >> > https://issues.apache.org/jira/browse/BEAM-6798 done until then.
>>> >> >
>>> >> > Thanks again,
>>> >> >
>>> >> > michel
>>> >> >
>>> >> > On Wed, Mar 13, 2019 at 8:13 PM Ahmet Altay >> >> > > wrote:
>>> >> >
>>> >> > Sounds great, thank you!
>>> >> >
>>> >> > On Wed, Mar 13, 2019 at 12:09 PM Andrew Pilloud <
>>> apill...@google.com
>>> >> > > wrote:
>>> >> >
>>> >> > Hello Beam community!
>>> >> >
>>> >> > Beam 2.12 release branch cut date is March 27th according
>>> to the
>>> >> > release calendar [1]. I would like to volunteer myself to do
>>> >> > this release. I intend to cut the branch as planned on March
>>> >> > 27th and cherrypick fixes if needed.
>>> >> >
>>> >> > If you have releasing blocking issues for 2.12 please mark
>>> their
>>> >> > "Fix Version" as 2.12.0. Kenn created a 2.13.0 release in
>>> JIRA
>>> >> > in case you would like to move any non-blocking issues to
>>> that
>>> >> > version.
>>> >> >
>>> >> > Does this sound reasonable?
>>> >> >
>>> >> > Andrew
>>> >> >
>>> >> > [1]
>>> >> >
>>> https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com=America%2FLos_Angeles
>>> >> >
>>>
>>

Re: [PROPOSAL] Preparing for Beam 2.12.0 release

[Kenneth Knowles]

We actually have some issues queued up for 2.7.1, and IMO it makes sense to
extend 2.7 since the 6 month period was just a pilot and like you say we
haven't really exercised LTS.

Re 2.12.0 I strongly feel LTS should be designated after a release has seen
some use. If we extend 2.7 for another while then we will have some
candidate by the time it expires. (2.8, 2.9, 2.10 all have major issues,
while 2.11 and 2.12 are untried)

Kenn

On Fri, Mar 15, 2019 at 7:50 AM Thomas Weise  wrote:

> Given no LTS activity for 2.7.x - do we really need it?
>
>
> On Fri, Mar 15, 2019 at 6:54 AM Ismaël Mejía  wrote:
>
>> After looking at the dates it seems that 2.12 should be the next LTS
>> since it will be exactly 6 months after the release of 2.7.0. Anyone
>> has comments, or prefer to do the LTS better for the next version
>> (2.13) ?
>>
>> On Thu, Mar 14, 2019 at 12:13 PM Michael Luckey 
>> wrote:
>> >
>> > @mxm
>> >
>> > Sure we should. Unfortunately the scripts to not have any '--dry-run'
>> toggle. Implementing this seemed not too easy on first sight, as those
>> release scripts do assume committed outputs of their predecessors and are
>> not yet in the shape to be parameterised.
>> >
>> > So here is what I did:
>> > 1. As I did not wanted the scripts to do 'sudo' installs on my machine,
>> I first created a docker image with required prerequisites.
>> > 2. Cloned beam to that machine (to get the release.scripts)
>> > 3. Edited the places which seemed to call to the outside
>> > - disabled any git push
>> > - changed git url to point to some copy on local filesystem to pull
>> required changes from there
>> > - changed './gradlew' build to './gradlew assemble' as build will
>> not work on docker anyway
>> > - changed publish to publishToMavenLocal
>> > - probably some more changes to ensure I do not write to remote
>> > 4. run the scripts
>> >
>> > What I missed out:
>> > 1. There is some communication with svn (signing artefacts downloaded
>> from svn and committing). I just skipped those steps, as I was just too
>> scared to miss some commit and doing an accidental push to some remote
>> system (where I am hopefully not authorised anyway without doing proper
>> authentication)
>> >
>> > If you believe I missed something which could be tested in advance, I d
>> happily do more testing to ensure a smooth release process.
>> >
>> > michel
>> >
>> > On Thu, Mar 14, 2019 at 11:23 AM Maximilian Michels 
>> wrote:
>> >>
>> >> Hi Andrew,
>> >>
>> >> Sounds good. Thank you for being the release manager.
>> >>
>> >> @Michael Shall we perform some dry-run release testing for ensuring
>> >> Gradle 5 compatibility?
>> >>
>> >> Thanks,
>> >> Max
>> >>
>> >> On 14.03.19 00:28, Michael Luckey wrote:
>> >> > Sounds good. Thanks for volunteering.
>> >> >
>> >> > Just as a side note: @aaltay had trouble releasing caused by the
>> switch
>> >> > to gradle5. Although that should be fixed now, you will be the first
>> >> > using those changes in production. So if you encounter any issues. do
>> >> > not hesitate to blame and contact me. Also I am currently looking
>> into
>> >> > some improvements to the process suggested by @kenn. So your
>> feedback on
>> >> > the current state would be greatly appreciated. I hope to get at
>> least
>> >> > https://issues.apache.org/jira/browse/BEAM-6798 done until then.
>> >> >
>> >> > Thanks again,
>> >> >
>> >> > michel
>> >> >
>> >> > On Wed, Mar 13, 2019 at 8:13 PM Ahmet Altay > >> > > wrote:
>> >> >
>> >> > Sounds great, thank you!
>> >> >
>> >> > On Wed, Mar 13, 2019 at 12:09 PM Andrew Pilloud <
>> apill...@google.com
>> >> > > wrote:
>> >> >
>> >> > Hello Beam community!
>> >> >
>> >> > Beam 2.12 release branch cut date is March 27th according to
>> the
>> >> > release calendar [1]. I would like to volunteer myself to do
>> >> > this release. I intend to cut the branch as planned on March
>> >> > 27th and cherrypick fixes if needed.
>> >> >
>> >> > If you have releasing blocking issues for 2.12 please mark
>> their
>> >> > "Fix Version" as 2.12.0. Kenn created a 2.13.0 release in
>> JIRA
>> >> > in case you would like to move any non-blocking issues to
>> that
>> >> > version.
>> >> >
>> >> > Does this sound reasonable?
>> >> >
>> >> > Andrew
>> >> >
>> >> > [1]
>> >> >
>> https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com=America%2FLos_Angeles
>> >> >
>>
>

Re: Apache Beam Newsletter - February/March 2019

[Matthias Baetens]

Adding my 2 cents:
+1 for adding News to the repository.
I agree with Alexey in that News is different from blogpost (tend to be
bigger and might have technical content and/or debriefs from events).
Tying newsletter to releases also don't fully make sense from an event
point of view - these don't tend to be scheduled 3 months in advance unless
they are very big. A cadence of once every other month like Pablo suggested
makes sense from that point of view.

On Wed, 13 Mar 2019 at 17:02, Ahmet Altay  wrote:

> Related to release notes, I do agree with cleaninp up the JIRA (as a
> community) and including the dependency changes. I find those information
> useful.
>
> If the proposal here is to loosely align newsletter timelines with release
> times, I agree with that. If it is a proposal to merge release notes with
> the newsletter, I will be worried it will make the release process more
> difficult than it is today. Simply because we will need more things to
> happen for a release to be completed.
>
> On Wed, Mar 13, 2019 at 2:37 AM Etienne Chauchot 
> wrote:
>
>> Hi,
>> +1 on the process
>> +1 on the new News section
>>
>> I also think that matching the newsletter pace with the release pace
>> makes it clearer for users. Also, as a general obvious comment, newsletter
>> have the interest of providing the ongoing work compared to release notes
>> that provide only the work that was done.
>>
>> Etienne
>>
>> Le mardi 12 mars 2019 à 21:50 -0700, Kenneth Knowles a écrit :
>>
>>
>>
>> On Tue, Mar 12, 2019 at 9:04 PM Thomas Weise  wrote:
>>
>> The release blog is already on the radar for improvement [1].
>>
>> I don't think there is a need to separate out the release blogs. That's
>> if they provide content that is valuable to users (IMO that's not exactly
>> the case today).
>>
>>
>> The case I am thinking about is a user searching the web for an issue and
>> figuring out what version it was fixed in. Dep upgrades being a primary
>> thing to notice.
>>
>>
>> If release blogs are just reformatted JIRA reports, then maybe we can
>> skip them altogether (release notes are already linked from download page).
>>
>>
>> One reason is that Jira remains mutable and I am not so sure of the SEO,
>> in terms of helping users figure out if an upgrade might solve their
>> problem.
>>
>>
>> In that case I would much rather like to see the original JIRA report
>> cleaned up as part of the release process.
>>
>>
>> Either way, this seems worthwhile. I think release managers often do this
>> anyhow. How could we formalize it? Do we need to? My favorite form of
>> formal process is just to get more eyes on some LGTM.
>>
>>
>> On the other hand, if release blogs are assembled similar to the
>> newsletter that we discuss here, through broader contributor input and with
>> the aim to provide more meaningful content to users, then I don't see why
>> we need to put them into a different area.
>>
>>
>> Same answer: people searching for issue resolution. I wonder if our
>> release notes should be a static copy/paste of the Jira list, deleting
>> things that really make no sense and editing everything else to be
>> meaningful. But I get the idea that we don't really need a separate area.
>>
>> What if newsletters matched releases, and could be drafted throughout the
>> 6 week period, with folks really trying to give a narrative to what they
>> contributed to a release?
>>
>> Kenn
>>
>>
>> Overall, given proposed newsletter cadence of 2 months and existing
>> release cadence, we would probably still end up with a monthly piece of
>> news.
>>
>> Thanks,
>> Thomas
>>
>>
>>
>> https://lists.apache.org/thread.html/224908775d8807eecc53e45f14f99d6a439beb8bb5165e3a813bf82d@%3Cdev.beam.apache.org%3E
>>
>>
>> On Tue, Mar 12, 2019 at 9:27 AM Rose Nguyen  wrote:
>>
>> Once every 2 months works. We include both big and small items. It'll
>> provide the focus Thomas mentions but still catches the frequency that
>> Pablo suggested for relevancy. To Alexey's point, it's difficult to
>> navigate the more recent non-release blog posts (<1 year) because they are
>> sprinkled in.
>>
>> A compromise for all of our points is to move the release notes to a
>> separate section, and have a single section that's blog/news.
>>
>> And thanks for wanting to contribute, Aizhamal! Teaming up with you will
>> make things run smoothly.
>>
>> On Tue, Mar 12, 2019 at 3:58 AM Alexey Romanenko <
>> aromanenko@gmail.com> wrote:
>>
>> +1 for “News” section. Though, I’d propose to exclude “release notes”
>> posts from “Blog” section list (since now we have quite regular releases,
>> it makes blog posts list not very readable for users) and move them to new
>> created News section. Also, this section could include the posts about
>> other Beam events, like meet-ups, conferences, project updates, etc. I’d
>> keep “Blog” section for more technical posts.
>>
>> On 12 Mar 2019, at 06:31, Pablo Estrada  wrote:
>>
>> I agree that the newsletter fits well as a blog post. I think 

Re: [PROPOSAL] Preparing for Beam 2.12.0 release

[Thomas Weise]

Given no LTS activity for 2.7.x - do we really need it?


On Fri, Mar 15, 2019 at 6:54 AM Ismaël Mejía  wrote:

> After looking at the dates it seems that 2.12 should be the next LTS
> since it will be exactly 6 months after the release of 2.7.0. Anyone
> has comments, or prefer to do the LTS better for the next version
> (2.13) ?
>
> On Thu, Mar 14, 2019 at 12:13 PM Michael Luckey 
> wrote:
> >
> > @mxm
> >
> > Sure we should. Unfortunately the scripts to not have any '--dry-run'
> toggle. Implementing this seemed not too easy on first sight, as those
> release scripts do assume committed outputs of their predecessors and are
> not yet in the shape to be parameterised.
> >
> > So here is what I did:
> > 1. As I did not wanted the scripts to do 'sudo' installs on my machine,
> I first created a docker image with required prerequisites.
> > 2. Cloned beam to that machine (to get the release.scripts)
> > 3. Edited the places which seemed to call to the outside
> > - disabled any git push
> > - changed git url to point to some copy on local filesystem to pull
> required changes from there
> > - changed './gradlew' build to './gradlew assemble' as build will
> not work on docker anyway
> > - changed publish to publishToMavenLocal
> > - probably some more changes to ensure I do not write to remote
> > 4. run the scripts
> >
> > What I missed out:
> > 1. There is some communication with svn (signing artefacts downloaded
> from svn and committing). I just skipped those steps, as I was just too
> scared to miss some commit and doing an accidental push to some remote
> system (where I am hopefully not authorised anyway without doing proper
> authentication)
> >
> > If you believe I missed something which could be tested in advance, I d
> happily do more testing to ensure a smooth release process.
> >
> > michel
> >
> > On Thu, Mar 14, 2019 at 11:23 AM Maximilian Michels 
> wrote:
> >>
> >> Hi Andrew,
> >>
> >> Sounds good. Thank you for being the release manager.
> >>
> >> @Michael Shall we perform some dry-run release testing for ensuring
> >> Gradle 5 compatibility?
> >>
> >> Thanks,
> >> Max
> >>
> >> On 14.03.19 00:28, Michael Luckey wrote:
> >> > Sounds good. Thanks for volunteering.
> >> >
> >> > Just as a side note: @aaltay had trouble releasing caused by the
> switch
> >> > to gradle5. Although that should be fixed now, you will be the first
> >> > using those changes in production. So if you encounter any issues. do
> >> > not hesitate to blame and contact me. Also I am currently looking into
> >> > some improvements to the process suggested by @kenn. So your feedback
> on
> >> > the current state would be greatly appreciated. I hope to get at least
> >> > https://issues.apache.org/jira/browse/BEAM-6798 done until then.
> >> >
> >> > Thanks again,
> >> >
> >> > michel
> >> >
> >> > On Wed, Mar 13, 2019 at 8:13 PM Ahmet Altay  >> > > wrote:
> >> >
> >> > Sounds great, thank you!
> >> >
> >> > On Wed, Mar 13, 2019 at 12:09 PM Andrew Pilloud <
> apill...@google.com
> >> > > wrote:
> >> >
> >> > Hello Beam community!
> >> >
> >> > Beam 2.12 release branch cut date is March 27th according to
> the
> >> > release calendar [1]. I would like to volunteer myself to do
> >> > this release. I intend to cut the branch as planned on March
> >> > 27th and cherrypick fixes if needed.
> >> >
> >> > If you have releasing blocking issues for 2.12 please mark
> their
> >> > "Fix Version" as 2.12.0. Kenn created a 2.13.0 release in JIRA
> >> > in case you would like to move any non-blocking issues to that
> >> > version.
> >> >
> >> > Does this sound reasonable?
> >> >
> >> > Andrew
> >> >
> >> > [1]
> >> >
> https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com=America%2FLos_Angeles
> >> >
>

Re: [PROPOSAL] Preparing for Beam 2.12.0 release

[Ismaël Mejía]

After looking at the dates it seems that 2.12 should be the next LTS
since it will be exactly 6 months after the release of 2.7.0. Anyone
has comments, or prefer to do the LTS better for the next version
(2.13) ?

On Thu, Mar 14, 2019 at 12:13 PM Michael Luckey  wrote:
>
> @mxm
>
> Sure we should. Unfortunately the scripts to not have any '--dry-run' toggle. 
> Implementing this seemed not too easy on first sight, as those release 
> scripts do assume committed outputs of their predecessors and are not yet in 
> the shape to be parameterised.
>
> So here is what I did:
> 1. As I did not wanted the scripts to do 'sudo' installs on my machine, I 
> first created a docker image with required prerequisites.
> 2. Cloned beam to that machine (to get the release.scripts)
> 3. Edited the places which seemed to call to the outside
> - disabled any git push
> - changed git url to point to some copy on local filesystem to pull 
> required changes from there
> - changed './gradlew' build to './gradlew assemble' as build will not 
> work on docker anyway
> - changed publish to publishToMavenLocal
> - probably some more changes to ensure I do not write to remote
> 4. run the scripts
>
> What I missed out:
> 1. There is some communication with svn (signing artefacts downloaded from 
> svn and committing). I just skipped those steps, as I was just too scared to 
> miss some commit and doing an accidental push to some remote system (where I 
> am hopefully not authorised anyway without doing proper authentication)
>
> If you believe I missed something which could be tested in advance, I d 
> happily do more testing to ensure a smooth release process.
>
> michel
>
> On Thu, Mar 14, 2019 at 11:23 AM Maximilian Michels  wrote:
>>
>> Hi Andrew,
>>
>> Sounds good. Thank you for being the release manager.
>>
>> @Michael Shall we perform some dry-run release testing for ensuring
>> Gradle 5 compatibility?
>>
>> Thanks,
>> Max
>>
>> On 14.03.19 00:28, Michael Luckey wrote:
>> > Sounds good. Thanks for volunteering.
>> >
>> > Just as a side note: @aaltay had trouble releasing caused by the switch
>> > to gradle5. Although that should be fixed now, you will be the first
>> > using those changes in production. So if you encounter any issues. do
>> > not hesitate to blame and contact me. Also I am currently looking into
>> > some improvements to the process suggested by @kenn. So your feedback on
>> > the current state would be greatly appreciated. I hope to get at least
>> > https://issues.apache.org/jira/browse/BEAM-6798 done until then.
>> >
>> > Thanks again,
>> >
>> > michel
>> >
>> > On Wed, Mar 13, 2019 at 8:13 PM Ahmet Altay > > > wrote:
>> >
>> > Sounds great, thank you!
>> >
>> > On Wed, Mar 13, 2019 at 12:09 PM Andrew Pilloud > > > wrote:
>> >
>> > Hello Beam community!
>> >
>> > Beam 2.12 release branch cut date is March 27th according to the
>> > release calendar [1]. I would like to volunteer myself to do
>> > this release. I intend to cut the branch as planned on March
>> > 27th and cherrypick fixes if needed.
>> >
>> > If you have releasing blocking issues for 2.12 please mark their
>> > "Fix Version" as 2.12.0. Kenn created a 2.13.0 release in JIRA
>> > in case you would like to move any non-blocking issues to that
>> > version.
>> >
>> > Does this sound reasonable?
>> >
>> > Andrew
>> >
>> > [1]
>> > 
>> > https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com=America%2FLos_Angeles
>> >