Re: [VOTE] Release 2.7.0, release candidate #3

Thank you everyone for your work on this release. I'm pleased to announce that the 2.7.0 RC3 is approved for release with 3 PMC +1 votes and no -1 votes. On Thu, Sep 27, 2018 at 5:31 AM Łukasz Gajowy wrote: > +1 > > I once again looked at the Nexmark dashboards, it seems that there are no >

Re: Tracking and resolving release blocking bugs

This sounds reasonable to me. On Fri, Sep 28, 2018 at 5:39 PM, Connell O'Callaghan wrote: > Ismaël and Ahmet thank you both for your responses!!! > > Can the text on the site today - reproduced below - be enhanced with the > text in bold? Note I would not expect this new text to be in bold once

Re: Tracking and resolving release blocking bugs

Ismaël and Ahmet thank you both for your responses!!! Can the text on the site today - reproduced below - be enhanced with the text in bold? Note I would not expect this new text to be in bold once deployed to the site. Triage release-blocking issues in JIRA There could be outstanding

Build failed in Jenkins: beam_Release_Gradle_NightlySnapshot #188

See -- [...truncated 24.43 MB...] > Could not write to resource

Re: Python typing library is not provisional in Python 3.7

Aha, I see. I'm coming from the future. Thanks Ahmet and Valentyn. On Sat, Sep 29, 2018 at 3:06 AM Valentyn Tymofieiev wrote: > Hi Manu, > > I second what Ahmet said - thanks for the pointers. Python 3.7 support can > come later down the road. > > Thanks, > Valentyn > > On Fri, Sep 28, 2018 at

Re: [DISCUSS] Committer Guidelines / Hygene before merging PRs

On Fri, Sep 28, 2018 at 10:29 AM Thomas Weise wrote: > +1 for stating the goal of clear provenance and granular rollback. > > > Also of course efficiency and quality of review (we don't to review tiny or out-of-context changes or huge mega-changes), efficiency of authoring (don't want to wait

Re: Python typing library is not provisional in Python 3.7

Hi Manu, I second what Ahmet said - thanks for the pointers. Python 3.7 support can come later down the road. Thanks, Valentyn On Fri, Sep 28, 2018 at 11:17 AM Ahmet Altay wrote: > Hi Manu, > > Currently, we use Python 3.5.2 on Jenkins for testing. Python tests print > out the python version

Re: Agenda for the Beam Summit London 2018

How exciting, can't wait to join you guys on Monday! :-) D. On 2018/09/27 22:03:16, Griselda Cuevas wrote: > Hi Beam Community, > > We have finalized the agenda for the Beam Summit London 2018, it's here: > https://www.linkedin.com/feed/update/urn:li:activity:6450125487321735168/ > > > We

Re: Why not adding all coders into ModelCoderRegistrar?

Thank you, Lukasz! Best, Shen On Fri, Sep 28, 2018 at 2:11 PM Lukasz Cwik wrote: > Runners can never know about every coder that a user may want to write > which is why we need to have a mechanism for Runners to be able to convert > any unknown coder to one it can handle. This is done via >

Re: Python typing library is not provisional in Python 3.7

Hi Manu, Currently, we use Python 3.5.2 on Jenkins for testing. Python tests print out the python version in the console logs and I found this information from one of the logs [1]. Initial proposal for the Python 3 support was to support a specific version of python 3 during the porting process

Re: Why not adding all coders into ModelCoderRegistrar?

Runners can never know about every coder that a user may want to write which is why we need to have a mechanism for Runners to be able to convert any unknown coder to one it can handle. This is done via WireCoders.instantiateRunnerWireCoder but this modifies the original coder which is why

Why not adding all coders into ModelCoderRegistrar?

Hi, I noticed that ModelCoderRegistrar only includes 9 out of ~40 coders. May I know the rationale behind this decision? https://github.com/apache/beam/blob/release-2.7.0/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ModelCoderRegistrar.java I think one

Re: [DISCUSS] Committer Guidelines / Hygene before merging PRs

+1 for stating the goal of clear provenance and granular rollback. I think this discussion helps to remind/identify best practices how to get there. Where appropriate we should augment guidelines for both, contributor and committer. Kenn: would you elaborate on the "1 commit = 1 review (and

Re: [DISCUSS] Committer Guidelines / Hygene before merging PRs

Anton makes a good point. We have been talking about policy for what we do, but the real issue is what we want to come out of it: a clear history for seeing where code came from and granular rollback. I think in both cases the key thing is that each commit is a single clear change. How they get

Re: [DISCUSS] Committer Guidelines / Hygene before merging PRs

+1 to Anton's points. It looks like the main concern with unsquashed commits is aesthetic, in that having "!fixup" commits produces noise and clutters the code tree. I would like to point out again for those unaware, that "git log --first-parent" filters the commit history so that each PR

Re: Agenda for the Beam Summit London 2018

Wow, this looks fantastic! Thanks to the organizers! On Thu, Sep 27, 2018 at 11:29 PM Andrew Psaltis wrote: > This is great. Any chance it will be recorded or at a minimum the slides > made available after. Unfortunately, I won't be able to make it to London > next week. > > Best, > Andrew > >

Re: [DISCUSS] Committer Guidelines / Hygene before merging PRs

Is there an actual problem caused by squashing or not squashing the commits that we face in the project? I personally have never needed to revert something complicated that would be problematic either way (and don't have a strong opinion about which way we should do it). From what I see so far in

Re: [DISCUSS] Committer Guidelines / Hygene before merging PRs

Thanks for linking the previous discussion. I have also seen a few cases where the intention was to make individual changes that can be applied independently. But why not create separate PRs for those, so they can also be reviewed and merged independently? Also, if the intention is to make

Re: [DISCUSS] Committer Guidelines / Hygene before merging PRs

Fully agree, if there is a logical commit history, we should keep it. I think this is speaking to the large number of PRs that have a single "real" commit, a bunch of fixups, and specifically authors who haven't gone through and cleaned up their history. (Now if the commits could logically be

Re: [DISCUSS] Committer Guidelines / Hygene before merging PRs

I brought up this discussion a few months ago from the other side: I don't like my commits being squashed. I try to create logical commits that each passes tests and could be broken up into multiple PRs. Keeping those changes intact is useful from a history perspective and squashing may break

Re: [DISCUSS] Committer Guidelines / Hygene before merging PRs

On Thu, Sep 27, 2018 at 9:51 AM Robert Bradshaw wrote: > I agree that we should create a good pointer for cleaning up PRs, and > request (though not require) that authors do it. It's unfortunate though > that squashing during a review makes things difficult to follow, so adds > one more round

Re: Portable Flink runner: Generator source for testing

Hi all, thank you, Thomas, for starting this discussion and Pablo for sharing the ideas. FWIW adding here, we discussed this in terms of Core Beam Transform Load Tests that we are working on right now [1]. If generating synthetic data will be possible for portable streaming pipelines, we could

Re: Re: How to optimize the performance of Beam on Spark(Internet mail)

Thanks for sharing those results. The second set (executors at 20-30) look similar to what I would have expected. BEAM-5036 definitely plays a part here as the data is not moved on HDFS efficiently (fix in PR awaiting review now [1]). To give an idea of the impact, here are some numbers from my

Build failed in Jenkins: beam_Release_Gradle_NightlySnapshot #186

See Changes: [shaun] added avroio package [shaun] updated read emits to support both string and custom type reflects [shaun] added avro write support [mergebot] [BEAM-5436] Improve docs for

Re: Re: How to optimize the performance of Beam on Spark(Internet mail)

Something here on the Beam side is clearly linear in the input size, as if there's a bottleneck where were' not able to get any parallelization. Is the spark variant running in parallel? On Fri, Sep 28, 2018 at 4:57 AM devinduan(段丁瑞) wrote: > Hi > I have completed my test. > 1. Spark

Re: Portable Flink runner: Generator source for testing

Hi Thomas, all, yes, this is quite important for testing, and in fact I'd think it's important to streamline the insertion of native sources from different runners to make the current runners more usable. But that's another topic. For generators of synthetic data, I had a couple ideas (and this