Re: Design Proposal for python wheels build process

Thank you Boyuan. This proposal looks good to me. I agree that it would be great to agree on this before the start of the next release. The reason is we built wheel files for the past 2 releases however we do not yet have a working process. Releases are slowed down because of that. If we can

Re: BiqQueryIO.write and Wait.on

Hi Carlos, Any updates / roadblocks you hit? On Tue, Jul 3, 2018 at 7:13 AM Eugene Kirpichov wrote: > Awesome!! Thanks for the heads up, very exciting, this is going to make a > lot of people happy :) > > On Tue, Jul 3, 2018, 3:40 AM Carlos Alonso wrote: > >> + dev@beam.apache.org >> >> Just

Re: CODEOWNERS for apache/beam repo

We did not, but I think we should. So far, in 100% of the PRs I've authored, the default functionality of CODEOWNERS did the wrong thing and I had to fix something up manually. On Mon, Jul 16, 2018 at 3:42 PM Andrew Pilloud wrote: > This sounds like a good plan. Did we want to rename the

Re: An update on Eugene

Eugene, Thanks for all your contributions to the project and especially the leadership in the IOs and SDF area. Congrats and best wishes for your new opportunity. Thomas On Mon, Jul 16, 2018 at 4:00 PM Ismaël Mejía wrote: > I am sad to read this, but at the same happy for you and your

Re: An update on Eugene

I am sad to read this, but at the same happy for you and your future adventures Eugene. Thanks a lot for all the work you have done in this project, all the work on SDF, the improvements on composability and the File-based IOs, and of course for your reviews that really helped improve the quality

Re: CODEOWNERS for apache/beam repo

This sounds like a good plan. Did we want to rename the CODEOWNERS file to disable github's mass adding of reviewers while we figure this out? Andrew On Mon, Jul 16, 2018 at 10:20 AM Jean-Baptiste Onofré wrote: > +1 > > Le 16 juil. 2018, à 19:17, Holden Karau a écrit: >> >> Ok if no one

Re: Beam Dependency Ownership

Thanks all for taking the ownership of Beam dependencies! This is an import step to keep our dependencies healthy and up-to-date. We will close this thread now. The next step would be integrating the ownership information into the Beam codebase and implementing a tool to create and manage the JIRA

Re: Let's start getting rid of BoundedSource

Hey all, The PR https://github.com/apache/beam/pull/5940 was merged, and now all runners at "master" support bounded-per-element SDFs! Thanks +Ismaël Mejía for the reviews. I have updated the Capability Matrix as well: https://beam.apache.org/documentation/runners/capability-matrix/ On Mon,

Design Proposal for python wheels build process

Hey all, After 2.4.0, we also publish python wheels as part of python artifacts. In order to make release process easier, we need a standardized way to build and stage python wheels as s part of release process. In this deign proposal[1], I debriefed how we built python wheels for 2.4.0 & 2.5.0,

Git Bisect broken on the beam repo after the beam-site merge

It appears that as of https://github.com/apache/beam/pull/5641 we have two root commits on the beam repo. This is breaking git bisect, as it expects repo to only have one initial commit. (git bisect is always following the doc tree to the root instead of going down the source tree.) I'm going to

Re: An update on Eugene

Thanks for all your work! On Mon, Jul 16, 2018, 9:17 PM Eugene Kirpichov wrote: > Hi beamers, > > After 5.5 years working on data processing systems at Google, several of > these years working on Dataflow and Beam, I am moving on to do something > new (also at Google) in the area of programming

An update on Eugene

Hi beamers, After 5.5 years working on data processing systems at Google, several of these years working on Dataflow and Beam, I am moving on to do something new (also at Google) in the area of programming models for machine learning. Anybody who worked with me closely knows how much I love

Re: CODEOWNERS for apache/beam repo

+1 Le 16 juil. 2018 à 19:17, à 19:17, Holden Karau a écrit: > >Ok if no one objects I'll create the INFRA ticket after OSCON and we >can test it for a week and decide if it helps or hinders. > > > > > On Mon, Jul 16, 2018, 7:12 PM Jean-Baptiste Onofré >

Re: CODEOWNERS for apache/beam repo

Ok if no one objects I'll create the INFRA ticket after OSCON and we can test it for a week and decide if it helps or hinders. On Mon, Jul 16, 2018, 7:12 PM Jean-Baptiste Onofré wrote: > Agree to test it for a week. > > Regards > JB > Le 16 juil. 2018, à 18:59, Holden Karau a écrit: >> >>

Re: CODEOWNERS for apache/beam repo

Agree to test it for a week. Regards JB Le 16 juil. 2018 à 18:59, à 18:59, Holden Karau a écrit: >Would folks be OK with me asking infra to turn on blame based >suggestions >for Beam and trying it out for a week? > >On Mon, Jul 16, 2018, 6:53 PM Rafael Fernandez >wrote: > >> +1 using blame --

Re: CODEOWNERS for apache/beam repo

Would folks be OK with me asking infra to turn on blame based suggestions for Beam and trying it out for a week? On Mon, Jul 16, 2018, 6:53 PM Rafael Fernandez wrote: > +1 using blame -- nifty :) > > On Mon, Jul 16, 2018 at 2:31 AM Huygaa Batsaikhan > wrote: > >> +1. This is great. >> >> On

Re: CODEOWNERS for apache/beam repo

I personally like blame-based suggestions. The downside is that you effectively become a owner of anything you touch. Most of the time blame based suggestions will return multiple candidates. Could we use the CODEOWNERS file to filter down the suggestions? Andrew On Mon, Jul 16, 2018 at 2:31 AM

Re: Beam site test/merge Jenkins issue

This is a really persistent flap in the website build. I would guess it hits 80% of the time, if you do enough builds it will eventually succeed. I opened an issue on it a while back: https://issues.apache.org/jira/browse/BEAM-4686 Andrew On Mon, Jul 16, 2018 at 5:05 AM Jean-Baptiste Onofré

Re: [DISCUSS] Use Confluence wiki for non-user-facing stuff

Your doc looks good to me. It looks like only one question remains: should it be a Confluence or Github wiki. I see other Apache projects using both, so it seems like either one is possible with support of the Beam community. It might be time to call a vote on this? Andrew On Fri, Jul 13, 2018

Re: Let's start getting rid of BoundedSource

Hi guys, I think it's the purpose of SDF to simplify the BoundedSource like writing. I agree that extended @SplitRestriction is a good approach. Regards JB On 16/07/2018 16:52, Eugene Kirpichov wrote: > Hi Etienne - thanks for catching this; indeed, I somehow missed that > actually several

Re: Let's start getting rid of BoundedSource

Hi Etienne - thanks for catching this; indeed, I somehow missed that actually several runners do this same thing - it seemed to me as something that can be done in user code (because it involves combining estimated size + split in pretty much the same way), but I'm not so sure: even though many

Beam Dependency Check Report (2018-07-16)

High Priority Dependency Updates Of Beam Python SDK: Dependency Name Current Version Latest Version Release Date Of the Current Used Version Release Date Of The Latest Release google-cloud-bigquery 0.25.0 1.4.0 2017-06-26

Re: Beam site test/merge Jenkins issue

Hi, let me take a look, it's maybe the client key auth which failing. Regards JB On 16/07/2018 13:02, Alexey Romanenko wrote: > Hi, > > From time to time, I observe *gpg key* issues in Jenkins job when I try > to test/merge Beam site PR. > For example: >

Beam site test/merge Jenkins issue

Hi, From time to time, I observe gpg key issues in Jenkins job when I try to test/merge Beam site PR. For example: https://builds.apache.org/job/beam_PreCommit_Website_Stage/1192/console It says the following: gpg:

Re: Let's start getting rid of BoundedSource

Hi, thanks Eugene for analyzing and sharing that.I have one comment inline Etienne Le dimanche 15 juillet 2018 à 14:20 -0700, Eugene Kirpichov a écrit : > Hey beamers, > I've always wondered whether the BoundedSource implementations in the Beam > SDK are worth their complexity, or whether > they

Re: CODEOWNERS for apache/beam repo

+1. This is great. On Sat, Jul 14, 2018 at 7:44 AM Udi Meiri wrote: > Mention bot looks cool, as it tries to guess the reviewer using blame. > I've written a quick and dirty script that uses only CODEOWNERS. > > Its output looks like: > $ python suggest_reviewers.py --pr 5940 >

Build failed in Jenkins: beam_Release_Gradle_NightlySnapshot #102

See Changes: [github] [BEAM-4432] Adding Sources to produce Synthetic output for Batch -- [...truncated 17.81 MB...]