Re: [PROPOSAL] Preparing 2.5.0 release next week

Sorry for the noise: this build error was due to a corrupted file in my .m2/repository. The HDFS extension build is now OK. I'm launching a full build. Regards JB On 05/06/2018 07:43, Jean-Baptiste Onofré wrote: > New failure on the build: > > FAILURE: Build failed with an exception. > > *

Re: [SQL] Unsupported features

FYI, Umbrella JIRA ticket: https://issues.apache.org/jira/browse/BEAM-4476 ᐧ ᐧ On Mon, Jun 4, 2018 at 3:08 PM Kai Jiang wrote: > Ismaël, I was running this naive code snippet > . > Yes, IT would be interesting. Next step, I was

Read from a Google Sheet based BigQuery table - Python SDK

Hi guys, just wanted to ask you if there is a chance to read from a Sheet based BigQuery table from a Beam pipeline running on Dataflow... I usually specify additional scopes to use through the authentication when running simple Python code to do the same, but I wasn't able to find a reference

Re: [PROPOSAL] Preparing 2.5.0 release next week

Have you cut the release branch? It is much easier to stabilize a cut branch that is separated from continued development on master. I think we have to cut it before continuing. Kenn On Tue, Jun 5, 2018 at 1:14 AM Jean-Baptiste Onofré wrote: > Sorry for the noise: this build error was due to a

Re: Portability and Timers

+1 the runner would "special-case" these timer PCollections to wire them with its native timer management. At least in the Java/FnDataReceiver case that looks straightforward. Thomas On Mon, Jun 4, 2018 at 3:46 PM, Lukasz Cwik wrote: > Fixed the permissions, feel free to comment on the doc. >

Re: [PROPOSAL] Preparing 2.5.0 release next week

On the way Le 5 juin 2018 à 18:23, à 18:23, Kenneth Knowles a écrit: >Have you cut the release branch? It is much easier to stabilize a cut >branch that is separated from continued development on master. I think >we >have to cut it before continuing. > >Kenn > >On Tue, Jun 5, 2018 at 1:14 AM

Re: [PROPOSAL] Preparing 2.5.0 release next week

Thanks, yeah I saw that but I'm planning to add some additional notes. Regards JB Le 5 juin 2018 à 19:02, à 19:02, Boyuan Zhang a écrit: >Hey JB, > >We have some updates in : >https://github.com/pabloem/beam-site/blob/372c93ecbafbf3a1440492df1e12050dfe939e91/src/contribute/release-guide.md >and

Re: Read from a Google Sheet based BigQuery table - Python SDK

See following regarding authenticating Dataflow jobs. https://cloud.google.com/dataflow/security-and-permissions I'm not sure about information specific to sheets, seems like there's some info in following. https://cloud.google.com/bigquery/external-data-drive On Tue, Jun 5, 2018 at 10:16 AM

Re: Read from a Google Sheet based BigQuery table - Python SDK

I don't think BQ federated tables support export jobs so reading directly from such tables likely will not work. But reading using a query should work if your job is authenticated properly (I haven't tested this). - Cham On Tue, Jun 5, 2018, 5:56 AM Leonardo Biagioli wrote: > Hi guys, > >

Re: [PROPOSAL] Preparing 2.5.0 release next week

Was checking if there was something like the release branch of the maven release plugin, but there's not with the gradle one. I'm creating the branch by hand and I'm updating the release guide in the mean time. Regards JB Le 5 juin 2018 à 18:44, à 18:44, "Jean-Baptiste Onofré" a écrit: >On

Re: [PROPOSAL] Preparing 2.5.0 release next week

Hey JB, We have some updates in : https://github.com/pabloem/beam-site/blob/372c93ecbafbf3a1440492df1e12050dfe939e91/src/contribute/release-guide.md and https://github.com/apache/beam-site/pull/424/files

Re: Proposal: keeping post-commit tests green

I've taken another pass over the doc, and it looks good to me. Thanks for driving this effort! On Mon, Jun 4, 2018 at 9:08 AM Mikhail Gryzykhin wrote: > Hello everyone, > > I have addressed comments on the proposal doc and updated it accordingly. > I have also added section on metrics that we

Re: [VOTE] Code Review Response-time SLO

Proposal 1: +1 Proposal 2: 0 Additional comments: Does this apply to committers only, or are contributors encourage to volunteer as an additional reviewer? I like the idea of documenting who is keen to review which area, plus reviewer availability. >From a contributor standpoint, it is worth

Re: Initial contributor experience

+user@ in case someone has had similar experiences. Thanks for documenting this Austin & Pablo! If any other folks would like to participate in improving the "First contribution experience" for Beam let us know in this thread. On Tue, 5 Jun 2018 at 13:40, Austin Bennett wrote: > Had been

Re: [VOTE] Code Review Response-time SLO

Proposal 1: +0 Proposal 2: +1 Additional Comments: I like the idea of providing a clear expectation to contributors on how promptly they can expect code review feedback. From your proposal, my main feedback would to prefer a single goal instead of two. Having a single goal seems simpler when

Re: [VOTE] Code Review Response-time SLO

Thanks Ken for pointing out that vote is a two step process. I will write an extended written doc about pros and cons of the SLO and any related topics. In the meantime, feel free to use this thread to express any suggestions and concerns. Thanks, Huygaa

Re: Initial contributor experience

Thanks Austin for taking the time to go through this! We came out with a few JIRAs to improve the documentation (see doc), and hopefully we'll keep iterating on this. Hopefully we can get more experiences from other people that start to approach Beam. Best -P. On Tue, Jun 5, 2018 at 1:49 PM

Re: Initial contributor experience

I was thinking of that being a portion of the first meet up for: https://www.meetup.com/San-Francisco-Apache-Beam/ so even if not as detailed, can go through this with many more people. Was thinking explaining open source contribution generally is something that would be great for lots of

Initial contributor experience

Had been meaning to get setup to contribute for a bit - so walked through with Pablo, to try to point out assumptions, lacking docs, etc. Write-up found--> https://docs.google.com/document/d/1hq-s3L676LkMTftvhv0eCkdwrRnZmCRiLdaQBWLHWWA/edit

Re: Initial contributor experience

This is great, thanks! I added some comments to the doc. On Tue, Jun 5, 2018 at 1:49 PM Griselda Cuevas wrote: > +user@ in case someone has had similar experiences. > > Thanks for documenting this Austin & Pablo! > > If any other folks would like to participate in improving the "First >

Re: [VOTE] Code Review Response-time SLO

I love that you are pushing this forward. Just a note on mailing list practice - I think we might want a round of discussion before declaring a vote. A vote is a formal thing and this seems a bit more open-ended still. Couple of thoughts: - What is the action when an SLO is missed? -

Re: Design Proposal: Beam-Site Automation Reliability

Thanks everyone; I've responded to feedback in the doc [1] and I believe we've reached consensus. I've added implementation tasks in JIRA under BEAM-4493 [2] and will start coding soon. As a recap, the high-level plan is: * Migrate website source code to the main apache/beam repository *

Re: Proposal for Beam Python User State and Timer APIs

Thanks everyone for contributing here. We've reached rough consensus on the approach we should take with this API, and I've summarized this in the new "Community consensus" sections I added to the doc ( https://s.apache.org/beam-python-user-state-and-timers). I will begin initial implementation

Re: Proposal: keeping precommit times fast

I've been having a separate discussion on the proposal doc, which is ready for another round of reviews. Change summary: - Changed fast requirement to be < 30 minutes and simplify the check as an aggregate for each precommit job type. - Updated slowness notification methods to include automated

Re: Proposal: keeping post-commit tests green

Thanks for taking this initiative. As the number of contributors grows, so does the cost of broken builds. I'm also in favor of locking master merges until related issues are fixed (short term pain for long term gain). It would penalize a few for the benefit of many. On that note, recently we

Re: [PROPOSAL] Preparing 2.5.0 release next week

JB, I believe many people are waiting on the release to happen and the release branch is yet to be cut. It has been almost a week since you said you would cut the release branch. It seems like your very busy, can you explain clearly what is slowing you down so people can help or would you like to

Re: Beam breaks when it isn't loaded via the Thread Context Class Loader

Perhaps we can also adopt a practice of making our own APIs explicitly pass a Classloader when appropriate so we only have to set this when we are entering code that does not have good hygiene. It might actually be nice to have a lightweight static analysis to forbid bad methods in our code. Kenn

[VOTE] [RESULT] Use probot/stale to automatically manage stale pull requests

This vote has passed with 17 approving votes (6 binding) and no disapproving votes. Alan - now we can go ahead and merge your config. Kenn On Mon, Jun 4, 2018 at 2:12 AM Jean-Baptiste Onofré wrote: > +1 > > Regards > JB > > On 01/06/2018 18:21, Kenneth Knowles wrote: > > Hi all, > > > >

Re: Managing outdated dependencies

Thanks everybody for all the comments in the doc. Based on the reaction so far, seems like community generally agrees to introducing policies around managing dependencies. I updated the suggested policies based on comments and rewrote them in a more concise form and added room for more human

Re: Existing transactionality inconsistency in the Beam Java State API

Thanks everyone for commenting and contributing to the discussion. There appears to be enough consensus on these points to start an initial implementation. Specifically, I'd like to highlight from the doc (

Re: [Call for items] Beam June Newsletter

Hi Everyone, Just a reminder to add items to the June Newsletter, the idea behind it is to summarize community efforts in the project to get others identify similar actions, opportunities to collaborate and ask questions. Folks in the mailing list have found these newsletters useful in the past,

Re: [VOTE] Code Review Response-time SLO

The one thing that seems a little odd to me is using business days. The beam community spans across company and country boundaries, which makes business days a slightly ambiguous concept. Not everyone's holidays and weekends line up. I also agree with Alan, we need to build in time to disconnect.

Re: Beam breaks when it isn't loaded via the Thread Context Class Loader

It is actually very localised in runner code where beam should reset the classloader when the deserialization happens and then the runner owns the classloader all the way in evaluators. If IO change the classloader they must indeed handle it too and patch the deserialization too. Here again (we

Re: [PROPOSAL] Preparing 2.5.0 release next week

Hi Luke, The delay was simply due to the fact that the build is not stable. I had many building issue on different tasks. I was not comfortable to cut a release without a roughly stable build. Anyway, I moved forward: - release-2.5.0 branch has been cut and updated - I also updated master I

Re: [PROPOSAL] Preparing 2.5.0 release next week

New issue during: ./gradlew publish -PisRelease > Task :beam-runners-apex:compileTestJava /home/jbonofre/Workspace/beam/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/utils/ApexStateInternalsTest.java:55: error: An unhandled exception was thrown by the Error Prone static