Re: [VOTE] Go SDK

Hi Henning, SGA has been filed for the entire project during the incubation period. Here, we have to check if SGA/IP donation is clean for the Go SDK. We don't have a lot to do, just checked that we are clean on this front. Regards JB On 22/05/2018 06:42, Henning Rohde wrote: Thanks

Re: [VOTE] Go SDK

+1 (binding) While this is the third language for Beam, I believe it's the first new one since Beam began. Including this in Beam will be a nice milestone for the project. Reuven On Mon, May 21, 2018 at 9:43 PM Henning Rohde wrote: > Thanks everyone! > > Davor -- regarding

Re: [VOTE] Go SDK

Thanks everyone! Davor -- regarding your two comments: * Robert mentioned that "SGA should have probably already been filed" in the previous thread. I got the impression that nothing further was needed. I'll follow up. * The standard Go tooling basically always pulls directly from github, so

Re: [VOTE] Go SDK

+1 (binding) I just want to check about SGA/IP/Headers. Thanks ! Regards JB On 22/05/2018 03:02, Henning Rohde wrote: Hi everyone, Now that the remaining issues have been resolved as discussed, I'd like to propose a formal vote on accepting the Go SDK into master. The main practical

Re: [VOTE] Go SDK

+1 (binding), with the following caveats: * [before closing the vote] Completion of IP clearance process, as we've been requested. It is easier to do it than having to argue why it is not necessary. * [at any time, possibly later] Figuring out the release mechanics. Great work; across the board!

Re: [VOTE] Go SDK

+1! So excited to have gotten to this point -- congrats to all. I've been excited to do some reviews of the Go SDK since becoming a committer; really happy about this. On Mon, May 21, 2018 at 6:03 PM Henning Rohde wrote: > Hi everyone, > > Now that the remaining issues have

Jenkins build is back to normal : beam_SeedJob #1753

See

[VOTE] Go SDK

Hi everyone, Now that the remaining issues have been resolved as discussed, I'd like to propose a formal vote on accepting the Go SDK into master. The main practical difference is that the Go SDK would be part of the Apache Beam release going forward. Highlights of the Go SDK: * Go user

Build failed in Jenkins: beam_SeedJob #1752

See -- GitHub pull request #5406 of commit 93fc6ef85c449a912affece91e26919a75041756, no merge conflicts. Setting status of 93fc6ef85c449a912affece91e26919a75041756 to PENDING with url

Proposal for Beam Python User State and Timer APIs

I want to share a proposal for adding user state and timer support to the Beam Python SDK and get the community's thoughts on how such an API should look: https://s.apache.org/beam-python-user-state-and-timers Let me know what you think and please add any comments and suggestions you may have.

Re: The Go SDK got accidentally merged - options to deal with the pain

Hi everyone, Thanks again for your patience. The last remaining Go SDK items are now resolved and the beam website has been updated! I'll start a separate thread for the formal vote shortly. Thanks, Henning On Thu, Apr 19, 2018 at 5:42 PM Henning Rohde wrote: > Hi

Re: Proposal: keeping precommit times fast

What we can do here is estimate how much effort we want to put in and set remote target. Such as: Third quarter 2018 -- 1hr SLO Forth quarter 2018 -- 30min SLO, etc. Combined with policy for newly added tests, this can give us some goal to aim for. --Mikhail Have feedback

Re: Proposal: keeping post-commit tests green

Hi Everyone, I've updated design doc according to comments. https://docs.google.com/document/d/1sczGwnCvdHiboVajGVdnZL0rfnr7ViXXAebBAf_uQME In general, ideas proposed seem to be appreciated. Still, some of sections require more discussion. Changes highlight: * Added roll-back first policy to

Re: Proposal: keeping precommit times fast

Thanks for the proposal, I left comments in the doc. Overall I think it's a great idea. I've seen other projects with much faster pre-commits, and it requires strict guidelines on unit test design and keeping tests isolated in-memory as much as possible. That's not currently the case in Java; we

Re: [SQL] Cross Join Operation

That seems very useful. For anything that is an equi-join and we can encode the things being compared, we should do an inner join that compiles to CoGroupByKey. We will probably need some costs to make this work right. There might be some needed changes to the join library to make this happen too,

Re: SQL shaded jars don't work. How to test?

Shading requires two pieces of information: 1) Which dependencies should be part of the shaded jar (controlled by includes/excludes) 2) How to relocate code within those dependencies (controlled by relocations) The reason why the exclude(".*") exists is because typically it is an error to produce

Re: Java PreCommit seems broken

The archetype projects are coupled to their parents but don't have any meaningful dependencies so a significantly simpler archetype could be used. The dependency that exists right now in the archetype is to provide specific build ordering which can easily be moved to Gradle directly. An

Re: What is the future of Reshuffle?

Filed https://issues.apache.org/jira/browse/BEAM-4372 (unassigned). On Mon, May 21, 2018 at 10:22 AM Raghu Angadi wrote: > > > On Mon, May 21, 2018 at 9:56 AM Robert Bradshaw > wrote: > >> We should probably keep the warning and all the caveats until we

Re: SQL shaded jars don't work. How to test?

The issue SQL is seeing is caused by a default dependency of exclude(".*") added in build_rules.gradle. This breaks the normal method of building shadow jars as everything must be explicitly included. SQL explicitly added calcite to the jar, but not calcite's dependencies. I've been told this is

Re: What is the future of Reshuffle?

On Mon, May 21, 2018 at 9:56 AM Robert Bradshaw wrote: > We should probably keep the warning and all the caveats until we introduce > the alternative (and migrate to it for the non-parallelism uses of > reshuffle). I was just proposing we do this via a separate transform

Re: What is the future of Reshuffle?

+1 to introducing alternative transforms even if they wrap Reshuffle The benefits of making them distinct is that we can put appropriate Javadoc in place and runners can figure out what the user is intending and whether Reshuffle or some other implementation is appropriate. We can also see which

Re: What is the future of Reshuffle?

We should probably keep the warning and all the caveats until we introduce the alternative (and migrate to it for the non-parallelism uses of reshuffle). I was just proposing we do this via a separate transform that just calls Reshuffle until we have the new story fully fleshed out (I don't know

Re: The full list of proposals / prototype documents

Alexey, I do not know where you can find all design docs, but I know a blog that has collected some of the major design docs. Hope it helps. https://wtanaka.com/beam/design-doc https://drive.google.com/drive/folders/0B-IhJZh9Ab52OFBVZHpsNjc4eXc On Mon, May 21, 2018 at 9:28 AM Alexey Romanenko

The full list of proposals / prototype documents

Hi all, Is it possible to obtain somewhere a list of all proposals / prototype documents that have been published as a technical / design documents for new features? I have links to only some of them (found in mail list discussions by chance) but I’m not aware of others. If yes, could someone

Re: Phrase triggering - is there a better way?

I don't recall any discussions about this issue - it has always been kind of messy, like you say. I have definitely broken jobs, broken the seed job, etc :-). That is why I added the backup seed job that does not refer to common_job_properties, so that editing that file can never break the whole

Phrase triggering - is there a better way?

Hello! Beam has a possibility to "Phrase trigger" Jenkins' jobs from Github. If a job has a special phrase defined in it, a contributor can type the phrase in a PR comment and the Job will run on Jenkins. A very popular usage is to run PreCommits this way (eg. by typing "Run JavaPreCommit" in a