Re: [VOTE] Apache Beam, version 2.6.0, release candidate #1

+1 non-binding 1. tested with beam samples 2. verified sigs and hashes of artifacts On Fri, Aug 3, 2018 at 12:43 AM, Jean-Baptiste Onofré wrote: > +1 (binding) > > Tested with beam-samples. > > I didn't have time to include three Jira, but 2.7.0 should be in vote in > soon ;) > > Regards > JB

Re: [VOTE] Apache Beam, version 2.6.0, release candidate #1

+1 (binding) Tested with beam-samples. I didn't have time to include three Jira, but 2.7.0 should be in vote in soon ;) Regards JB On 01/08/2018 01:50, Pablo Estrada wrote: > Hello everyone! > > I have been able to prepare a release candidate for Beam 2.6.0. : D > > Please review and vote on

Re: Nexmark PostCommit tests fail

Thank you for looking into this Andrew. --Mikhail Have feedback ? On Thu, Aug 2, 2018 at 5:17 PM Andrew Pilloud wrote: > These are timeouts at 4 hours. Dataflow is slow to start compared with > other runners. https://github.com/apache/beam/pull/6127 will run tests

Re: Removing documentation for old Beam versions

[image: pr-520.png] (trying that image again) On Thu, Aug 2, 2018 at 7:00 PM Udi Meiri wrote: > Alright, created https://github.com/apache/beam-site/pull/520 > [image: pr-520.png] > Reduces staging upload from 500M down to 270M, and halves the number of > files from ~22k to 11k. > > > > On Thu,

Re: Removing documentation for old Beam versions

Alright, created https://github.com/apache/beam-site/pull/520 [image: pr-520.png] Reduces staging upload from 500M down to 270M, and halves the number of files from ~22k to 11k. On Thu, Aug 2, 2018 at 6:58 PM Pablo Estrada wrote: > I believe tags will be necessarily because for anyone looking

Re: Removing documentation for old Beam versions

I believe tags will be necessarily because for anyone looking for old docs that have been removed, they will need to browse back in history, not just browse the tree of directories. -P. On Thu, Aug 2, 2018, 6:46 PM Mikhail Gryzykhin wrote: > Last time I talked with Scott I brought this idea in.

Re: Removing documentation for old Beam versions

Last time I talked with Scott I brought this idea in. I believe the plan was either to publish compiled site to website directly, or keep it in separate storage from apache/beam repo. One of the main reasons not to check in compiled version of website is that every developer will have to pull all

Re: Removing documentation for old Beam versions

Pablo, the docs are generated into versioned paths, e.g., https://beam.apache.org/documentation/sdks/javadoc/2.5.0/ so tags are not necessary? Also, once apache/beam-site is merged with apache/beam the release branch should have the relevant docs (although perhaps it's better to put them in a

Re: [Call for items] Beam August Newsletter

Thanks for doing this Rose! I'll add a couple of things. -P. On Thu, Aug 2, 2018, 4:18 PM Rose Nguyen wrote: > Hi all: > > Here's > > [1] > the template for the August Beam Newsletter! > > *Add the

Re: Nexmark PostCommit tests fail

These are timeouts at 4 hours. Dataflow is slow to start compared with other runners. https://github.com/apache/beam/pull/6127 will run tests in parallel and fix the issue. Andrew On Thu, Aug 2, 2018, 5:05 PM Reuven Lax wrote: > I'm having a hard time understanding what the failure is. I see a

Re: Nexmark PostCommit tests fail

I'm having a hard time understanding what the failure is. I see a bunch of Jenkins errors in the log. Did the actual Beam pipeline fail, or is this a Jenkins-level failure? Reuven On Thu, Aug 2, 2018 at 4:54 PM Mikhail Gryzykhin wrote: > Hi everyone, > > We have Nexmark PostCommit tests fail.

Nexmark PostCommit tests fail

Hi everyone, We have Nexmark PostCommit tests fail. Should we disable or fix these? https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ Regards, --Mikhail Have feedback ?

[Call for items] Beam August Newsletter

Hi all: Here's [1] the template for the August Beam Newsletter! *Add the highlights from June and July that you want to share with the community by 8/8 11:59 p.m. **PDT.* I'm working with Gris--we've heard

Re: Parallelizing test runs

I've disabled concurrency for auto-triggered post-commits job. That should reduce job scheduling considerably. I believe that this change should resolve quota issue we have seen this time. I'll monitor if problem reappears. --Mikhail Have feedback ? On Wed, Aug 1,

Re: Live coding & reviewing adventures

Ok Gris has an even more delayed laptop so I'm going to push it out a week and hope it shows up for then. Sorry about that one and thanks for everyone who tuned in for the Go SDK one :) On Mon, Jul 30, 2018 at 1:54 PM, Holden Karau wrote: > So small schedule changes. > I’ll be doing some poking

Re: [VOTE] Apache Beam, version 2.6.0, release candidate #1

+1 Verified with SQL component. On Thu, Aug 2, 2018 at 10:05 AM, Thomas Weise wrote: > It does include *some* of the portable Flink runner (you will be able to > run wordcount as documented on https://beam.apache.org/ > contribute/portability/#status). > > I would recommend to continue using

Re: Removing documentation for old Beam versions

+1 for removing pre 2.0 documentation (as well as the entries from https://beam.apache.org/get-started/downloads/) Isn't it part of the beam-site changes that we will no longer check in generated documentation into the repository? Those can be generated and deployed independently (when a commit

Re: Removing documentation for old Beam versions

Is it worth adding a tag / branch to the repositories every time we make a release, so that people are able to dive in and find the docs? Best -P. On Thu, Aug 2, 2018 at 12:09 PM Ahmet Altay wrote: > I would guess that users are still using some of these old releases. It is > unclear from Beam

Re: Removing documentation for old Beam versions

I would guess that users are still using some of these old releases. It is unclear from Beam website which releases are still supported or not. It probably makes sense to drop documentation for releases < 2.0. (I would suggest keeping docs for 2.0). For the future I can work on updating the Beam

Re: Removing documentation for old Beam versions

The older docs are not directly linked to and are in Github commit history. If there are no objections I'm going to delete javadocs and pydocs for releases older than 1 year, meaning 2.0.0 and older (going by the dates here ). On Thu, Aug 2, 2018

Re: Removing documentation for old Beam versions

The older docs should be recorded in the commit history of the website repository, right? If they're not currently used in the website and they're in the commit history then I don't see a reason to save them. On Tue, Jul 31, 2018 at 1:51 PM Udi Meiri wrote: > Hi all, > I'm writing a PR for

Re: Community Examples Repository

I like he initiative but I feel that fragmenting the codebase will make it harder to discover examples. Having examples in a separate repo makes it easier to forget that examples should get the same love as the rest of the codebase. The other challenge is the tooling and integration which is

Re: Community Examples Repository

Agreed with Rui. We could also add more SQL examples (like, different IOs ) for everyone to get started with. Best, Kai On 2018/08/02 17:40:32, Rui Wang wrote: > I might miss it: are examples to be moved including those which are not > under example/? For example there are some BeamSQL

Re: Community Examples Repository

I might miss it: are examples to be moved including those which are not under example/? For example there are some BeamSQL examples in org/apache/beam/sdk/extensions/sql/example

Re: Cleanup resources on pipeline cancelation

Romain is correct, you would need some global reference counting here to use the close() callback. The problem is that the input subscription is a pipeline-wide resource, it's not a per-reader resource. On Thu, Aug 2, 2018 at 10:07 AM Romain Manni-Bucau wrote: > > > Le jeu. 2 août 2018 18:32,

Re: Cleanup resources on pipeline cancelation

Le jeu. 2 août 2018 18:32, Andrew Pilloud a écrit : > The subscriptions I want to clean up are ones that are implicitly created > by the PubsubIO. These subscriptions are created then leaked, they aren't > reused in future pipelines so the data loss issues are moot here. I agree > that we don't

Re: [VOTE] Apache Beam, version 2.6.0, release candidate #1

It does include *some* of the portable Flink runner (you will be able to run wordcount as documented on https://beam.apache.org/contribute/portability/#status). I would recommend to continue using master though, as we are still not fully at MVP, adding test coverage and also Flink version update.

Re: Cleanup resources on pipeline cancelation

The subscriptions I want to clean up are ones that are implicitly created by the PubsubIO. These subscriptions are created then leaked, they aren't reused in future pipelines so the data loss issues are moot here. I agree that we don't want to tear down user supplied subscriptions. I've been

Re: Community Examples Repository

Robert, I agree with you in general. However there is also a second motivation. There is an increase in new PRs that are coming to add new examples. This is great however the core code (including distributions) is not a great place to host such examples. An examples repo would help in this case.

Re: Cleanup resources on pipeline cancelation

Actually I think SDF is the right way to fix this. The SDF can set a timer at infinity (which will only fires when the pipeline shuts down). I believe that SDF support is being added to the portability layer now, so eventually all portable runners will support it, and maybe we can live with the

Re: Community Examples Repository

I have to admit I'm generally -1 on moving examples to a separate repository. In particular, I think it would actually inhibit the stated goals of increasing visibility and better keeping them up to date, and for all the reasons we just migrated the beam-site directory in. It seems the primary