[Perk] Sharing the love for Flink Forward

Hi Beam Community! As you know, Apache Beam is a big supporter of Apache Flink - pun intended ;) - and because Google Cloud is a proud sponsor of Flink Forward we have 10 passes for members of our Apache Beam community to attend the event in Berlin this Sept. 3rd & 4th for free. If you're interes

Re: [PROPOSAL] Prepare Beam 2.7.0 release

Thanks everyone. Again, we will proceed with the initial release cut on August 29. A reminder to please tag any blocking issues as "Priority: Blocker" and "Fix version: 2.7.0" in JIRA. We recently resolved https://issues.apache.org/jira/browse/BEAM-5180, and there are no other blocking bugs at t

Re: BEAM-5180 for 2.7.0 ?

Thank you for getting the partial rollback in. I will close https://issues.apache.org/jira/browse/BEAM-5180 as fixed. Ankur: if you have a more nuanced fix in mind, please open a new JIRA ticket to track and update us on this thread. On Fri, Aug 24, 2018 at 10:42 AM Ankur Goenka wrote: > Repli

Re: Process JobBundleFactory for portable runner

> Do we expect pipelines to always have a single environment for each PTransform, thus the SDK is dictating how it is launched/managed or do we expect for each SDK to say here are a couple of ways to run me, letting the runner to decide? Good point. I think we should allow multiple environments in

Re: Gradle Races in beam-examples-java, beam-runners-apex

I'm seeing failures due to this on 12 of the last 16 PostCommits. Precommits take about 22 minutes run in parallel, so at a 25% pass rate that puts the expected time to a good test run at 264 minutes assuming you immediately restart on each failure. We are looking at 56 minutes for a precommit that

Re: INFO:root:Executing Error when executing a pipeline on dataflow

It seems like we only mention the need for pip 7.0.0 on the python quickstart page https://beam.apache.org/get-started/quickstart-py/ Would you like to submit a change to update it? On Wed, Aug 22, 2018 at 9:31 AM OrielResearch Eila Arich-Landkof < e...@orielresearch.org> wrote: > The issue was

Re: Process JobBundleFactory for portable runner

Do we expect pipelines to always have a single environment for each PTransform, thus the SDK is dictating how it is launched/managed or do we expect for each SDK to say here are a couple of ways to run me, letting the runner to decide? What does providing the target os/arch provide in beam:env:pro

Re: Publishing release artifacts to custom artifactory

Alexey, publishing to custom repo with authentication is now possible, see https://github.com/apache/beam/pull/6230 with example. On Fri, Aug 24, 2018 at 1:08 PM Lukasz Cwik wrote: > *"-Poffline-repository" *controls the addition of another maven repo to > read dependencies from, it doesn't imp

Re: Publishing release artifacts to custom artifactory

*"-Poffline-repository" *controls the addition of another maven repo to read dependencies from, it doesn't impact project publishing and shouldn't be needed. On Fri, Aug 24, 2018 at 5:28 AM Alexey Romanenko wrote: > Maybe my answer is not 100% relevant to initial topic (sorry for that in > advan

Re: [Proposal] Track non-code contributions in Jira

+1 On Fri, Aug 24, 2018 at 11:50 AM Henning Rohde wrote: > +1 > > On Fri, Aug 24, 2018 at 11:44 AM Rose Nguyen wrote: > >> +1 Great idea >> >> On Fri, Aug 24, 2018 at 10:01 AM Mikhail Gryzykhin >> wrote: >> >>> +1. Idea sounds great. >>> >>> --Mikhail >>> >>> Have feedback

Re: [Proposal] Track non-code contributions in Jira

+1 On Fri, Aug 24, 2018 at 11:44 AM Rose Nguyen wrote: > +1 Great idea > > On Fri, Aug 24, 2018 at 10:01 AM Mikhail Gryzykhin > wrote: > >> +1. Idea sounds great. >> >> --Mikhail >> >> Have feedback ? >> >> >> On Fri, Aug 24, 2018 at 7:19 AM Maximilian Michels >> wro

Re: Removing documentation for old Beam versions

Hi Udi, Good to know you will continue this work. Let me know if you want to try the buildbot route (which does not require generated documentation to be checked into the repo). Happy to help with that. Thomas On Fri, Aug 24, 2018 at 11:36 AM Udi Meiri wrote: > I'm picking up the website migr

Re: Removing documentation for old Beam versions

Git is really efficient at things it can perform diffs on. Generated source code tends to be fine as long as it has reasonably short lines. It becomes an issue when you are checking in binaries, images, and compressed files (jars for example). On Fri, Aug 24, 2018 at 11:36 AM Udi Meiri wrote: >

Re: [Proposal] Track non-code contributions in Jira

+1 Great idea On Fri, Aug 24, 2018 at 10:01 AM Mikhail Gryzykhin wrote: > +1. Idea sounds great. > > --Mikhail > > Have feedback ? > > > On Fri, Aug 24, 2018 at 7:19 AM Maximilian Michels wrote: > >> +1 Code is just one part of a successful open-source project. As lon

Re: Removing documentation for old Beam versions

I'm picking up the website migration. The plan is to not include generated files in the master branch. However, I've been told that even putting generated files a separate branch could blow up the git repository for all (e.g. make git pulls a lot longer?). Not sure if this is a real issue or not.

Re: Gradle Races in beam-examples-java, beam-runners-apex

I believe it would mitigate the issue but also make the jobs take much longer to complete. On Thu, Aug 23, 2018 at 2:44 PM Andrew Pilloud wrote: > There seems to be a misconfiguration of gradle that is causing a high rate > of failure for the last several weeks in building beam-examples-java and

Re: BEAM-5180 for 2.7.0 ?

Replies on the Jira and PR. For now we should go ahead with rollback to unblock 2.7 On Fri, Aug 24, 2018 at 10:21 AM Udi Meiri wrote: > +Ankur Goenka (Kenneth is out of office) > > On Fri, Aug 24, 2018 at 3:20 AM Tim Robertson > wrote: > >> Thanks Jozef for bringing this to dev@ and your work

Re: BEAM-5180 for 2.7.0 ?

+Ankur Goenka (Kenneth is out of office) On Fri, Aug 24, 2018 at 3:20 AM Tim Robertson wrote: > Thanks Jozef for bringing this to dev@ and your work in reporting Jiras > and offering fixes. > > I propose we consider BEAM-5180, BEAM-2277 blockers on 2.7.0. They break > word count and file IO wri

Re: [Proposal] Track non-code contributions in Jira

+1. Idea sounds great. --Mikhail Have feedback ? On Fri, Aug 24, 2018 at 7:19 AM Maximilian Michels wrote: > +1 Code is just one part of a successful open-source project. As long as > the tasks are properly labelled and actionable, I think it works to put > them int

Re: [Proposal] Track non-code contributions in Jira

+1 Code is just one part of a successful open-source project. As long as the tasks are properly labelled and actionable, I think it works to put them into JIRA. On 24.08.18 15:09, Matthias Baetens wrote: I fully agree and think it is a great idea. I think that, next to visibility and keeping

Re: [Proposal] Track non-code contributions in Jira

Jira is basically a fancy TODO list; if folks think it would be helpful for tracking these kinds of contributions (e.g. there's a lot of stuff that needs to be done for a successful meetup, or things like "write a blog post about X") I think it's worth a try. I don't know how useful it'd be for ope

Re: [Proposal] Track non-code contributions in Jira

I fully agree and think it is a great idea. I think that, next to visibility and keeping track of everything that is going on in the community, the other goal would be documenting best practices for future use. I am also not sure, though, if JIRA is the best place to do so, as Austin raised. Intr

Re: Publishing release artifacts to custom artifactory

Maybe my answer is not 100% relevant to initial topic (sorry for that in advance) but it took me quite a time to find out how to properly install artefacts into local maven repository with gradle. Finally, I came to this command (additional flags are skipped for the sake of simplicity). ./gradl

Re: Beam Summit London 2018

Hi, here is a copy of my proposal so that anyone interested can give also feedback here: Title: "Lesson Learned from Migrating to Apache Beam for Geo-Data Visualisation" Summary: "Any company that wants to establish data-driven processes needs to set up specific tools for its data infrastructur

Re: BEAM-5180 for 2.7.0 ?

Thanks Jozef for bringing this to dev@ and your work in reporting Jiras and offering fixes. I propose we consider BEAM-5180, BEAM-2277 blockers on 2.7.0. They break word count and file IO writing on HDFS unless the workaround is used (see BEAM-2277 commentary). In addition the performance of writ

Re: Beam Summit London 2018

Thanks. I'll stick to 20+10 then when I send my proposal. Cheers! On Wed, Aug 22, 2018 at 10:58 AM Matthias Baetens wrote: > Hi Pascal, Javier, > > Thanks for your interest in submitting a talk! > @Pascal: I am happy to check for you if what you have in mind is already > being covered in anothe

Re: Process JobBundleFactory for portable runner

I think "external" still needs some way (I was suggesting grpc) to pass the control address, etc. to whatever starts up the workers. Also, +1 to making this a URN. Embedded makes sense too. On Fri, Aug 24, 2018 at 6:00 AM Thomas Weise wrote: > > Option #3 "external" would fit the Kubernetes use c

Build failed in Jenkins: beam_Release_Gradle_NightlySnapshot #151

See Changes: [batbat] Added @onWindowExpiration annotation to use for annotating a user [batbat] Incorporated the OutputReceiver changes. [pablo] Adding numeric support to BQ Sink [github] Up