Re: [DISCUSS] Committer Guidelines / Hygene before merging PRs

Generally, +1 for isolated commits which are easy to revert. I don't think it's actually harder to roll back a set of commits that are merged together. I think Thomas was mainly concerned about "fixup" commits to land in master (as part of a merge). These indeed make reverting commits more

Re: [Discuss] Upgrade story for Beam's execution engines

FYI, I opened a PR with a compatibility table for the Flink Runner page: https://github.com/apache/beam-site/pull/553 On 17.09.18 09:31, Robert Bradshaw wrote: On Mon, Sep 17, 2018 at 2:02 AM Austin Bennett mailto:whatwouldausti...@gmail.com>> wrote: Do we currently maintain a finer

Re: [VOTE] Release 2.7.0, release candidate #1

Hi Charles, Thanks for driving the release. Here are the cherrypicks I mentioned: https://github.com/apache/beam/pull/6427 Thanks, Max On 17.09.18 19:17, Charles Chen wrote: Luke, Maximillian, Raghu, can you please propose cherry-pick PRs to the release-2.7.0 for your issues and add me as a

Re: [Discuss] Upgrade story for Beam's execution engines

[Copying this also to the dev list] +1. A version compatibility table would be great! > I don't know if Flink could do something like this (become a provided > dep) in particular for the current case where there seems not to be > API breaking changes. That doesn't work. The Flink Runner is too

Re: [Discuss] Upgrade story for Beam's execution engines

rth supporting as well. Agree. Apart from the LTS, we want to minimize upgrade pain for new Beam versions as much as possible. On 17.09.18 09:30, Robert Bradshaw wrote: On Sun, Sep 16, 2018 at 12:59 PM Maximilian Michels <mailto:m...@apache.org>> wrote: > If I understand t

Re: Beam dashboards

Thanks Mikhail, that will help to identify flaky or slow tests. At the size of the Beam code base, such statistics are extremely helpful. If we had a list of test cases ordered by test duration, that would be a great addition. On 14.09.18 00:30, Connell O'Callaghan wrote: Thank you Mikhail

Re: [Discuss] Upgrade story for Beam's execution engines

story. Absolutely. If we find that we can improve compatibility with upstream changes, we should go that path. Even if we don't have a dedicated compatibility layer upstream yet. On 13.09.18 19:34, Thomas Weise wrote: On Thu, Sep 13, 2018 at 9:49 AM Maximilian Michels <mailto:m...@apache.

Re: Java postcommit tests timeout due to flink jobs

culprit test, I think it is a good candidate to disable while you're working on proper fix. Regards, --Mikhail Have feedback <http://go/migryz-feedback>? On Wed, Sep 12, 2018 at 10:47 AM Maximilian Michels <mailto:m...@apache.org>> wrote: Missed that issue. I can look int

Re: [VOTE] Donating the Dataflow Worker code to Apache Beam

+1 (binding) On 15.09.18 20:07, Reuven Lax wrote: +1 On Sat, Sep 15, 2018 at 9:40 AM Rui Wang > wrote: +1 -Rui On Sat, Sep 15, 2018 at 12:32 AM Robert Bradshaw mailto:rober...@google.com>> wrote: +1 (binding) On Sat, Sep 15, 2018

Re: Donating the Dataflow Worker code to Apache Beam

If anything, merging the Dataflow Worker code shows Google's commitment to the Beam project. Yes, it does solve internal issues with syncing their runtime with Beam, but Beam was always about the programming model for data processing, not about a specific type of execution engine. Like any

Re: [Discuss] Upgrade story for Beam's execution engines

e chose not to do this on spark runner. WDYT ? Best Etienne Le mardi 11 septembre 2018 à 15:42 +0200, Maximilian Michels a écrit : Hi Beamers, In the light

Re: Java postcommit tests timeout due to flink jobs

Missed that issue. I can look into this tomorrow. On 12.09.18 19:27, Mikhail Gryzykhin wrote: +Aljoscha for real. On Wed, Sep 12, 2018 at 10:25 AM Mikhail Gryzykhin > wrote: Hi everyone, We have ~50% java post-commit jobs timeout due to Flink tests. Can

Re: [VOTE] Release 2.7.0, release candidate #1

Could we still include some fixes for the RC2? I just discovered two JIRA issues which were not properly marked with "Fix Version". https://issues.apache.org/jira/browse/BEAM-5239 https://issues.apache.org/jira/browse/BEAM-5246 They are not show-stoppers, so also fine with me if we don't

Re: Gradle Races in beam-examples-java, beam-runners-apex

Do we have inotifywait available on Travis and could set it up to log concurrent access to the relevant Jar files? On 10.09.18 22:41, Lukasz Cwik wrote: I had originally suggested to use some Linux kernel tooling such as inotifywait[1] to watch what is happening. It is likely that we have

Re: SplittableDoFn

Thanks for moving forward with this, Lukasz! Unfortunately, can't make it on Friday but I'll sync with somebody on the call (e.g. Ryan) about your discussion. On 08.09.18 02:00, Lukasz Cwik wrote: Thanks for everyone who wanted to fill out the doodle poll. The most popular time was Friday

Re: [Call for items] September Beam Newsletter

Good stuff! Left some items for the Flink Runner. On 08.09.18 02:14, Rose Nguyen wrote: *bump* Celebrate the weekend by sharing with the community your talks, contributions, plans, etc! On Wed, Sep 5, 2018 at 10:25 AM Rose Nguyen > wrote: Hi Beamers:

Re: [FYI] Paper of Building Beam Runner for IBM Streams

Excellent write-up. Thank you! On 09.09.18 20:43, Jean-Baptiste Onofré wrote: Good idea. It could also help people who wants to create runners. Regards JB On 09/09/2018 13:00, Manu Zhang wrote: Hi all, I've spent the weekend reading Challenges and Experiences in Building an Efficient Apache

Re: PTransforms and Fusion

A) What should we do with these "empty" PTransforms? We can't translate them, so dropping them seems the most reasonable choice. Should we throw an error/warning to make the user aware of this? Otherwise might be unexpected for the user. A3) Handle the "empty" PTransform case within all of

Re: PR/6343: Adding support for MustFollow

This is a great idea but I share Lukasz' doubts about this being a universal solution for awaiting some action in a pipeline. I wonder, wouldn't it work to not pass in a PCollection, but instead wrap a DoFn which internally ensures the correct triggering behavior? All runners which correctly

Re: [NEW CONTRIBUTOR] ElasticsearchIO now supports Elasticsearch v6.x

Well done. Thank you, Dat! On 06.09.18 22:47, Trần Thành Đạt wrote: Thank you. Etienne Chauchot and Tim Robertson helped me a lot to get familiar with Beam code. On Fri, Sep 7, 2018 at 2:59 AM Thomas Weise > wrote: Support for Elastic 6.x is really good to have.

Re: Python 3: final step

This has been requested multiple times. Thanks for working on the Python 3 story. Let me know if I can help out in any way! On 05.09.18 19:01, Valentyn Tymofieiev wrote: This is awesome! Kudos to Robbe and Matthias who have been pushing this forward! On Wed, Sep 5, 2018 at 9:45 AM Charles

Re: Beam Schemas: current status

this should only do its magic if there is only one possible way to feed data to the constructor. That's why a dedicated interface would be the easier and safer way to opt-in. On 31.08.18 11:27, Robert Bradshaw wrote: On Fri, Aug 31, 2018 at 11:22 AM Maximilian Michels <mailto:m...@apache.org>&

Re: Beam Schemas: current status

also assume that there's a default constructor).  I can remove this restriction if there is an appropriate constructor or builder interface that lets us construct the object directly. Reuven On Thu, Aug 30, 2018 at 6:51 AM Maximilian Michels <mailto:m...@apache.org>> wrote:

Re: builds.apache.org refused connections since last night

Jenkins is up again! (woho!) On 30.08.18 20:23, Thomas Weise wrote: I would be concerned with multiple folks running the Jekyll build locally to end up with inconsistent results. But if Jenkins stays down for longer, then maybe one of us can be the Jenkins substitute :) On Thu, Aug 30, 2018

Re: Status of IntelliJ with Gradle

ult, I have to run and debug only using gradle for now. Thanks, Xinyu On Wed, Aug 22, 2018 at 1:45 AM, Maximilian Michels wrote: Thanks Lukasz. I also found that I can never fix all import errors by manually adding jars to the IntelliJ library list. It is also not a good solution because it bre

Re: Beam Schemas: current status

That's a cool feature. Are there any limitations for the schema inference apart from being a Pojo/Bean? Does it supported nested PoJos, e.g. "wrapper.field"? -Max On 29.08.18 07:40, Reuven Lax wrote: I wanted to send a quick note to the community about the current status of schema-aware

Re: [Proposal] Creating a reproducible environment for Beam Jenkins Tests

Hi Yifan, Thanks for the proposal. I like the idea of unifying test environments via Docker. It would be great if we could still easily run tests without Docker. Best, Max On 27.08.18 19:53, Yifan Zou wrote: Hi, I have a proposal for creating a reproducible environment for Jenkins tests

Re: Bootstrapping Beam's Job Server

endpoints). This could be implemented as a script that goes and makes the call and exits, but I think this would be common enough it'd be worth building in, and also useful enough for testing that it should be very lightweight. On Mon, Aug 27, 2018 at 10:51 AM Maximilian Michels wrote: Robert, just t

Re: Process JobBundleFactory for portable runner

onfigured, additional steps such as virtualenv activate or setting of other environment variables can be included as well. On Thu, Aug 23, 2018 at 5:15 AM Maximilian Michels mailto:m...@apache.org>> wrote: Just to recap:  From this and the other thread ("Boot

Re: Bootstrapping Beam's Job Server

figuration. On 23.08.18 17:07, Robert Bradshaw wrote: On Thu, Aug 23, 2018 at 3:47 PM Maximilian Michels wrote: > Going down this path may start to get fairly involved, with an almost > endless list of features that could be requested. Instead, I would > suggest we keep process-based ex

Re: [Perk] Sharing the love for Flink Forward

Just wanted to chime in here and say that Flink Forward is a great conference. You get to meet lots of people from the Flink community from all over the world, committers as well as end users. There are awesome talks as well. Plus, you get to travel to Berlin which, if you haven't been, I

Re: [Proposal] Track non-code contributions in Jira

+1 Code is just one part of a successful open-source project. As long as the tasks are properly labelled and actionable, I think it works to put them into JIRA. On 24.08.18 15:09, Matthias Baetens wrote: I fully agree and think it is a great idea. I think that, next to visibility and

Re: Bootstrapping Beam's Job Server

ger worker creation > (deletion?), passing the requisite parameters (e.g. the fn api > endpoints). Aren't you making up more features now? :) Couldn't this be also handled by the shell script? On 23.08.18 14:13, Robert Bradshaw wrote: On Thu, Aug 23, 2018 at 1:54 PM Maximilian Michels wro

Re: Process JobBundleFactory for portable runner

y add the     external idea where the SDK/user starts the SDK harnesses     instead of the runner. Each runner may not support all types     of environments.     Henning     On Tue, Aug 21, 2018 at 2:52 AM Maximilian Michels     mailto:m...@apache.org>&

Re: Process JobBundleFactory for portable runner

dea where the SDK/user starts the SDK harnesses instead of the runner. Each runner may not support all types of environments. Henning On Tue, Aug 21, 2018 at 2:52 AM Maximilian Michels mailto:m...@apache.org>> wrote:

Re: Bootstrapping Beam's Job Server

enabling docker in their clusters. On Tue, Aug 21, 2018 at 11:50 AM Maximilian Michels mailto:m...@apache.org>> wrote: > > Thanks Henning and Thomas. It looks like > > a) we want to keep the Docker Job Server Docker container and

Re: Status of IntelliJ with Gradle

iml beforehand to add the vendored jar file as the top dependency (jar never appears in the modules dependencies) On Mon, Aug 20, 2018 at 8:36 AM Maximilian Michels <mailto:m...@apache.org>> wrote: Thank you Etienne for opening the issue. Anyone else having problems with t

Re: Process JobBundleFactory for portable runner

For reference, here is corresponding JIRA issue for this thread: https://issues.apache.org/jira/browse/BEAM-5187 On 16.08.18 11:15, Maximilian Michels wrote: Makes sense to have an option to run the SDK harness in a non-dockerized environment. I'm in the process of creating a Docker entry

Re: Bootstrapping Beam's Job Server

ture plans to deploy Flink TMs via Kubernetes. Thanks, Thomas [1] https://lists.apache.org/thread.html/d8b81e9f74f77d74c8b883cda80fa48efdcaf6ac2ad313c4fe68795a@%3Cdev.beam.apache.org%3E On Mon, Aug 20, 2018 at 3:00 PM Maximilian Michels <mailto:m...@apache.org>> wrote:

Re: Beam Docs Contributor

That sounds great, Rose. Welcome! On 21.08.18 09:21, Etienne Chauchot wrote: > Welcome Rose ! > > Etienne > > Le lundi 30 juillet 2018 à 10:10 -0700, Thomas Weise a écrit : >> Welcome Rose, and looking forward to the docs update! >> >> On Mon, Jul 30, 2018 at 9:15 AM Henning Rohde >

Re: Beam application upgrade on Flink crashes

AFAIK the serializer used here is the CoderTypeSerializer which may not be recoverable because of changes to the contained Coder (TaggedKvCoder). It doesn't currently have a serialVersionUID, so even small changes could break serialization backwards-compatibility. As of now Beam doesn't offer the

Re: Status of IntelliJ with Gradle

about gradle > improvements and I just split it in several tickets. Here is the one > concerning the same issue: https://issues.apache.org/jira/browse/BEAM-5176 > > Etienne > > Le lundi 20 août 2018 à 15:51 +0200, Maximilian Michels a écrit : >> Hi Beamers, >> >> It

Bootstrapping Beam's Job Server

Hi everyone, I wanted to get your opinion on the Job-Server startup [1] which is part of the portability story. I've created a docker container to bring up Beam's Job Server, which is the entry point for pipeline execution. Generally, this works fine when the backend (Flink in this case) runs

Re: Status of IntelliJ with Gradle

Sorry, please disregard this duplicate mail. The Apache mail relay was flaky and my client doesn't seem to handle it particularly well. On 20.08.18 15:51, Maximilian Michels wrote: > Hi Beamers, > > It's great to see the Beam build system overhauled. Thank you for all > t

Status of IntelliJ with Gradle

Hi Beamers, It's great to see the Beam build system overhauled. Thank you for all the hard work. That said, I've just started contributing to Beam again and I feel really stupid for not having a fully-functional IDE. I've closely followed the IntelliJ/Gradle instructions [1]. In the terminal

Status of IntelliJ with Gradle

Hi Beamers, It's great to see the Beam build system overhauled. Thank you for all the hard work. That said, I've just started contributing to Beam again and I feel really stupid for not having a fully-functional IDE. I've closely followed the IntelliJ/Gradle instructions [1]. In the terminal

Re: Discussion: Scheduling across runner and SDKHarness in Portability framework

Hi Ankur, Thanks for looking into this problem. The cause seems to be Flink's pipelined execution mode. It runs multiple tasks in one task slot and produces a deadlock when the pipelined operators schedule the SDK harness DoFns in non-topological order. The problem would be resolved if we

Re: Metrics architecture inside the runners

the regular website would be needed. > You're right, please fill a jira. > > Etienne > > Le jeudi 16 août 2018 à 18:24 +0200, Maximilian Michels a écrit : >> Hi Etienne, >> >> Great overview. Thank you! >> >> When do we plan to document Metrics for u

Re: Metrics architecture inside the runners

Hi Etienne, Great overview. Thank you! When do we plan to document Metrics for users? Perhaps I should open a JIRA issue. Cheers, Max On 16.08.18 12:22, Etienne Chauchot wrote: > Hi folks ! > > I've created a page in the new Beam wiki for contributors: > >

Re: Process JobBundleFactory for portable runner

Makes sense to have an option to run the SDK harness in a non-dockerized environment. I'm in the process of creating a Docker entry point for Flink's JobServer[1]. I suppose you would also prefer to execute that one standalone. We should make sure this is also an option. [1]

Re: Test failures list

Thank you Mikhail for looking into test failures and compiling the list! > I cannot access this link. Is it publicly accessible? Works for me but it takes a while to show results. > One general question: maybe it's a good idea to assign change > authors/code owners to the issues? Or just reach

Re: Apache Beam Python Wheels Repository

+1 Travis for building the Python wheels looks fine to me. Many Apache projects use Travis in addition to Jenkins. Apache is also invested in Travis [1] to ensure the build capacity is sufficient. In any case, we could migrate away from Travis if it doesn't work out as expected. We don't have to

Re: [VOTE] Fixing @yyy.com.INVALID mailing addresses

+1 Thanks for looking into it! On 23.11.17 00:25, Lukasz Cwik wrote: > I have noticed that some e-mail addresses (notably @google.com) get > .INVALID suffixed onto it so per...@yyy.com become per...@yyy.com.INVALID > in the From: header. > > I have figured out that this is an issue with the way

Re: On my activity at the project

Frances! > > On Tue, 17 Jan 2017 at 18:53 Kenneth Knowles <k...@google.com.invalid> wrote: > >> Great to work with you so far, and looking forward to it in the future. >> Enjoy your time off! >> >> Kenn >> >> On Sat, Jan 14, 2017 at 12:0

<    1   2   3   4   5   6