Re: Python 3: final step

Does anyone know how to set up python version on Jenkins ? It’s Python 3.5.2 now. Thanks, Manu Zhang On Oct 5, 2018, 9:24 AM +0800, Valentyn Tymofieiev , wrote: > I have put together a guide [1] to help get started with investigating Python > 3-related test failures that may be helpful for new

BEAM-2953, Timeseries library

RE: Pull Request : https://github.com/apache/beam/pull/6540 I have been doing some work on a generalized set of timeseries transforms, with the goal to abstract the user from the process of dealing with some of the common problems when working with timeseries in BEAM batch / stream mode.

Re: Beam Samza Runner status update

Interesting On Wed, Oct 10, 2018, 3:49 PM Kenneth Knowles wrote: > Welcome, Hai! > > On Wed, Oct 10, 2018 at 3:46 PM Hai Lu wrote: > >> Hi, all >> >> This is Hai from LinkedIn. As Xinyu mentioned, I have been working on >> portable API for Samza runner and made some solid progress. It's been a

Re: Log output from Dataflow tests

Hi Max, I don't have edit privileges for the project so can't modify user. On Wed, Oct 10, 2018 at 9:02 AM Maximilian Michels wrote: > Thank you Scott! Ismael also sent me the logs and I could fix the error. > > It seems we have granted read-only access to project members in the > past. I just

Re: Java > 8 support

Hello all, If I understand you correctly Ismael, a good amount of 'beam-sdks-java-core' tests are already passing with Java 11, so the amount of work necessary on the core module should be relatively small. Is this correct? Are there improvements that may be missing in terms of modularization?

Re: Beam Samza Runner status update

Welcome, Hai! On Wed, Oct 10, 2018 at 3:46 PM Hai Lu wrote: > Hi, all > > This is Hai from LinkedIn. As Xinyu mentioned, I have been working on > portable API for Samza runner and made some solid progress. It's been a > very smooth process (although not effortless for sure) and I'm really >

Re: Beam Samza Runner status update

Hi, all This is Hai from LinkedIn. As Xinyu mentioned, I have been working on portable API for Samza runner and made some solid progress. It's been a very smooth process (although not effortless for sure) and I'm really grateful for the great platform that you all have built. I'm very impressed.

Re: [Proposal] Euphoria DSL - looking for reviewers

Anton: All of the points are be correct, with one minor exception. We are currently moving our production workloads from Euphoria to Beam (using the DSL), but we are hitting scalability issues of the current spark runner, so it is not technically used in

Re: [Proposal] Euphoria DSL - looking for reviewers

I just glanced through it to make sure things are in the right place and build set up right and that all LGTM. We need to file the IP Clearance to finish the process that Davor started. Please fill the XML template at

Re: [DISCUSS] Gradle for the build ?

Thank you JB for starting this discussion. Others comment on many of these points far better than I can, but my experience is similar to JB. 1. IDEA integration (and laptop slowing like crazy) being the biggest contributor to my feeling of being unproductive 2. Not knowing the correct way to

Re: [Proposal] Euphoria DSL - looking for reviewers

I think the code looks good and we should probably just merge it (unless there are other blockers, e.g. formal approvals), considering: - it has been reviewed; - it is tested and used in production; - it was discussed on the list and there were no objections to having it as part of Beam; - it

Re: Fwd: Slack invitation

I got it, thank you! On Wed, 10 Oct 2018 at 16:17, Jean-Baptiste Onofré wrote: > You didn't receive it ? > > Let me try another time. > > Regards > JB > Le 10 oct. 2018, à 17:15, "Filip Popić" a écrit: >> >> Any news regarding invitation? >> >> On Mon, 8 Oct 2018 at 17:24, Jean-Baptiste Onofré

Re: [PROPOSAL] Prepare Beam 2.8.0 release

Given the number of open issues, I will re-cut the release branch once the blocking issues are resolved. Don't worry about cherry picking changes to directly to the release branch for now. I will continue to update this thread. On Wed, Oct 10, 2018 at 12:12 PM, Niel Markwick wrote: > The 3

Re: [DISCUSS] Beam public roadmap

What about a link in the menu. It should contain a list of features and estimate date with probable error (like "in 5 months +- 1 months) otherwise it does not bring much IMHO. Le mer. 10 oct. 2018 23:32, Kenneth Knowles a écrit : > Hi all, > > We made an attempt at putting together a sort of

[DISCUSS] Beam public roadmap

Hi all, We made an attempt at putting together a sort of roadmap [1] in the past and also some wide-ranging threads about what could be on it [2]. and I think we should pick it up again. The description I really liked was "strategic and user impacting initiatives (ongoing and future) in an easy

Re: Portable Flink runner: Generator source for testing

I've opened a JIRA for adding the generator source (BEAM-5707) and sent out a very rough PR (https://github.com/apache/beam/pull/6637). Would appreciate any feedback. On Mon, Oct 8, 2018 at 9:43 AM, Thomas Weise wrote: > The portable runner does not support metrics yet: https://s.apache.org/ >

Re: Beam Samza Runner status update

Clarification: Thomas Groh wrote the fuser, not me! Thanks for the sharing all this. Really cool. Kenn On Wed, Oct 10, 2018 at 11:17 AM Rui Wang wrote: > Thanks for sharing! it's so exciting to hear that Beam is being used on > Samza in production @LinkedIn! Your feedback will be helpful to

Re: Splitting the repo

I think Robert's initial question needs to be focused on a particular split. I agree that a "single project spanning multiple repos" does not make sense. But separate projects in separate repos is pretty widely used :-). The point of separate repos IMO would be to empower (and force) them to act

Re: Does anyone have a strong intelliJ setup?

I left my tips to run *Java* unit tests in Intellij (work for me all the time). I assumed that people mostly use intellij for Java development. If there are some cases when people use Intellij to develop other languages (maybe because of the power of plugins?), we might need to create separate

Re: [PROPOSAL] Prepare Beam 2.8.0 release

Thank you JB. It turns out there are 2 more blocker issues. I will look at them now first. (So, I am not rushing towards cutting RC1 yet.) On Wed, Oct 10, 2018 at 11:42 AM, Jean-Baptiste Onofré wrote: > Hey > > Etienne should do a new pass soon. I do my best to cherry pick RabbitMQIO. > >

Re: Does anyone have a strong intelliJ setup?

Last week I migrated all previous content from the website into wiki pages for IntelliJ [1] and Eclipse [2] (thanks Thomas Weise for the pointers). The next step is to incorporate all the tips that people have mentioned here and fill in any other gaps we have. Here's how I'd like to get started:

Re: [PROPOSAL] Prepare Beam 2.8.0 release

Hey Etienne should do a new pass soon. I do my best to cherry pick RabbitMQIO. Thanks Regards JB Le 10 oct. 2018 à 21:25, à 21:25, Ahmet Altay a écrit: >Update: > >I started cutting the branch. There are 2 open issues: >- RabbitMQIO - JB, if you plan to complete this soon I can cherry pick >to

Re: Splitting the repo

Le mer. 10 oct. 2018 21:31, Robert Bradshaw a écrit : > > > On Wed, Oct 10, 2018, 4:56 PM Romain Manni-Bucau > wrote: > >> >> >> >> Le mer. 10 oct. 2018 à 14:59, Maximilian Michels a >> écrit : >> >>> Hi, >>> >>> I agree that splitting up Beam into separate repositories would cause >>> more

Re: Splitting the repo

On Wed, Oct 10, 2018, 4:56 PM Romain Manni-Bucau wrote: > > > > Le mer. 10 oct. 2018 à 14:59, Maximilian Michels a > écrit : > >> Hi, >> >> I agree that splitting up Beam into separate repositories would cause >> more pain than gain. >> >> To a large degree we already have independent modules,

Re: [PROPOSAL] Prepare Beam 2.8.0 release

Update: I started cutting the branch. There are 2 open issues: - RabbitMQIO - JB, if you plan to complete this soon I can cherry pick to the branch. - One new issue related to release process changes with respect to beam-site deprecation. On Tue, Oct 9, 2018 at 11:38 AM, Jean-Baptiste Onofré

Re: Beam Samza Runner status update

Thanks for sharing! it's so exciting to hear that Beam is being used on Samza in production @LinkedIn! Your feedback will be helpful to Beam community! Besides, Beam supports SQL right now and hopefully Beam community could also receive feedback on BeamSQL

Re: Splitting the repo

Hi, I think the subtext here is that development is hard in general. I agree to it. And a major cause of it is diversity of languages, complexity of the project and legacy code. To alleviate language related issues, we are trying to have modular code which we already have to a certain extent. On

Re: Beam Samza Runner status update

Thanks for sharing and congrats for this great work ! Regards JB Le 10 oct. 2018 à 20:23, à 20:23, Xinyu Liu a écrit: >Hi, All, > >It's been over four months since we added the Samza Runner to Beam, and >we've been making a lot of progress after that. Here I would like to >update >your guys and

Beam Samza Runner status update

Hi, All, It's been over four months since we added the Samza Runner to Beam, and we've been making a lot of progress after that. Here I would like to update your guys and share some really good news happening here at LinkedIn: 1) First Beam job in production @LInkedIn! After a few rounds of

Re: [DISCUSS] Gradle for the build ?

> Perhaps we should go through and prioritize (and add missing items to) BEAM-4045 +1. It's hard to know where to start when there's such a laundry list of tasks. If you're having build issues, will you make sure it is represented in BEAM-4045, and "Vote" for the issues that you believe are the

Re: Log output from Dataflow tests

Thank you Scott! Ismael also sent me the logs and I could fix the error. It seems we have granted read-only access to project members in the past. I just checked back with Ankur, he might be able to grant access for my GCP account. -Max On 10.10.18 17:26, Scott Wegner wrote: I'm not sure

Re: Log output from Dataflow tests

I'm not sure how apache-beam-testing permissions are managed; Kenn, could we grant read-access for contributors who need it for testing? Here are two logs from the job that seem relevant: 2018-10-08 14:44:45.381 PDT Parsing unknown args:

Re: What is required for LTS releases? (was: [PROPOSAL] Prepare Beam 2.8.0 release)

some times Ago JB spoke about Beam roadmap. I tend to think this discussion does no make any sense without a clear roadmap. The rational here is that a roadmap will give you the future changes and the potential future versions (we spoke a few times of Beam 3). This does not have to be very

Re: What is required for LTS releases? (was: [PROPOSAL] Prepare Beam 2.8.0 release)

On Wed, Oct 10, 2018 at 2:56 AM Robert Bradshaw wrote: > On Wed, Oct 10, 2018 at 9:37 AM Ismaël Mejía wrote: > >> The simplest thing we can do is just to pin all the deps of the LTS >> and not move them in any maintenance release if not a strong reason to >> do so. >> >> The next subject is to

Re: Splitting the repo

Le mer. 10 oct. 2018 à 14:59, Maximilian Michels a écrit : > Hi, > > I agree that splitting up Beam into separate repositories would cause > more pain than gain. > > To a large degree we already have independent modules, e.g. runners/* or > sdks/*. Although this is not the case for the core. It

Re: Fwd: Slack invitation

You didn't receive it ? Let me try another time. Regards JB Le 10 oct. 2018 à 17:15, à 17:15, "Filip Popić" a écrit: >Any news regarding invitation? > >On Mon, 8 Oct 2018 at 17:24, Jean-Baptiste Onofré >wrote: > >> Ok I will send it to you as well. >> >> Regards >> JB >> Le 8 oct. 2018, à

Re: Fwd: Slack invitation

Any news regarding invitation? On Mon, 8 Oct 2018 at 17:24, Jean-Baptiste Onofré wrote: > Ok I will send it to you as well. > > Regards > JB > Le 8 oct. 2018, à 18:23, Emmanuel Bastien a écrit: >> >> Hello, >> I would like to join the Beam Slack channel. Could someone send me an >> invitation?

Re: Splitting the repo

Hi, I agree that splitting up Beam into separate repositories would cause more pain than gain. To a large degree we already have independent modules, e.g. runners/* or sdks/*. Although this is not the case for the core. It would be desirable to break it up further. > possibly even with

Re: [Proposal] Euphoria DSL - looking for reviewers

Hello Max, It would be great if you can do more of a "general" review, the code base is fairly large, well tested and it was already reviewed internally by several people. We would like to have the overall approach and design decisions validated by the community and get some inputs on what could

Re: [Proposal] Euphoria DSL - looking for reviewers

That is a huge PR! :) Euphoria looks great. Especially for people coming from Flink/Spark. I'll check out the documentation. Do you have any specific code parts which you want to have reviewed? Thanks, Max On 10.10.18 10:30, Jean-Baptiste Onofré wrote: Hi, Thanks for all the work you are

Re: What is required for LTS releases? (was: [PROPOSAL] Prepare Beam 2.8.0 release)

On Wed, Oct 10, 2018 at 9:37 AM Ismaël Mejía wrote: > The simplest thing we can do is just to pin all the deps of the LTS > and not move them in any maintenance release if not a strong reason to > do so. > > The next subject is to make maintainers aware of which release will be > the LTS in

Re: [DISCUSS] Gradle for the build ?

Some rough stats (because I was curious): The gradle files have been edited by ~79 unique contributors over 696 distinct commits, whereas the maven ones were edited (over a longer time period) by ~130 unique contributors over 1389 commits [1]. This doesn't capture how much effort was put into

Re: Splitting the repo

This looks functionnal whereas the split is more about languages and making the build smooth and efficient to work with to get back up to speed. Runners can stay in java land/subproject while they are not in other languages for instance so the api between core and runner can stay as it for that

Re: Splitting the repo

On Wed, Oct 10, 2018 at 10:25 AM Romain Manni-Bucau wrote: > On the split point: a mono-repo works for me as well. The main point is "N > separate builds". > > On the portable thing: currently runner integrates with portable api. It > impacts all runner. The needed code is the same everywhere

Re: Splitting the repo

Yep for the split For the clean point it is quite linked to the build tools and fake env for not native modules for the build tool (go for gradle which is java first for instance). This is why having a real build which is natural per language would be beneficial IMO. Le mer. 10 oct. 2018 11:38,

Re: Splitting the repo

Correct, it's more "module splitting" than repositories indeed. Regards JB On 10/10/2018 10:35, Robert Bradshaw wrote: > Gotcha. So this is more about dividing the code (particularly core) into > finer modules, rather than splitting the modules into separate > repositories, right?  > > On Wed,

Re: Splitting the repo

On Wed, Oct 10, 2018 at 10:35 AM Romain Manni-Bucau wrote: > Also we can get a more adapted build tool by area and not break the repo > for each build. Go and python build always need a git clean for java users > which is a big issue so let's build each subproject - that is what beam is > today

Re: Splitting the repo

Gotcha. So this is more about dividing the code (particularly core) into finer modules, rather than splitting the modules into separate repositories, right? On Wed, Oct 10, 2018 at 10:29 AM Jean-Baptiste Onofré wrote: > The purpose is that we have a monolithic core today mostly providing >

Re: Splitting the repo

Also we can get a more adapted build tool by area and not break the repo for each build. Go and python build always need a git clean for java users which is a big issue so let's build each subproject - that is what beam is today - as they should with an adapted tool. It requires very few

Re: [DISCUSS] Gradle for the build ?

Hi all, I must admit that I agree on the status especially regarding 2 points: 1. new contributors obstacles: gradle learning curve might be too long for spare-time contributors, also complex scripted build takes time to understand comparing to self-descriptive one. 2. IDE integration kind of

Re: [Proposal] Euphoria DSL - looking for reviewers

Hi, Thanks for all the work you are doing on this DSL ! I tried to follow the features branch for a while. I'm still committed to move forward on that front, but more reviewers would be great. Regards JB On 10/10/2018 10:26, Plajt, Vaclav wrote: > Hello Beam devs, > we finished our main

Re: Splitting the repo

The purpose is that we have a monolithic core today mostly providing abstract classes. The idea is to have something more API oriented with interface/SPI. Our users would then be able to pick the part of the core they want, resulting with lighter artifacts, and for us, it gives a more flexible

Re: Splitting the repo

My question was not whether we should split the repo, but why? (Dividing things into more (or fewer) modules withing a single repo is a separate question.) Maybe I'm just not following what you mean by "more API oriented." It would force stabler APIs. On Wed, Oct 10, 2018 at 10:18 AM

[Proposal] Euphoria DSL - looking for reviewers

Hello Beam devs, we finished our main goals in development of Euphoria DSL. It is Easy to use Java 8 API build on top of the Beam's Java SDK. API provides a high-level abstraction of data transformations, with focus on the Java 8 language features (e.g. lambdas and streams). It is fully

Re: Splitting the repo

On the split point: a mono-repo works for me as well. The main point is "N separate builds". On the portable thing: currently runner integrates with portable api. It impacts all runner. The needed code is the same everywhere since it is mainly a DoFn at the end (a bit caricatural but that is the

Re: [DISCUSS] Gradle for the build ?

On Wed, Oct 10, 2018 at 8:03 AM Jean-Baptiste Onofré wrote: > Hi Robert, > > about your point about we never fully build the project, even if I > agree, it's what we "sold" with Gradle. > Because, with Maven you can also build a single module without problem. > Good incremental support for the

Re: Splitting the repo

Hi, +1, even I think we could split the core even deeper. I discussed with Luke and Reuven to introduce core-sql, core-schema, core-sdf, ... It's not a huge effort, and would allow us to move forward on Beam "more API oriented" approach. Regards JB On 10/10/2018 10:12, Robert Bradshaw wrote:

Splitting the repo

Hi everyone, While IMHO it's too early to even be able to split the repo, it's not to early to talk about it, and I wanted to spin this off to keep the other thread focused. In particular, I am trying to figure out exactly what is hoped to be gained by splitting things up. In my experience, a

Re: Log output from Dataflow tests

Would be great to provide access to Dataflow build logs. In the meantime, could someone with access send me the logs for the job below? https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-10-08_14_41_03-9578125971484804239?project=apache-beam-testing Thanks,

Re: Java > 8 support

Thanks for the clarification Ismaël. * • **Arif Kasim* * • * Strategic Cloud Engineer * • *Google, Inc. • arifka...@google.com On Wed, Oct 10, 2018 at 9:41 AM Ismaël Mejía wrote: > Just wanted to clarify, there is already a JIRA for ongoing work on > Java 11 support. >

Re: Java > 8 support

Just wanted to clarify, there is already a JIRA for ongoing work on Java 11 support. https://issues.apache.org/jira/browse/BEAM-2530 I led the initial work on supporting what at the time was Java 9/10, so far the biggest blockers were around the ApiSurface tests (not at all compatible with these

Re: What is required for LTS releases? (was: [PROPOSAL] Prepare Beam 2.8.0 release)

The simplest thing we can do is just to pin all the deps of the LTS and not move them in any maintenance release if not a strong reason to do so. The next subject is to make maintainers aware of which release will be the LTS in advance so they decide what to do with the dependencies versions. In

Re: [DISCUSS] Gradle for the build ?

JB mentioned some factual points, I think most of the community embraced the move to gradle with the best hopes about the promised improvements, but with the perspective of time, it is clear that we have not delivered on many of them. I can easily understand the growing bitterness about the

Re: [DISCUSS] Gradle for the build ?

@Reuven: any idea what is missing? I don't expect it to be ready very quickly but having 2 repos does not hurt that much if both are working better than a single one so can be worth a try maybe? Romain Manni-Bucau @rmannibucau | Blog

Re: [DISCUSS] Gradle for the build ?

Unrelated to the Maven/Gradle discussion, but I do somewhat agree with Romain that Beam could be split into separate repos, however I don't think it's quite ready yet. AFAIK the portability interfaces are still being modified, and until these interfaces are fixed it will be hard to split the

Re: [DISCUSS] Gradle for the build ?

Hi Robert, about your point about we never fully build the project, even if I agree, it's what we "sold" with Gradle. Because, with Maven you can also build a single module without problem. So, if I don't get the argument about build time for single module: in that case, build speed was not a