Jira Integration with Github

Hi all, Github has announced the official support for Jira integraton https://blog.github.com/2018-10-04-announcing-the-new-github-and-jira-software-cloud-integration/ . Is it possible to enable it for Apache Beam's Jira tickets? It could help with automation of issue workflows in Jira. Maybe,

Re: [PROPOSAL] Move testing utilities to a common place.

On Mon, Oct 8, 2018 at 9:43 AM Łukasz Gajowy wrote: > > Therefore, I propose one of the following: > >1. creating a module called "sdks/tests" and place "load-tests" and >"nexmark" modules inside it. Place the common code in "sdks/tests/common" >(or similar) and reuse it in every

Re: Rethinking Timers as PCollections

Really love this thread. The analysis is really educational. Seems like the pun of "PCollection" for so many purposes is hitting its limit. Timers should fire according to just the watermark of the data input, but nevertheless are a hold on GC and also output watermark. Kenn On Thu, Oct 4, 2018

Monitoring Url

Hello, I am interested in exposing a monitoring url through the PipelineResult object. All of the documentation pages in the runners tab of beam.apache.org have a section dedicated to ‘Monitoring your job’ and most of those reference a web interface. If we were able to get a link to this web

Re: Beam Summit community feedback

Hi Alex, Would be great to have someone experienced with Kubernetes. Not sure if it would require a custom Kubernetes Operator. It would probably suffice to have a dedicated Kubernetes mode which starts the Beam environment including Runner and dependencies. From there on, we wouldn't have

Re: Beam Summit community feedback

Related thread: https://lists.apache.org/thread.html/d6b6fde764796de31996db9bb5f9de3e7aaf0ab29b99d0adb52ac508@%3Cdev.beam.apache.org%3E Kubernetes is otherwise more of a runner deployment concern. There are efforts in the Flink community underway to make deployment on Kubernetes easier. Max:

[Call for items] October Beam Newsletter

Hi Beamers: So much has been going on that it's time to sync up again in the October Beam Newsletter [1]! :) *Add the highlights from September to now (or planned events and talks) that you want to share with the community by 10/14 11:59 p.m. **PDT.* We will collect the notes via Google docs

Re: 2.7.0 release notes inconsistent with released code

Thanks Andrew - and sorry folks, that was simply me doing a bad email search as I looked into the issue. I don't think there is much more to do communicate. I just corrected the label in Jira and I see the release notes are updated automatically - I'm not sure how we can include a note to say

Re: Portable Flink runner: Generator source for testing

The portable runner does not support metrics yet: https://s.apache.org/apache-beam-portability-support-table There is also no JIRA referenced in the table, would be good to locate/create it. On Mon, Oct 8, 2018 at 9:11 AM Łukasz Gajowy wrote: > Does anyone know what is the status of metrics

[PROPOSAL] Move testing utilities to a common place.

Hi all, we recently started implementing load tests for core apache beam operations[1] and noticed that there's some code that could be placed in a common module and used by Nexmark, IOIT and the load tests (and therefore not duplicated in multiple places). The examples are: - code for

Re: 2.7.0 release notes inconsistent with released code

As I understand, Tim's concern is the accuracy of the release notes, and +1 for correcting them. In the end it does not matter that much to the users when a release was proposed to be cut vs. when it actually happened, but what they get with the release. Perhaps between the contributors we could

Re: python post-commit failures

Is it possible to add component/unit test for this case? I believe we should aim for all precommits to be executable on isolated singlebox. Regards, --Mikhail On Mon, Oct 8, 2018, 09:27 Udi Meiri It's the current status: I believe having a basic wordcount integration > test in precommit would

Re: python post-commit failures

It's the current status: I believe having a basic wordcount integration test in precommit would have caught this issue, since it seems to have broken all tests using the Dataflow service. On Sun, Oct 7, 2018 at 9:06 PM Kenneth Knowles wrote: > Out of curiosity - is it a logical necessity, or

Re: Portable Flink runner: Generator source for testing

Does anyone know what is the status of metrics support for Flink Portable Runner? I think we need them to be used in such tests to at least collect time metric that does not contain cluster warm up time, staging resources time and other things that can disturb the actual run time metric. We

Re: 2.7.0 release notes inconsistent with released code

Keep reading the proposal thread and you'll find: "We should follow the calendar and aim to cut on 8/29, not 9/7 as I incorrectly wrote earlier." There were several folowup emails in the thread with reminders of the release cut date. Is there something we should do to better communicate release

Re: Metrics Pusher support on Dataflow

If we implement MetricsPusher to run from the local job submission JVM for Dataflow jobs and the JVM dies: the Dataflow job would continue to completion, but the MetricsPusher would not restart, so the exported metrics would be stale. I like that the Spark implementation has the ability to elect

Re: SF Meetup(s)

Great! Given the responses, seems a wealth of suitable locations. It sounds like this would be great to be a roaming meet up, to not be too tied to SF or peninsula location nor a specific company/office. I'll be in touch with the individuals who responded to me (both on-list and off), with aims

Re: Fwd: Slack invitation

Ok I will send it to you as well. Regards JB Le 8 oct. 2018 à 18:23, à 18:23, Emmanuel Bastien a écrit: >Hello, >I would like to join the Beam Slack channel. Could someone send me an >invitation? >Thanks in advance! >Emmanuel

Fwd: Slack invitation

Hello, I would like to join the Beam Slack channel. Could someone send me an invitation? Thanks in advance! Emmanuel

Re: 2.7.0 release notes inconsistent with released code

On Mon, Oct 8, 2018 at 3:17 AM Tim Robertson wrote: > > Do we correct release notes when mistakes are spotted? > I think we should, yes. Release notes are long-lived documents that should be accurate. The correction should be mentioned in the notes so that anyone who previously read the

Beam Dependency Check Report (2018-10-08)

High Priority Dependency Updates Of Beam Python SDK: Dependency Name Current Version Latest Version Release Date Of the Current Used Version Release Date Of The Latest Release JIRA Issue google-cloud-bigquery 0.25.0 1.6.0

2.7.0 release notes inconsistent with released code

Hi folks Our release notes [1] for 2.7.0 say that Beam supports Elasticsearch 6 ( BEAM-5107 ). The 2.7.0 code [2] however does not seem to, while master does [3]. The PR [4] was merged on the 6th September and in the 2.7.0 chat I see that Charles

Re: Portable Flink runner: Generator source for testing

This is correct. However, the example code is only part of Lyft's code base. Until timer support is done, we would have to do something similar in our code base. On 08.10.18 02:34, Łukasz Gajowy wrote: Hi, just to clarify, judging from the above snippets: it seems that we are able now to

Re: Is there any way to ask the runner to call finalizeCheckpoint() method before it closed the Reader?

For the UnboundedSource interface, this depends on the Runner. Generally, close() should be called when no more data will be read from the Reader. The FlinkRunner calls `close()` on the Readers when it closes the operator (see UnboundedSourceWrapper). The best documentation we have for this

Re: Beam website sources migrated to apache/beam

Thank you for pushing this through; this is a great improvement! On Sat, Oct 6, 2018 at 1:52 PM Alexey Romanenko wrote: > This is great news! Many thanks to all who contributed to this! > > I believe that as the process of website update will be clearer and > simpler, as our documentation will