[VOTE] Release 2.23.0, release candidate #1

Hi everyone, Please review and vote on the release candidate #1 for the version 2.23.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1],

Season of Docs 2020 Proposal for Apache Beam (Ayeshmantha)

Below is a project proposal from a technical writer (bcc'd) who wants to work with your organization on a Season of Docs project. Please assess the proposal and ensure that you have a mentor to work with the technical writer. If you want to accept the proposal, please submit the technical writing

Season of Docs 2020 Proposal for Apache Beam (Basavraj)

Below is a project proposal from a technical writer (bcc'd) who wants to work with your organization on a Season of Docs project. Please assess the proposal and ensure that you have a mentor to work with the technical writer. If you want to accept the proposal, please submit the technical writing

Season of Docs 2020 Proposal for Apache Beam (Sruthi Sree Kumar)

Below is a project proposal from a technical writer (bcc'd) who wants to work with your organization on a Season of Docs project. Please assess the proposal and ensure that you have a mentor to work with the technical writer. If you want to accept the proposal, please submit the technical writing

Re: Streaming pipeline "most-recent" join

Hya, I never got a chance to finish this one, maybe I will get some time in the summer break... but I think it will help with your use case...

Streaming pipeline "most-recent" join

Hi Beam devs, I'm working on a streaming pipeline where we need to do a "most-recent" join between two PCollections. Specifically, something like: out = pcoll1 | beam.Map(lambda a,b: (a,b), b=beam.pvalue.AsSingleton(pcoll2)) The goal is to join each value in pcoll1 with only the most recent

Re: Versioning published Java containers

My main question is, are we confident the Java 11 container is ready to release? AFAIK there are still a number of issues blocking full Java 11 support (cf [1] ; not sure how many of these, if any, affect the SDK harness specifically though.) For

Interactive Beam Side Panel in JupyterLab

Hi everyone, Here is a design doc about adding a JupyterLab extension for Interactive Beam

Versioning published Java containers

Hi all, I'm getting ramped up on contributing and was looking into adding the Java 11 harness container to releases ( https://issues.apache.org/jira/browse/BEAM-8106) - should I rename the current java container so we have two new images `beam_java8_sdk` and `beam_java11_sdk` or hold off on

[Proposal] - Publish Content for Apache Beam Channels

Hi folks, I wanted to share some exciting news that Google is starting to leverage Hootsuite and Brandwatch as social listening and engagement tools to streamline engagement and measure impact of content produced by open source projects. We will do this to listen to our user and contributor

Re: Season of Docs Interest

Hey Sharon! Thank you so much for your interest to contribute to Beam's documentation. It is a big help that you have knowledge and experience with Spark and Dataflow already. In order to be considered for the program, you'll need to submit a proposal with a summary of the documentation work

Re: Finer-grained test runs?

It does sound like we're generally on the same page. Minor comments below. On Thu, Jul 9, 2020 at 1:00 PM Kenneth Knowles wrote: > > On Thu, Jul 9, 2020 at 11:47 AM Robert Bradshaw wrote: >> >> On Thu, Jul 9, 2020 at 8:40 AM Luke Cwik wrote: >> > >> >> If Brian's: it does not result in

Re: Finer-grained test runs?

No, not without doing the research myself to see what is the current tooling available. On Thu, Jul 9, 2020 at 1:17 PM Kenneth Knowles wrote: > > > On Thu, Jul 9, 2020 at 1:10 PM Luke Cwik wrote: > >> The budget would represent some criteria that we need from tests (e.g. >> percent passed, max

Re: Contributor permission for Beam Jira ticket

Welcome. I have added you to Beam's Jira. On Thu, Jul 9, 2020 at 9:30 AM Jiahao Wu wrote: > Hi, > > This is Jiahao from Google. I am working in the Google Cloud HCLS team > this summer and we want to add ab IO connector for our API to better > support our customers. Can someone add me as a

Re: KinesisIO Tests - are they run anywhere?

It has come up a few times[1, 2, 3, 4] and there have also been a few comments over time about whether someone could donate AWS resources to the project. 1: https://issues.apache.org/jira/browse/BEAM-601 2: https://issues.apache.org/jira/browse/BEAM-3373 3:

Re: Finer-grained test runs?

On Thu, Jul 9, 2020 at 1:10 PM Luke Cwik wrote: > The budget would represent some criteria that we need from tests (e.g. > percent passed, max num skipped tests, test execution time, ...). If we > fail the criteria then there must be actionable work (such as fix tests) > followed with something

Re: Finer-grained test runs?

The budget would represent some criteria that we need from tests (e.g. percent passed, max num skipped tests, test execution time, ...). If we fail the criteria then there must be actionable work (such as fix tests) followed with something that prevents the status quo from continuing (such as

Re: KinesisIO Tests - are they run anywhere?

Have you guys considered using localstack to run AWS service based integration tests? https://github.com/localstack/localstack On Thu, 9 Jul, 2020, 5:25 PM Piotr Szuberski, wrote: > Yeah, I meant KinesisIOIT tests. I'll do the same with the cross-language > it tests then. Thanks for your reply

Re: Finer-grained test runs?

On Thu, Jul 9, 2020 at 11:47 AM Robert Bradshaw wrote: > On Thu, Jul 9, 2020 at 8:40 AM Luke Cwik wrote: > > > >> If Brian's: it does not result in redundant build (if plugin works) > since it would be one Gradle build process. But it does do a full build if > you touch something at the root of

Re: Monitoring performance for releases

Not yet, I just learned about the migration to a new frontend, including a new backend (InfluxDB instead of BigQuery). - Are the metrics available on metrics.beam.apache.org? Is http://metrics.beam.apache.org online? I was never able to access it. - What is the feature delta between

Re: Finer-grained test runs?

On Thu, Jul 9, 2020 at 8:40 AM Luke Cwik wrote: > >> If Brian's: it does not result in redundant build (if plugin works) since it >> would be one Gradle build process. But it does do a full build if you touch >> something at the root of the ancestry tree like core SDK or model. I would >> like

Re: contributor permission for Beam Jira tickets

Done! On Thu, Jul 9, 2020 at 11:28 AM Damian Gadomski wrote: > Hi, > > Can I be added to the JIRA contributors so I can assign tickets to > myself, please? > > my Jira username: damgad > > Thanks, > Damian >

contributor permission for Beam Jira tickets

Hi, Can I be added to the JIRA contributors so I can assign tickets to myself, please? my Jira username: damgad Thanks, Damian

Re: Monitoring performance for releases

Questions: - Are the metrics available on metrics.beam.apache.org? - What is the feature delta between usinig metrics.beam.apache.org (much better UI) and using apache-beam-testing.appspot.com? - Can we notice regressions faster than release cadence? - Can we get automated alerts? Kenn On

Re: Finer-grained test runs?

On Thu, Jul 9, 2020 at 8:40 AM Luke Cwik wrote: > On Wed, Jul 8, 2020 at 9:22 PM Kenneth Knowles wrote: > >> I like your use of "ancestor" and "descendant". I will adopt it. >> >> On Wed, Jul 8, 2020 at 4:53 PM Robert Bradshaw >> wrote: >> >>> On Wed, Jul 8, 2020 at 4:44 PM Luke Cwik wrote:

Monitoring performance for releases

Hi, We recently saw an increase in latency migrating from Beam 2.18.0 to 2.21.0 (Python SDK with Flink Runner). This proofed very hard to debug and it looks like each version in between the two versions let to increased latency. This is not the first time we saw issues when migrating,

Contributor permission for Beam Jira ticket

Hi, This is Jiahao from Google. I am working in the Google Cloud HCLS team this summer and we want to add ab IO connector for our API to better support our customers. Can someone add me as a contributor for Beam's Jira issue tracker so I can create/assign tickets for my work? My Jira username is:

Re: KinesisIO Tests - are they run anywhere?

Yeah, I meant KinesisIOIT tests. I'll do the same with the cross-language it tests then. Thanks for your reply :) On 2020/07/08 17:13:11, Alexey Romanenko wrote: > If you mean Java KinesisIO tests, then unit tests are running on Jenkins [1] > and ITs are not running since it requires AWS

Re: Finer-grained test runs?

On Wed, Jul 8, 2020 at 9:22 PM Kenneth Knowles wrote: > I like your use of "ancestor" and "descendant". I will adopt it. > > On Wed, Jul 8, 2020 at 4:53 PM Robert Bradshaw > wrote: > >> On Wed, Jul 8, 2020 at 4:44 PM Luke Cwik wrote: >> > >> > I'm not sure that breaking it up will be