Re: [VOTE] Extension name of Interactive Beam Side Panel in JupyterLab

+1 for [3] as well. On Wed, Jul 15, 2020 at 5:40 PM Ahmet Altay wrote: > > I agree with Kyle. [3] sounds more accurate. > > On Wed, Jul 15, 2020 at 3:00 PM Kyle Weaver wrote: >> >> I prefer [3]. >> >> On Tue, Jul 14, 2020 at 10:53 AM Ning Kang wrote: >>> >>> Hi everyone, >>> >>> Last week, I

Re: [VOTE] Release 2.23.0, release candidate #1

I agree. I think Dataflow x-lang users could run into flaky pipelines due to this. Valentyn, are you OK with creating a new RC that includes the fix (already merged - https://github.com/apache/beam/pull/12164) and preferably https://github.com/apache/beam/pull/12196 ? Thanks, Cham On Wed, Jul

Re: [VOTE] Extension name of Interactive Beam Side Panel in JupyterLab

I agree with Kyle. [3] sounds more accurate. On Wed, Jul 15, 2020 at 3:00 PM Kyle Weaver wrote: > I prefer [3]. > > On Tue, Jul 14, 2020 at 10:53 AM Ning Kang wrote: > >> Hi everyone, >> >> Last week, I sent a design doc >>

Re: Chronically flaky tests

I think it will be reasonable to disable/sickbay any flaky test that is actively blocking people. Collective cost of flaky tests for such a large group of contributors is very significant. Most of these issues are unassigned. IMO, it makes sense to assign these issues to the most relevant person

Re: [VOTE] Release 2.23.0, release candidate #1

I think we need to cherry-pick https://issues.apache.org/jira/browse/BEAM-10397 which fixes missing environment errors for Dataflow xlang pipelines. Internally, we have a flaky xlang kafkaio test because of missing environment errors and any xlang pipelines using GroupByKey could encounter this.

Re: [VOTE] Release 2.23.0, release candidate #1

On Wed, Jul 15, 2020 at 4:55 PM Robert Bradshaw wrote: > All the artifacts, signatures, and hashes look good. > > I would like to understand the severity of > https://issues.apache.org/jira/browse/BEAM-10397 before giving my > vote. > +Heejong Lee to comment on this. > > On Wed, Jul 15, 2020

Re: Chronically flaky tests

The situation is much worse than that IMO. My experience of the last few days is that a large portion of time went to *just connecting failing runs with the corresponding Jira tickets or filing new ones*. Summarized on PRs: - https://github.com/apache/beam/pull/12272#issuecomment-659050891 -

Re: [VOTE] Release 2.23.0, release candidate #1

All the artifacts, signatures, and hashes look good. I would like to understand the severity of https://issues.apache.org/jira/browse/BEAM-10397 before giving my vote. On Wed, Jul 15, 2020 at 10:51 AM Pablo Estrada wrote: > > +1 > I was able to run the python 3.8 quickstart from wheels on

Chronically flaky tests

We have two test suites that are responsible for a large percentage of our flaky tests and both have bugs open for about a year without being fixed. These suites are ParDoLifecycleTest (BEAM-8101 ) in Java and BigQueryWriteIntegrationTests in

Re: [VOTE] Extension name of Interactive Beam Side Panel in JupyterLab

I prefer [3]. On Tue, Jul 14, 2020 at 10:53 AM Ning Kang wrote: > Hi everyone, > > Last week, I sent a design doc > > and proposals in this email thread >

Re: Versioning published Java containers

It makes sense to me that the snapshot should be everything needed for a release. Definitely containers fit that. Kenn On Wed, Jul 15, 2020 at 11:37 AM Chamikara Jayalath wrote: > > > On Wed, Jul 15, 2020 at 11:17 AM Kyle Weaver wrote: > >> Thanks everyone for the details. Seems like Java 11

Re: Versioning published Java containers

On Wed, Jul 15, 2020 at 11:17 AM Kyle Weaver wrote: > Thanks everyone for the details. Seems like Java 11 support is farther > along than I had imagined :) > > > Is there any progress into getting > > back, any ticket people can follow if interested? > >

Re: Versioning published Java containers

Thanks everyone for the details. Seems like Java 11 support is farther along than I had imagined :) > Is there any progress into getting > back, any ticket people can follow if interested? https://issues.apache.org/jira/browse/BEAM-10049 > I understand that a user can publish their own versions

Re: [VOTE] Release 2.23.0, release candidate #1

+1 I was able to run the python 3.8 quickstart from wheels on DirectRunner. I verified hashes for Python files. -P. On Fri, Jul 10, 2020 at 4:34 PM Ahmet Altay wrote: > I validated the python 3 quickstarts. I had issues with running > with python 3.8 wheel files, but did not have issues with

Re: [PROPOSAL] Make PBegin and PDone public in the Python SDK

With splittable DoFns, we should be aiming to have 'sources' take PCollections for input as the default implementation. The common 'Read' with no inputs will still exist and it would make sense to have PBegin. We have found that most sinks actually could have meaningful output (e.g. how many

Re: Versioning published Java containers

Can we consider regularly publishing HEAD containers as well (for example, we publish SNAPSHOT jars daily) ? I understand that a user can publish their own versions of HEAD containers but this does not work well when developing automated tests for distributed runners. Apologies if this was

Re: Use Coder message for cross-lang ExternalConfigurationPayload?

On Fri, Jul 10, 2020 at 4:47 PM Robert Bradshaw wrote: > On Fri, Jul 10, 2020 at 4:36 PM Brian Hulette wrote: > > > > Ah yes I'm +1 for that approach too - it would let us leverage all the > schema-inference already in the Java SDK for translating configuration > objects which would be great. >

Re: [VOTE] Release 2.23.0, release candidate #1

+1. Thanks. Tried several Python batch examples and a streaming pipeline with x-lang Kafka with Dataflow. - Cham On Fri, Jul 10, 2020 at 4:34 PM Ahmet Altay wrote: > I validated the python 3 quickstarts. I had issues with running > with python 3.8 wheel files, but did not have issues with

RE: NanosInstant not being recognised by BigQueryIO.Write

Hi Cham, Yes, I’m happy to when I get a moment. Kind regards, Rob From: Chamikara Jayalath [mailto:chamik...@google.com] Sent: 15 July 2020 16:36 To: dev Subject: Re: NanosInstant not being recognised by BigQueryIO.Write * "This is an external

Re: NanosInstant not being recognised by BigQueryIO.Write

On Wed, Jul 8, 2020 at 3:30 AM wrote: > Hi All, > > > > I am posting this to the dev (as opposed to user channel) as I believe it > will be of interest to the those working on either Schemas or BigQuery > > > > I have a pipeline based on BEAM 2.22 that is ingesting data into > BigQuery.

Re: Versioning published Java containers

Thanks Robert for the explanation. Is there any progress into getting back, any ticket people can follow if interested? On Wed, Jul 15, 2020 at 12:13 AM Robert Burke wrote: > > Disallowing the go containers was largely due to not having a simple check on > the go boot code's licenses which is

Re: [DISCUSS] ReadAll pattern and consistent use in IO connectors

Vincent, I will be out in the sense that I cannot really engage myself into more activities because I have apart of your review two more pending + other work to finish so I prefer not to add more work I cannot finish. I am still available for the review however so let’s get this finally finished