Forward StackOverflow questions with the apache-beam tag to a new mailing list

Hi folks, It seems like many of the questions posted to StackOverflow with the apache-beam tag [1] go unanswered or take more than they should to receive an acceptable answer. What do you all think about creating a new mailing list, stackoverf...@beam.apache.org (assuming Apache Infra is OK with

Re: [VOTE] Release 2.41.0, release candidate #1

I don't think so. The vendored version that Beam depends on has been published and shouldn't need to be built again. Not sure if our build has changed so that it builds the vendored stuff as part of the main build as well, vs pulling it from maven central. Kenn On Tue, Aug 16, 2022 at 2:42 PM

Re: [VOTE] Release 2.41.0, release candidate #1

If we roll back, do we need to roll back +Lukasz Cwik 's change [1] on master as well? Liam, are we okay to roll back this change for this release? [1] https://github.com/apache/beam/pull/22594 On Tue, Aug 16, 2022 at 2:25 PM Kenneth Knowles wrote: > Two options to unblock the release are: >

Re: [VOTE] Release 2.41.0, release candidate #1

Two options to unblock the release are: 1. Roll back https://github.com/apache/beam/pull/17317/files 2. Downgrade unvendored bytebuddy to 1.11.0 (if the above doesn't work or is too high risk) And as a follow up we should make sure there is some test that would exercise this, since that PR was

Re: [VOTE] Release 2.41.0, release candidate #1

cc: @Liam Miller-Cushon , who worked on the bytebuddy update. Liam, do you have any context on this error? On Tue, Aug 16, 2022 at 10:11 AM Alexey Romanenko wrote: > I tested with "beam-samples" [1] and found that a rather simple test > pipeline fails [2] with this runtime error: > > Error: >

Re: Beam BigtableIO versus Google CloudBigtableIO

Sounds good! In regards to the second paragraph, it is true that there was a recent change to the amount of mutations in a batch. I would still recommend using bulkOptions and withBigtableOptionsConfigurator(), I believe that the field 'BIGTABLE_BULK_MAX_ROW_KEY_COUNT_DEFAULT' may be what you are

Re: Beam BigtableIO versus Google CloudBigtableIO

Hello Diego, Right now we are using BigtableIO so I will continue to use that one! For the second part, Ill explain a bit more what we saw as I simplified a bit in my original email. At some point we had two streaming pipelines writing to bigtable and we decided to combine these into one

Re: [VOTE] Release 2.41.0, release candidate #1

I tested with "beam-samples" [1] and found that a rather simple test pipeline fails [2] with this runtime error: Error: Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.776 s <<< FAILURE! - in SerializationTest 5809 Error: SerializationTest.nonSerilizableTest Time

Re: Beam BigtableIO versus Google CloudBigtableIO

Hello Sahith, We recommend using BigtableIO over CloudBigtableIO. Both of them have similar performances and main differences being than CloudBigtableIO uses HBase Result and Puts, while BigtableIO uses protos to read results and mutations. The two connectors should result in similar spending on

Re: [VOTE] Release 2.41.0, release candidate #1

+1 (non-binding), Validated Go SDK Quickstart on Direct and Dataflow runner On Tue, Aug 16, 2022 at 4:26 AM Jan Lukavský wrote: > +1 (non-binding) > > Validated Java SDK with classical Flink Runner. > On 8/15/22 23:06, Chamikara Jayalath via dev wrote: > > +1 as well > (I believe Kiley is

Beam BigtableIO versus Google CloudBigtableIO

Hello, I see that there are two implementations of reading and writing from Bigtable, one in beam and one that is references in Google cloud documentation. Is one preferred over the other? We often use the Beam BigtableIO to write to bigtable but I have found that sometimes the default

Benchmark tests for the Beam RunInference API

Hi, I created a doc [1] which outlines the plan for the RunInference API[2] benchmark/performance tests. I would appreciate feedback on the following, - Models used for the benchmark tests. - Metrics

Beam High Priority Issue Report (68)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/22642 [Bug]: Dataflow

Re: [VOTE] Release 2.41.0, release candidate #1

+1 (non-binding) Validated Java SDK with classical Flink Runner. On 8/15/22 23:06, Chamikara Jayalath via dev wrote: +1 as well (I believe Kiley is addressing the container tags issue) Thanks, Cham On Mon, Aug 15, 2022 at 1:00 PM Robert Bradshaw wrote: +1 (binding). I verified