Re: [PROPOSAL] Vendored bytebuddy dependency release

Microbenchmarks are tough for these benchmarks. In the past, we've had changes that increase the time it took to generate bytcode. While this his minimal impact on real pipelines (since bytecode is generated on worker startup), it has an outsized impact on microbencmark run time. On Wed, May 12,

Re: Removing deprecated oauth2client dependency for Python SDK

+Valentyn Tymofieiev might have an idea. On Mon, May 3, 2021 at 4:12 PM Chuck Yang wrote: > Hi Beam devs, > > I saw there has been some previous discussion [1][2] around removing > the deprecated oauth2client dependency and using the supported > google-auth dependency instead. A portion of

Re: [PROPOSAL] Preparing for Beam 2.30.0 release

Please note that I created https://issues.apache.org/jira/browse/BEAM-12338 as a potential blocker. Fix was merged to master and sent a cherry-pick: https://github.com/apache/beam/pull/14821 Thanks, Cham On Fri, May 14, 2021 at 7:59 AM Evan Galpin wrote: > Not sure if it's on the radar but

Re: [EXT] Re: [EXT] Re: [EXT] Re: [EXT] Re: [EXT] Re: Beam Dataframe - sort and grouping

Thank you for the clarification! Is there a way to control the number of shards, i.e. the bundle? I know in pure Beam IO connectors, we have num_shards supported, for example, WriteToParquet

Flaky test issue report

This is your daily summary of Beam's current flaky tests. These are P1 issues because they have a major negative impact on the community and make it hard to determine the quality of the software. BEAM-12322: FnApiRunnerTestWithGrpcAndMultiWorkers flaky (py precommit)

P1 issues report

This is your daily summary of Beam's current P1 issues, not including flaky tests. See https://beam.apache.org/contribute/jira-priorities/#p1-critical for the meaning and expectations around P1 issues. BEAM-12338: Some BQ DisplayData are missing for the Python FnAPI path

Beam Learning Month - Looking for speakers

Hi Apache Beam Community! :beam's-mascot-1: We are organizing a Beam learning month, that will happen during the Wednesdays of June, and we are looking for speakers who want to share their use cases on Apache Beam. Remember that to share your experience will help others to learn and solve

Re: [DISCUSS] Client SDK/Job Server/Worker Pool Lifecycle Management on Kubernetes

1. and 2. are both facilitated by GRPC, which takes care of most of the retry/wait logic. In some places we have a configurable timeout (which defaults to 60s) [1], while in other places we block [2][3]. [1] https://issues.apache.org/jira/browse/BEAM-7933 [2]

Re: BEAM-11838: An x-lang wrapper for DebeziumIO

Hi Pablo! Did you have a chance to execute the test to find something rare? Do you know if there is a way to use breakpoints when running this kind of test I mean breakpoints in java code when running a pipeline from python? On Fri, May 7, 2021 at 4:54 PM Benjamin Gonzalez Delgado <

[DISCUSS] Client SDK/Job Server/Worker Pool Lifecycle Management on Kubernetes

Hello All, I came across this question when I am reading Beam on Flink on Kubernetes and flink-on-k8s-operator

Re: Extremely Slow DirectRunner

Any further thoughts here? Or tips on profiling Beam DirectRunner? Thanks, Evan On Wed, May 12, 2021 at 6:22 PM Evan Galpin wrote: > Ok gotcha. In my tests, all sdk versions 2.25.0 and higher exhibit slow > behaviour regardless of use_deprecated_reads. Not sure if that points to > something

Re: [PROPOSAL] Preparing for Beam 2.30.0 release

Not sure if it's on the radar but there was mention on this Reshuffle PR[1] of cherry-picking it to 2.30.0 [1] https://github.com/apache/beam/pull/14720 Thanks, Evan On Thu, May 13, 2021 at 6:50 PM Heejong Lee wrote: > UPDATE: > > All precommit and postcommit tests are passed now: >