Re: [ANNOUNCE] New PMC Member: Alex Van Boxel

Congratulations! On Tue, Oct 3, 2023 at 3:48 PM Byron Ellis via dev wrote: > Congrats! > > On Tue, Oct 3, 2023 at 12:40 PM Danielle Syse via dev > wrote: > >> Congratulations Alex!! Definitely well deserved! >> >> On Tue, Oct 3, 2023 at 2:57 PM Ahmet Altay via dev >> wrote: >> >>>

Re: [ANNOUNCE] New committer: Ahmed Abualsaud

Thanks to the PMC for these responsibilities, and thank you all for guiding me along this journey. I'm looking forward to helping this community however I can :) Best, Ahmed On Sun, Aug 27, 2023 at 8:48 PM Reza Rokni via dev wrote: > Congrats Ahmed! > > On Fri, Aug 25, 2023 at 2:34 PM John

Re: [ANNOUNCE] New committer: John Casey

Congrats John, what a great addition! On Fri, Jul 29, 2022 at 4:56 PM Kerry Donny-Clark via dev < dev@beam.apache.org> wrote: > John, you have made a huge impact on the many, many users of Kafka and > other IOs. This is great recognition of your commitment to Beam. > Kerry > > On Fri, Jul 29,

A lesson about DoFn retries

Hi all, TLDR: When writing IO connectors, be wary of how bundle retries can affect the work flow. A faulty implementation of a step in BigQuery batch loads was discovered recently. I raised an issue [1] but also wanted to mention it here as a potentially helpful lesson for those developing

Re: A lesson about DoFn retries

gt; > On Thu, Sep 1, 2022 at 1:43 PM Ahmed Abualsaud via dev < > dev@beam.apache.org> wrote: > >> Hi all, >> >> TLDR: When writing IO connectors, be wary of how bundle retries can >> affect the work flow. >> >> A faulty implementation of a s

Re: [Proposal] Change to Default PubsubMessage Coder

+1 to looking into using RowCoder, this may help avoid creating more specialized coders in the future (which is mentioned as a pain point in the issue you linked [1]). [1] https://github.com/apache/beam/issues/23525#issuecomment-1281294275 On Tue, Dec 20, 2022 at 3:00 AM Andrew Pilloud via dev

Re: SchemaTransformProvider | Java class naming convention

Thank you for the informative email Damon! I am in favor of setting an intuitive naming convention early on to reduce confusion when Schema Transforms become more widespread. I like the proposed name in your email and I think this convention should also apply to the rest of the classes involved

Re: SchemaTransformProvider | Java class naming convention

> > Schema-aware transforms are not restricted to I/Os. An arbitrary transform > can be a Schema-Transform. Also, designation Read/Write does not map to an > arbitrary transform. Probably we should try to make this more generic ? > Agreed, I suggest keeping everything on the left side of the

Re: One Configuration, Many File Write Formats

Thanks for drafting this Damon! I left some comments on the doc. It's really cool that users can go to one source with a specified file format (json, avro, xml, csv, parquet) and retrieve the relevant file writing PTransform. I also like how the same configuration can be re-used for different file

Re: [ANNOUNCE] New committer: Ritesh Ghorse

Congrats Ritesh! On Fri, Nov 4, 2022 at 10:29 AM Andy Ye via dev wrote: > Congrats Ritesh! > > On Fri, Nov 4, 2022 at 9:26 AM Kerry Donny-Clark via dev < > dev@beam.apache.org> wrote: > >> Congratulations Ritesh, I'm happy to see your hard work and community >> spirit recognized! >> >> On Fri,

Support existing IOs with Schema Transforms

Hi all, There has been an effort to add SchemaTransform capabilities to our connectors to facilitate the use of multi-lang pipelines. I've drafted a document below that provides guidelines and examples of how to support IOs with SchemaTransforms. Please take a look and share your thoughts and

Re: [ANNOUNCE] New committer: Yi Hu

Congrats Yi! On Wed, Nov 9, 2022 at 1:33 PM Sachin Agarwal via dev wrote: > Congratulations Yi! > > On Wed, Nov 9, 2022 at 10:32 AM Kenneth Knowles wrote: > >> Hi all, >> >> Please join me and the rest of the Beam PMC in welcoming a new >> committer: Yi Hu (y...@apache.org) >> >> Yi started

Re: [PROPOSAL] Preparing for 2.47.0 Release

Thanks for stepping up Jack! On Thu, Mar 23, 2023 at 12:43 PM Ahmet Altay via dev wrote: > Thank you Jack! > > On Wed, Mar 22, 2023 at 8:39 AM Jack McCluskey via dev < > dev@beam.apache.org> wrote: > >> Hey all, >> >> The next (2.47.0) release branch cut is scheduled for April 5th, 2023, >>

Re: Refactor BigQuery SchemaTransforms naming

Thank you Damon, I left a few comments. On Fri, Mar 3, 2023 at 11:14 AM Damon Douglas via dev wrote: > Hello Everyone, > > This PR brings BigQuery Schema transforms in line with the others in terms > of naming conventions and use of AutoService. > > https://github.com/apache/beam/pull/25706 > >

Re: [ANNOUNCE] New committer: Anand Inguva

Congrats Anand! On Fri, Apr 21, 2023 at 3:18 PM Anand Inguva via dev wrote: > Thanks everyone. Really excited to be a part of Beam Committers. > > On Fri, Apr 21, 2023 at 3:07 PM XQ Hu via dev wrote: > >> Congratulations, Anand!!! >> >> On Fri, Apr 21, 2023 at 2:31 PM Jack McCluskey via dev <

Re: [ANNOUNCE] New committer: Damon Douglas

Congrats Damon! On Mon, Apr 24, 2023 at 5:05 PM Kerry Donny-Clark via dev < dev@beam.apache.org> wrote: > Damon, you have done outstanding work to grow and improve Beam and the > Beam community. Well done, well deserved! > > On Mon, Apr 24, 2023 at 4:39 PM XQ Hu via dev wrote: > >> Congrats

Re: [VOTE] Release 2.46.0, release candidate #1

@Danny McCormick @Reuven Lax sorry it's been a while since you looked into this, but do you remember if the fix in #25642 issue is related to the recent "ALREADY_EXISTS: The offset is within stream, expected offset..." errors? On Fri, Mar 10, 2023 at

Proposal to reduce the steps to make a Java transform portable

Hey everyone, I was looking at how we use SchemaTransforms in our expansion service. From what I see, there may be a redundant step in developing SchemaTransforms. Currently, we have 3 pieces: - SchemaTransformProvider [1] - A configuration object - SchemaTransform [2] The API is generally used

Re: Proposal to reduce the steps to make a Java transform portable

gt; DRY. >>>>>> >>>>>> On Tue, May 30, 2023 at 10:59 AM Robert Bradshaw via dev < >>>>>> dev@beam.apache.org> wrote: >>>>>> >>>>>>> +1 to this simplification, it's a historical artifact that provides >>&

Re: Proposal to reduce the steps to make a Java transform portable

gt;>>>> >>>>>>> +1 but I think the hard part today is to convert existing >>>>>>> PTransforms to be schema-aware transform compatible (for example, change >>>>>>> input/output types and make sure parameters take Beam Schema

[Design Doc] Auto-generating SchemaTransform wrappers

Hey everyone, I've written a design doc for automatically generating SchemaTransform wrappers. The document is focused on generating Python SDK wrappers, but the framework would be generic enough to be easily applicable to other SDKs. I'd like to share this with you all and gather any feedback.

Re: Bigquery Connector Rate limits

Hey Taher, Regarding the first question about what API Beam uses, that depends on the BigQuery method you set in the connector's configuration. We have 4 different write methods, and a high-level description of each can be found in the documentation:

Re: Proposal to reduce the steps to make a Java transform portable

e. >>>>>>>>> >>>>>>>> >>>>>>>> +1 but I think the hard part today is to convert existing >>>>>>>> PTransforms to be schema-aware transform compatible (for example, >>>>>>>> change >>

Supporting Dynamic Destinations in a portable context

Hey all, There have been some conversations lately about how best to enable dynamic destinations in a portable context. Usually, this comes up for cross-language transforms and more recently for Beam YAML. I've started a short doc outlining some routes we could take. The purpose is to establish

Re: Supporting Dynamic Destinations in a portable context

., dest_info=...} and the elide-fields >> approaches, as the former is nicer when one has a fixed representation for >> the output record (e.g. a proto or avro schema) and the flattened form for >> ease of use in more free-form contexts (e.g. when producing records from >> YAML and