Re: Patch release proposal

+1 to a targeted patch release. We did the same for the Go SDK a little while back. It would be good to see what's different for a different SDK. On Wed, Mar 27, 2024, 4:01 PM Robert Bradshaw via dev wrote: > Given the severity of the breakage, and the simplicity of the workaround, > I'm in

Re: container dev environment: go get issue

ook "just > delete the problematic line" > > Thanks for the quick response. I am unblocked now :) > > On Fri, Mar 22, 2024 at 8:47 AM Robert Burke wrote: > >> It's not clear to me why that's even requesting that package at all. I >> would remove that 'go

Re: container dev environment: go get issue

It's not clear to me why that's even requesting that package at all. I would remove that 'go get' line. There's a different issue at play here too since it was written for pre-module Go in mind. I'm unfamiliar with that script though. I'll take a proper look in a few hours. On Fri, Mar 22,

Re: [DISCUSS] Processing time timers in "batch" (faster-than-wall-time [re]processing)

primitive where such a Cross Worker Communication Pair (the processor/server + DoFn client) could be built, but not purely be limited to Rate limiting/Throttling. Possibly mumble mumble StatePipe? But that feels like a harder problem for the time being. Robert Burke On 2024/02/28 08:25:35 Jan

Re: [DISCUSS] Processing time timers in "batch" (faster-than-wall-time [re]processing)

ng the timers into two distinct features? >> >> Jan >> On 2/26/24 21:22, Kenneth Knowles wrote: >> >> Yea I like DelayTimer, or SleepTimer, or WaitTimer or some such. >> >> OutputTime is always an event time timestamp so it isn't even allowed to >> be set

Re: [DISCUSS] Processing time timers in "batch" (faster-than-wall-time [re]processing)

hat long > > The main reason we created timers: to take action in the absence of data. > The archetypal use case for processing time timers was/is "flush data from > state if it has been sitting there too long". For this use case, the right > behavior for batch is to skip the

Re: [DISCUSS] Processing time timers in "batch" (faster-than-wall-time [re]processing)

While I'm currently on the other side of the fence, I would not be against changing/requiring the semantics of ProcessingTime constructs to be "must wait and execute" as such a solution, and enables the Proposed "batch" process continuation throttling mechanism to work as hypothesized for both

Re: [DISCUSS] Processing time timers in "batch" (faster-than-wall-time [re]processing)

't Block Execution" and "We should figure out the best central primitive to manage this class of concept". Robert Burke Beam Go Busybody [0] https://github.com/apache/beam/blob/11f9bce485c4f6fe466ff4bf5073d2414e43678c/sdks/go/pkg/beam/runners/prism/internal/engine/elementmanager.go

Re: Throttle PTransform

question, I do not >> >> > remember what arguments were used against enforcing sorting inputs by >> >> > timestamp in the batch stateful DoFn as a requirement in the model. >> That >> >> > would enable the appropriate firing of timers while preserving the >>

Re: Throttle PTransform

://s.apache.org/beam-design-docs if not already. Robert Burke Beam Go Busybody On 2024/02/20 14:00:00 Damon Douglas wrote: > Hello Everyone, > > The following describes a Throttle PTransform that holds element throughput > to minimize downstream API overusage. Thank you for read

Re: [API PROPOSAL] PTransform.getURN, toProto, etc, for Java

t specific languages affordances. It's definitely a big plus to be able to see all the bits for a single transform in one file, instead of trying to find the 5-8 different places once must add a registration for it. More so in Java where such handler registrations can be done via class annotations! Ro

Re: [RESULT] [VOTE] Vendored Dependencies Release beam-vendor-grpc-1-60-1:0.2

chamikara@ Chamikara Madhusanka Jayalath > > * kenn@ Kenneth Knowles > > * lostluck@ Robert Burke > > * tvalentyn@ Valentyn Tymofieiev > > * yhu@ Yi Hu (non-binding) > > There are no disapproving votes. > > Thanks everyone! >

[ANNOUNCE] Beam 2.54.0 Released

/ and the Github release page https://github.com/apache/beam/releases/tag/v2.54.0 Thanks to everyone who contributed to this release, and we hope you enjoy using Beam 2.54.0. -- Robert Burke, on behalf of the Apache Beam Team.

Re: [RESULT] [VOTE] Release 2.54.0, release candidate #2

The release is now complete https://beam.apache.org/blog/beam-2.54.0/ Please share and promote. I'll be working on the last odds and ends, but the release is now out. Robert Burke Beam 2.54.0 Release Manager On 2024/02/14 17:36:31 Robert Burke wrote: > I'm happy to announce that we h

[RESULT] [VOTE] Release 2.54.0, release candidate #2

I'm happy to announce that we have unanimously approved this release. There are 9 approving votes, 5 of which are binding: * Jan Lukavský * Chamikara Jayalath * Valentyn Tymofieiev * Robert Bradshaw * Robert Burke There are no disapproving votes. Thanks everyone! Robert Burke Beam 2.54.0

Re: [VOTE] Release 2.54.0, release candidate #2

And with that, we have sufficient conditions to declare that RC2 has met community approval. The vote is now closed. Robert Burke Beam 2.54.0 Release Manager On 2024/02/14 17:20:46 Robert Bradshaw via dev wrote: > +1 (binding) > > We've done the validation we can for now, let's no

Re: [VOTE] Vendored Dependencies Release

+1 (binding) On Wed, Feb 14, 2024, 7:35 AM Yi Hu via dev wrote: > +1 (non-binding) > > checked artifact packages not leaking namespace (or under > org.apache.beam.vendor.grpc.v1p60p1) and the tests in > https://github.com/apache/beam/pull/30212 > > > > > On Tue, Feb 13, 2024 at 4:29 AM Sam

Re: [VOTE] Release 2.54.0, release candidate #2

the validation sheets and updated my own Beam Go code. Other than a non-blocking update to the Go wordcount quickstart, I didn't run into any issues. Robert Burke Beam 2.54.0 Release Manager On 2024/02/10 01:41:12 Robert Bradshaw via dev wrote: > I validated that the release artifacts are all corr

Re: Playground: File Explorer?

Tran wrote: > Ah that makes sense. Does the new version of Playground get staged for > release validation? > > On Thu, Feb 8, 2024 at 12:08 PM Robert Burke wrote: > >> We redeploy the playground along with the release, so once 2.54.0 RC2 has >> been validated and vo

Re: [DESIGN PROPOSAL] Reshuffle Allowing Duplicates

tion added accordingly if they need that property per previous discussions. On Thu, Feb 8, 2024, 10:31 AM Kenneth Knowles wrote: > > > On Wed, Feb 7, 2024 at 5:15 PM Robert Burke wrote: > >> OK, so my stance is a configurable Reshuffle might be interesting, so my >> vot

Re: Playground: File Explorer?

We redeploy the playground along with the release, so once 2.54.0 RC2 has been validated and voted on, I'll be redeploying it with 2.54.0. On Thu, Feb 8, 2024, 7:18 AM Joey Tran wrote: > Here's two: > > https://play.beam.apache.org/?path=SDK_PYTHON_MultipleOutputPardo=python >

Re: [DESIGN PROPOSAL] Reshuffle Allowing Duplicates

ements as the input PCollection". It remains an open question about what that means for checkpointing/durability behavior, but that's largely been runner dependent anyway. I admit the above definition is biased by the uses of Reshuffle I'm aware of, which largely are to incur a fusion break i

[VOTE] Release 2.54.0, release candidate #2

nes on how to try the release in your projects, check out our RC testing guide [13]. Thanks, Robert Burke Beam 2.54.0 Release Manager [1] https://github.com/apache/beam/milestone/18?closed=1 [2] https://dist.apache.org/repos/dist/dev/beam/2.54.0/ [3] https://dist.apache.org/repos/dist/release/be

[VOTE] Release 2.54.0, release candidate #2

nes on how to try the release in your projects, check out our RC testing guide [13]. Thanks, Robert Burke Beam 2.54.0 Release Manager [1] https://github.com/apache/beam/milestone/18?closed=1 [2] https://dist.apache.org/repos/dist/dev/beam/2.54.0/ [3] https://dist.apache.org/repos/dist/release/be

Re: [VOTE] Release 2.54.0, release candidate #1

a working website when we publish the release blog, and likely digging into the current issue with the GoUsingJava suite issues. Absolute last call for cherry picks for RC2. I will be doing my final cleanup and then start the RC build in an hour or two. Robert Burke Beam 2.54.0 Release Manager

Re: [VOTE] Release 2.54.0, release candidate #1

Fri, Feb 2, 2024 at 6:01 PM XQ Hu via dev wrote: > > > +1 validated by running the simple RunInference ML pipeline: > > https://github.com/google/dataflow-ml-starter/actions/runs/7761835540/job/21171080332 > > > > On Fri, Feb 2, 2024 at 4:10 PM Robert Burke wrote: > >

[VOTE] Release 2.54.0, release candidate #1

t 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes. For guidelines on how to try the release in your projects, check out our RC testing guide [13]. Thanks, Robert Burke Beam 2.54.0 Release Manager [1] https://github.com/apache/beam/milestone/18?closed=1

Re: [DESIGN PROPOSAL] Reshuffle Allowing Duplicates

e I looked) that the Java reshuffle promises GBK-like side effects. But that's a long deprecated transform without a satisfying replacement for it's usage, so it may be moot. Robert Burke On Tue, Jan 30, 2024, 1:34 PM Kenneth Knowles wrote: > Hi all, > > Just when you thought I had squeez

Re: [Release 2.54.0] Release Branch has been Cut!

pass until the Google internal containers have been generated (the ones that do pass at this stage tend to be validating the container, and build their own.) Thank you for your patience and cooperation. Robert Burke Beam 2.54.0 Release Manager On 2024/01/24 22:55:32 Robert Burke wrote: > He

[Release 2.54.0] Release Branch has been Cut!

to be resolved before we can cut an RC1. If so, I'll be adding them to this thread. Thank you very much for your cooperation and support. Robert Burke Your friendly neighbourhood Beam 2.54.0 release manager [0] https://github.com/apache/beam/tree/release-2.54.0 [1] https://github.com/apache/beam

Re: Google Artifact Registry detects critical vuln CVE-2023-45853 in beam dataflow

Thanks for the shout out XQ! And thanks for bringing this up. Moving to a Distroless base for Go SDK images should reduce the vulnerability surface to whichever version of glibc we have packaged in . I do have some concerns around if a user would like to extend the image (not having shells or

Re: @RequiresTimeSortedInput adoption by runners

t; pre-portability runners to still work, which is the same prereqs as > pre-portable "Override" implementations to still work. > > TBH I'm 50/50 on the idea. If something is going to be implemented more > slowly or less scalably as a fallback, I think it may be best to sim

Re: @RequiresTimeSortedInput adoption by runners

lback" protocol described above, so this thread is still useful for my Prism work, especially if I see any similar situations once I start on the Java Validates Runner suite. Robert Burke Beam Go Busybody On Fri, Jan 19, 2024, 6:41 AM Jan Lukavský wrote: > I was primarily focuse

Re: @RequiresTimeSortedInput adoption by runners

adoption beyond "This Exists, and we can tell you about it if you ask.". AFAICT this is just one of those features we built, but then proceeded not to use within Beam, and evangelize. This is a point we could certainly do better on in Beam as a whole. Robert Burke Beam Go Busy

Re: Beam 2.54.0 Release

Not sure why newlines were eaten. Hopefully reflowed inline below. On 2024/01/10 17:53:56 Robert Burke wrote: > Hey everyone, Happy New Year! > > The next release (2.54.0) branch cut is scheduled for Jan 24, 2024, 2 weeks > from today, according to the release calendar [1]. I'd lik

Re: ByteBuddy DoFnInvokers Write Up

That's neat! Thanks for writing that up! On Wed, Jan 10, 2024, 11:12 AM John Casey via dev wrote: > The team at Google recently held an internal hackathon, and my hack > involved modifying how our ByteBuddy DoFnInvokers work. My hack didn't end > up going anywhere, but I learned a lot about how

Beam 2.54.0 Release

currently 8 release blockers. Let me know if you have any comments/objections/questions. Thanks, Robert Burke [1] https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com [2] https://github.com/apache/beam/milestone/18 [3] https://beam.apache.org/contribute/release-blocking/

Re: [RESULT] [VOTE] Release 2.53.0, release candidate #2

Done! On Fri, Jan 5, 2024, 11:30 AM Robert Burke wrote: > Going to try to get this done. Will report back when completed (or I get > pulled elsewhere). > > On Thu, Jan 4, 2024, 11:23 AM Jack McCluskey via dev > wrote: > >> Hey everyone, >> >> Following u

Re: [RESULT] [VOTE] Release 2.53.0, release candidate #2

approved this release. >> >> There are nine approving votes, three of which are binding: >> * Jan Lukavský (binding) >> * Chamikara Jayalath (binding) >> * Robert Burke (binding) >> * XQ Hu >> * Danny McCormick >> * Bruno Volpato >> * Svetak

Re: [VOTE] Release 2.53.0, release candidate #2

+1 (binding) Validated the Go SDK against my own pipleines. Robert Burke On Wed, Jan 3, 2024, 7:52 AM Chamikara Jayalath via dev wrote: > +1 (binding) > > Validated Java/Python x-lang jobs. > > - Cham > > On Tue, Jan 2, 2024 at 7:35 AM Jack McCluskey via dev > wr

Re: How do side inputs relate to stage fusion?

e fusion I was expecting. I'm I'm looking into how to > make these hints mergeable now. > > On Thu, Dec 14, 2023 at 7:46 PM Robert Burke wrote: > >> Building on what Robert Bradshaw has said, basically, if these fusion >> breaks don't exist, the pipeline can live loc

Re: How do side inputs relate to stage fusion?

Building on what Robert Bradshaw has said, basically, if these fusion breaks don't exist, the pipeline can live lock, because the side input is unable to finish computing for a given input element's window. I have recently added fusion to the Go Prism runner based on the python side input

Re: Beam 2.53.0 Release

Thanks Jack! On Wed, Nov 29, 2023, 10:01 AM Jack McCluskey via dev wrote: > Hey everyone, the next release (2.53.0) branch cut is scheduled for Dec 13, > 2023, 2 weeks from today, according to the release calendar [1]. I'd like > to perform this release; I will cut the branch on that date, and

Re: [YAML] Aggregations

I came across Edge DB, and it has a novel syntax moving away from SQL with their EdgeQL. https://www.edgedb.com/ Eg. Heere are two equivalent "nested" queries. # EdgeQL select Movie { title, actors: { name }, rating := math::mean(.reviews.score) } filter "Zendaya" in .actors.name;

Re: Streaming update compatibility

On Fri, Oct 27, 2023, 9:09 AM Robert Bradshaw via dev wrote: > On Fri, Oct 27, 2023 at 7:50 AM Kellen Dye via dev > wrote: > > > > > Auto is hard, because it would involve > > > querying the runner before pipeline construction, and we may not even > > > know what the runner is at this point > >

Re: Streaming update compatibility

You raise a very good point: https://github.com/apache/beam/blob/master/model/job-management/src/main/proto/org/apache/beam/model/job_management/v1/beam_job_api.proto#L54 The job management API does allow for the pipeline proto to be returned. So one could get the live job, so the SDK could

Re: Streaming update compatibility

Regarding 3. I suspect Go wasn't changed because the PR is centering around implementations of the Expansion Service server, not client callers. The Go SDK doesn't yet have an expansion service. On Thu, Oct 26, 2023, 3:59 AM Johanna Öjeling via dev wrote: > Hi, > > I like this idea of making it

Re: [YAML] Aggregations

MongoDB has its own concept of aggregation pipelines as well. https://www.mongodb.com/docs/manual/core/aggregation-pipeline/#std-label-aggregation-pipeline On Wed, Oct 18, 2023, 6:07 PM Robert Bradshaw via dev wrote: > On Wed, Oct 18, 2023 at 5:06 PM Byron Ellis wrote: > > > > Is it worth

Re: [Question] Bundle finalization callback

continuous stream of messages that > keeps the ProcessElement active. Then I will want to interrupt processing > of new messages and self-checkpoint before those 30s have passed, if the > runner hasn't initiated it within that time frame. > > Johanna > > On Sun, Oct 15, 2023, 21:

Re: [Question] Bundle finalization callback

Hi! Nswers inline. On Sun, Oct 15, 2023, 11:48 AM Johanna Öjeling via dev wrote: > Hi, > > I'm working on a native streaming IO connector for the Go SDK to enable > reads and writes from/to NATS (#29000 > ) and would like to better > understand how

Re: [PROPOSAL] [Nice-to-have] CI job names and commands that match

nnerV2Streaming". > > Last thing I'll add - this is true for you and probably many contributors, > but is less friendly for new folks who are less familiar with the project > IMO (especially when the filepath/command is less obvious). > > On Tue, Oct 10, 2023 at 12:29 PM Robert B

Re: [PROPOSAL] [Nice-to-have] CI job names and commands that match

+1 to the general proposal. I'm not bothered if something says a gradle command and in execution, that task ends up running multiple different commands. Arguably, if we're running a gradle task manualy to prepare for a subsequent task that latter task should be adding the former to it's

Re: [YAML] Fileio sink parameterization (streaming, sharding, and naming)

ou mention.) But it's just as likely I missed a document somewhere. It has been a while since I last searched for this, let alone have time to do the deep dives required to produce it. Robert Burke Beam Go Busybody On Mon, Oct 9, 2023, 12:37 PM Robert Bradshaw via dev wrote: > Currently the various

Re: Reshuffle PTransform Design Doc

actually be leaving unsafe code in user's >>>> hands) and/or separate @RequiresStableInput/checkpointing uses of Reshuffle >>>> from redistribution-only uses of Reshuffle. >>>> >>>> Any other thoughts on this one high level bit? >>>> >&g

Re: Reshuffle PTransform Design Doc

the SDKs dictated reference implementation for the composite(s), and any "fast" or "prod" variant would simply do the current implementation. Robert Burke Beam Go Busybody [0]: https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/graphx/translate.go#L46C3-L

Re: Runner Bundling Strategies

Oh neat, Preserving Keys. I didn't think we had/provided a mechanism for declaring that. Good doc. I do know there's no annotation for FinishBundle. It's generally optional and SDK side only as a concern. Finalize Bundle is a different mechanism which does have an annotation, and requires both

Re: Runner Bundling Strategies

ends to do hundreds of single elements bundles per worker to reduce processing latency. I can't speak to the Flink and Spark strategies. Robert Burke Beam Go Busybody On Thu, Sep 21, 2023, 4:24 PM Joey Tran wrote: > Writing a runner and the first strategy for determining bundling size was &g

Re: User-facing website vs. contributor-facing website

TBH i find the wiki to be entirely unfriendly. It hard to find things in it and isn't discoverable. The syntax is archaic and the UI is wonky. There's no "flow" to it. No common entry point etc. I'd rather the release guide remain in Github as markdown, even if it's not on the website anymore.

Re: [PROPOSAL] Preparing for 2.51.0 Release

Thanks Kenn! On Wed, Sep 13, 2023, 6:20 PM Kenneth Knowles wrote: > Hello Beam community! > > The next release (2.51.0) branch cut is scheduled for September 20, 2023, > one week from today, according to the release calendar [1]. > > I'd like to volunteer to perform this release. My plan is to

Re: Contribution of Asgarde: Error Handling for Beam?

I would say until we *require* Asgard on a core transform, it shouldn't be in the main repo. Incorporating something before there's a need for it is premature abstraction. We can't do things because they *might* be useful. Let's see concrete places where they are useful, or we're already having a

Re: [ANNOUNCE] Beam 2.50.0 Released

Of course I miss replacing one of them. Thanks! On Wed, Aug 30, 2023 at 11:43 AM Robert Burke wrote: > The Apache Beam Team is pleased to announce the release of version 2.50.0. > > You can download the release here: > > https://beam.apache.org/get-started/downloads/ > >

[ANNOUNCE] Beam 2.50.0 Released

/ and the Github release page https://github.com/apache/beam/releases/tag/v2.50.0 Thanks to everyone who contributed to this release, and we hope you enjoy using Beam 2.50.0. -- Robert Burke, on behalf of the Apache Beam Team.

[RESULT] [VOTE] Release 2.50.0, release candidate #2

I'm happy to announce that we have unanimously approved this release. There are 7 approving votes, 3 of which are binding: * Ahmet Altay * Chamikara Jayalath * Robert Bradshaw There are no disapproving votes. Thanks everyone!

Re: [VOTE] Release 2.50.0, release candidate #2

""" On Tue, Aug 29, 2023, 12:27 PM Robert Bradshaw wrote: +1 (binding) Verified the artifacts and signatures, they all look good. Tried a simple Python pipeline in a fresh install that worked fine. Thanks for putting this together. """ On Tue, Aug 29, 202

Re: [VOTE] Release 2.50.0, release candidate #2

resh install that worked fine. Thanks for putting > this together. > > On Tue, Aug 29, 2023 at 10:51 AM Robert Burke wrote: > >> I was encouraged to ping y'all for Beam release validation and binding >> votes. So far it's looking good, but only 2 PMC votes. >> &g

Re: [VOTE] Release 2.50.0, release candidate #2

Current status is six +1 votes, two binding. Thanks to everyone so far who has participated in validating the release! Voting for RC2 will remain open until there's further PMC engagement for binding votes, or we find cause for a RC3. Thanks Robert Burke Apache Beam 2.50.0 Release Manager

Re: [VOTE] Release 2.50.0, release candidate #2

, Aug 25, 2023 at 3:05 PM Bruno Volpato via dev < >> dev@beam.apache.org> wrote: >> >>> +1 (non-binding). >>> >>> Tested with https://github.com/GoogleCloudPlatform/DataflowTemplates >>> (Java SDK 11, Dataflow runner). >>> >>> Thanks Robert!

Re: [ANNOUNCE] New committer: Ahmed Abualsaud

Congratulations Ahmed!! On Thu, Aug 24, 2023, 4:08 PM Chamikara Jayalath via dev < dev@beam.apache.org> wrote: > Congrats Ahmed!! > > On Thu, Aug 24, 2023 at 4:06 PM Bruno Volpato via dev > wrote: > >> Congratulations, Ahmed! >> >> Very well deserved! >> >> >> On Thu, Aug 24, 2023 at 6:09 PM XQ

Re: [VOTE] Release 2.50.0, release candidate #2

Two minor erata from the previous email: The validation spreadsheet link should be: https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1014811464 And the source code tag is: "v2.50.0-RC2" On 2023/08/24 23:09:23 Robert Burke wrote: >

[VOTE] Release 2.50.0, release candidate #2

e.org/blog/validate-beam-release/. Thanks, Robert Burke Apache Beam 2.50.0 Release Manager [1] https://github.com/apache/beam/milestone/14 [2] https://dist.apache.org/repos/dist/dev/beam/2.50.0/ [3] https://dist.apache.org/repos/dist/release/beam/KEYS [4] https://repository.apache.org/content/re

Re: [VOTE] Release 2.50.0, release candidate #1

continue validation with those using RC1 in the mean time. Robert Burke Apache Beam 2.50.0 Release Manager On 2023/08/22 01:35:45 Valentyn Tymofieiev via dev wrote: > I tried running a Dataflow Python pipeline on RC1 and got an error: > > Pipeline construction environment and pipelin

Re: [VOTE] Release 2.50.0, release candidate #1

Hi Beamers, Today I'm working on the aforementioned gaps in this RC blocking. However, it's still valuable to validate and vote on the remainder of the RC in order to ensure a timely 2.50.0 release, and finding whether we'll need an RC2 or not. Robert Burke Apache Beam 2.50.0 Release Manager

Re: [PROPOSAL] Preparing for 2.50.0 Release

RC1 is sufficiently ready for testing and validation. Please vote and discuss at that thread. https://lists.apache.org/thread/xgx49zshms7253lfx6d6lsnvwf7tyyfp Robert Burke 2.50.0 Release Manager On 2023/08/17 02:30:00 Robert Burke wrote: > Despite my best efforts, python continues to vex

[VOTE] Release 2.50.0, release candidate #1

t 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes. For guidelines on how to try the release in your projects, check out our blog post at https://beam.apache.org/blog/validate-beam-release/. Thanks, Robert Burke Apache Beam 2.50.0 Release Manager [1] https://

Re: [PROPOSAL] Preparing for 2.50.0 Release

Despite my best efforts, python continues to vex me. RC1 is almost ready, just missing the beam site and doc updates PR, and (optionally) the typescript container. So I'm calling it a night, and will build and send out a partial docs PR in the morning. Robert Burke 2.50.0 Release Manager On Wed

Re: [RFC] Bootloader Buffered Logging

I've added some comments but generally +1 on this. A later change might be able to build from this to ensure the various STDErr and STDOut logs from the SDK harness executions are always plumbed as described. But that would take more thought since other incidental logs from the users worker

Re: [PROPOSAL] Preparing for 2.50.0 Release

build the non-portable Dataflow Java workers after the first RC is tagged and the open side is on its way. The hope is RC1 will be available tonight. Either way, this thread will be updated with the status. Robert Burke Beam 2.50.0 Release Manager On 2023/08/14 21:51:47 Robert Burke wrote

Re: [PROPOSAL] Preparing for 2.50.0 Release

can help with validation? Is there a wiki page with >> the expected validations I can help with? >> >> Best >> Hong >> >> On 14 Aug 2023, at 14:34, Robert Burke wrote: >> >>  >> The release branch was cut. Before yhe weekend, I was working on

Re: [PROPOSAL] Preparing for 2.50.0 Release

release date for the 2.50 release? > > Best, > Hong > > On Thu, Aug 10, 2023 at 3:08 AM Robert Burke wrote: > >> I'm in the process of producing the Cut branch, but due to various delays >> on my part, it will not be cut today. >> >> There are two outstand

Re: [PROPOSAL] Preparing for 2.50.0 Release

. Remaining new issues will be cherry picked as required. Thanks Robert Burke Beam 2.50.0 Release Manager On 2023/07/26 15:49:37 Robert Burke wrote: > Hey Beam community, > > The next release (2.50.0) branch cut is scheduled on August 9th, 2023, > according to > the release calendar [1]. &

Re: [Discuss] Get rid of OWNERS files

Either we keep OWNERS and have the review bot use them, or we remove them and use the reviews bot config as the single source of truth. The bot is less likely to go out of date since it's at least active in how it behaves. I agree it doesn't necessarily solve the problem of things getting out of

[PROPOSAL] Preparing for 2.50.0 Release

at [3]. Let me know if you have any comments/objections/questions. Thanks, Robert Burke (he/him) Beam Go Busybody [1] https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com [2] https://github.com/apache/beam/milestone/14 [3] https://beam.apache.or

Re: [DISCUSS] Enable Github Discussions?

I'm -1 on GitHub discussions. If anything the user can just file a GitHub issue for the same purpose, if they prefer the GitHub interface over emails. In theory, GHDiscussions can be better for very active topics, but honestly I don't think we have that sort of throughput. Having payed

Re: Rust SDK design docs/notes

I threw in some Go specific comments and links. Especially needed to call out that we try to avoid the Magic Type Synthesis Path for types and Casting pointers to functions after looking Them Up In the Debug symbol table. For Go at least, both had robustness or performance issues that may or may

Beam Go now has a v2.48.2 release.

error. The 2.48.0 release blog and notes have been updated to note the discrepancy as well. Thank you for your understanding and time, Robert Burke Beam Go Busybody [0] https://github.com/apache/beam/issues/27064 [1] https://beam.apache.org/contribute/release-guide/#git-tag [2] https://github.com/apache/beam/pull/27070

Re: Client-Side Throttling in Apache Beam

Great article! Though it's depressing to see we have a pair of magic counter names to help modulate scaling behavior. On Tue, May 30, 2023, 11:42 AM Jack McCluskey via dev wrote: > Hey everyone, > > While working on some remote model handler code I hit a point where I > needed to understand

Re: [Proposal] Automate Release Signing

s://www.apache.org/legal/release-policy.html#release-signing >> >> "All supplied packages MUST be cryptographically signed by the Release >> Manager with a detached signature" >> >> Kenn >> >> On Wed, May 3, 2023 at 12:14 PM John Casey via d

Re: [Proposal] Automate Release Signing

+1 to simplifying release processes, since it leads to a more consistent experience. If we continue to reduce release overhead we'll be able to react with more agility when CVEs come a knocking. On Wed, May 3, 2023, 12:08 PM Jack McCluskey via dev wrote: > +1 to automating release signing. As

Re: [ANNOUNCE] New committer: Damon Douglas

Congratulations Damon!!! On Mon, Apr 24, 2023, 12:52 PM Kenneth Knowles wrote: > Hi all, > > Please join me and the rest of the Beam PMC in welcoming a new committer: > Damon Douglas (damondoug...@apache.org) > > Damon has contributed widely: Beam Katas, playground, infrastructure, and > many

Re: [ANNOUNCE] New committer: Anand Inguva

Congratulations Anand! On Fri, Apr 21, 2023, 10:55 AM Danny McCormick via dev wrote: > Woohoo, congrats Anand! This is very well deserved! > > On Fri, Apr 21, 2023 at 1:54 PM Chamikara Jayalath > wrote: > >> Hi all, >> >> Please join me and the rest of the Beam PMC in welcoming a new

Re: [DISCUSS] @Experimental, @Internal, @Stable, etc annotations

g any real breaking changes that would be healthy to make for Beam. But critically, what are the concrete benefits to pipeline authors in such a move. Otherwise it's self serving churn. On Tue, Apr 18, 2023, 9:08 AM Alexey Romanenko wrote: > > On 17 Apr 2023, at 21:14, Robert Burke wrot

Re: [DISCUSS] @Experimental, @Internal, @Stable, etc annotations

+1 on how to iterate without a Beam 3.0 Often that just means, write the new thing, "support both for a while",make it clear how to migrate to the new thing, and the next Major Version just drops everything that doesn't cut the mustard anymore. On Mon, Apr 17, 2023, 11:54 AM Ahmet Altay via dev

Re: Jenkins Flakes

The coverage issue is only with the Java builds in specific. Go abd Python have their coverage numbers codecov uploads done in GitHub Actions instead. On Tue, Apr 11, 2023, 8:14 AM Moritz Mack wrote: > Thanks so much for looking into this! > > I’m absolutely +1 for removing Jenkins related

Re: Jenkins Flakes

+1 SGTM Remember, if an issue is being investigated, a committer can always mark a build to be retained longer in the Jenkins UI. Just be sure to clean it up once it's resolved though. (TBH there may also be some old retained builds like that, but I doubt there's a good way to see which are

Re: [DISCUSS] @Experimental, @Internal, @Stable, etc annotations

I've been thinking a similar thing for the Go SDK for a while, though Go doesn't have annotations in the way lJava does. I agree that explicitly Stable/ default Evolving is probably the best way to go. Even for SDK devs, it's better the default isn't something that needs to be remembered to be

[Proposal] State and Timer Composites (Go SDK)

specific, as we're finishing off being able have timer support there, and would require a small non-breaking change on how state is handled to enable. Please take a look, and let me know what you think (here or on the issue). Robert Burke Beam Go Busybody

Re: direct runner OOM issue

Which direct runner? They are language specific. On Mon, Mar 13, 2023, 11:27 AM wilsonny...@gmail.com wrote: > Hi guys, > > We are trying to run our pipeline using direct runner and the input > dataset is a large amount of HDFS files (few hundred of GB data) > > We experienced OOM issue crash.

Re: [DISCUSS] Provide MultimapUserStateHandler interface in StateRequestHandlers

The runners should be able to support Multimap User State portably over the FnApi already. https://github.com/apache/beam/blob/master/model/fn-execution/src/main/proto/org/apache/beam/model/fn_execution/v1/beam_fn_api.proto#L937 How that's supported on each SDK is a different matter though. On

Re: [Go SDK] Direct Runner Replacement: Prism

though. :) Thanks again, and lets keep making Beam easier to use! Robert Burke Beam Go Busybody On 2023/02/20 23:30:06 Robert Burke wrote: > Johanna managed to take a look at the penultimate PR, so that's now in, and > finally, here's the last one. > > https://github.com/apache/bea

Re: new contributor messaging: behaviorbot/welcome

ontent (and triggers) would > be. > > On Tue, Feb 21, 2023 at 1:36 PM Austin Bennett > wrote: > > > > It is fantastic if generally able to address welcoming newcomers > manually [ @Robert Burke ! ] . Community communication, human connection [ > ex: community > code ] idea

  1   2   3   4   5   >