Re: Getting Started With Implementing a Runner

I took a first pass at https://github.com/apache/beam/blob/be19140f3e9194721f36e57f4a946adc6c43971a/website/www/site/content/en/contribute/runner-guide.md https://github.com/apache/beam/blob/1cfc0fdc6ff27ad70365683fdc8264f42642f6e9/sdks/python/apache_beam/runners/trivial_runner.py may also be of

Re: Getting Started With Implementing a Runner

Could you let me know when you update it? I would be interested in rereading after the rewrite. Thanks! Joey On Fri, Jul 14, 2023 at 4:38 PM Robert Bradshaw wrote: > I'm taking an action item to update that page, as it is *way* out of date. > > On Thu, Jul 13, 2023 at 6:54 PM Joey Tran >

Re: Getting Started With Implementing a Runner

I'm taking an action item to update that page, as it is *way* out of date. On Thu, Jul 13, 2023 at 6:54 PM Joey Tran wrote: > I see. I guess I got a little confused since these are mentioned in the > Authoring > a Runner >

Re: Getting Started With Implementing a Runner

I see. I guess I got a little confused since these are mentioned in the Authoring a Runner docs page which implied to me that they'd be safe to use. I'll check out the bundle_processor. Thanks! On Mon, Jul 10, 2023 at 1:07 

Re: Getting Started With Implementing a Runner

On Sun, Jul 9, 2023 at 9:22 AM Joey Tran wrote: > Working on this on and off now and getting some pretty good traction. > > One thing I'm a little worried about is all the classes that are marked > "internal use only". A lot of these seem either very useful or possibly > critical to writing a

Re: Getting Started With Implementing a Runner

Working on this on and off now and getting some pretty good traction. One thing I'm a little worried about is all the classes that are marked "internal use only". A lot of these seem either very useful or possibly critical to writing a runner. How strictly should I interpret these private

Re: Getting Started With Implementing a Runner

Another advantage of a portable runner would be that it will be using well defined and backwards compatible Beam portable APIs to communicate with SDKs. I think this is specially important for runners that do not live in the Beam repo since otherwise future SDK releases could break your runner in

Re: Getting Started With Implementing a Runner

On Fri, Jun 23, 2023 at 1:43 PM Joey Tran wrote: > Totally doable by one person, especially given the limited feature set you >> mention above. >> https://docs.google.com/presentation/d/1Cso0XP9dmj77OD9Bd53C1M3W1sPJF0ZnA20gzb2BPhE >> is >> a good starting point as to what the relationship

Re: Getting Started With Implementing a Runner

> > Totally doable by one person, especially given the limited feature set you > mention above. > https://docs.google.com/presentation/d/1Cso0XP9dmj77OD9Bd53C1M3W1sPJF0ZnA20gzb2BPhE > is > a good starting point as to what the relationship between a Runner and the > SDK is at a level of detail

Re: Getting Started With Implementing a Runner

On Fri, Jun 23, 2023 at 11:15 AM Joey Tran wrote: > Thanks all for the responses! > > If Beam Runner Authoring Guide is rather high-level for you, then, at >> fist, I’d suggest to answer two questions for yourself: >> - Am I going to implement a portable runner or native one? >> > > Portable

Re: Getting Started With Implementing a Runner

Thanks all for the responses! If Beam Runner Authoring Guide is rather high-level for you, then, at fist, > I’d suggest to answer two questions for yourself: > - Am I going to implement a portable runner or native one? > Portable sounds great, but the answer depends on how much additional cost

Re: Getting Started With Implementing a Runner

> On 23 Jun 2023, at 17:40, Robert Bradshaw via user > wrote: > > On Fri, Jun 23, 2023, 7:37 AM Alexey Romanenko > wrote: >> If Beam Runner Authoring Guide is rather high-level for you, then, at fist, >> I’d suggest to answer two questions for yourself: >> -

Re: Getting Started With Implementing a Runner

On Fri, Jun 23, 2023, 7:37 AM Alexey Romanenko wrote: > If Beam Runner Authoring Guide is rather high-level for you, then, at > fist, I’d suggest to answer two questions for yourself: > - Am I going to implement a portable runner or native one? > The answer to this should be portable, as

Re: Getting Started With Implementing a Runner

If Beam Runner Authoring Guide is rather high-level for you, then, at fist, I’d suggest to answer two questions for yourself: - Am I going to implement a portable runner or native one? - Which SDK I should use for this runner? Then, depending on answers, I’d suggest to take as an example one of

Re: Getting Started With Implementing a Runner

Thanks Jack! I've tried that Slack link but it requires an account with a @apache email On Thu, Jun 22, 2023 at 10:08 AM Jack McCluskey via user < user@beam.apache.org> wrote: > Hey Joey, > > The best resource to look at, at the moment, is likely Robert Burke's > Prism runner that he is

Re: Getting Started With Implementing a Runner

Hey Joey, The best resource to look at, at the moment, is likely Robert Burke's Prism runner that he is implementing ( https://github.com/apache/beam/tree/master/sdks/go/pkg/beam/runners/prism). Runners are pretty complicated and there are a number of primitives that you need to have implemented

Getting Started With Implementing a Runner

Hi Beam community! I'm interested in trying to implement a runner with my company's execution environment but I'm struggling to get started. I've read the docs page on implementing a runner but it's quite high level. Anyone