It's now all in! Thank you for following. Big thanks to Johanna, Jack, and
Ritesh for doing reviews of the PRs.
My current plan is to not think about it for a bit, and resolve the blocker
from adding Timers to the SDK.
After that, the priorities are focused on resolving the Side Input memory l
Johanna managed to take a look at the penultimate PR, so that's now in, and
finally, here's the last one.
https://github.com/apache/beam/pull/25568 - Add in the execution stack
connecting preprocessing and execution. Includes all pipeline execution
unit tests.
On Mon, 20 Feb 2023 at 10:26, Robert
We're in the home stretch! Only 2 PRs to go, and then Prism will be
available for use from the Beam Repo, hopefully with the 2.46 cut. Also, it
would be available for others to improve and meaningfully extend.
https://github.com/apache/beam/pull/25565 - Adds in the primary element
manager. This is
I had to scratch the itch this weekend, and now have Prism able to pay
attention to Estimated Output Watermarks, and send elements downstream if
the transform is not blocked
by watermarks somewhere. Woohoo! Is it fully correct? Probably not, but
it's a start.
Transforms are still executed one at
Next up:
https://github.com/apache/beam/pull/25478 - Large PR for initial handling of
worker FnAPI surfaces.
https://github.com/apache/beam/pull/25518 - Tiny PR for handling basic
windowing strategies.
https://github.com/apache/beam/pull/25520 - Medium PR for adding the graph
preprocessor scaf
Here are the next two chunks!
https://github.com/apache/beam/pull/25476 - Coder / element / bytes
handling internally for prism.
https://github.com/apache/beam/pull/25478 - Worker fnAPI handling.
Took a bit to get a baseline of unit testing in for these, since they were
covered by whole pipeline
Just a +100 to the idea of this runner. Having an easy-to-read,
portable-execution, batch & streaming, parallel, local runner, that
exercises plenty of advanced model features... solid gold!
On Thu, Feb 9, 2023 at 12:01 PM Robert Burke via dev
wrote:
> Here are the first of the smaller PRs:
>
>
Here are the first of the smaller PRs:
https://github.com/apache/beam/pull/25404 -> Adds READMEs and updates
go.mod so later changes don't collide there.
https://github.com/apache/beam/pull/25405 -> Adds internal/urns package for
extracting URNs from the protos.
https://github.com/apache/beam/pull
Thanks for the work on this; a very welcomed feature/contribution!
On Thu, Feb 9, 2023 at 7:36 AM Jack McCluskey via dev
wrote:
> Congratulations on getting the runner to a state you're happy contributing
> to the main repo! I'm happy to help review PRs and get sub-packages in.
> Anything that h
Congratulations on getting the runner to a state you're happy contributing
to the main repo! I'm happy to help review PRs and get sub-packages in.
Anything that helps developers and users test Beam pipelines more
effectively is a welcome inclusion.
Thanks,
Jack McCluskey
P.S. I'm glad the Prism
Hello Beam!
== tl;dr; ==
I wrote a local, portable Beam runner in Go to replace the Go direct
runner. I'd like to contribute it to the Beam Repo. The Big PR with
everything is here: https://github.com/apache/beam/pull/25391
I'll be sending smaller PRs out for review to get it into the repo. Tak
11 matches
Mail list logo
