damccorm opened a new pull request #17045: URL: https://github.com/apache/beam/pull/17045
This is a continuation of the effort to add Bundle Finalization started in #16980 **Summary of Overall Changes** Bundle finalization enables a DoFn to perform side effects after a runner has acknowledged that it has durably persisted the output. Right now, Java and Python support bundle finalization by allowing a user to register a callback function which is invoked when the runner acknowledges that it has persisted all output, but Go does not have any such support. This is part of a larger change to add support to the Go Sdk as outlined in this [design doc](https://docs.google.com/document/d/1dLylt36oFhsWfyBaqPayYXqYHCICNrSZ6jmr51eqZ4k/edit#). **Summary of Changes in this PR** I completed most of the execution changes in the last pr, leaving this PR to handle the user experience and plumbing the user's bundle finalization parameter through to the execution layer. **Additional testing done** On top of the units added, I also was able to run an E2E example on Dataflow (FWIW, [only Dataflow currently has bundle finalization support](https://beam.apache.org/documentation/runners/capability-matrix/bounded-splittable-dofn-support-status/)). In that example, I hijacked the wordcount example and added a bundleFinalizer to write a file to persistent storage for each line that had at least 3 words (chosen pretty randomly to minimize the chances of collisions). I'll omit the whole sample since its long, but it produced a bunch of files like this: <img width="259" alt="image" src="https://user-images.githubusercontent.com/42773683/156254068-adab3a92-7978-4284-9963-8b80c8ecbf59.png";> This indeed ran after the other data was persisted. I decided not to add an integration test because of the complexity involved. Because only the dataflow runner supports finalization, any integration test would require finding some way to (a) modify dataflow state and query that state (probably not a great option since it requires modifying a devs personal GCP account, and its not obvious what the best thing to do actually would be without knowing more about their config), (b) creating some sort of local endpoint for dataflow to talk back to (technically feasible, but definitely non-trivial - would also add significant complexity outside of what we're actually testing), or (c) using some 3rd party to communicate between the 2 (non-ideal since it adds an extra dependency that isn't part of what's being tested just for this scenario). I'm definitely open to doing this, but at the moment with the information I have it doesn't feel worth it. **Next Steps** After this, I'll update the documentation here to include an example https://github.com/apache/beam/blob/6438626c059c19ff9ca32cd834d0aa62253e531b/website/www/site/content/en/documentation/programming-guide.md#127-bundle-finalization-bundle-finalization ------------------------ Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md) GitHub Actions Tests Status (on master branch) ------------------------------------------------------------------------------------------------ [](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule) [](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule) [](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule) See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
![https://user-images.githubusercontent.com/42773683/156254068-adab3a92-7978-4284-9963-8b80c8ecbf59.png"](https://user-images.githubusercontent.com/42773683/156254068-adab3a92-7978-4284-9963-8b80c8ecbf59.png"){kind=link}
