damondouglas commented on issue #31912: URL: https://github.com/apache/beam/issues/31912#issuecomment-2294432467
# Q&A ## Is all that is needed in the DoFn to add BundleFinalization to the ProcessElement? Answer: All that's necessary for the Go SDK is to declare a BundleFinalizer parameter, in the appropriate order, and to actually register the closure for the BundleFinalizer. In hindsight we'd want this to be not on ProcessElement, because for large bundles, it's going to keep a lot of things in scope, but probably fine in practice. (eg keeping a slice of IDs to ACK is smaller than a pile of closured IDs) Oh well. Nothing for it now. ## Is this statement true? "Bundle finalization is not limited to SDFs but is called out here since this is the primary use case" Answer: Yes. It isn't limited to Splittable DoFns, but they do apply most commonly to such sources. Basically any worthwhile source would be an SDF anyway. ## I see this phrase "durably persisted". Does that refer to after "PersistBundle"? Answer: In this case, yes. In Prism, the call to the ElementManager's PersistBundle method is where "durably persisted" is supposed to, and will eventually occur. As a technical note, we don't do anything "durably" in Prism, due to doing everything in memory. If the program is shut down, it's all lost. But as we're doing this for testing purposes that's fine. ## The phrase "The callback is invoked" in https://beam.apache.org/documentation/programming-guide/#bundle-finalization. When Runner sends the wk.sendInstruction(ctx, &fnpb.InstructionRequest{}) with the bundle finalization request, does that trigger the SDK to execute the callback? Answer: Yes. When the SDK receives the control instruction request to finalize a specific bundle the SDK should be executing all callbacks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
