lostluck commented on issue #23893: URL: https://github.com/apache/beam/issues/23893#issuecomment-1786091487
This has been re-prioritized (admittedly, due to a need within Google), but I have an approach I like well enough for parity with both Environment Hints and Annotations on Composites. In particular, it's a mild bit of indirection through a context.Context, which do match against the Hints and annotations well enough. A new package called contextreg, where extractor functions can be added. Ones for ResourceHints on environments, and ones for Annotations for composites. Both kinds are take the "refinement" approach. The closest version is what applies to a leaf transform. eg. Whole Pipeline -> Composite A -> Composite B -> Leaf Transform C So if a given annotation is applied both Composite A and Leaf C, C will have it's own annotation attached, and should take priority for whatever purposes. This ultimately allows annotations to be attached to a Functional DoFn, which wasn't previously possible. Similarly, for Resource Hints, whole Pipeline hints adjust the default environment. But if a composite has hints sets, they form a new derived environment which has both the defaults, and the new overrides set. (At which point, runners ([like dataflow](https://cloud.google.com/dataflow/docs/guides/right-fitting#go)) do what they need to do to union environments together.) ------- Hanging on context avoids adding additional APIs for both Annotations and Resource Hints, but also allows for other bits to be added in as needed to the graph from context.Context in the future. The rest ends up in how runners treat the Annotations and Hints, which is not something the SDK should get in the way of, since runners are already free to ignore hints as they like. Conversely, but out of scope for now, would be to possibly re-hydrate these into context.Context that are available on execution time for access in user transforms. This would let the SDK access these in both Structural and Functional DoFns as the user sees fit. But it must be noted that neither should be used for dictating correct behavior of the DoFn if that ever comes to pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
