lostluck opened a new issue, #23893: URL: https://github.com/apache/beam/issues/23893
### What would you like to happen? Support [Resource Hints](https://beam.apache.org/documentation/runtime/resource-hints/). Specifically being able to apply min_ram and accelerator hints at minimum at the whole pipeline level. This would enable a certain amount of [right fitting support on Google Cloud Dataflow](https://cloud.google.com/dataflow/docs/guides/right-fitting#go). While the intent is currently to support them only at the pipeline level, it makes sense for the initial infrastructure to be plumbed in at the scope level instead, to avoid a whole re-write for transform level support. Resource Hints are part of the [Environment Proto](https://github.com/apache/beam/blob/2c8e7eb7a39cbe3a1678a5c6b8b3f8700d4d8706/model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/beam_runner_api.proto#L1556) and have [standard URNs](https://github.com/apache/beam/blob/2c8e7eb7a39cbe3a1678a5c6b8b3f8700d4d8706/model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/beam_runner_api.proto#L1962) and behavior (described on the [site](https://beam.apache.org/documentation/runtime/resource-hints/).) The Standard urns don't indicate the format of their payloads however, so part of this work will be to see what Java and Python are doing, and documenting that in the protos. Specify your byte formats kids. For reference, I wrote a doc on this earlier this year, where I had a few open questions: https://docs.google.com/document/d/1fHIIdY1pXrSc0GmQqdGlw3Y7UOuaVbusprTEtCUGS0I/edit# Notably, it looks like space has stabilized for Beam at the moment. ------ Current implementation plan is as follows: V0: Simple flag, and populate the "top level" graph.Scope with this information at pipeline Build time. Translate these hints to the sole Go environment in the correct format. Current plan is "simple" where there's a hint interface. ``` type Hint interface{ URN() string Payload() []byte Merge(Hint) Hint } ``` With some underlying concrete implementations, notionally the "defaultHint" type which has the "most specific" merge behavior currently for the accelerators, and "maxHint" for currently the min_ram, where the maximum in a given scope is preserved. (See hint documentation for details). Those will likely have their own constructor helper methods for customization, and with specific functions for AcceleratorHint and MinRamHint in the same package. This would enable new standard hints to be added, or for supporting new non-standard hints if needed. The only mechanism at this stage would be to set the hint with a flag, similar to the existing python flag. Converting the hints into the map entry for the environment would simply be to call the URN and Payload methods. The actual specifics for dealing with merges are moot until v1 however. This approach implies that the resolution to how to deal with other "metadata" is to have additional packages for Annotations (for PTransforms) and for Display Data (for Pipeline, PTransforms, Environments, PCollections). At this stage, it would be best if the methods to apply those weren't added to the beam.Scope, but remain separate if possible at the moment. The Scope ends up as proxy for the environment (new usecase) and composite PTransforms (existing usecase), and we shouldn't simply be exposing analogs for the underlying protocol buffers. V1: Needs to make choices on Go environment, since presently a single Go environment is assumed. This additional complexity is why it's useful to split the "whole pipeline" effort from "Transforms" level efforts. In particular, hints at different scopes requires supporting multiple environments in a single pipeline, which the Go SDK hasn't needed to do. This requires investigating the Python and Java behavior. Current expectation is that "it's the runner's job to figure them out" and consolidate compatible environments. Shouldn't be too difficult, but it would best be verified with a thorough suite of proto inspecting code to ensure the new environments are broken up and populated correctly (eg. for any docker image swaps or similar). It should also be said that this approach continues to hide the Environment from users of the Go SDK, as a beam implementation detail, allowing the framework to construct the protos as appropriate. ----- There's also a potential for a "modular" approach for dealing with hints, but we don't have sufficient complexity nor variety in hints to warrant a level of indirection that some registry system would enable. As such a single "beam/resources" package would be sufficient. ### Issue Priority Priority: 2 ### Issue Component Component: sdk-go -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
