The intention is that these kinds of hints such as CPU and/or memory should be embedded in the environment specification that is associated with the transforms that need resource hints.
The environment spec is woefully ill prepared as it only has a docker URL right now. On Thu, Mar 28, 2019 at 8:45 AM Robert Burke <rob...@frantil.com> wrote: > A question came over the beam-go slack that I wasn't able to answer, in > particular for Dataflow*, is there a way to increase how much of a Portable > FnAPI worker is dedicated for the SDK side, vs the Runner side? > > My assumption is that runners should manage it, and have the Runner > Harness side be as lightweight as possible, to operate under reasonable > memory bounds, allowing the user-code more room to spread, since it's > largely unknown. > > I saw there's the Provisioning API > <https://github.com/apache/beam/blob/master/model/fn-execution/src/main/proto/beam_provision_api.proto#L52> > which to communicates resource limits to the SDK side, but is there a way > to make the request (probably on job start up) in the other direction? > > I imagine it has to do with the container boot code, but I have only vague > knowledge of how that works at present. > > If there's a portable way for it, that's ideal, but I suspect this will be > require a Dataflow specific answer. > > Thanks! > Robert B > > *Dataflow doesn't support the Go SDK, but the Go SDK supports Dataflow. >
