On Thu, Mar 28, 2019 at 9:30 AM Lukasz Cwik <lc...@google.com> wrote:
> The intention is that these kinds of hints such as CPU and/or memory > should be embedded in the environment specification that is associated with > the transforms that need resource hints. > > The environment spec is woefully ill prepared as it only has a docker URL > right now. > FWIW I think this is actually "extremely well prepared" :-) Protobuf is great for adding fields when you need more but removing is nearly impossible once deployed, so it is best to do the absolute minimum until you need to expand. Kenn > > On Thu, Mar 28, 2019 at 8:45 AM Robert Burke <rob...@frantil.com> wrote: > >> A question came over the beam-go slack that I wasn't able to answer, in >> particular for Dataflow*, is there a way to increase how much of a Portable >> FnAPI worker is dedicated for the SDK side, vs the Runner side? >> >> My assumption is that runners should manage it, and have the Runner >> Harness side be as lightweight as possible, to operate under reasonable >> memory bounds, allowing the user-code more room to spread, since it's >> largely unknown. >> >> I saw there's the Provisioning API >> <https://github.com/apache/beam/blob/master/model/fn-execution/src/main/proto/beam_provision_api.proto#L52> >> which to communicates resource limits to the SDK side, but is there a way >> to make the request (probably on job start up) in the other direction? >> >> I imagine it has to do with the container boot code, but I have only >> vague knowledge of how that works at present. >> >> If there's a portable way for it, that's ideal, but I suspect this will >> be require a Dataflow specific answer. >> >> Thanks! >> Robert B >> >> *Dataflow doesn't support the Go SDK, but the Go SDK supports Dataflow. >> >
