Thomas Weise created BEAM-5520: ---------------------------------- Summary: Flink runner per operator SDK harness option Key: BEAM-5520 URL: https://issues.apache.org/jira/browse/BEAM-5520 Project: Beam Issue Type: Improvement Components: runner-flink Reporter: Thomas Weise Assignee: Thomas Weise
For streaming pipelines, the runner currently uses a single SDK harness process for all subtasks of a job that get deployed on the same task manager. In common deployments with 16 or more tasks slots, many executable stage operators all use the same SDK harness process. To scale, we need an option to run separate harness processes per subtask. -- This message was sent by Atlassian JIRA (v7.6.3#76005)