Flink has a set of workers, each worker has a number of task slots. A pipeline will use the number of slots based upon what it was configured to run with.
Are you trying to get the total number of workers, total number of tasks slots, number of task slots your pipeline is using or number of workers your pipeline is executing on? I was under the impression that the first two were properties of the Flink cluster and don't change while the third property is configured at job submission time and also doesn't change. I may not be understanding what you're trying to measure and why at pipeline runtime for Flink since many of these values don't change through the lifetime of the cluster and/or job. On Mon, Aug 8, 2022 at 4:59 PM aryan m <maryan8...@gmail.com> wrote: > Hi Luke! > Thanks !! We use the Flink Runner and run SDK workers as processes [1] > within a k8s pod. Can you please share broad steps on how one can do in the > runner ? > > > [1] > https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ProcessEnvironmentFactory.java > > > On Mon, Aug 8, 2022 at 8:51 AM Luke Cwik via user <user@beam.apache.org> > wrote: > >> That code only executes within a runner and is only used by certain >> runners and wouldn't work in general from user code that is monitoring the >> job or user code executing within one of the workers. >> >> You would need to author code that is likely runner specific to look up >> the number of workers associated with a job as I don't believe there is a >> general way to do this for an arbitrary Apache Beam runner. >> >> Which runner would you most likely want to use? >> >> On Sun, Aug 7, 2022 at 1:02 PM aryan m <maryan8...@gmail.com> wrote: >> >>> Hi Users! >>> Is there a recommended approach to publish metrics on the number of >>> sdk workers available/running as a gauge ? >>> >>> >>> [1] >>> https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java#L267 >>> [2] >>> https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java#L148 >>> >>> >>> -- Aryan >>> >>