Hi Kyle, Thank you for the prompt response and apologize for the late reply.
[1] seems to be only available in python portable_runner but not java PortableRunner, is it intended or we could add similar changes in java as well? [2] makes sense to block since the wait/retry is handled in the previous prepare(), however, is there any specific reason why we do not want to support timeout in start worker request? Best, Ke > On May 14, 2021, at 11:25 AM, Kyle Weaver <[email protected]> wrote: > > 1. and 2. are both facilitated by GRPC, which takes care of most of the > retry/wait logic. In some places we have a configurable timeout (which > defaults to 60s) [1], while in other places we block [2][3]. > > [1] https://issues.apache.org/jira/browse/BEAM-7933 > <https://issues.apache.org/jira/browse/BEAM-7933> > [2] > https://github.com/apache/beam/blob/51541a595b09751dd3dde2c50caf2a968ac01b68/sdks/python/apache_beam/runners/portability/portable_runner.py#L238-L242 > > <https://github.com/apache/beam/blob/51541a595b09751dd3dde2c50caf2a968ac01b68/sdks/python/apache_beam/runners/portability/portable_runner.py#L238-L242> > [3] > https://github.com/apache/beam/blob/9601bdef8870bc6acc7895c06252e43ec040bd8c/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ExternalEnvironmentFactory.java#L115 > > <https://github.com/apache/beam/blob/9601bdef8870bc6acc7895c06252e43ec040bd8c/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ExternalEnvironmentFactory.java#L115> > On Fri, May 14, 2021 at 10:51 AM Ke Wu <[email protected] > <mailto:[email protected]>> wrote: > Hello All, > > I came across this question when I am reading Beam on Flink on Kubernetes > <https://docs.google.com/document/d/1z3LNrRtr8kkiFHonZ5JJM_L4NWNBBNcqRc_yAf6G0VI/edit#heading=h.x9qy4wlfgc1g> > and flink-on-k8s-operator > <https://github.com/GoogleCloudPlatform/flink-on-k8s-operator/tree/0310df76d6e2128cd5d2bc51fae4e842d370c463> > and realized that there seems no retry/wait logic built in PortableRunner > nor ExternalEnvironmentFactory, (correct me if I am wrong) which creates > implications that: > > 1. Job Server needs to be ready to accept request before SDK Client could > submit request. > 2. External Worker Pool Service needs to be ready to accept start/stop worker > request before runner starts to request. > > This may bring some challenges on k8s since Flink opt to use multi containers > pattern when bringing up a beam portable pipeline, in addition, I don’t find > any special lifecycle management in place to guarantee the order, e.g. > External Worker Pool Service container to start and ready before the task > manager container to start making requests. > > I am wondering if I missed anything to guarantee the readiness of the > dependent service or we are relying on that dependent containers are much > lighter weigh so it should, in most time, be ready before the other container > start to make requests. > > Best, > Ke >
