cameronlee314 commented on pull request #1499: URL: https://github.com/apache/samza/pull/1499#issuecomment-832298098
> > More context about how this can be used: > > When using Kubernetes, we can create a "Service" which provides a consistent coordinator url, even when the job coordinator switches pods. The port is one piece of information needed to create this "Service". If we want to use a Kubernetes controller to create the "Service" (per the Kubernetes operator pattern), then we don't need to retrieve the port from the job coordinator in order to create the "Service". > > This may not bring a lot of benefit, especially for portable beam jobs, as there are multiple port numbers need to be communicated between JC Pod and Worker Pod and preconfigure all of them may not be practical. > > Is this just for ease of creating "Service" or w/o predefined port number, we cannot create "Service" at all? It might be hard to statically specify many ports, but this does give a bit more flexibility in some cases. If we only have the option to dynamically allocate ports, then there is an ordering dependency for resource management between the job coordinator and the worker containers, because the worker containers need to know the port that the job coordinator is using. This configuration does need to be used with care, because it does introduce another failure case (i.e. port conflict) if it is not used properly. If we dynamically allocate the port, then the job coordinator needs to expose the port first before the "Service" can be created. This creates more of an ordering dependency. The job coordinator would need to trigger the creation of the "Service" or the job coordinator needs to pass the port to somewhere (e.g. coordinator stream, config map) before something else can create the "Service". -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
