Re: Flink YARN job manager web port

[Till Rohrmann]

Hi Shannon,

if you need this feature (assigning range of web server ports) for your use
case, then we would have to add it. If you want to do it, then it would
help us a lot.

I think the documentation is a bit outdated here. The port is either chosen
from the range of ports or a ephemeral port is picked up. I'll create a
JIRA to fix the documentation.

Cheers,
Till

On Thu, Apr 21, 2016 at 10:55 PM, Shannon Carey  wrote:

> The documentation states: "The ports Flink is using for its services are
> the standard ports configured by the user + the application id as an offset"
>
> When I launch Flink via YARN in an AWS EMR cluster, stdout says:
> JobManager Web Interface:
> http://ip-xxx.us-west-2.compute.internal:20888/proxy/application_1461178294210_0010/
>
> I need to be able to create an IAM Security Group that allows access to
> the JobManager web interface so that I can make use of it. However, I am
> confused about how port 20888 is chosen. Based on the code, I would have
> guessed that it would use the same port as given by: "yarn application
> -status application_1461178294210_0010". However, that's not the case
> (they don't match). It gives "Tracking-URL :
> http://ip-xxx.us-west-2.compute.internal:36495;
>
> On the other hand, I see that YarnApplicationMasterRunner sets the port to
> 0, which InetSocketAddress says results in "A port number of zero will
> let the system pick up an ephemeral port in a bind operation."
>
> I couldn't find anything in the code that adds an offset to a port. Changing
> the value of "jobmanager.web.port" appears to have no effect. The
> documentation on "Running Flink on YARN behind Firewalls" only talks about
> the JobManager and BlobServer ports.
>
> Does Flink need logic to allow users to specify a range of ports for
> jobmanager.web.port in the same way as is done in
> BootstrapTools#startActorSystem? If so, I am happy to make that
> contribution!
>
> -Shannon
>

Flink YARN job manager web port

[Shannon Carey]

The documentation states: "The ports Flink is using for its services are the 
standard ports configured by the user + the application id as an offset"

When I launch Flink via YARN in an AWS EMR cluster, stdout says:
JobManager Web Interface: 
http://ip-xxx.us-west-2.compute.internal:20888/proxy/application_1461178294210_0010/

I need to be able to create an IAM Security Group that allows access to the 
JobManager web interface so that I can make use of it. However, I am confused 
about how port 20888 is chosen. Based on the code, I would have guessed that it 
would use the same port as given by: "yarn application -status 
application_1461178294210_0010". However, that's not the case (they don't 
match). It gives "Tracking-URL : http://ip-xxx.us-west-2.compute.internal:36495;

On the other hand, I see that YarnApplicationMasterRunner sets the port to 0, 
which InetSocketAddress says results in "A port number of zero will let the 
system pick up an ephemeral port in a bind operation."

I couldn't find anything in the code that adds an offset to a port. Changing 
the value of "jobmanager.web.port" appears to have no effect. The documentation 
on "Running Flink on YARN behind Firewalls" only talks about the JobManager and 
BlobServer ports.

Does Flink need logic to allow users to specify a range of ports for 
jobmanager.web.port in the same way as is done in 
BootstrapTools#startActorSystem? If so, I am happy to make that contribution!

-Shannon