Re: high-availability.jobmanager.port vs jobmanager.rpc.port

2017-09-26 Thread Elias Levy
I presume then that the Job Managers and Task Managers are performing service discovery via Zookeeper in HA mode, rather than from the config file or the masters file. Yes? On Mon, Sep 25, 2017 at 11:14 PM, Till Rohrmann wrote: > Because a single port could easily lead to

Re: high-availability.jobmanager.port vs jobmanager.rpc.port

2017-09-26 Thread Till Rohrmann
Because a single port could easily lead to clashes if there is another JobManager running on the same machine with the same port (e.g. due to standby JobManagers). Cheers, Till On Sep 26, 2017 03:20, "Elias Levy" wrote: > Why a range instead of just a single port

Re: high-availability.jobmanager.port vs jobmanager.rpc.port

2017-09-25 Thread Elias Levy
Why a range instead of just a single port in HA mode? On Mon, Sep 25, 2017 at 1:49 PM, Till Rohrmann wrote: > Yes, with Flip-6 it will most likely look like how Stephan described it. > We need the explicit port in standalone mode so that TMs can connect to the > JM. In the

Re: high-availability.jobmanager.port vs jobmanager.rpc.port

2017-09-25 Thread Till Rohrmann
Yes, with Flip-6 it will most likely look like how Stephan described it. We need the explicit port in standalone mode so that TMs can connect to the JM. In the other deployment scenarios, the port can be randomly picked unless you want to specify a port range, e.g. for firewall configuration

Re: high-availability.jobmanager.port vs jobmanager.rpc.port

2017-09-25 Thread Stephan Ewen
/cc Till for real this time ;-) Hi! I think that can probably be simplified in the FLIP-6 case: - All RPC is only between JM and TM and the port should be completely random (optionally within a range). TM and JM discover each other via HA (ZK) or the TM gets the JM RPC port as a parameter

Re: high-availability.jobmanager.port vs jobmanager.rpc.port

2017-09-25 Thread Stephan Ewen
Hi! I think that can probably be simplified in the FLIP-6 case: - All RPC is only between JM and TM and the port should be completely random (optionally within a range). TM and JM discover each other via HA (ZK) or the TM gets the JM RPC port as a parameter when the container is started.

Re: high-availability.jobmanager.port vs jobmanager.rpc.port

2017-09-25 Thread Nico Kruber
Hi Elias, indeed that looks strange but was introduced with FLINK-3172 [1] with an argument about using the same configuration key (as opposed to having two different keys as mentioned) starting at https://issues.apache.org/jira/browse/FLINK-3172? focusedCommentId=15091940#comment-15091940

high-availability.jobmanager.port vs jobmanager.rpc.port

2017-09-23 Thread Elias Levy
I am wondering why HA mode there is a need for a separate config parameter to set the JM RPC port (high-availability.jobmanager.port) and why this parameter accepts a range, unlike jobmanager.rpc.port.