[jira] [Commented] (FLINK-938) Change start-cluster.sh script so that users don't have to configure the JobManager address
[ https://issues.apache.org/jira/browse/FLINK-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14626096#comment-14626096 ] ASF GitHub Bot commented on FLINK-938: -- Github user uce commented on the pull request: https://github.com/apache/flink/pull/248#issuecomment-121178791 I waited to merge this, because of the recent ZooKeeper integration (to see whether this would be affected by it or not). Turns out this change is orthogonal, so I think we can finally merge this now if there are no objections. Change start-cluster.sh script so that users don't have to configure the JobManager address --- Key: FLINK-938 URL: https://issues.apache.org/jira/browse/FLINK-938 Project: Flink Issue Type: Improvement Components: Build System Reporter: Robert Metzger Assignee: Mingliang Qi Priority: Minor Fix For: 0.9 To improve the user experience, Flink should not require users to configure the JobManager's address on a cluster. In combination with FLINK-934, this would allow running Flink with decent performance on a cluster without setting a single configuration value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-938) Change start-cluster.sh script so that users don't have to configure the JobManager address
[ https://issues.apache.org/jira/browse/FLINK-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14626106#comment-14626106 ] ASF GitHub Bot commented on FLINK-938: -- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/248#issuecomment-121180165 Are we sure this helps, or will it add yet another way of configuring and starting the cluster. Are we getting to the point that it becomes hard to understand what happens in which situation, with all the different variants? Change start-cluster.sh script so that users don't have to configure the JobManager address --- Key: FLINK-938 URL: https://issues.apache.org/jira/browse/FLINK-938 Project: Flink Issue Type: Improvement Components: Build System Reporter: Robert Metzger Assignee: Mingliang Qi Priority: Minor Fix For: 0.9 To improve the user experience, Flink should not require users to configure the JobManager's address on a cluster. In combination with FLINK-934, this would allow running Flink with decent performance on a cluster without setting a single configuration value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-938) Change start-cluster.sh script so that users don't have to configure the JobManager address
[ https://issues.apache.org/jira/browse/FLINK-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14626201#comment-14626201 ] ASF GitHub Bot commented on FLINK-938: -- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/248#issuecomment-121203637 How would you give a good error message? It would manifest itself as the fact that the TaskManagers cannot connect to the JobManager. I would actually agree to not add this for now. Change start-cluster.sh script so that users don't have to configure the JobManager address --- Key: FLINK-938 URL: https://issues.apache.org/jira/browse/FLINK-938 Project: Flink Issue Type: Improvement Components: Build System Reporter: Robert Metzger Assignee: Mingliang Qi Priority: Minor Fix For: 0.9 To improve the user experience, Flink should not require users to configure the JobManager's address on a cluster. In combination with FLINK-934, this would allow running Flink with decent performance on a cluster without setting a single configuration value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-938) Change start-cluster.sh script so that users don't have to configure the JobManager address
[ https://issues.apache.org/jira/browse/FLINK-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14626131#comment-14626131 ] ASF GitHub Bot commented on FLINK-938: -- Github user uce commented on the pull request: https://github.com/apache/flink/pull/248#issuecomment-121183870 Yeah, I was also thinking about this. This will result in a total of 4 combinations: HA mode | JM address configured | JM address behaviour - | - | - 0 | 0 | Use hostname of where the JM is started (new behaviour, this PR) 0 | 1 | Use configured hostname (current behaviour) 1 | 0 | Ignored (current behaviour) 1 | 1 | Ignored (current behaviour) We would comment out the JM RPC address in the default config and most cluster setups w/o HA would require one less configuration value. What do you say? I'm slightly leaning towards adding this, but if your gut feeling is that this complicates things I'm also happy to close this PR. :) Change start-cluster.sh script so that users don't have to configure the JobManager address --- Key: FLINK-938 URL: https://issues.apache.org/jira/browse/FLINK-938 Project: Flink Issue Type: Improvement Components: Build System Reporter: Robert Metzger Assignee: Mingliang Qi Priority: Minor Fix For: 0.9 To improve the user experience, Flink should not require users to configure the JobManager's address on a cluster. In combination with FLINK-934, this would allow running Flink with decent performance on a cluster without setting a single configuration value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-938) Change start-cluster.sh script so that users don't have to configure the JobManager address
[ https://issues.apache.org/jira/browse/FLINK-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14626177#comment-14626177 ] ASF GitHub Bot commented on FLINK-938: -- Github user uce commented on the pull request: https://github.com/apache/flink/pull/248#issuecomment-121195803 I've tested this with default EC2 and GCE instances, which by default return the cloud internal hostname. I would say that this is a reasonable configuration. If it is not configured this way, the address needs to be configured (which is the current behavior). With a good startup/error message this should be fine. Still, I start to feel like we are discussing this issue for too long w/o anybody asking for it. Since we have to rebase this on the recent big changes anyways, let's close this PR and reconsider it the issue if someone complains/asks for it. Agree? Change start-cluster.sh script so that users don't have to configure the JobManager address --- Key: FLINK-938 URL: https://issues.apache.org/jira/browse/FLINK-938 Project: Flink Issue Type: Improvement Components: Build System Reporter: Robert Metzger Assignee: Mingliang Qi Priority: Minor Fix For: 0.9 To improve the user experience, Flink should not require users to configure the JobManager's address on a cluster. In combination with FLINK-934, this would allow running Flink with decent performance on a cluster without setting a single configuration value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-938) Change start-cluster.sh script so that users don't have to configure the JobManager address
[ https://issues.apache.org/jira/browse/FLINK-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14626214#comment-14626214 ] ASF GitHub Bot commented on FLINK-938: -- Github user uce commented on the pull request: https://github.com/apache/flink/pull/248#issuecomment-121208526 Something like Using address XYZ. Configure via ABC. at startup and a similar error messasge if connecting fails. You could also try to connect from the remote host before starting the task managers. But as I've said, let's not add it for now. @qmlmoon, can you close this PR? Change start-cluster.sh script so that users don't have to configure the JobManager address --- Key: FLINK-938 URL: https://issues.apache.org/jira/browse/FLINK-938 Project: Flink Issue Type: Improvement Components: Build System Reporter: Robert Metzger Assignee: Mingliang Qi Priority: Minor Fix For: 0.9 To improve the user experience, Flink should not require users to configure the JobManager's address on a cluster. In combination with FLINK-934, this would allow running Flink with decent performance on a cluster without setting a single configuration value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-938) Change start-cluster.sh script so that users don't have to configure the JobManager address
[ https://issues.apache.org/jira/browse/FLINK-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14626226#comment-14626226 ] ASF GitHub Bot commented on FLINK-938: -- Github user qmlmoon closed the pull request at: https://github.com/apache/flink/pull/248 Change start-cluster.sh script so that users don't have to configure the JobManager address --- Key: FLINK-938 URL: https://issues.apache.org/jira/browse/FLINK-938 Project: Flink Issue Type: Improvement Components: Build System Reporter: Robert Metzger Assignee: Mingliang Qi Priority: Minor Fix For: 0.9 To improve the user experience, Flink should not require users to configure the JobManager's address on a cluster. In combination with FLINK-934, this would allow running Flink with decent performance on a cluster without setting a single configuration value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-938) Change start-cluster.sh script so that users don't have to configure the JobManager address
[ https://issues.apache.org/jira/browse/FLINK-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14595513#comment-14595513 ] ASF GitHub Bot commented on FLINK-938: -- Github user uce commented on the pull request: https://github.com/apache/flink/pull/248#issuecomment-114042530 OK going to merge and adding the warning. Makes it more comfortable to run a stand-alone cluster in the cloud. :) Thanks for the PR. Change start-cluster.sh script so that users don't have to configure the JobManager address --- Key: FLINK-938 URL: https://issues.apache.org/jira/browse/FLINK-938 Project: Flink Issue Type: Improvement Components: Build System Reporter: Robert Metzger Assignee: Mingliang Qi Priority: Minor Fix For: 0.9 To improve the user experience, Flink should not require users to configure the JobManager's address on a cluster. In combination with FLINK-934, this would allow running Flink with decent performance on a cluster without setting a single configuration value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-938) Change start-cluster.sh script so that users don't have to configure the JobManager address
[ https://issues.apache.org/jira/browse/FLINK-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14511291#comment-14511291 ] ASF GitHub Bot commented on FLINK-938: -- Github user uce commented on the pull request: https://github.com/apache/flink/pull/248#issuecomment-95981935 I've just checked EC2. The default AMIs all return the cloud internal hostname, which is accessible (given the correct security setup). I count this as an indicator that just using `hostname` is a reasonable approach. Before merging, I would like to hear another opinion on this though. Change start-cluster.sh script so that users don't have to configure the JobManager address --- Key: FLINK-938 URL: https://issues.apache.org/jira/browse/FLINK-938 Project: Flink Issue Type: Improvement Components: Build System Reporter: Robert Metzger Assignee: Mingliang Qi Priority: Minor Fix For: 0.9 To improve the user experience, Flink should not require users to configure the JobManager's address on a cluster. In combination with FLINK-934, this would allow running Flink with decent performance on a cluster without setting a single configuration value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-938) Change start-cluster.sh script so that users don't have to configure the JobManager address
[ https://issues.apache.org/jira/browse/FLINK-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14511329#comment-14511329 ] ASF GitHub Bot commented on FLINK-938: -- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/248#issuecomment-95990793 This will work for most use cases where the JobManager is started within the same cluster as the TaskManagers. It will certainly fail if the JobManager resides in a different network environment where the hostname cannot be resolved by the TaskManagers. I think it's ok to merge if we print a big warning when the job manager Change start-cluster.sh script so that users don't have to configure the JobManager address --- Key: FLINK-938 URL: https://issues.apache.org/jira/browse/FLINK-938 Project: Flink Issue Type: Improvement Components: Build System Reporter: Robert Metzger Assignee: Mingliang Qi Priority: Minor Fix For: 0.9 To improve the user experience, Flink should not require users to configure the JobManager's address on a cluster. In combination with FLINK-934, this would allow running Flink with decent performance on a cluster without setting a single configuration value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-938) Change start-cluster.sh script so that users don't have to configure the JobManager address
[ https://issues.apache.org/jira/browse/FLINK-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509010#comment-14509010 ] ASF GitHub Bot commented on FLINK-938: -- Github user uce commented on the pull request: https://github.com/apache/flink/pull/48#issuecomment-95579798 Thanks for the PR. I've looked into the other approach (#248) and I think you can safely close this on now. Change start-cluster.sh script so that users don't have to configure the JobManager address --- Key: FLINK-938 URL: https://issues.apache.org/jira/browse/FLINK-938 Project: Flink Issue Type: Improvement Components: Build System Reporter: Robert Metzger Assignee: Mingliang Qi Priority: Minor Fix For: 0.9 To improve the user experience, Flink should not require users to configure the JobManager's address on a cluster. In combination with FLINK-934, this would allow running Flink with decent performance on a cluster without setting a single configuration value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-938) Change start-cluster.sh script so that users don't have to configure the JobManager address
[ https://issues.apache.org/jira/browse/FLINK-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509013#comment-14509013 ] ASF GitHub Bot commented on FLINK-938: -- Github user qmlmoon closed the pull request at: https://github.com/apache/flink/pull/48 Change start-cluster.sh script so that users don't have to configure the JobManager address --- Key: FLINK-938 URL: https://issues.apache.org/jira/browse/FLINK-938 Project: Flink Issue Type: Improvement Components: Build System Reporter: Robert Metzger Assignee: Mingliang Qi Priority: Minor Fix For: 0.9 To improve the user experience, Flink should not require users to configure the JobManager's address on a cluster. In combination with FLINK-934, this would allow running Flink with decent performance on a cluster without setting a single configuration value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-938) Change start-cluster.sh script so that users don't have to configure the JobManager address
[ https://issues.apache.org/jira/browse/FLINK-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509047#comment-14509047 ] ASF GitHub Bot commented on FLINK-938: -- Github user uce commented on the pull request: https://github.com/apache/flink/pull/248#issuecomment-95586174 I've rebased this on the current master, added log messages when falling back to the default address (which I've renamed as Till suggested), and tested it. There is one remaining issue though: When starting the task manager, you use `hostname` as the default JM address. As @StephanEwen pointed out in another thread [1], this will only work for certain setups. Then the question is whether we really improve deployment with this or not. @rmetzger, you reported the original issue. Do you (and others of course) have an idea about this? [1] https://github.com/apache/flink/pull/581#issuecomment-94911176 PS: The rebased branch is here, if you have time to pick it up from there: https://github.com/uce/incubator-flink/tree/jmaddress Change start-cluster.sh script so that users don't have to configure the JobManager address --- Key: FLINK-938 URL: https://issues.apache.org/jira/browse/FLINK-938 Project: Flink Issue Type: Improvement Components: Build System Reporter: Robert Metzger Assignee: Mingliang Qi Priority: Minor Fix For: 0.9 To improve the user experience, Flink should not require users to configure the JobManager's address on a cluster. In combination with FLINK-934, this would allow running Flink with decent performance on a cluster without setting a single configuration value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-938) Change start-cluster.sh script so that users don't have to configure the JobManager address
[
https://issues.apache.org/jira/browse/FLINK-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293520#comment-14293520
]
ASF GitHub Bot commented on FLINK-938:
--
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/248#discussion_r23607471
--- Diff:
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
---
@@ -519,6 +519,12 @@ object JobManager {
configuration.setString(ConfigConstants.FLINK_BASE_DIR_PATH_KEY,
config.configDir + /..)
}
+if
(GlobalConfiguration.getString(ConfigConstants.JOB_MANAGER_IPC_ADDRESS_KEY,
+ null) == null) {
+
configuration.setString(ConfigConstants.JOB_MANAGER_IPC_ADDRESS_KEY,
+InetAddress.getLocalHost.getHostName)
+}
+
val hostname =
configuration.getString(ConfigConstants.JOB_MANAGER_IPC_ADDRESS_KEY, null)
--- End diff --
Can't we just set InetAddress.getLocalHost.getHostName as the default value
here instead of having the if block above?
Change start-cluster.sh script so that users don't have to configure the
JobManager address
---
Key: FLINK-938
URL: https://issues.apache.org/jira/browse/FLINK-938
Project: Flink
Issue Type: Improvement
Components: Build System
Reporter: Robert Metzger
Assignee: Mingliang Qi
Priority: Minor
Fix For: 0.9
To improve the user experience, Flink should not require users to configure
the JobManager's address on a cluster.
In combination with FLINK-934, this would allow running Flink with decent
performance on a cluster without setting a single configuration value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-938) Change start-cluster.sh script so that users don't have to configure the JobManager address
[
https://issues.apache.org/jira/browse/FLINK-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293525#comment-14293525
]
ASF GitHub Bot commented on FLINK-938:
--
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/248#discussion_r23607703
--- Diff:
flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala
---
@@ -596,6 +596,10 @@ object TaskManager {
opt[String](tempDir) optional() action { (x, c) =
c.copy(tmpDir = x)
} text (Specify temporary directory.)
+
+ opt[String](defaultJobManagerAdd) optional() action { (x, c) =
--- End diff --
I'm slightly in favour of writing the option completely out:
defaultJobManagerAddress
Change start-cluster.sh script so that users don't have to configure the
JobManager address
---
Key: FLINK-938
URL: https://issues.apache.org/jira/browse/FLINK-938
Project: Flink
Issue Type: Improvement
Components: Build System
Reporter: Robert Metzger
Assignee: Mingliang Qi
Priority: Minor
Fix For: 0.9
To improve the user experience, Flink should not require users to configure
the JobManager's address on a cluster.
In combination with FLINK-934, this would allow running Flink with decent
performance on a cluster without setting a single configuration value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-938) Change start-cluster.sh script so that users don't have to configure the JobManager address
[
https://issues.apache.org/jira/browse/FLINK-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293561#comment-14293561
]
ASF GitHub Bot commented on FLINK-938:
--
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/248#discussion_r23609808
--- Diff:
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
---
@@ -519,6 +519,12 @@ object JobManager {
configuration.setString(ConfigConstants.FLINK_BASE_DIR_PATH_KEY,
config.configDir + /..)
}
+if
(GlobalConfiguration.getString(ConfigConstants.JOB_MANAGER_IPC_ADDRESS_KEY,
+ null) == null) {
+
configuration.setString(ConfigConstants.JOB_MANAGER_IPC_ADDRESS_KEY,
+InetAddress.getLocalHost.getHostName)
+}
+
val hostname =
configuration.getString(ConfigConstants.JOB_MANAGER_IPC_ADDRESS_KEY, null)
--- End diff --
That is true for the TaskManager but not for the JobManager. For the
TaskManager, we could replace the null default value with
```InetAddress.getLocalHost.getHostName``` at both places.
Change start-cluster.sh script so that users don't have to configure the
JobManager address
---
Key: FLINK-938
URL: https://issues.apache.org/jira/browse/FLINK-938
Project: Flink
Issue Type: Improvement
Components: Build System
Reporter: Robert Metzger
Assignee: Mingliang Qi
Priority: Minor
Fix For: 0.9
To improve the user experience, Flink should not require users to configure
the JobManager's address on a cluster.
In combination with FLINK-934, this would allow running Flink with decent
performance on a cluster without setting a single configuration value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-938) Change start-cluster.sh script so that users don't have to configure the JobManager address
[
https://issues.apache.org/jira/browse/FLINK-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293588#comment-14293588
]
ASF GitHub Bot commented on FLINK-938:
--
Github user qmlmoon commented on a diff in the pull request:
https://github.com/apache/flink/pull/248#discussion_r23610509
--- Diff:
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
---
@@ -519,6 +519,12 @@ object JobManager {
configuration.setString(ConfigConstants.FLINK_BASE_DIR_PATH_KEY,
config.configDir + /..)
}
+if
(GlobalConfiguration.getString(ConfigConstants.JOB_MANAGER_IPC_ADDRESS_KEY,
+ null) == null) {
+
configuration.setString(ConfigConstants.JOB_MANAGER_IPC_ADDRESS_KEY,
+InetAddress.getLocalHost.getHostName)
+}
+
val hostname =
configuration.getString(ConfigConstants.JOB_MANAGER_IPC_ADDRESS_KEY, null)
--- End diff --
In local mode, jobmanager will start an internal taskmanager and call the
`TaskManager.parseConfiguration`. For TaskManager, we can replace the null
default value with the `config.defaultJobManagerAdd` but not the
`InetAddress.getLocalHost.getHostName`. I think the question is whether we add
an additional parameter to the `parseConfiguration` for the default jobmanager
address or put this in the configuration.
Change start-cluster.sh script so that users don't have to configure the
JobManager address
---
Key: FLINK-938
URL: https://issues.apache.org/jira/browse/FLINK-938
Project: Flink
Issue Type: Improvement
Components: Build System
Reporter: Robert Metzger
Assignee: Mingliang Qi
Priority: Minor
Fix For: 0.9
To improve the user experience, Flink should not require users to configure
the JobManager's address on a cluster.
In combination with FLINK-934, this would allow running Flink with decent
performance on a cluster without setting a single configuration value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-938) Change start-cluster.sh script so that users don't have to configure the JobManager address
[ https://issues.apache.org/jira/browse/FLINK-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293481#comment-14293481 ] ASF GitHub Bot commented on FLINK-938: -- Github user qmlmoon commented on the pull request: https://github.com/apache/flink/pull/248#issuecomment-71645508 ok. I rebased the PR and modified it with scala implementation. Change start-cluster.sh script so that users don't have to configure the JobManager address --- Key: FLINK-938 URL: https://issues.apache.org/jira/browse/FLINK-938 Project: Flink Issue Type: Improvement Components: Build System Reporter: Robert Metzger Assignee: Mingliang Qi Priority: Minor Fix For: 0.9 To improve the user experience, Flink should not require users to configure the JobManager's address on a cluster. In combination with FLINK-934, this would allow running Flink with decent performance on a cluster without setting a single configuration value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
