[jira] [Commented] (FLINK-6487) Remove JobManager local mode
[
https://issues.apache.org/jira/browse/FLINK-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892285#comment-16892285
]
TisonKun commented on FLINK-6487:
-
Thank [~StephanEwen]. I just found when execute the main method of Flink
program directly we use MiniCluster.
> Remove JobManager local mode
>
>
> Key: FLINK-6487
> URL: https://issues.apache.org/jira/browse/FLINK-6487
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Reporter: Stephan Ewen
>Priority: Major
>
> We should remove the "local" mode from the JobManager.
> Currently, the JobManager has the strange "local" mode where it also starts
> an embedded Task Manager.
> I think that mode has caused confusion / problems:
> - No TaskManagers can join the cluster
> - TaskManagers do not support querable state
> - It is redundant code to maintain
> At the same time, the mode does not help at all:
> - The MiniCluster does not use that mode
> - Starting from scripts, the {{start-cluster.sh}} works out of the box,
> creating a proper local cluster, but in two processes, rather than one.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (FLINK-6487) Remove JobManager local mode
[
https://issues.apache.org/jira/browse/FLINK-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892039#comment-16892039
]
Stephan Ewen commented on FLINK-6487:
-
The mini cluster is also used behind the Local[Stream]ExecutionEnvironment, so
for examples, testing, debugging, etc.
It is definitely meant to be used by users, not just by tests.
> Remove JobManager local mode
>
>
> Key: FLINK-6487
> URL: https://issues.apache.org/jira/browse/FLINK-6487
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Reporter: Stephan Ewen
>Priority: Major
>
> We should remove the "local" mode from the JobManager.
> Currently, the JobManager has the strange "local" mode where it also starts
> an embedded Task Manager.
> I think that mode has caused confusion / problems:
> - No TaskManagers can join the cluster
> - TaskManagers do not support querable state
> - It is redundant code to maintain
> At the same time, the mode does not help at all:
> - The MiniCluster does not use that mode
> - Starting from scripts, the {{start-cluster.sh}} works out of the box,
> creating a proper local cluster, but in two processes, rather than one.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (FLINK-6487) Remove JobManager local mode
[
https://issues.apache.org/jira/browse/FLINK-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891630#comment-16891630
]
TisonKun commented on FLINK-6487:
-
Hi [~StephanEwen], as you mentioned above, "The MiniCluster does not use that
mode". Is our MiniCluster currently used for testing purpose only(including
user scope testing/getting start)?
> Remove JobManager local mode
>
>
> Key: FLINK-6487
> URL: https://issues.apache.org/jira/browse/FLINK-6487
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Reporter: Stephan Ewen
>Priority: Major
>
> We should remove the "local" mode from the JobManager.
> Currently, the JobManager has the strange "local" mode where it also starts
> an embedded Task Manager.
> I think that mode has caused confusion / problems:
> - No TaskManagers can join the cluster
> - TaskManagers do not support querable state
> - It is redundant code to maintain
> At the same time, the mode does not help at all:
> - The MiniCluster does not use that mode
> - Starting from scripts, the {{start-cluster.sh}} works out of the box,
> creating a proper local cluster, but in two processes, rather than one.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (FLINK-6487) Remove JobManager local mode
[
https://issues.apache.org/jira/browse/FLINK-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16001269#comment-16001269
]
Patrick Lucas commented on FLINK-6487:
--
This appears to work:
{code}
#!/bin/bash
JM_PID=
TM_PID=
function die() {
kill $JM_PID $TM_PID
}
trap die SIGINT SIGTERM
jobmanager.sh start-foreground cluster &
JM_PID=$!
taskmanager.sh start-foreground &
TM_PID=$!
wait $JM_PID $TM_PID
wait $JM_PID $TM_PID
{code}
The double wait is intentional. When a trap fires it immediately returns all
pending {{wait}}s, so you have to {{wait}} again on the same pids to actually
block until they exit. There might be a nicer way to do that.
> Remove JobManager local mode
>
>
> Key: FLINK-6487
> URL: https://issues.apache.org/jira/browse/FLINK-6487
> Project: Flink
> Issue Type: Bug
> Components: JobManager
>Reporter: Stephan Ewen
>
> We should remove the "local" mode from the JobManager.
> Currently, the JobManager has the strange "local" mode where it also starts
> an embedded Task Manager.
> I think that mode has caused confusion / problems:
> - No TaskManagers can join the cluster
> - TaskManagers do not support querable state
> - It is redundant code to maintain
> At the same time, the mode does not help at all:
> - The MiniCluster does not use that mode
> - Starting from scripts, the {{start-cluster.sh}} works out of the box,
> creating a proper local cluster, but in two processes, rather than one.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6487) Remove JobManager local mode
[
https://issues.apache.org/jira/browse/FLINK-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16001228#comment-16001228
]
Patrick Lucas commented on FLINK-6487:
--
I think we have to be cleverer (more clever?) than that—to propagate those
signals in bash properly you need to exec (which of course you could only do
once) or use trap... which might actually work okay. I'll give it a shot.
Killing PID 1 with this approach as-is will orphan the cat and a bash that has
two java child processes.
> Remove JobManager local mode
>
>
> Key: FLINK-6487
> URL: https://issues.apache.org/jira/browse/FLINK-6487
> Project: Flink
> Issue Type: Bug
> Components: JobManager
>Reporter: Stephan Ewen
>
> We should remove the "local" mode from the JobManager.
> Currently, the JobManager has the strange "local" mode where it also starts
> an embedded Task Manager.
> I think that mode has caused confusion / problems:
> - No TaskManagers can join the cluster
> - TaskManagers do not support querable state
> - It is redundant code to maintain
> At the same time, the mode does not help at all:
> - The MiniCluster does not use that mode
> - Starting from scripts, the {{start-cluster.sh}} works out of the box,
> creating a proper local cluster, but in two processes, rather than one.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6487) Remove JobManager local mode
[
https://issues.apache.org/jira/browse/FLINK-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16001120#comment-16001120
]
Stephan Ewen commented on FLINK-6487:
-
I think what we want can be done via
{code}
{ jobmanager.sh & taskmanager.sh; } | cat
{code}
> Remove JobManager local mode
>
>
> Key: FLINK-6487
> URL: https://issues.apache.org/jira/browse/FLINK-6487
> Project: Flink
> Issue Type: Bug
> Components: JobManager
>Reporter: Stephan Ewen
>
> We should remove the "local" mode from the JobManager.
> Currently, the JobManager has the strange "local" mode where it also starts
> an embedded Task Manager.
> I think that mode has caused confusion / problems:
> - No TaskManagers can join the cluster
> - TaskManagers do not support querable state
> - It is redundant code to maintain
> At the same time, the mode does not help at all:
> - The MiniCluster does not use that mode
> - Starting from scripts, the {{start-cluster.sh}} works out of the box,
> creating a proper local cluster, but in two processes, rather than one.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6487) Remove JobManager local mode
[
https://issues.apache.org/jira/browse/FLINK-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16000980#comment-16000980
]
Patrick Lucas commented on FLINK-6487:
--
Something like that should be fine. Even just a supervisord.conf that runs both
with the right options could work, though unfortunately installing supervisord
adds ~22MB to the image.
We really don't need anything fancy, just something that spawns two processes,
propagates signals to them properly, and multiplexes their stdouts into its own
stdout. I wonder what the simplest approach for that would be.
> Remove JobManager local mode
>
>
> Key: FLINK-6487
> URL: https://issues.apache.org/jira/browse/FLINK-6487
> Project: Flink
> Issue Type: Bug
> Components: JobManager
>Reporter: Stephan Ewen
>
> We should remove the "local" mode from the JobManager.
> Currently, the JobManager has the strange "local" mode where it also starts
> an embedded Task Manager.
> I think that mode has caused confusion / problems:
> - No TaskManagers can join the cluster
> - TaskManagers do not support querable state
> - It is redundant code to maintain
> At the same time, the mode does not help at all:
> - The MiniCluster does not use that mode
> - Starting from scripts, the {{start-cluster.sh}} works out of the box,
> creating a proper local cluster, but in two processes, rather than one.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6487) Remove JobManager local mode
[
https://issues.apache.org/jira/browse/FLINK-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16000960#comment-16000960
]
Stephan Ewen commented on FLINK-6487:
-
I see. I would vote to have a different form of entry-point then, specifically
not to use the JobManager local mode (which starts a semi-functional
TaskManager), but to have a dedicated entry point that simply forks two threads
that run a proper JobManager and a proper TaskManager.
> Remove JobManager local mode
>
>
> Key: FLINK-6487
> URL: https://issues.apache.org/jira/browse/FLINK-6487
> Project: Flink
> Issue Type: Bug
> Components: JobManager
>Reporter: Stephan Ewen
>
> We should remove the "local" mode from the JobManager.
> Currently, the JobManager has the strange "local" mode where it also starts
> an embedded Task Manager.
> I think that mode has caused confusion / problems:
> - No TaskManagers can join the cluster
> - TaskManagers do not support querable state
> - It is redundant code to maintain
> At the same time, the mode does not help at all:
> - The MiniCluster does not use that mode
> - Starting from scripts, the {{start-cluster.sh}} works out of the box,
> creating a proper local cluster, but in two processes, rather than one.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6487) Remove JobManager local mode
[
https://issues.apache.org/jira/browse/FLINK-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16000909#comment-16000909
]
Patrick Lucas commented on FLINK-6487:
--
One nice reason to have local mode is for docker, so users can, for example,
run {{docker run flink local}} to get a local single-taskmanager cluster. (In
[{{docker-entrypoint.sh}}|https://github.com/docker-flink/docker-flink/blob/e92920ff33107e6f01abaddf7b0195570f4d06e1/docker-entrypoint.sh#L56]
this is turned into a call to {{jobmanager.sh start-foreground local}}.
Right now, {{start-cluster.sh}} has the flaws that it doesn't have an option to
run in the foreground and there is no real process management for the two
processes it spawns.
I think to properly replicate the behavior of {{jobmanager.sh start-foreground
local}} we either need to inline something like
[supervisord|http://supervisord.org/], making sure to multiplex [the processes'
stdouts|https://github.com/Supervisor/supervisor/issues/511] or have the
jobmanager fork (to keep the two-process paradigm).
> Remove JobManager local mode
>
>
> Key: FLINK-6487
> URL: https://issues.apache.org/jira/browse/FLINK-6487
> Project: Flink
> Issue Type: Bug
> Components: JobManager
>Reporter: Stephan Ewen
>
> We should remove the "local" mode from the JobManager.
> Currently, the JobManager has the strange "local" mode where it also starts
> an embedded Task Manager.
> I think that mode has caused confusion / problems:
> - No TaskManagers can join the cluster
> - TaskManagers do not support querable state
> - It is redundant code to maintain
> At the same time, the mode does not help at all:
> - The MiniCluster does not use that mode
> - Starting from scripts, the {{start-cluster.sh}} works out of the box,
> creating a proper local cluster, but in two processes, rather than one.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
