[GitHub] flink pull request #5528: [FLINK-6489] [FLINK-8696] [shell scripts] Remove J...
Github user StephanEwen closed the pull request at: https://github.com/apache/flink/pull/5528 ---
[GitHub] flink pull request #5528: [FLINK-6489] [FLINK-8696] [shell scripts] Remove J...
Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/5528#discussion_r169622022 --- Diff: flink-dist/src/main/flink-bin/bin/start-cluster.bat --- @@ -0,0 +1,77 @@ +::### +:: Licensed to the Apache Software Foundation (ASF) under one +:: or more contributor license agreements. See the NOTICE file +:: distributed with this work for additional information +:: regarding copyright ownership. The ASF licenses this file +:: to you under the Apache License, Version 2.0 (the +:: "License"); you may not use this file except in compliance +:: with the License. You may obtain a copy of the License at +:: +:: http://www.apache.org/licenses/LICENSE-2.0 +:: +:: Unless required by applicable law or agreed to in writing, software +:: distributed under the License is distributed on an "AS IS" BASIS, +:: WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +:: See the License for the specific language governing permissions and +:: limitations under the License. +::### + +@echo off +setlocal EnableDelayedExpansion + +SET bin=%~dp0 +SET FLINK_ROOT_DIR=%bin%.. +SET FLINK_LIB_DIR=%FLINK_ROOT_DIR%\lib +SET FLINK_CONF_DIR=%FLINK_ROOT_DIR%\conf +SET FLINK_LOG_DIR=%FLINK_ROOT_DIR%\log + +SET JVM_ARGS=-Xms1024m -Xmx1024m + +SET FLINK_CLASSPATH=%FLINK_LIB_DIR%\* + +SET logname_jm=flink-%username%-jobmanager-%computername%.log +SET logname_tm=flink-%username%-taskmanager-%computername%.log +SET log_jm=%FLINK_LOG_DIR%\%logname_jm% +SET log_tm=%FLINK_LOG_DIR%\%logname_tm% +SET outname_jm=flink-%username%-jobmanager-%computername%.out +SET outname_tm=flink-%username%-taskmanager-%computername%.out +SET out_jm=%FLINK_LOG_DIR%\%outname_jm% +SET out_tm=%FLINK_LOG_DIR%\%outname_tm% + +SET log_setting_jm=-Dlog.file="%log_jm%" -Dlogback.configurationFile=file:"%FLINK_CONF_DIR%/logback.xml" -Dlog4j.configuration=file:"%FLINK_CONF_DIR%/log4j.properties" +SET log_setting_tm=-Dlog.file="%log_tm%" -Dlogback.configurationFile=file:"%FLINK_CONF_DIR%/logback.xml" -Dlog4j.configuration=file:"%FLINK_CONF_DIR%/log4j.properties" + +:: Log rotation (quick and dirty) +CD "%FLINK_LOG_DIR%" +for /l %%x in (5, -1, 1) do ( +SET /A y = %%x+1 +RENAME "%logname_jm%.%%x" "%logname_jm%.!y!" 2> nul +RENAME "%logname_tm%.%%x" "%logname_tm%.!y!" 2> nul +RENAME "%outname_jm%.%%x" "%outname_jm%.!y!" 2> nul +RENAME "%outname_tm%.%%x" "%outname_tm%.!y!" 2> nul +) +RENAME "%logname_jm%" "%logname_jm%.0" 2> nul +RENAME "%logname_tm%" "%logname_tm%.0" 2> nul +RENAME "%outname_jm%" "%outname_jm%.0" 2> nul +RENAME "%outname_tm%" "%outname_tm%.0" 2> nul +DEL "%logname_jm%.6" 2> nul +DEL "%logname_tm%.6" 2> nul +DEL "%outname_jm%.6" 2> nul +DEL "%outname_tm%.6" 2> nul + +for %%X in (java.exe) do (set FOUND=%%~$PATH:X) +if not defined FOUND ( +echo java.exe was not found in PATH variable +goto :eof +) + +echo Starting a local cluster with one JobManager process and one TaskManager process. + +echo You can terminate the processes via CTRL-C in the spawned shell windows. + +echo Web interface by default on http://localhost:8081/. --- End diff -- ok ð ---
[GitHub] flink pull request #5528: [FLINK-6489] [FLINK-8696] [shell scripts] Remove J...
Github user StephanEwen commented on a diff in the pull request: https://github.com/apache/flink/pull/5528#discussion_r169612616 --- Diff: flink-dist/src/main/flink-bin/bin/start-cluster.bat --- @@ -0,0 +1,77 @@ +::### +:: Licensed to the Apache Software Foundation (ASF) under one +:: or more contributor license agreements. See the NOTICE file +:: distributed with this work for additional information +:: regarding copyright ownership. The ASF licenses this file +:: to you under the Apache License, Version 2.0 (the +:: "License"); you may not use this file except in compliance +:: with the License. You may obtain a copy of the License at +:: +:: http://www.apache.org/licenses/LICENSE-2.0 +:: +:: Unless required by applicable law or agreed to in writing, software +:: distributed under the License is distributed on an "AS IS" BASIS, +:: WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +:: See the License for the specific language governing permissions and +:: limitations under the License. +::### + +@echo off +setlocal EnableDelayedExpansion + +SET bin=%~dp0 +SET FLINK_ROOT_DIR=%bin%.. +SET FLINK_LIB_DIR=%FLINK_ROOT_DIR%\lib +SET FLINK_CONF_DIR=%FLINK_ROOT_DIR%\conf +SET FLINK_LOG_DIR=%FLINK_ROOT_DIR%\log + +SET JVM_ARGS=-Xms1024m -Xmx1024m + +SET FLINK_CLASSPATH=%FLINK_LIB_DIR%\* + +SET logname_jm=flink-%username%-jobmanager-%computername%.log +SET logname_tm=flink-%username%-taskmanager-%computername%.log +SET log_jm=%FLINK_LOG_DIR%\%logname_jm% +SET log_tm=%FLINK_LOG_DIR%\%logname_tm% +SET outname_jm=flink-%username%-jobmanager-%computername%.out +SET outname_tm=flink-%username%-taskmanager-%computername%.out +SET out_jm=%FLINK_LOG_DIR%\%outname_jm% +SET out_tm=%FLINK_LOG_DIR%\%outname_tm% + +SET log_setting_jm=-Dlog.file="%log_jm%" -Dlogback.configurationFile=file:"%FLINK_CONF_DIR%/logback.xml" -Dlog4j.configuration=file:"%FLINK_CONF_DIR%/log4j.properties" +SET log_setting_tm=-Dlog.file="%log_tm%" -Dlogback.configurationFile=file:"%FLINK_CONF_DIR%/logback.xml" -Dlog4j.configuration=file:"%FLINK_CONF_DIR%/log4j.properties" + +:: Log rotation (quick and dirty) +CD "%FLINK_LOG_DIR%" +for /l %%x in (5, -1, 1) do ( +SET /A y = %%x+1 +RENAME "%logname_jm%.%%x" "%logname_jm%.!y!" 2> nul +RENAME "%logname_tm%.%%x" "%logname_tm%.!y!" 2> nul +RENAME "%outname_jm%.%%x" "%outname_jm%.!y!" 2> nul +RENAME "%outname_tm%.%%x" "%outname_tm%.!y!" 2> nul +) +RENAME "%logname_jm%" "%logname_jm%.0" 2> nul +RENAME "%logname_tm%" "%logname_tm%.0" 2> nul +RENAME "%outname_jm%" "%outname_jm%.0" 2> nul +RENAME "%outname_tm%" "%outname_tm%.0" 2> nul +DEL "%logname_jm%.6" 2> nul +DEL "%logname_tm%.6" 2> nul +DEL "%outname_jm%.6" 2> nul +DEL "%outname_tm%.6" 2> nul + +for %%X in (java.exe) do (set FOUND=%%~$PATH:X) +if not defined FOUND ( +echo java.exe was not found in PATH variable +goto :eof +) + +echo Starting a local cluster with one JobManager process and one TaskManager process. + +echo You can terminate the processes via CTRL-C in the spawned shell windows. + +echo Web interface by default on http://localhost:8081/. --- End diff -- I would like to keep it at 8081, because the flip-6 endpoint will switch to that in the next days anyways and they will probably not be aware that this would need to be updated again ;-) ---
[GitHub] flink pull request #5528: [FLINK-6489] [FLINK-8696] [shell scripts] Remove J...
Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/5528#discussion_r169595831 --- Diff: flink-dist/src/main/flink-bin/bin/start-cluster.bat --- @@ -0,0 +1,77 @@ +::### +:: Licensed to the Apache Software Foundation (ASF) under one +:: or more contributor license agreements. See the NOTICE file +:: distributed with this work for additional information +:: regarding copyright ownership. The ASF licenses this file +:: to you under the Apache License, Version 2.0 (the +:: "License"); you may not use this file except in compliance +:: with the License. You may obtain a copy of the License at +:: +:: http://www.apache.org/licenses/LICENSE-2.0 +:: +:: Unless required by applicable law or agreed to in writing, software +:: distributed under the License is distributed on an "AS IS" BASIS, +:: WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +:: See the License for the specific language governing permissions and +:: limitations under the License. +::### + +@echo off +setlocal EnableDelayedExpansion + +SET bin=%~dp0 +SET FLINK_ROOT_DIR=%bin%.. +SET FLINK_LIB_DIR=%FLINK_ROOT_DIR%\lib +SET FLINK_CONF_DIR=%FLINK_ROOT_DIR%\conf +SET FLINK_LOG_DIR=%FLINK_ROOT_DIR%\log + +SET JVM_ARGS=-Xms1024m -Xmx1024m + +SET FLINK_CLASSPATH=%FLINK_LIB_DIR%\* + +SET logname_jm=flink-%username%-jobmanager-%computername%.log +SET logname_tm=flink-%username%-taskmanager-%computername%.log +SET log_jm=%FLINK_LOG_DIR%\%logname_jm% +SET log_tm=%FLINK_LOG_DIR%\%logname_tm% +SET outname_jm=flink-%username%-jobmanager-%computername%.out +SET outname_tm=flink-%username%-taskmanager-%computername%.out +SET out_jm=%FLINK_LOG_DIR%\%outname_jm% +SET out_tm=%FLINK_LOG_DIR%\%outname_tm% + +SET log_setting_jm=-Dlog.file="%log_jm%" -Dlogback.configurationFile=file:"%FLINK_CONF_DIR%/logback.xml" -Dlog4j.configuration=file:"%FLINK_CONF_DIR%/log4j.properties" +SET log_setting_tm=-Dlog.file="%log_tm%" -Dlogback.configurationFile=file:"%FLINK_CONF_DIR%/logback.xml" -Dlog4j.configuration=file:"%FLINK_CONF_DIR%/log4j.properties" + +:: Log rotation (quick and dirty) +CD "%FLINK_LOG_DIR%" +for /l %%x in (5, -1, 1) do ( +SET /A y = %%x+1 +RENAME "%logname_jm%.%%x" "%logname_jm%.!y!" 2> nul +RENAME "%logname_tm%.%%x" "%logname_tm%.!y!" 2> nul +RENAME "%outname_jm%.%%x" "%outname_jm%.!y!" 2> nul +RENAME "%outname_tm%.%%x" "%outname_tm%.!y!" 2> nul +) +RENAME "%logname_jm%" "%logname_jm%.0" 2> nul +RENAME "%logname_tm%" "%logname_tm%.0" 2> nul +RENAME "%outname_jm%" "%outname_jm%.0" 2> nul +RENAME "%outname_tm%" "%outname_tm%.0" 2> nul +DEL "%logname_jm%.6" 2> nul +DEL "%logname_tm%.6" 2> nul +DEL "%outname_jm%.6" 2> nul +DEL "%outname_tm%.6" 2> nul + +for %%X in (java.exe) do (set FOUND=%%~$PATH:X) +if not defined FOUND ( +echo java.exe was not found in PATH variable +goto :eof +) + +echo Starting a local cluster with one JobManager process and one TaskManager process. + +echo You can terminate the processes via CTRL-C in the spawned shell windows. + +echo Web interface by default on http://localhost:8081/. --- End diff -- This should be updated to `9065`. ---
[GitHub] flink pull request #5528: [FLINK-6489] [FLINK-8696] [shell scripts] Remove J...
GitHub user StephanEwen opened a pull request: https://github.com/apache/flink/pull/5528 [FLINK-6489] [FLINK-8696] [shell scripts] Remove JobManager local mode from the Shell Scripts ## What is the purpose of the change The JobManager local mode is problematic: - The JobManager does not start all components properly, making some operations fail/timeout unexpectedly - It introduces code complexity in maintaining the JobManager entry point that also spawns an embedded TaskManager The JobManager local mode is also unnecessary. The `start-cluster.sh` script sets up a proper local cluster without additional configuration. This pull request removes the local mode from all UNIX scripts and changes the Windows `start-local.bat` to a `start-cluster.bat` that also starts two separate processes for JobManager and TaskManager. ## Brief change log - Changes the `start-local.bat` to a `start-cluster.bat` which spawns two processes (using `start` command). The two processes show up as separate windows and can be stopped separately. - Removes the `start-local-sh` and `stop-local.sh` scripts. - Removes the passing of the *local* parameter between the shell scripts and always passes *cluster* as the execution mode parameter to the JobManager. The next step would be to remove the execution mode from the JobManagers command line argument parsing and setup logic. ## Verifying this change - Building Flink, starting a cluster via the shell scripts (Linux, MacOS, Windows) and checking the WebUI that the cluster is properly up. No actual job execution needed. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no)** - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no)** - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no)** - If yes, how is the feature documented? (not applicable / **docs** / JavaDocs / not documented) Docs are updated to not refer to `start-local.xy` any more. You can merge this pull request into a Git repository by running: $ git pull https://github.com/StephanEwen/incubator-flink remove_local_mode Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/5528.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5528 commit 99d564928f8299d2ad868ed5c23a9c88d8899a47 Author: Stephan Ewen Date: 2018-02-16T20:14:41Z [FLINK-6489] [shell scripts] Remove local mode from Windows start-local.bat Instead, this uses 'start' to start JobManager and TaskManager background processes. commit f142deebd951a498f2247b2ce243c15b19a81e84 Author: Stephan Ewen Date: 2018-02-16T20:15:49Z [FLINK-8696] [shell scripts] Remove JobManager local mode from shell scripts commit 11e87c4e2e879afb753354660ce3e33fb38a60d8 Author: Stephan Ewen Date: 2018-02-19T18:13:12Z [FLINK-6489] [FLINK-8696] [docs] Update docs to use 'start-cluster.sh' instead of 'start-local.sh' (and likewise for 'start-cluster.bat' vs. 'start-local.bat') ---
