[GitHub] flink pull request #5745: [FLINK-8976] [test] Add end-to-end test for resumi...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5745 ---
[GitHub] flink pull request #5745: [FLINK-8976] [test] Add end-to-end test for resumi...
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5745#discussion_r176678588 --- Diff: flink-end-to-end-tests/test-scripts/test_resume_savepoint.sh --- @@ -17,11 +17,25 @@ # limitations under the License. +if [ -z $1 ] || [ -z $2 ]; then + echo "Usage: ./test_resume_savepoint.sh " + exit 1 +fi + source "$(dirname "$0")"/common.sh -# modify configuration to have 2 slots +ORIGINAL_DOP=$1 +NEW_DOP=$2 + +if (( $ORIGINAL_DOP >= $NEW_DOP )); then + NUM_SLOTS=$(( $ORIGINAL_DOP + 1 )) --- End diff -- ð Yes, it is for the Kafka event generator job. ---
[GitHub] flink pull request #5745: [FLINK-8976] [test] Add end-to-end test for resumi...
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5745#discussion_r176678540 --- Diff: flink-end-to-end-tests/test-scripts/test_resume_savepoint.sh --- @@ -17,11 +17,25 @@ # limitations under the License. +if [ -z $1 ] || [ -z $2 ]; then + echo "Usage: ./test_resume_savepoint.sh " + exit 1 +fi + source "$(dirname "$0")"/common.sh -# modify configuration to have 2 slots +ORIGINAL_DOP=$1 +NEW_DOP=$2 + +if (( $ORIGINAL_DOP >= $NEW_DOP )); then + NUM_SLOTS=$(( $ORIGINAL_DOP + 1 )) +else + NUM_SLOTS=$(( $NEW_DOP + 1 )) +fi + +# modify configuration to have enough slots cp $FLINK_DIR/conf/flink-conf.yaml $FLINK_DIR/conf/flink-conf.yaml.bak -sed -i -e 's/taskmanager.numberOfTaskSlots: 1/taskmanager.numberOfTaskSlots: 2/' $FLINK_DIR/conf/flink-conf.yaml +sed -i -e "s/taskmanager.numberOfTaskSlots: 1/taskmanager.numberOfTaskSlots: $NUM_SLOTS/" $FLINK_DIR/conf/flink-conf.yaml # modify configuration to use SLF4J reporter; we will be using this to monitor the state machine progress cp $FLINK_DIR/opt/flink-metrics-slf4j-1.6-SNAPSHOT.jar $FLINK_DIR/lib/ --- End diff -- That makes sense, will fix this. ---
[GitHub] flink pull request #5745: [FLINK-8976] [test] Add end-to-end test for resumi...
Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/5745#discussion_r176677259 --- Diff: flink-end-to-end-tests/test-scripts/test_resume_savepoint.sh --- @@ -17,11 +17,25 @@ # limitations under the License. +if [ -z $1 ] || [ -z $2 ]; then + echo "Usage: ./test_resume_savepoint.sh " + exit 1 +fi + source "$(dirname "$0")"/common.sh -# modify configuration to have 2 slots +ORIGINAL_DOP=$1 +NEW_DOP=$2 + +if (( $ORIGINAL_DOP >= $NEW_DOP )); then + NUM_SLOTS=$(( $ORIGINAL_DOP + 1 )) +else + NUM_SLOTS=$(( $NEW_DOP + 1 )) +fi + +# modify configuration to have enough slots cp $FLINK_DIR/conf/flink-conf.yaml $FLINK_DIR/conf/flink-conf.yaml.bak -sed -i -e 's/taskmanager.numberOfTaskSlots: 1/taskmanager.numberOfTaskSlots: 2/' $FLINK_DIR/conf/flink-conf.yaml +sed -i -e "s/taskmanager.numberOfTaskSlots: 1/taskmanager.numberOfTaskSlots: $NUM_SLOTS/" $FLINK_DIR/conf/flink-conf.yaml # modify configuration to use SLF4J reporter; we will be using this to monitor the state machine progress cp $FLINK_DIR/opt/flink-metrics-slf4j-1.6-SNAPSHOT.jar $FLINK_DIR/lib/ --- End diff -- can we define the version as an environment variable in the common.sh? ---
[GitHub] flink pull request #5745: [FLINK-8976] [test] Add end-to-end test for resumi...
Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/5745#discussion_r176677065 --- Diff: flink-end-to-end-tests/test-scripts/test_resume_savepoint.sh --- @@ -17,11 +17,25 @@ # limitations under the License. +if [ -z $1 ] || [ -z $2 ]; then + echo "Usage: ./test_resume_savepoint.sh " + exit 1 +fi + source "$(dirname "$0")"/common.sh -# modify configuration to have 2 slots +ORIGINAL_DOP=$1 +NEW_DOP=$2 + +if (( $ORIGINAL_DOP >= $NEW_DOP )); then + NUM_SLOTS=$(( $ORIGINAL_DOP + 1 )) --- End diff -- would be good to explain where the +1 comes from. (i guess it is for the kafka job) ---
[GitHub] flink pull request #5745: [FLINK-8976] [test] Add end-to-end test for resumi...
GitHub user tzulitai opened a pull request: https://github.com/apache/flink/pull/5745 [FLINK-8976] [test] Add end-to-end test for resuming savepoints with different parallelism ## What is the purpose of the change This PR adds end-to-end tests for resuming a savepoint with different parallelisms. ## Brief change log The changes are based on the new `test_resume_savepoint.sh` test script in #5733, so only the last commit is relevant. That script has been modified to be able to specify parallelism of the state machine job, before and after the savepoint restore. - Adapt `test_resume_savepoint.sh` to be able to use different parallelism after the savepoint restore. - Add scale up / scale down tests to be executed by Travis. ## Verifying this change This PR adds new tests. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) You can merge this pull request into a Git repository by running: $ git pull https://github.com/tzulitai/flink FLINK-8976 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/5745.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5745 commit 529e060cb05fd723b8656dcc9ef48f8011282dd8 Author: Tzu-Li (Gordon) TaiDate: 2018-03-21T08:25:37Z [FLINK-8975] [test] Add Kafka events generator job for StateMachineExample commit 6b5126c006c752c6c1bee2699d429927e74587c9 Author: Tzu-Li (Gordon) Tai Date: 2018-03-21T08:32:51Z [FLINK-8975] [test] Add resume from savepoint end-to-end test commit 475ef4de0f8abbd3bb485e0e6fefcce2612074d3 Author: Tzu-Li (Gordon) Tai Date: 2018-03-22T07:34:49Z fixup! Use SLF4J reporter to monitor state machine progress commit 55d69567bc4cb61cd08ad852126c9c2c39b8feb4 Author: Tzu-Li (Gordon) Tai Date: 2018-03-22T09:09:53Z [FLINK-8976] [test] Add end-to-end tests for resuming savepoint with differrent parallelism ---