[jira] [Commented] (FLINK-8977) End-to-end test: Manually resume job after terminal failure

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16483661#comment-16483661
 ] 

ASF GitHub Bot commented on FLINK-8977:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/6004


> End-to-end test: Manually resume job after terminal failure
> ---
>
> Key: FLINK-8977
> URL: https://issues.apache.org/jira/browse/FLINK-8977
> Project: Flink
>  Issue Type: Sub-task
>  Components: Tests
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Blocker
> Fix For: 1.6.0, 1.5.1
>
>
> We should add an end-to-end test which verifies that a job can be resumed 
> manually after a terminal job failure if there is a checkpoint. In order to 
> do that we should 
> # run the general purpose testing job FLINK-8971 
> # wait for the completion of a checkpoint
> # Trigger a failure which leads to a terminal failure
> # Resume the job from the retained checkpoint
> This end-to-end test should run with all state backend combinations: RocksDB 
> (incremental/full, async/sync), FsStateBackend (async/sync).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8977) End-to-end test: Manually resume job after terminal failure

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473749#comment-16473749
 ] 

ASF GitHub Bot commented on FLINK-8977:
---

GitHub user tzulitai opened a pull request:

https://github.com/apache/flink/pull/6004

[FLINK-8977] [e2e] End-to-end test for manual job resume after terminal 
failure

## What is the purpose of the change

This PR is based on new e2e features introduced by #5941, #5990, and #5969.
Only the last two commits are relevant to FLINK-8977.

This PR adds e2e test coverage for the case that after a terminal failure 
caused by the user job code, manually resuming from a retained checkpoint works 
correctly.

This is achieved by extending the `test_resume_externalized_checkpoints.sh` 
test script to accept a `SIMULATE_FAILURE` flag.

## Brief change log

- 9360ea9 Extend the general purpose DataStream job to allow configuring 
restart strategies.
- b5d713c Extend `test_resume_externalized_checkpoints.sh` to allow 
simulating the job failure + manual resume case.

## Verifying this change

Verifiable by running locally the following e2e test script:
`SIMULATE_FAILURE=true 
flink-end-to-end-tests/test-scripts/test_resume_externalized_checkpoints.sh`

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (yes / **no**)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (yes / **no**)
  - The serializers: (yes / **no** / don't know)
  - The runtime per-record code paths (performance sensitive): (yes / 
**no** / don't know)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
  - The S3 file system connector: (yes / **no** / don't know)

## Documentation

  - Does this pull request introduce a new feature? (yes / **no**)
  - If yes, how is the feature documented? (**not applicable** / docs / 
JavaDocs / not documented)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tzulitai/flink FLINK-8977

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/6004.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #6004


commit 8db7f894b67b00f94148e0314a1c10d76266a350
Author: Tzu-Li (Gordon) Tai 
Date:   2018-04-30T10:04:43Z

[hotfix] [e2e-tests] Make SequenceGeneratorSource usable for 0-size key 
ranges

commit c8e14673e58aed0f9625e38875ec85a776282ad4
Author: Tzu-Li (Gordon) Tai 
Date:   2018-04-30T10:05:46Z

[FLINK-8971] [e2e-tests] Include broadcast / union state in general purpose 
DataStream job

commit 78354b295832fa2ec5d829ec4ac21150ecac1231
Author: Tzu-Li (Gordon) Tai 
Date:   2018-05-08T03:44:13Z

PR review - refactor source run function

commit f346fd0958e7c3361886680912630fe22761a63d
Author: Tzu-Li (Gordon) Tai 
Date:   2018-05-08T04:39:40Z

PR review - simplify broadcast / union state verification

commit b01cfda7d77723e8ded2ce99ee12f17352a3ca1f
Author: Tzu-Li (Gordon) Tai 
Date:   2018-05-11T03:51:12Z

[FLINK-9322] [e2e] Add failure simulation to the general purpose DataStream 
job

commit 0931f6ed48523ca46e2c99adc24950777d843ac8
Author: Tzu-Li (Gordon) Tai 
Date:   2018-05-11T07:09:00Z

[FLINK-9320] [e2e] Update test_ha e2e to use general purpose DataStream job

commit a819e56e0998e09bb6461b6c76be0807d83a1ef5
Author: Tzu-Li (Gordon) Tai 
Date:   2018-05-09T03:40:24Z

[FLINK-9074] [e2e] Allow configuring externalized checkpoints for the 
general purpose DataStream job

commit 3d0c83a991ab78d03c3cc1c9ff2abb61e0329d9d
Author: Tzu-Li (Gordon) Tai 
Date:   2018-05-09T04:17:25Z

[FLINK-9074] [e2e] Add e2e test for resuming jobs from retained checkpoints

commit 9360ea9ad0db858e7fdeecb54b1918e6b84cae1d
Author: Tzu-Li (Gordon) Tai 
Date:   2018-05-14T03:56:07Z

[FLINK-8977] [e2e] Allow configuring restart strategy for general purpose 
DataStream job

commit b5d713cf19290be437286e152d921b23ff532c7d
Author: Tzu-Li (Gordon) Tai 
Date:   2018-05-14T03:56:41Z

[FLINK-8977] [e2e] Extend externalized checkpoint e2e to simulate job 
failures




> End-to-end test: Manually resume job after terminal failure
> ---
>
> Key: FLINK-8977
> URL: https://issues.apache.org/jira/browse/FLINK-8977
> Project: Flink
>  Issue Type: Sub-task
>  Components: Tests
>