Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2618#discussion_r83022623
--- Diff:
flink-tests/src/test/java/org/apache/flink/test/checkpointing/ContinuousFileProcessingCheckpointITCase.java
---
@@ -257,74 +190,158 @@ public void open(Configuration parameters) throws
Exception {
long failurePosMax = (long) (0.7 * LINES_PER_FILE);
elementsToFailure = (new Random().nextLong() %
(failurePosMax - failurePosMin)) + failurePosMin;
-
- if (elementCounter >= NO_OF_FILES * LINES_PER_FILE) {
- finalCollectedContent = new HashMap<>();
- for (Map.Entry<Integer, Set<String>> result:
collectedContent.entrySet()) {
-
finalCollectedContent.put(result.getKey(), new ArrayList<>(result.getValue()));
- }
- throw new SuccessException();
- }
- }
-
- @Override
- public void close() {
- try {
- super.close();
- } catch (Exception e) {
- e.printStackTrace();
- }
}
@Override
public void invoke(String value) throws Exception {
- int fileIdx =
Character.getNumericValue(value.charAt(0));
+ int fileIdx = getFileIdx(value);
- Set<String> content = collectedContent.get(fileIdx);
+ Set<String> content = actualContent.get(fileIdx);
if (content == null) {
content = new HashSet<>();
- collectedContent.put(fileIdx, content);
+ actualContent.put(fileIdx, content);
}
+ // detect duplicate lines.
if (!content.add(value + "\n")) {
fail("Duplicate line: " + value);
System.exit(0);
}
-
elementCounter++;
+
+ // this is termination
if (elementCounter >= NO_OF_FILES * LINES_PER_FILE) {
- finalCollectedContent = new HashMap<>();
- for (Map.Entry<Integer, Set<String>> result:
collectedContent.entrySet()) {
-
finalCollectedContent.put(result.getKey(), new ArrayList<>(result.getValue()));
- }
+ actualCollectedContent = actualContent;
throw new SuccessException();
}
- count++;
- if (!hasFailed) {
+ // add some latency so that we have at least one
checkpoint in
+ if (!hasFailed && successfulCheckpoints == 0) {
Thread.sleep(2);
- if (numSuccessfulCheckpoints >= 1 && count >=
elementsToFailure) {
- hasFailed = true;
- throw new Exception("Task Failure");
- }
+ }
+
+ // simulate a node failure
+ if (!hasFailed && successfulCheckpoints > 0 &&
elementCounter >= elementsToFailure) {
--- End diff --
Is it assured that a checkpoint has been taken place? Or is the exception
simply skipped here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
