[jira] [Commented] (FLINK-4000) Exception: Could not restore checkpointed state to operators and functions; during Job Restart (Job restart is triggered due to one of the task manager failure)
[ https://issues.apache.org/jira/browse/FLINK-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15320527#comment-15320527 ] ASF GitHub Bot commented on FLINK-4000: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/2062 > Exception: Could not restore checkpointed state to operators and functions; > during Job Restart (Job restart is triggered due to one of the task manager > failure) > - > > Key: FLINK-4000 > URL: https://issues.apache.org/jira/browse/FLINK-4000 > Project: Flink > Issue Type: Bug > Components: DataStream API >Affects Versions: 1.0.3 > Environment: //Fault Tolerance Configuration of the Job > env.enableCheckpointing(5000); > env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE); > env.getCheckpointConfig().setMaxConcurrentCheckpoints(1); > env.setRestartStrategy(RestartStrategies.fixedDelayRestart( 3,1)); >Reporter: Aride Chettali > > java.lang.Exception: Could not restore checkpointed state to operators and > functions > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:457) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:209) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.Exception: Failed to restore state to function: null > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:168) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:449) > ... 3 more > Caused by: java.lang.NullPointerException > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:184) > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:80) > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:165) > ... 4 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4000) Exception: Could not restore checkpointed state to operators and functions; during Job Restart (Job restart is triggered due to one of the task manager failure)
[ https://issues.apache.org/jira/browse/FLINK-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318783#comment-15318783 ] ASF GitHub Bot commented on FLINK-4000: --- Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2062 Thanks, this is a critical fix. +1 Merging this... > Exception: Could not restore checkpointed state to operators and functions; > during Job Restart (Job restart is triggered due to one of the task manager > failure) > - > > Key: FLINK-4000 > URL: https://issues.apache.org/jira/browse/FLINK-4000 > Project: Flink > Issue Type: Bug > Components: DataStream API >Affects Versions: 1.0.3 > Environment: //Fault Tolerance Configuration of the Job > env.enableCheckpointing(5000); > env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE); > env.getCheckpointConfig().setMaxConcurrentCheckpoints(1); > env.setRestartStrategy(RestartStrategies.fixedDelayRestart( 3,1)); >Reporter: Aride Chettali > > java.lang.Exception: Could not restore checkpointed state to operators and > functions > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:457) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:209) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.Exception: Failed to restore state to function: null > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:168) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:449) > ... 3 more > Caused by: java.lang.NullPointerException > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:184) > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:80) > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:165) > ... 4 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4000) Exception: Could not restore checkpointed state to operators and functions; during Job Restart (Job restart is triggered due to one of the task manager failure)
[ https://issues.apache.org/jira/browse/FLINK-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15311404#comment-15311404 ] ASF GitHub Bot commented on FLINK-4000: --- Github user rekhajoshm closed the pull request at: https://github.com/apache/flink/pull/2061 > Exception: Could not restore checkpointed state to operators and functions; > during Job Restart (Job restart is triggered due to one of the task manager > failure) > - > > Key: FLINK-4000 > URL: https://issues.apache.org/jira/browse/FLINK-4000 > Project: Flink > Issue Type: Bug > Components: DataStream API >Affects Versions: 1.0.3 > Environment: //Fault Tolerance Configuration of the Job > env.enableCheckpointing(5000); > env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE); > env.getCheckpointConfig().setMaxConcurrentCheckpoints(1); > env.setRestartStrategy(RestartStrategies.fixedDelayRestart( 3,1)); >Reporter: Aride Chettali >Assignee: Rekha Joshi > > java.lang.Exception: Could not restore checkpointed state to operators and > functions > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:457) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:209) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.Exception: Failed to restore state to function: null > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:168) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:449) > ... 3 more > Caused by: java.lang.NullPointerException > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:184) > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:80) > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:165) > ... 4 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4000) Exception: Could not restore checkpointed state to operators and functions; during Job Restart (Job restart is triggered due to one of the task manager failure)
[ https://issues.apache.org/jira/browse/FLINK-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15311403#comment-15311403 ] ASF GitHub Bot commented on FLINK-4000: --- Github user rekhajoshm commented on the issue: https://github.com/apache/flink/pull/2061 aha, in one of my intermediate runs had done just initializing idsProcessedButNotAcknowledged and retain pendingCheckpoints , but in last run changed it calling open() :-( . makes sense @asavartsov. closing. thanks > Exception: Could not restore checkpointed state to operators and functions; > during Job Restart (Job restart is triggered due to one of the task manager > failure) > - > > Key: FLINK-4000 > URL: https://issues.apache.org/jira/browse/FLINK-4000 > Project: Flink > Issue Type: Bug > Components: DataStream API >Affects Versions: 1.0.3 > Environment: //Fault Tolerance Configuration of the Job > env.enableCheckpointing(5000); > env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE); > env.getCheckpointConfig().setMaxConcurrentCheckpoints(1); > env.setRestartStrategy(RestartStrategies.fixedDelayRestart( 3,1)); >Reporter: Aride Chettali >Assignee: Rekha Joshi > > java.lang.Exception: Could not restore checkpointed state to operators and > functions > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:457) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:209) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.Exception: Failed to restore state to function: null > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:168) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:449) > ... 3 more > Caused by: java.lang.NullPointerException > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:184) > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:80) > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:165) > ... 4 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4000) Exception: Could not restore checkpointed state to operators and functions; during Job Restart (Job restart is triggered due to one of the task manager failure)
[ https://issues.apache.org/jira/browse/FLINK-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15311395#comment-15311395 ] ASF GitHub Bot commented on FLINK-4000: --- Github user asavartsov commented on the issue: https://github.com/apache/flink/pull/2061 Take a look at my pull request at https://github.com/apache/flink/pull/2062 > Exception: Could not restore checkpointed state to operators and functions; > during Job Restart (Job restart is triggered due to one of the task manager > failure) > - > > Key: FLINK-4000 > URL: https://issues.apache.org/jira/browse/FLINK-4000 > Project: Flink > Issue Type: Bug > Components: DataStream API >Affects Versions: 1.0.3 > Environment: //Fault Tolerance Configuration of the Job > env.enableCheckpointing(5000); > env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE); > env.getCheckpointConfig().setMaxConcurrentCheckpoints(1); > env.setRestartStrategy(RestartStrategies.fixedDelayRestart( 3,1)); >Reporter: Aride Chettali >Assignee: Rekha Joshi > > java.lang.Exception: Could not restore checkpointed state to operators and > functions > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:457) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:209) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.Exception: Failed to restore state to function: null > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:168) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:449) > ... 3 more > Caused by: java.lang.NullPointerException > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:184) > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:80) > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:165) > ... 4 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4000) Exception: Could not restore checkpointed state to operators and functions; during Job Restart (Job restart is triggered due to one of the task manager failure)
[ https://issues.apache.org/jira/browse/FLINK-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15311375#comment-15311375 ] ASF GitHub Bot commented on FLINK-4000: --- Github user rekhajoshm commented on the issue: https://github.com/apache/flink/pull/2061 @asavartsov Ok. Please let me know how you propose to solve this? thanks > Exception: Could not restore checkpointed state to operators and functions; > during Job Restart (Job restart is triggered due to one of the task manager > failure) > - > > Key: FLINK-4000 > URL: https://issues.apache.org/jira/browse/FLINK-4000 > Project: Flink > Issue Type: Bug > Components: DataStream API >Affects Versions: 1.0.3 > Environment: //Fault Tolerance Configuration of the Job > env.enableCheckpointing(5000); > env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE); > env.getCheckpointConfig().setMaxConcurrentCheckpoints(1); > env.setRestartStrategy(RestartStrategies.fixedDelayRestart( 3,1)); >Reporter: Aride Chettali >Assignee: Rekha Joshi > > java.lang.Exception: Could not restore checkpointed state to operators and > functions > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:457) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:209) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.Exception: Failed to restore state to function: null > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:168) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:449) > ... 3 more > Caused by: java.lang.NullPointerException > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:184) > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:80) > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:165) > ... 4 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4000) Exception: Could not restore checkpointed state to operators and functions; during Job Restart (Job restart is triggered due to one of the task manager failure)
[ https://issues.apache.org/jira/browse/FLINK-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15311362#comment-15311362 ] ASF GitHub Bot commented on FLINK-4000: --- Github user asavartsov commented on the issue: https://github.com/apache/flink/pull/2061 No, it does not make any sense and even makes things worse, sorry. > Exception: Could not restore checkpointed state to operators and functions; > during Job Restart (Job restart is triggered due to one of the task manager > failure) > - > > Key: FLINK-4000 > URL: https://issues.apache.org/jira/browse/FLINK-4000 > Project: Flink > Issue Type: Bug > Components: DataStream API >Affects Versions: 1.0.3 > Environment: //Fault Tolerance Configuration of the Job > env.enableCheckpointing(5000); > env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE); > env.getCheckpointConfig().setMaxConcurrentCheckpoints(1); > env.setRestartStrategy(RestartStrategies.fixedDelayRestart( 3,1)); >Reporter: Aride Chettali >Assignee: Rekha Joshi > > java.lang.Exception: Could not restore checkpointed state to operators and > functions > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:457) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:209) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.Exception: Failed to restore state to function: null > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:168) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:449) > ... 3 more > Caused by: java.lang.NullPointerException > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:184) > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:80) > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:165) > ... 4 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4000) Exception: Could not restore checkpointed state to operators and functions; during Job Restart (Job restart is triggered due to one of the task manager failure)
[ https://issues.apache.org/jira/browse/FLINK-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15311356#comment-15311356 ] ASF GitHub Bot commented on FLINK-4000: --- GitHub user asavartsov opened a pull request: https://github.com/apache/flink/pull/2062 [FLINK-4000] Fix for checkpoint state restore at MessageAcknowledgingSourceBase As says documentation for MessageAcknowledgingSourceBase.restoreState() This method is invoked when a function is executed as part of a recovery run. Note that restoreState() is called before open(). So current implementation 1. Fails on restoreState with NullPointerException, jobs fail to restart. 2. Does not restore anything because following open erases all checkpoint data immediately. 3. As consequence, violates exactly once rule because processed but not acknowledged list erased. Proposed change fixes that. You can merge this pull request into a Git repository by running: $ git pull https://github.com/asavartsov/flink FLINK-4000 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2062.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2062 commit 58fb7df0b2d1dd68a90f4088fbde10874eb290b6 Author: Alexey SavartsovDate: 2016-06-01T23:23:53Z [FLINK-4000] Fix for checkpoint state restore at MessageAcknowledgingSourceBase As says documentation for MessageAcknowledgingSourceBase.restoreState() This method is invoked when a function is executed as part of a recovery run. Note that restoreState() is called before open(). So current implementation 1. Fails on restoreState with NullPointerException, jobs fail to restart. 2. Does not restore anything because following open erases all checkpoint data immediately. 3. As consequence, violates exactly once rule because processed but not acknowledged list erased. Proposed change fixes that. > Exception: Could not restore checkpointed state to operators and functions; > during Job Restart (Job restart is triggered due to one of the task manager > failure) > - > > Key: FLINK-4000 > URL: https://issues.apache.org/jira/browse/FLINK-4000 > Project: Flink > Issue Type: Bug > Components: DataStream API >Affects Versions: 1.0.3 > Environment: //Fault Tolerance Configuration of the Job > env.enableCheckpointing(5000); > env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE); > env.getCheckpointConfig().setMaxConcurrentCheckpoints(1); > env.setRestartStrategy(RestartStrategies.fixedDelayRestart( 3,1)); >Reporter: Aride Chettali >Assignee: Rekha Joshi > > java.lang.Exception: Could not restore checkpointed state to operators and > functions > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:457) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:209) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.Exception: Failed to restore state to function: null > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:168) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:449) > ... 3 more > Caused by: java.lang.NullPointerException > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:184) > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:80) > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:165) > ... 4 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4000) Exception: Could not restore checkpointed state to operators and functions; during Job Restart (Job restart is triggered due to one of the task manager failure)
[ https://issues.apache.org/jira/browse/FLINK-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15311206#comment-15311206 ] ASF GitHub Bot commented on FLINK-4000: --- Github user rekhajoshm commented on the issue: https://github.com/apache/flink/pull/2061 I agree @asavartsov , that was a quick look and was working to reproduce.Does the updated make sense? thank you. > Exception: Could not restore checkpointed state to operators and functions; > during Job Restart (Job restart is triggered due to one of the task manager > failure) > - > > Key: FLINK-4000 > URL: https://issues.apache.org/jira/browse/FLINK-4000 > Project: Flink > Issue Type: Bug > Components: DataStream API >Affects Versions: 1.0.3 > Environment: //Fault Tolerance Configuration of the Job > env.enableCheckpointing(5000); > env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE); > env.getCheckpointConfig().setMaxConcurrentCheckpoints(1); > env.setRestartStrategy(RestartStrategies.fixedDelayRestart( 3,1)); >Reporter: Aride Chettali >Assignee: Rekha Joshi > > java.lang.Exception: Could not restore checkpointed state to operators and > functions > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:457) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:209) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.Exception: Failed to restore state to function: null > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:168) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:449) > ... 3 more > Caused by: java.lang.NullPointerException > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:184) > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:80) > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:165) > ... 4 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4000) Exception: Could not restore checkpointed state to operators and functions; during Job Restart (Job restart is triggered due to one of the task manager failure)
[ https://issues.apache.org/jira/browse/FLINK-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15311152#comment-15311152 ] ASF GitHub Bot commented on FLINK-4000: --- Github user asavartsov commented on the issue: https://github.com/apache/flink/pull/2061 This kind of check is might be useless and probably wouldn't fix the issue. My debugging shows that the list `idsProcessedButNotAcknowledged` is null on recovery, not checkpoints itself. This list is initialized in `open` method, but somehow it doesn't get called in such scenario. > Exception: Could not restore checkpointed state to operators and functions; > during Job Restart (Job restart is triggered due to one of the task manager > failure) > - > > Key: FLINK-4000 > URL: https://issues.apache.org/jira/browse/FLINK-4000 > Project: Flink > Issue Type: Bug > Components: DataStream API >Affects Versions: 1.0.3 > Environment: //Fault Tolerance Configuration of the Job > env.enableCheckpointing(5000); > env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE); > env.getCheckpointConfig().setMaxConcurrentCheckpoints(1); > env.setRestartStrategy(RestartStrategies.fixedDelayRestart( 3,1)); >Reporter: Aride Chettali >Assignee: Rekha Joshi > > java.lang.Exception: Could not restore checkpointed state to operators and > functions > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:457) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:209) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.Exception: Failed to restore state to function: null > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:168) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:449) > ... 3 more > Caused by: java.lang.NullPointerException > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:184) > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:80) > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:165) > ... 4 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4000) Exception: Could not restore checkpointed state to operators and functions; during Job Restart (Job restart is triggered due to one of the task manager failure)
[ https://issues.apache.org/jira/browse/FLINK-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15311087#comment-15311087 ] ASF GitHub Bot commented on FLINK-4000: --- GitHub user rekhajoshm opened a pull request: https://github.com/apache/flink/pull/2061 [FLINK-4000] Checkpoint dictionaries null after taskmgr failures Fix for exception during job restart after task mgr failure, at which point restoreState fails as checkpoint dictionaries can be null. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rekhajoshm/flink FLINK-4000 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2061.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2061 commit 1e1ce9efefccf5a5585be802af4200e9d1ae7a98 Author: Rekha JoshiDate: 2016-06-01T20:03:50Z Merge pull request #1 from apache/master Apache Flink master pull commit 88929dcd989c34c3232f29a96af1b32ec1315911 Author: Joshi Date: 2016-06-01T20:56:52Z [FLINK-4000] Checkpoint dictionaries null after taskmgr failures > Exception: Could not restore checkpointed state to operators and functions; > during Job Restart (Job restart is triggered due to one of the task manager > failure) > - > > Key: FLINK-4000 > URL: https://issues.apache.org/jira/browse/FLINK-4000 > Project: Flink > Issue Type: Bug > Components: DataStream API >Affects Versions: 1.0.3 > Environment: //Fault Tolerance Configuration of the Job > env.enableCheckpointing(5000); > env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE); > env.getCheckpointConfig().setMaxConcurrentCheckpoints(1); > env.setRestartStrategy(RestartStrategies.fixedDelayRestart( 3,1)); >Reporter: Aride Chettali >Assignee: Rekha Joshi > > java.lang.Exception: Could not restore checkpointed state to operators and > functions > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:457) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:209) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.Exception: Failed to restore state to function: null > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:168) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:449) > ... 3 more > Caused by: java.lang.NullPointerException > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:184) > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:80) > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:165) > ... 4 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4000) Exception: Could not restore checkpointed state to operators and functions; during Job Restart (Job restart is triggered due to one of the task manager failure)
[ https://issues.apache.org/jira/browse/FLINK-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15310328#comment-15310328 ] Alexey Savartsov commented on FLINK-4000: - I can confirm this issue with RMQSource, with zookeeper state too. Looks like restoreState on MessageAcknowledgingSourceBase is called before open, so dictionaries for checkpoints are null. > Exception: Could not restore checkpointed state to operators and functions; > during Job Restart (Job restart is triggered due to one of the task manager > failure) > - > > Key: FLINK-4000 > URL: https://issues.apache.org/jira/browse/FLINK-4000 > Project: Flink > Issue Type: Bug > Components: DataStream API >Affects Versions: 1.0.3 > Environment: //Fault Tolerance Configuration of the Job > env.enableCheckpointing(5000); > env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE); > env.getCheckpointConfig().setMaxConcurrentCheckpoints(1); > env.setRestartStrategy(RestartStrategies.fixedDelayRestart( 3,1)); >Reporter: Aride Chettali > > java.lang.Exception: Could not restore checkpointed state to operators and > functions > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:457) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:209) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.Exception: Failed to restore state to function: null > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:168) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:449) > ... 3 more > Caused by: java.lang.NullPointerException > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:184) > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:80) > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:165) > ... 4 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4000) Exception: Could not restore checkpointed state to operators and functions; during Job Restart (Job restart is triggered due to one of the task manager failure)
[ https://issues.apache.org/jira/browse/FLINK-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15310323#comment-15310323 ] Aride Chettali commented on FLINK-4000: --- Thank you for the quick response. Yes I am using the default state checkpoint (in-memory). Looks like the error is in source function,I am using Rabbit MQ source with correlationIds enabled. Below is the dependency I have org.apache.flink flink-connector-rabbitmq_2.10 1.0.3 > Exception: Could not restore checkpointed state to operators and functions; > during Job Restart (Job restart is triggered due to one of the task manager > failure) > - > > Key: FLINK-4000 > URL: https://issues.apache.org/jira/browse/FLINK-4000 > Project: Flink > Issue Type: Bug > Components: DataStream API >Affects Versions: 1.0.3 > Environment: //Fault Tolerance Configuration of the Job > env.enableCheckpointing(5000); > env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE); > env.getCheckpointConfig().setMaxConcurrentCheckpoints(1); > env.setRestartStrategy(RestartStrategies.fixedDelayRestart( 3,1)); >Reporter: Aride Chettali > > java.lang.Exception: Could not restore checkpointed state to operators and > functions > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:457) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:209) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.Exception: Failed to restore state to function: null > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:168) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:449) > ... 3 more > Caused by: java.lang.NullPointerException > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:184) > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:80) > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:165) > ... 4 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4000) Exception: Could not restore checkpointed state to operators and functions; during Job Restart (Job restart is triggered due to one of the task manager failure)
[ https://issues.apache.org/jira/browse/FLINK-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15310251#comment-15310251 ] Aride Chettali commented on FLINK-4000: --- I was using the default state check-points (in-memory). And I have figured out the issue. The issue was user function did not implement the "Checkpointed" interface. Once I implemented the methods snapshotState() and restoreState() then my job got restarted successfully. Thanks for your help.. > Exception: Could not restore checkpointed state to operators and functions; > during Job Restart (Job restart is triggered due to one of the task manager > failure) > - > > Key: FLINK-4000 > URL: https://issues.apache.org/jira/browse/FLINK-4000 > Project: Flink > Issue Type: Bug > Components: DataStream API >Affects Versions: 1.0.3 > Environment: //Fault Tolerance Configuration of the Job > env.enableCheckpointing(5000); > env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE); > env.getCheckpointConfig().setMaxConcurrentCheckpoints(1); > env.setRestartStrategy(RestartStrategies.fixedDelayRestart( 3,1)); >Reporter: Aride Chettali > > java.lang.Exception: Could not restore checkpointed state to operators and > functions > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:457) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:209) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.Exception: Failed to restore state to function: null > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:168) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:449) > ... 3 more > Caused by: java.lang.NullPointerException > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:184) > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:80) > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:165) > ... 4 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4000) Exception: Could not restore checkpointed state to operators and functions; during Job Restart (Job restart is triggered due to one of the task manager failure)
[ https://issues.apache.org/jira/browse/FLINK-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15310250#comment-15310250 ] Aride Chettali commented on FLINK-4000: --- I was using the default state check-points (in-memory). And I have figured out the issue. The issue was user function did not implement the "Checkpointed" interface. Once I implemented the functions _snapshotState_ and _restoreState_ then my job got restarted successfully. Thanks for your help.. > Exception: Could not restore checkpointed state to operators and functions; > during Job Restart (Job restart is triggered due to one of the task manager > failure) > - > > Key: FLINK-4000 > URL: https://issues.apache.org/jira/browse/FLINK-4000 > Project: Flink > Issue Type: Bug > Components: DataStream API >Affects Versions: 1.0.3 > Environment: //Fault Tolerance Configuration of the Job > env.enableCheckpointing(5000); > env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE); > env.getCheckpointConfig().setMaxConcurrentCheckpoints(1); > env.setRestartStrategy(RestartStrategies.fixedDelayRestart( 3,1)); >Reporter: Aride Chettali > > java.lang.Exception: Could not restore checkpointed state to operators and > functions > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:457) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:209) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.Exception: Failed to restore state to function: null > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:168) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:449) > ... 3 more > Caused by: java.lang.NullPointerException > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:184) > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:80) > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:165) > ... 4 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4000) Exception: Could not restore checkpointed state to operators and functions; during Job Restart (Job restart is triggered due to one of the task manager failure)
[ https://issues.apache.org/jira/browse/FLINK-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15310007#comment-15310007 ] Ufuk Celebi commented on FLINK-4000: Thanks for reporting the issue. Did you configure a specific state backend or are you using the default one? > Exception: Could not restore checkpointed state to operators and functions; > during Job Restart (Job restart is triggered due to one of the task manager > failure) > - > > Key: FLINK-4000 > URL: https://issues.apache.org/jira/browse/FLINK-4000 > Project: Flink > Issue Type: Bug > Components: DataStream API >Affects Versions: 1.0.3 > Environment: //Fault Tolerance Configuration of the Job > env.enableCheckpointing(5000); > env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE); > env.getCheckpointConfig().setMaxConcurrentCheckpoints(1); > env.setRestartStrategy(RestartStrategies.fixedDelayRestart( 3,1)); >Reporter: Aride Chettali > > java.lang.Exception: Could not restore checkpointed state to operators and > functions > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:457) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:209) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.Exception: Failed to restore state to function: null > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:168) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:449) > ... 3 more > Caused by: java.lang.NullPointerException > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:184) > at > org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase.restoreState(MessageAcknowledgingSourceBase.java:80) > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:165) > ... 4 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)