[jira] [Commented] (FLINK-2221) Checkpoints to "file://" are not cleaned up
[ https://issues.apache.org/jira/browse/FLINK-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15571757#comment-15571757 ] Aljoscha Krettek commented on FLINK-2221: - +1 for "won't fix" ([~StephanEwen]) > Checkpoints to "file://" are not cleaned up > --- > > Key: FLINK-2221 > URL: https://issues.apache.org/jira/browse/FLINK-2221 > Project: Flink > Issue Type: Bug > Components: Streaming >Reporter: Aljoscha Krettek > > If you think about it, this could never work. The state handle cleanup logic > happens purely on the JobManager. So what happens is that the TaskManagers > create state in some directory, let's say /tmp/checkpoints, on the > TaskManager. For cleanup, the JobManager gets the state handle and calls > discard (on the JobManager), this tries to cleanup the state in > /tmp/checkpoints, but of course, there is nothing there since we are still on > the JobManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2221) Checkpoints to "file://" are not cleaned up
[ https://issues.apache.org/jira/browse/FLINK-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15569386#comment-15569386 ] Stephan Ewen commented on FLINK-2221: - I would like to close this as "won't fix", because it is expected. In a distributed system, checkpointing to locations that are only locally accessible cannot work. Supporting "file://" is still valid, because many NFS-like filesystems would be addressed via that scheme. > Checkpoints to "file://" are not cleaned up > --- > > Key: FLINK-2221 > URL: https://issues.apache.org/jira/browse/FLINK-2221 > Project: Flink > Issue Type: Bug > Components: Streaming >Reporter: Aljoscha Krettek > > If you think about it, this could never work. The state handle cleanup logic > happens purely on the JobManager. So what happens is that the TaskManagers > create state in some directory, let's say /tmp/checkpoints, on the > TaskManager. For cleanup, the JobManager gets the state handle and calls > discard (on the JobManager), this tries to cleanup the state in > /tmp/checkpoints, but of course, there is nothing there since we are still on > the JobManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2221) Checkpoints to "file://" are not cleaned up
[ https://issues.apache.org/jira/browse/FLINK-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14595044#comment-14595044 ] ASF GitHub Bot commented on FLINK-2221: --- Github user mbalassi commented on the pull request: https://github.com/apache/flink/pull/747#issuecomment-113898364 Thanks for the design outline guys, it looks great. Two minor comments in terms of implementation: * Let us make some more emphasis on the architecture of discarding or compaction old checkpoint data. This seems a minor issue, but we have seen during the recent release testing that it has implications. [1] In the current version the `JobManager` discards old state, so it needs access to it. If I understand it correctly that behavior is intact after this PR. * API: `KeyedDataStream` should be feasible if we really make it a more general version of data stream. The interplay with windowed and connected streams is interesting. The reason why e.g. a windowed and grouping fits nicely is that the windowed groups contain only elements from one group. You propose the same design for keyed streams as well? [1] https://issues.apache.org/jira/browse/FLINK-2221 > Checkpoints to "file://" are not cleaned up > --- > > Key: FLINK-2221 > URL: https://issues.apache.org/jira/browse/FLINK-2221 > Project: Flink > Issue Type: Bug > Components: Streaming >Reporter: Aljoscha Krettek > > If you think about it, this could never work. The state handle cleanup logic > happens purely on the JobManager. So what happens is that the TaskManagers > create state in some directory, let's say /tmp/checkpoints, on the > TaskManager. For cleanup, the JobManager gets the state handle and calls > discard (on the JobManager), this tries to cleanup the state in > /tmp/checkpoints, but of course, there is nothing there since we are still on > the JobManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2221) Checkpoints to "file://" are not cleaned up
[ https://issues.apache.org/jira/browse/FLINK-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587883#comment-14587883 ] Márton Balassi commented on FLINK-2221: --- A documentation for 0.9 has been pushed, the issue itself remains open. We should discuss whether this architecture is desirable. > Checkpoints to "file://" are not cleaned up > --- > > Key: FLINK-2221 > URL: https://issues.apache.org/jira/browse/FLINK-2221 > Project: Flink > Issue Type: Bug > Components: Streaming >Reporter: Aljoscha Krettek > > If you think about it, this could never work. The state handle cleanup logic > happens purely on the JobManager. So what happens is that the TaskManagers > create state in some directory, let's say /tmp/checkpoints, on the > TaskManager. For cleanup, the JobManager gets the state handle and calls > discard (on the JobManager), this tries to cleanup the state in > /tmp/checkpoints, but of course, there is nothing there since we are still on > the JobManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2221) Checkpoints to "file://" are not cleaned up
[ https://issues.apache.org/jira/browse/FLINK-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587872#comment-14587872 ] ASF GitHub Bot commented on FLINK-2221: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/839 > Checkpoints to "file://" are not cleaned up > --- > > Key: FLINK-2221 > URL: https://issues.apache.org/jira/browse/FLINK-2221 > Project: Flink > Issue Type: Bug > Components: Streaming >Reporter: Aljoscha Krettek > > If you think about it, this could never work. The state handle cleanup logic > happens purely on the JobManager. So what happens is that the TaskManagers > create state in some directory, let's say /tmp/checkpoints, on the > TaskManager. For cleanup, the JobManager gets the state handle and calls > discard (on the JobManager), this tries to cleanup the state in > /tmp/checkpoints, but of course, there is nothing there since we are still on > the JobManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2221) Checkpoints to "file://" are not cleaned up
[ https://issues.apache.org/jira/browse/FLINK-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586450#comment-14586450 ] ASF GitHub Bot commented on FLINK-2221: --- Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/839#issuecomment-112161893 Looks good +1 > Checkpoints to "file://" are not cleaned up > --- > > Key: FLINK-2221 > URL: https://issues.apache.org/jira/browse/FLINK-2221 > Project: Flink > Issue Type: Bug > Components: Streaming >Reporter: Aljoscha Krettek > > If you think about it, this could never work. The state handle cleanup logic > happens purely on the JobManager. So what happens is that the TaskManagers > create state in some directory, let's say /tmp/checkpoints, on the > TaskManager. For cleanup, the JobManager gets the state handle and calls > discard (on the JobManager), this tries to cleanup the state in > /tmp/checkpoints, but of course, there is nothing there since we are still on > the JobManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2221) Checkpoints to "file://" are not cleaned up
[ https://issues.apache.org/jira/browse/FLINK-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586311#comment-14586311 ] ASF GitHub Bot commented on FLINK-2221: --- GitHub user mbalassi opened a pull request: https://github.com/apache/flink/pull/839 [FLINK-2221] [docs] Docs for not using local filesystem on the cluste… …r as state backup This is just a clear documentation of the problem for the 0.9 release. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mbalassi/flink flink-2221 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/839.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #839 commit 79f0db81c8eda3296bc0f42f9191ea2d0790132c Author: mbalassi Date: 2015-06-15T16:53:08Z [FLINK-2221] [docs] Docs for not using local filesystem on the cluster as state backup This is just a clear documentation of the problem for the 0.9 release. > Checkpoints to "file://" are not cleaned up > --- > > Key: FLINK-2221 > URL: https://issues.apache.org/jira/browse/FLINK-2221 > Project: Flink > Issue Type: Bug > Components: Streaming >Reporter: Aljoscha Krettek > > If you think about it, this could never work. The state handle cleanup logic > happens purely on the JobManager. So what happens is that the TaskManagers > create state in some directory, let's say /tmp/checkpoints, on the > TaskManager. For cleanup, the JobManager gets the state handle and calls > discard (on the JobManager), this tries to cleanup the state in > /tmp/checkpoints, but of course, there is nothing there since we are still on > the JobManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2221) Checkpoints to "file://" are not cleaned up
[ https://issues.apache.org/jira/browse/FLINK-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586154#comment-14586154 ] Márton Balassi commented on FLINK-2221: --- Ok, so for the release I am adding some documentation, highlighting that the local flesystem should not be used as a state backend in a distributed setting. We might address the issue later. > Checkpoints to "file://" are not cleaned up > --- > > Key: FLINK-2221 > URL: https://issues.apache.org/jira/browse/FLINK-2221 > Project: Flink > Issue Type: Bug > Components: Streaming >Reporter: Aljoscha Krettek > > If you think about it, this could never work. The state handle cleanup logic > happens purely on the JobManager. So what happens is that the TaskManagers > create state in some directory, let's say /tmp/checkpoints, on the > TaskManager. For cleanup, the JobManager gets the state handle and calls > discard (on the JobManager), this tries to cleanup the state in > /tmp/checkpoints, but of course, there is nothing there since we are still on > the JobManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2221) Checkpoints to "file://" are not cleaned up
[ https://issues.apache.org/jira/browse/FLINK-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586148#comment-14586148 ] Márton Balassi commented on FLINK-2221: --- After a bit more careful consideration and checking other parts of the code I feel that it does not make sense to implement this. A clash in twenty byte long random string has to be enough. > Checkpoints to "file://" are not cleaned up > --- > > Key: FLINK-2221 > URL: https://issues.apache.org/jira/browse/FLINK-2221 > Project: Flink > Issue Type: Bug > Components: Streaming >Reporter: Aljoscha Krettek > > If you think about it, this could never work. The state handle cleanup logic > happens purely on the JobManager. So what happens is that the TaskManagers > create state in some directory, let's say /tmp/checkpoints, on the > TaskManager. For cleanup, the JobManager gets the state handle and calls > discard (on the JobManager), this tries to cleanup the state in > /tmp/checkpoints, but of course, there is nothing there since we are still on > the JobManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2221) Checkpoints to "file://" are not cleaned up
[ https://issues.apache.org/jira/browse/FLINK-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586149#comment-14586149 ] Márton Balassi commented on FLINK-2221: --- After a bit more careful consideration and checking other parts of the code I feel that it does not make sense to implement this. A clash in twenty byte long random string has to be enough. > Checkpoints to "file://" are not cleaned up > --- > > Key: FLINK-2221 > URL: https://issues.apache.org/jira/browse/FLINK-2221 > Project: Flink > Issue Type: Bug > Components: Streaming >Reporter: Aljoscha Krettek > > If you think about it, this could never work. The state handle cleanup logic > happens purely on the JobManager. So what happens is that the TaskManagers > create state in some directory, let's say /tmp/checkpoints, on the > TaskManager. For cleanup, the JobManager gets the state handle and calls > discard (on the JobManager), this tries to cleanup the state in > /tmp/checkpoints, but of course, there is nothing there since we are still on > the JobManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2221) Checkpoints to "file://" are not cleaned up
[ https://issues.apache.org/jira/browse/FLINK-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586135#comment-14586135 ] Márton Balassi commented on FLINK-2221: --- We can go down the check and retry route, but as it is distributed systems that is still not safe. My current approach is to add some prefix consisting of jobname, datetime, operatorname and parallel instance name that makes it more unique. Or we could also generate a UUID. > Checkpoints to "file://" are not cleaned up > --- > > Key: FLINK-2221 > URL: https://issues.apache.org/jira/browse/FLINK-2221 > Project: Flink > Issue Type: Bug > Components: Streaming >Reporter: Aljoscha Krettek > > If you think about it, this could never work. The state handle cleanup logic > happens purely on the JobManager. So what happens is that the TaskManagers > create state in some directory, let's say /tmp/checkpoints, on the > TaskManager. For cleanup, the JobManager gets the state handle and calls > discard (on the JobManager), this tries to cleanup the state in > /tmp/checkpoints, but of course, there is nothing there since we are still on > the JobManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2221) Checkpoints to "file://" are not cleaned up
[ https://issues.apache.org/jira/browse/FLINK-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586134#comment-14586134 ] Márton Balassi commented on FLINK-2221: --- We can go down the check and retry route, but as it is distributed systems that is still not safe. My current approach is to add some prefix consisting of jobname, datetime, operatorname and parallel instance name that makes it more unique. Or we could also generate a UUID. > Checkpoints to "file://" are not cleaned up > --- > > Key: FLINK-2221 > URL: https://issues.apache.org/jira/browse/FLINK-2221 > Project: Flink > Issue Type: Bug > Components: Streaming >Reporter: Aljoscha Krettek > > If you think about it, this could never work. The state handle cleanup logic > happens purely on the JobManager. So what happens is that the TaskManagers > create state in some directory, let's say /tmp/checkpoints, on the > TaskManager. For cleanup, the JobManager gets the state handle and calls > discard (on the JobManager), this tries to cleanup the state in > /tmp/checkpoints, but of course, there is nothing there since we are still on > the JobManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2221) Checkpoints to "file://" are not cleaned up
[ https://issues.apache.org/jira/browse/FLINK-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586126#comment-14586126 ] Gyula Fora commented on FLINK-2221: --- A workaround would be to actually discard the state where it was created. But this adds a large overhead of sending state handles around. > Checkpoints to "file://" are not cleaned up > --- > > Key: FLINK-2221 > URL: https://issues.apache.org/jira/browse/FLINK-2221 > Project: Flink > Issue Type: Bug > Components: Streaming >Reporter: Aljoscha Krettek > > If you think about it, this could never work. The state handle cleanup logic > happens purely on the JobManager. So what happens is that the TaskManagers > create state in some directory, let's say /tmp/checkpoints, on the > TaskManager. For cleanup, the JobManager gets the state handle and calls > discard (on the JobManager), this tries to cleanup the state in > /tmp/checkpoints, but of course, there is nothing there since we are still on > the JobManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2221) Checkpoints to "file://" are not cleaned up
[ https://issues.apache.org/jira/browse/FLINK-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586119#comment-14586119 ] Ufuk Celebi commented on FLINK-2221: @Marton: how are you fixing this? Check and retry with a different name? I think it is a valid use case to do the checkpoints on the local FS. I will sync with [~StephanEwen] and look into this later. > Checkpoints to "file://" are not cleaned up > --- > > Key: FLINK-2221 > URL: https://issues.apache.org/jira/browse/FLINK-2221 > Project: Flink > Issue Type: Bug > Components: Streaming >Reporter: Aljoscha Krettek > > If you think about it, this could never work. The state handle cleanup logic > happens purely on the JobManager. So what happens is that the TaskManagers > create state in some directory, let's say /tmp/checkpoints, on the > TaskManager. For cleanup, the JobManager gets the state handle and calls > discard (on the JobManager), this tries to cleanup the state in > /tmp/checkpoints, but of course, there is nothing there since we are still on > the JobManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2221) Checkpoints to "file://" are not cleaned up
[ https://issues.apache.org/jira/browse/FLINK-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586091#comment-14586091 ] Aljoscha Krettek commented on FLINK-2221: - Then we should add it to the configuration, that the user should not user file:// there when running on a cluster. > Checkpoints to "file://" are not cleaned up > --- > > Key: FLINK-2221 > URL: https://issues.apache.org/jira/browse/FLINK-2221 > Project: Flink > Issue Type: Bug > Components: Streaming >Reporter: Aljoscha Krettek > > If you think about it, this could never work. The state handle cleanup logic > happens purely on the JobManager. So what happens is that the TaskManagers > create state in some directory, let's say /tmp/checkpoints, on the > TaskManager. For cleanup, the JobManager gets the state handle and calls > discard (on the JobManager), this tries to cleanup the state in > /tmp/checkpoints, but of course, there is nothing there since we are still on > the JobManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2221) Checkpoints to "file://" are not cleaned up
[ https://issues.apache.org/jira/browse/FLINK-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586078#comment-14586078 ] Márton Balassi commented on FLINK-2221: --- So the current architecture has the assumption that the JobManager has access to the checkpoint files the same way a TaskManager does. We should make this more clear for the user. An additional fragility factor can be that the file state provider simply adds a random string to the directory name and expects it to be an unused name, or simply overwrites it. It is highly unlikely that this behaviur would cause a trouble, but let us still avoid it. Let me add a quick fix for the latter issue. > Checkpoints to "file://" are not cleaned up > --- > > Key: FLINK-2221 > URL: https://issues.apache.org/jira/browse/FLINK-2221 > Project: Flink > Issue Type: Bug > Components: Streaming >Reporter: Aljoscha Krettek > > If you think about it, this could never work. The state handle cleanup logic > happens purely on the JobManager. So what happens is that the TaskManagers > create state in some directory, let's say /tmp/checkpoints, on the > TaskManager. For cleanup, the JobManager gets the state handle and calls > discard (on the JobManager), this tries to cleanup the state in > /tmp/checkpoints, but of course, there is nothing there since we are still on > the JobManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2221) Checkpoints to "file://" are not cleaned up
[ https://issues.apache.org/jira/browse/FLINK-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586066#comment-14586066 ] Gyula Fora commented on FLINK-2221: --- For the time being let's assume that the JM can actually execute the discard logic. Using file:// is useless anyways. > Checkpoints to "file://" are not cleaned up > --- > > Key: FLINK-2221 > URL: https://issues.apache.org/jira/browse/FLINK-2221 > Project: Flink > Issue Type: Bug > Components: Streaming >Reporter: Aljoscha Krettek > > If you think about it, this could never work. The state handle cleanup logic > happens purely on the JobManager. So what happens is that the TaskManagers > create state in some directory, let's say /tmp/checkpoints, on the > TaskManager. For cleanup, the JobManager gets the state handle and calls > discard (on the JobManager), this tries to cleanup the state in > /tmp/checkpoints, but of course, there is nothing there since we are still on > the JobManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2221) Checkpoints to "file://" are not cleaned up
[ https://issues.apache.org/jira/browse/FLINK-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586060#comment-14586060 ] Aljoscha Krettek commented on FLINK-2221: - Yes, for those it seems to work, although I'm a bit afraid about name clashes. The name of the checkpoint file should also include the operator-id, name and parallel subtask is. > Checkpoints to "file://" are not cleaned up > --- > > Key: FLINK-2221 > URL: https://issues.apache.org/jira/browse/FLINK-2221 > Project: Flink > Issue Type: Bug > Components: Streaming >Reporter: Aljoscha Krettek > > If you think about it, this could never work. The state handle cleanup logic > happens purely on the JobManager. So what happens is that the TaskManagers > create state in some directory, let's say /tmp/checkpoints, on the > TaskManager. For cleanup, the JobManager gets the state handle and calls > discard (on the JobManager), this tries to cleanup the state in > /tmp/checkpoints, but of course, there is nothing there since we are still on > the JobManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2221) Checkpoints to "file://" are not cleaned up
[ https://issues.apache.org/jira/browse/FLINK-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586052#comment-14586052 ] Gyula Fora commented on FLINK-2221: --- Does the cleanup work for HDFS and other distributed file systems that the JM sees? > Checkpoints to "file://" are not cleaned up > --- > > Key: FLINK-2221 > URL: https://issues.apache.org/jira/browse/FLINK-2221 > Project: Flink > Issue Type: Bug > Components: Streaming >Reporter: Aljoscha Krettek > > If you think about it, this could never work. The state handle cleanup logic > happens purely on the JobManager. So what happens is that the TaskManagers > create state in some directory, let's say /tmp/checkpoints, on the > TaskManager. For cleanup, the JobManager gets the state handle and calls > discard (on the JobManager), this tries to cleanup the state in > /tmp/checkpoints, but of course, there is nothing there since we are still on > the JobManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
