Github user zentol commented on the pull request:
https://github.com/apache/flink/pull/339#issuecomment-71672497
Whenever I look more closely at the DC I'm always left wondering how it can
work at all.
About your first point, i don't think thats enough. there is a more
fundamental flaw, we need another counter for delete processes.
consider the following 2 scenarios with 2 tasks distributing the same file.
C denotes the creating of a copying process, D denotes deleting process. #
denotes the count variable, O the oldCount variable.
```
1): I II III IV
T1:---C--------D--------
T2:-------C--------D---
# 1 2 2 2
O 2 2
2) I II III IV
T1:---C------------D---
T2:-------C----D--------
# 1 2 2 2
O 2 2
```
In both scenarios, D at III should not delete the file, but all D's have
the very same information.
instead, i propose having 2 counters, one counting the # of copy
operations; and one counting the # of delete operations, with the current value
(at process creation) stored in the process. when executing, if the current
value is equal to the copy count, files may be deleted, since this means that
this delete process was the last to be started.
let's make another fancy schema to illustrate the point:
```
1): I II III IV
T1:---C--------D--------
T2:-------C--------D---
# 1 2 2 2
O 1 2
2) I II III IV
T1:---C------------D---
T2:-------C----D--------
# 1 2 2 2
O 1 2
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
