[GitHub] flink issue #2797: [FLINK-5056] Makes the BucketingSink rescalable.

2016-11-16 Thread kl0u
Github user kl0u commented on the issue: https://github.com/apache/flink/pull/2797 Done @zentol --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] flink issue #2797: [FLINK-5056] Makes the BucketingSink rescalable.

2016-11-16 Thread kl0u
Github user kl0u commented on the issue: https://github.com/apache/flink/pull/2797 Hi @zentol . I integrated your last comments. Now for the your last question, the answer is that the only way to distinguish between the two types of files is by their filename (prefix and suf

[GitHub] flink issue #2797: [FLINK-5056] Makes the BucketingSink rescalable.

2016-11-15 Thread zentol
Github user zentol commented on the issue: https://github.com/apache/flink/pull/2797 Do we provide anything that allows the user to identify files that weren't cleaned up but can be removed? Something like a list of "active" pending files? --- If your project is set up for it, you ca

[GitHub] flink issue #2797: [FLINK-5056] Makes the BucketingSink rescalable.

2016-11-15 Thread kl0u
Github user kl0u commented on the issue: https://github.com/apache/flink/pull/2797 Sorry I just amended my previous commit with the comment integration. :S --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project d

[GitHub] flink issue #2797: [FLINK-5056] Makes the BucketingSink rescalable.

2016-11-15 Thread zentol
Github user zentol commented on the issue: https://github.com/apache/flink/pull/2797 That makes sense, thanks for the explanation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] flink issue #2797: [FLINK-5056] Makes the BucketingSink rescalable.

2016-11-15 Thread kl0u
Github user kl0u commented on the issue: https://github.com/apache/flink/pull/2797 The subtask matters for files that were opened but they did not make it into a checkpoint because a failure occurred after they were opened but before a checkpoint barrier arrived. These are the on

[GitHub] flink issue #2797: [FLINK-5056] Makes the BucketingSink rescalable.

2016-11-15 Thread zentol
Github user zentol commented on the issue: https://github.com/apache/flink/pull/2797 No, it's not just that one method. The method is on a whole other level; it is so easy to edit a commit there simply is no reason for such a thing to exist in the *initial PR*. So, let's go t

[GitHub] flink issue #2797: [FLINK-5056] Makes the BucketingSink rescalable.

2016-11-15 Thread kl0u
Github user kl0u commented on the issue: https://github.com/apache/flink/pull/2797 Thanks for the first review @zentol . On the comment about the diff reduction, you are referring to the removal of the method in the intermediate commits that was not used? If yes, then this i

[GitHub] flink issue #2797: [FLINK-5056] Makes the BucketingSink rescalable.

2016-11-15 Thread zentol
Github user zentol commented on the issue: https://github.com/apache/flink/pull/2797 I also haven't checked the tests yet. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled