Re: Empty directories left over from checkpointing

Stephan, Thanks for taking care of this. We'll give it a try once 1.4 drops. On Sat, Oct 14, 2017 at 1:25 PM, Stephan Ewen wrote: > Some updates on this: > > Aside from reworking how the S3 directory handling is done, we also looked > into supporting S3 different than we

Re: Empty directories left over from checkpointing

Some updates on this: Aside from reworking how the S3 directory handling is done, we also looked into supporting S3 different than we currently do. Currently support goes strictly through Hadoop's S3 file systems, which we need to change, because we want it to be possible to use Flink without

Re: Empty directories left over from checkpointing

Hi, We recently removed some cleanup code, because it involved checking some store meta data to check when we can delete a directory. For certain stores (like S3), requesting this meta data whenever we delete a file was so expensive that it could bring down the job because removing state could