What should be the behavior of BucketingSink vis a vis state ( pending ,
inprogess and finalization ) when we suspend and resume ?

So I did this

* I had a pipe writing to hdfs suspend and resume using

--allowNonRestoredState as in I had added a harmless MapOperator (
stateless ).


* I see that a file on hdfs, the file being written to ( before the cancel
with save point )  go into a pending state  _part-0-21.pending


* I see a new file being written to in the resumed pipe
_part-0-22.in-progress.


What  I do not see is the file in  _part-0-21.pending being finalized ( as
in renamed to a just part-0-21. I would have assumed that would be the case
in this controlled suspend/resume circumstance. Further it is a rename and
hdfs mv is not an expensive operation.



Am I understanding the process correct and it yes any pointers ?


Regards,


Vishal

Reply via email to