Hi Taher, Well, I would say there is no single class that implements it. In a nutshell, it is the StreamingFileSink that (through Buckets) tells the responsible Bucket what to do at each step of the lifecycle of the Flink operator (mainly on element, on checkpoint, on checkpoint completed and on restore).
So I would suggest that you should have a look in these classes. In essence, the main steps of the "choreography" are similar to the ones that the older BucketingSink was doing. Cheers, Kostas On Thu, Jan 10, 2019 at 10:47 AM Taher Koitawala <taher.koitaw...@gslab.com> wrote: > Hi Kostas, > Thanks you for the clarification, also can you please point > how StreamingFileSink uses TwoPhaseCommit. Can you also point out the > implementing class for that? > > > Regards, > Taher Koitawala > GS Lab Pune > +91 8407979163 > > > On Thu, Jan 10, 2019 at 3:10 PM Kostas Kloudas <kklou...@gmail.com> wrote: > >> That is correct. >> The StreamingFileSink does not extend the TwoPhaseCommitSinkFunction >> because of some internal implementation details. >> But this, does not mean that it does not implement a two phase commit >> protocol (which is independent of the implementation). >> >> Cheers, >> Kostas >> >> On Thu, Jan 10, 2019 at 10:33 AM Taher Koitawala < >> taher.koitaw...@gslab.com> wrote: >> >>> StreamingFileSink extends RichSinkFunction and implements >>> CheckpointedFunction, CheckpointListener and ProcessingTimeCallback however >>> TwoPhaseCommitSinkFunction is never used by StreamingFileSink. Hence I had >>> a question if the sink uses the TwoPhaseCommit protocol or not. >>> >>> Regards, >>> Taher Koitawala >>> >>> On Thu 10 Jan, 2019, 2:40 PM Kostas Kloudas <kklou...@gmail.com wrote: >>> >>>> Hi Taher, >>>> >>>> The StreamingFileSink implements a version of TwoPhaseCommit. Can you >>>> elaborate a bit on what do you mean by " TwoPhaseCommit is not being >>>> used"? >>>> >>>> Cheers, >>>> Kostas >>>> >>>> On Thu, Jan 10, 2019 at 9:29 AM Taher Koitawala < >>>> taher.koitaw...@gslab.com> wrote: >>>> >>>>> Hi All, >>>>> As per my understanding and the API of StreamingFileSink, >>>>> TwoPhaseCommit is not being used. Can someone please confirm is that's >>>>> right? Also if StreamingFileSink does not support >>>>> TwoPhaseCommits what is the best way to implement this? >>>>> >>>>> >>>>> Regards, >>>>> Taher Koitawala >>>>> GS Lab Pune >>>>> +91 8407979163 >>>>> >>>>
