Re: [BucketingSink] notify on moving into pending/ final state

Hi Rinat, I have commented on your PR and on the JIRA. Let me know what you think. Cheers, Kostas > On Oct 11, 2018, at 4:45 PM, Dawid Wysakowicz wrote: > > Hi Ribat, > I haven't checked your PR but we introduced a new connector in flink 1.6 > called StreamingFileSink that is supposed to

Re: [BucketingSink] notify on moving into pending/ final state

Hi Ribat, I haven't checked your PR but we introduced a new connector in flink 1.6 called StreamingFileSink that is supposed to replace BucketingSink long term. I think it might solve a few problems of yours. Have you checked it by chance? Best, Dawid On Thu, 11 Oct 2018, 14:10 Rinat, wrote: >

Re: [BucketingSink] notify on moving into pending/ final state

Hi Piotr, during the migration to the latest Flink version, we’ve decided to try to contribute this functionality to the master branch. PR is available here https://github.com/apache/flink/pull/6824 More details about hooking the state changes in BucketingSink are available in

Re: [BucketingSink] notify on moving into pending/ final state

Hi Piotr, I’ve create an issue https://issues.apache.org/jira/browse/FLINK-9592 The third proposal looks great, may I try to contribute this issue ? > On 14 Jun 2018, at 12:29, Piotr Nowojski wrote: > > Hi, > > Couple of things: > > 1.

Re: [BucketingSink] notify on moving into pending/ final state

Hi, Couple of things: 1. Please create a Jira ticket with this proposal, so we can move discussion from user mailing list. I haven’t thought it through, so take my comments with a grain of salt, however: 2. If we were to go with such callback, I would prefer to have one

Re: [BucketingSink] notify on moving into pending/ final state

Hi guys, thx for your reply. The following code info is actual for release-1.5.0 tag, org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink class For now, BucketingSink has the following lifecycle of files When moving files from opened to pending state: on each item (method

Re: [BucketingSink] notify on moving into pending/ final state

Hi, I see that could be a useful feature. What exactly now is preventing you from inheriting from BucketingSink? Maybe it would be just enough to make the BucketingSink easier extendable. One thing now that could collide with such feature is that Kostas is now working on larger BucketingSink

[BucketingSink] notify on moving into pending/ final state

Hi mates, I got a proposal about functionality of BucketingSink. During implementation of one of our tasks we got the following need - create a meta-file, with the path and additional information about the file, created by BucketingSink, when it’s been moved into final place. Unfortunately such