[
https://issues.apache.org/jira/browse/FLINK-9609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangminglei updated FLINK-9609:
--------------------------------
Description:
Currently, BucketingSink only support {{notifyCheckpointComplete}}. But users
want to do some extra work when a bucket is ready. It would be nice if we can
support {{BucketReady}} mechanism for users or we can tell users when a bucket
is ready for use. For example, One bucket is created for every 5 minutes, at
the end of 5 minutes before creating the next bucket, the user might need to do
something as the previous bucket ready, like sending the timestamp of the
bucket ready time to a server or do some other stuff.
Here, Bucket ready means all the part files name suffix under a bucket neither
{{.pending}} nor {{.in-progress}}. Then we can think this bucket is ready for
user use. Like a watermark means no elements with a timestamp older or equal to
the watermark timestamp should arrive at the window. We can also refer to the
concept of watermark here, or we can call this *BucketWatermark* if we could.
was:
Currently, BucketingSink only support {{notifyCheckpointComplete}}. But users
want to do some extra work when a bucket is ready. It would be nice if we can
support {{BucketReady}} mechanism for users or we can tell users when a bucket
is ready for use. For example, One bucket is created for every 5 minutes, at
the end of 5 minutes before creating the next bucket, the user might need to do
something as the previous bucket ready, like sending the timestamp of the
bucket ready time to a server or do some other stuff.
Here, Bucket ready means all the part files name suffix under a bucket neither
{{.pending}} nor {{.in-progress}}. Then we can think this bucket is ready for
user use. Like a watermark means no elements with a timestamp older or equal to
the watermark timestamp should arrive at the window. We can also refer to the
concept of watermark here.
> Add bucket ready mechanism for BucketingSink when checkpoint complete
> ---------------------------------------------------------------------
>
> Key: FLINK-9609
> URL: https://issues.apache.org/jira/browse/FLINK-9609
> Project: Flink
> Issue Type: New Feature
> Components: filesystem-connector, Streaming Connectors
> Affects Versions: 1.5.0, 1.4.2
> Reporter: zhangminglei
> Assignee: zhangminglei
> Priority: Major
>
> Currently, BucketingSink only support {{notifyCheckpointComplete}}. But users
> want to do some extra work when a bucket is ready. It would be nice if we can
> support {{BucketReady}} mechanism for users or we can tell users when a
> bucket is ready for use. For example, One bucket is created for every 5
> minutes, at the end of 5 minutes before creating the next bucket, the user
> might need to do something as the previous bucket ready, like sending the
> timestamp of the bucket ready time to a server or do some other stuff.
> Here, Bucket ready means all the part files name suffix under a bucket
> neither {{.pending}} nor {{.in-progress}}. Then we can think this bucket is
> ready for user use. Like a watermark means no elements with a timestamp older
> or equal to the watermark timestamp should arrive at the window. We can also
> refer to the concept of watermark here, or we can call this *BucketWatermark*
> if we could.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
