[
https://issues.apache.org/jira/browse/FLINK-9118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lakshmi Rao updated FLINK-9118:
-------------------------------
Priority: Major (was: Minor)
> Support a time based rollover of part file in Bucketing Sink
> ------------------------------------------------------------
>
> Key: FLINK-9118
> URL: https://issues.apache.org/jira/browse/FLINK-9118
> Project: Flink
> Issue Type: Improvement
> Components: filesystem-connector
> Reporter: Lakshmi Rao
> Priority: Major
>
> In the current implementation, the BucketingSink rolls over a part file based
> on a _batchSize_
> ([here|https://github.com/apache/flink/blob/release-1.4.0/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java#L459]).
> Can we also support a roll over based on a constant time interval? This is
> not the same as the _inactiveBucketCheckInterval_ as this bucket is not truly
> inactive, it's still being written to but just has to be flushed every X
> minutes, where X is a user-specified time interval.
> The change would involve tracking a _bucketCreationTime_ in the BucketState
> (much like the _lastWrittenToTime_) whenever a new part file is opened and
> would include a condition to check _currentProcessingTime_ -
> _bucketCreationTime_ > X in the _shouldRoll_ method.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
