Lakshmi Rao created FLINK-9118:
----------------------------------
Summary: Support a time based rollover of part file in Bucketing
Sink
Key: FLINK-9118
URL: https://issues.apache.org/jira/browse/FLINK-9118
Project: Flink
Issue Type: Improvement
Components: filesystem-connector
Reporter: Lakshmi Rao
In the current implementation, the BucketingSink rolls over a part file based
on a _batchSize_
([here|https://github.com/apache/flink/blob/release-1.4.0/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java#L459]).
Can we also support a roll over based on a constant time interval? This is
not the same as the _inactiveBucketCheckInterval_ as this bucket is not truly
inactive, it's still being written to but just has to be flushed every X
minutes, where X is a user-specified time interval.
The change would involve tracking a _bucketCreationTime_ in the BucketState
(much like the _lastWrittenToTime_) whenever a new part file is opened and
would include a condition to check _currentProcessingTime_ -
_bucketCreationTime_ > X in the _shouldRoll_ method.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
