Xiaowei Jiang updated FLINK-4854:
    Component/s: DataStream API

> Efficient Batch Operator in Streaming
> -------------------------------------
>                 Key: FLINK-4854
>                 URL: https://issues.apache.org/jira/browse/FLINK-4854
>             Project: Flink
>          Issue Type: Improvement
>          Components: DataStream API
>            Reporter: Xiaowei Jiang
>            Assignee: MaGuowei
>              Labels: features
>   Original Estimate: 168h
>  Remaining Estimate: 168h
> Very often, it's more efficient to process a batch of records at once instead 
> of processing them one by one. We can use window to achieve this 
> functionality. However, window will store all records in states, which can be 
> costly. It's desirable to have an efficient implementation of batch operator. 
> The batch operator works per task and behave similarly to aligned windows. 
> Here is an example of how the interface looks like to a user.
> interface BatchFunction {
>     // add the record to the buffer
>     // returns if the batch is ready to be flushed
>     boolean addRecord(T record);
>     // process all pending records in the buffer
>     void flush(Collector collector) ;
> }
> DataStream ds = ...
> BatchFunction func = ...
> ds.batch(func);
> The operator calls addRecord for each record. The batch function saves the 
> record in its own buffer. The addRecord returns if the pending buffer should 
> be flushed. In that case, the operator invokes flush.

This message was sent by Atlassian JIRA

Reply via email to