Xiaowei Jiang created FLINK-4854:

             Summary: Efficient Batch Operator in Streaming
                 Key: FLINK-4854
                 URL: https://issues.apache.org/jira/browse/FLINK-4854
             Project: Flink
          Issue Type: Improvement
            Reporter: Xiaowei Jiang
            Assignee: MaGuowei

Very often, it's more efficient to process a batch of records at once instead 
of processing them one by one. We can use window to achieve this functionality. 
However, window will store all records in states, which can be costly. It's 
desirable to have an efficient implementation of batch operator. The batch 
operator works per task and behave similarly to aligned windows. Here is an 
example of how the interface looks like to a user.

interface BatchFunction {
    // add the record to the buffer
    // returns if the batch is ready to be flushed
    boolean addRecord(T record);

    // process all pending records in the buffer
    void flush(Collector collector) ;

DataStream ds = ...
BatchFunction func = ...

The operator calls addRecord for each record. The batch function saves the 
record in its own buffer. The addRecord returns if the pending buffer should be 
flushed. In that case, the operator invokes flush.

This message was sent by Atlassian JIRA

Reply via email to