In the meantime, if you are interested, you can read the design doc in the
corresponding JIRA - https://issues.apache.org/jira/browse/SPARK-18124

On Thu, Dec 1, 2016 at 12:53 AM, Tathagata Das <tathagata.das1...@gmail.com>
wrote:

> That feature is coming in 2.1.0. We have added watermarking, that will
> track the event time of the data and accordingly close old windows, output
> its corresponding aggregate and then drop its corresponding state. But in
> that case, you will have to use append mode, and aggregated data of a
> particular window will be evicted only when the windows is closed. You will
> be able to control the threshold on how long to wait for late, out-of-order
> data before closing a window.
>
> We will be updated the docs soon to explain this.
>
> On Tue, Nov 29, 2016 at 8:30 PM, Xinyu Zhang <wsz...@163.com> wrote:
>
>> Hi
>>
>> I want to use window operations. However, if i don't remove any data, the
>> "complete" table will become larger and larger as time goes on. So I want
>> to remove some outdated data in the complete table that I would never use.
>> Is there any method to meet my requirement?
>>
>> Thanks!
>>
>>
>>
>>
>>
>
>

Reply via email to