In the meantime, if you are interested, you can read the design doc in the corresponding JIRA - https://issues.apache.org/jira/browse/SPARK-18124
On Thu, Dec 1, 2016 at 12:53 AM, Tathagata Das <tathagata.das1...@gmail.com> wrote: > That feature is coming in 2.1.0. We have added watermarking, that will > track the event time of the data and accordingly close old windows, output > its corresponding aggregate and then drop its corresponding state. But in > that case, you will have to use append mode, and aggregated data of a > particular window will be evicted only when the windows is closed. You will > be able to control the threshold on how long to wait for late, out-of-order > data before closing a window. > > We will be updated the docs soon to explain this. > > On Tue, Nov 29, 2016 at 8:30 PM, Xinyu Zhang <wsz...@163.com> wrote: > >> Hi >> >> I want to use window operations. However, if i don't remove any data, the >> "complete" table will become larger and larger as time goes on. So I want >> to remove some outdated data in the complete table that I would never use. >> Is there any method to meet my requirement? >> >> Thanks! >> >> >> >> >> > >
