Re: Spark 2.0 Streaming and Event Time

2016-06-26 Thread Chang Lim
Here is an update to my question: = Tathagata Das Jun 9 to me Event time is part of windowed aggregation API. See my slides - https://www.slideshare.net/mobile/databricks/a-deep-dive-into-structured-streaming Let me know if it helps you to find it. Keeping it short as I am o

Re: Spark 2.0 Streaming and Event Time

2016-06-09 Thread Chang Lim
Yes, now I realized that. I did exchanged emails with TD on this topic. The Microsoft presentation at Spark summit ("reorder" function) would be a good addition to Spark. Would this feature be on the road map? On Thu, Jun 9, 2016 at 9:56 AM, Michael Armbrust wrote: > There is no special settin

Re: Spark 2.0 Streaming and Event Time

2016-06-09 Thread Michael Armbrust
There is no special setting for event time (though we will be adding one for setting a watermark in 2.1 to allow us to reduce the amount of state that needs to be kept around). Just window/groupBy on the on the column that is your event time. On Wed, Jun 8, 2016 at 4:12 PM, Chang Lim wrote: > H

Spark 2.0 Streaming and Event Time

2016-06-08 Thread Chang Lim
Hi All, Does Spark 2.0 Streaming [sqlContext.read.format(...).stream(...)] support Event Time? In TD's Spark Summit talk yesterday, this is listed as a 2.0 feature. Of so, where is the API or how to set it? Thanks in advanced, Chang -- View this message in context: http://apache-spark-user-