Thanks for sharing your proposal as well as implementation. Looks like your proposal is more like focused to design details: I may be better to write one more for design details and share it as well. Stay tuned!
Btw, I'm trying out your patch to see whether it passes the tests I've added, and looks like it fails on below UT: https://github.com/apache/spark/blob/ad0b7466ef3f79354a99bd1b95c23e4c308502d5/sql/core/src/test/scala/org/apache/spark/sql/streaming/EventTimeWatermarkSuite.scala#L475-L573 Could you take a look at UT and see whether I'm missing here or the UT is correct? (Actually most of UTs I've added fail but some UTs are for update mode, and the patch doesn't provide same experience with select only session window, so I'm pointing only one UT which is testing basic session window.) -Jungtaek Lim (HeartSaVioR) 2018년 9월 28일 (금) 오후 9:22, Yuanjian Li <xyliyuanj...@gmail.com>님이 작성: > Hi Jungtaek: > > We also meet this problem during migration of streaming application to > Structure Streaming in Baidu practice, we solved this in our folk and just > steady running in product. > As the initial plan we are doing the code clean work and preparing give > a SPIP in Oct, happy to see your proposal. Hope we share some spots > together. > Here’s the PR and doc: > https://github.com/apache/spark/pull/22583 > > https://docs.google.com/document/d/1zeAc7QKSO7J4-Yk06kc76kvldl-QHLCDJuu04d7k2bg/edit?usp=sharing > > Thanks, > Yuanjian Li > > > 在 2018年9月28日，06:22，Jungtaek Lim <kabh...@gmail.com> 写道： > > Hi all, > > I would like to initiate discussion thread to discuss "Native support of > session window". > Origin issue is filed to SPARK-10816 [1] but I can file another one for > representing SPIP if necessary. WIP but working PR is available as well, so > we can even test it directly or see the difference if some of us feel more > convenient to go through the source code instead of doc. > > I've attached PDF version of SPIP in SPARK-10816, but adding Google Docs > link [2] for who feel convenient to comment in doc. > > Please let me know if we would like to see also technical design for this. > I avoid to go too deep on SPIP doc so anyone could review and see the > benefit of adopting this. > > Looking forward to hear your feedback. > > Thanks, > Jungtaek Lim (HeartSaVioR) > > 1. https://issues.apache.org/jira/browse/SPARK-10816 > 2. > https://docs.google.com/document/d/1_rMLmUSyGzb62RnP2A3WX6D6uRxox8Q_7WcoI_HrTw4/edit?usp=sharing > 3. https://github.com/apache/spark/pull/22482 > > > >
