Btw, just wrote up detailed design doc on existing patch: https://docs.google.com/document/d/1tUO29BDXb9127RiivUS7Hv324dC0YHuokYvyQRpurDY/edit?usp=sharing
This doc is a wall of text, since I guess we already imagine how session window works (and I showed a simple example in SPIP doc), so try to avoid drawing something which would take non-trivial efforts. New classes are linked to the actual source code so that we can read the code directly whenever curious/wonders about something. Please let me know anytime if something is unclear and need elaboration. -Jungtaek Lim (HeartSaVioR) 2018년 9월 28일 (금) 오후 10:18, Jungtaek Lim <kabh...@gmail.com>님이 작성: > Thanks for sharing your proposal as well as implementation. Looks like > your proposal is more like focused to design details: I may be better to > write one more for design details and share it as well. Stay tuned! > > Btw, I'm trying out your patch to see whether it passes the tests I've > added, and looks like it fails on below UT: > > https://github.com/apache/spark/blob/ad0b7466ef3f79354a99bd1b95c23e4c308502d5/sql/core/src/test/scala/org/apache/spark/sql/streaming/EventTimeWatermarkSuite.scala#L475-L573 > Could you take a look at UT and see whether I'm missing here or the UT is > correct? > > (Actually most of UTs I've added fail but some UTs are for update mode, > and the patch doesn't provide same experience with select only session > window, so I'm pointing only one UT which is testing basic session window.) > > -Jungtaek Lim (HeartSaVioR) > > 2018년 9월 28일 (금) 오후 9:22, Yuanjian Li <xyliyuanj...@gmail.com>님이 작성: > >> Hi Jungtaek: >> >> We also meet this problem during migration of streaming application to >> Structure Streaming in Baidu practice, we solved this in our folk and just >> steady running in product. >> As the initial plan we are doing the code clean work and preparing >> give a SPIP in Oct, happy to see your proposal. Hope we share some spots >> together. >> Here’s the PR and doc: >> https://github.com/apache/spark/pull/22583 >> >> https://docs.google.com/document/d/1zeAc7QKSO7J4-Yk06kc76kvldl-QHLCDJuu04d7k2bg/edit?usp=sharing >> >> Thanks, >> Yuanjian Li >> >> >> 在 2018年9月28日，06:22，Jungtaek Lim <kabh...@gmail.com> 写道： >> >> Hi all, >> >> I would like to initiate discussion thread to discuss "Native support of >> session window". >> Origin issue is filed to SPARK-10816 [1] but I can file another one for >> representing SPIP if necessary. WIP but working PR is available as well, so >> we can even test it directly or see the difference if some of us feel more >> convenient to go through the source code instead of doc. >> >> I've attached PDF version of SPIP in SPARK-10816, but adding Google Docs >> link [2] for who feel convenient to comment in doc. >> >> Please let me know if we would like to see also technical design for >> this. I avoid to go too deep on SPIP doc so anyone could review and see the >> benefit of adopting this. >> >> Looking forward to hear your feedback. >> >> Thanks, >> Jungtaek Lim (HeartSaVioR) >> >> 1. https://issues.apache.org/jira/browse/SPARK-10816 >> 2. >> https://docs.google.com/document/d/1_rMLmUSyGzb62RnP2A3WX6D6uRxox8Q_7WcoI_HrTw4/edit?usp=sharing >> 3. https://github.com/apache/spark/pull/22482 >> >> >> >>
