Hi Rui, Thanks for your advice!
After reading Chapter 2&3 of *Streaming Systems* and some other materials, eventually I make it work! It indeed turned out to be an issue of not setting the trigger correctly. Previously, I didn't set the trigger & watermark so it would use the default settings. After I added `.withAllowedLateness()`, it can correctly materialize the window output as expected. Thank you so much for your help! Thanks & Regards, Minreng On Mon, Aug 24, 2020 at 1:58 PM Rui Wang <[email protected]> wrote: > Hi, > > I checked the query in your SO question and I think the SQL usage is > correct. > > My current guess is that the problem is how does watermark generate and > advance in KafkaIO. It could be either the watermark didn't pass the end of > your SQL window for aggregation or the data was lagging behind the > watermark so they are considered late data. > > One way to verify it is you can try to use TestStream as the source to > evaluate your pipeline and see whether it works well. > > -Rui > > On Mon, Aug 24, 2020 at 11:06 AM Minreng Wu <[email protected]> wrote: > >> Hi contributors, >> >> Sorry to bother you! I met a problem when I was trying to apply a >> windowing aggregation Beam SQL query to a Kafka input source. >> >> The details of the question are in the following link: >> https://stackoverflow.com/questions/63566057/how-to-integrate-beam-sql-windowing-query-with-kafkaio. >> And the version of the Beam Java SDK I used is *2.23.0* >> >> Really appreciate your help and advice! Stay safe and happy! >> >> Thanks and regards, >> Minreng >> >
