HeartSaVioR commented on pull request #35362: URL: https://github.com/apache/spark/pull/35362#issuecomment-1030927553
https://github.com/HeartSaVioR/spark/commit/d532b6f6bcdd80cdaac520b21587ebb69ff2df8f I just played with my own simple benchmark (in the commit above), and the gain is much more than the PR description. It's up to 30% for tumble window to 60% for sliding window. (I expect the gain gets bigger if maxNumOverlapping is higher.) I'll update the PR description to contain the benchmark result. I also did some calculations based on the new math to create sliding windows with offset by hand, and it seemed OK. I can't think of cases the new math may miss. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
