HeartSaVioR commented on issue #24457: [SPARK-27340][SS] Alias on TimeWindow expression may cause watermark metadata lost URL: https://github.com/apache/spark/pull/24457#issuecomment-571372782 Looking at the test code, the issue seems to be valid and PR fixes the issue correctly. But I'm not sure about the side effect, as @xuanyuanking commented. Btw I think this has been known issue and underlying issue may not just be missing copying metadata. I'm not sure Spark can ensure metadata is propagated correctly during any multiple transformations, including typed -> untyped, and vice versa. It doesn't seem to be a thing we can rely on. I think the root issue is that the event time column and value are open to modify. Other streaming frameworks provide the way to specify the event time per row, and the value is treated as special column which cannot be modified (both column and value) during transformation. I've had a long discussion with @echauchot (working with Spark runner in Beam) regarding this. Please follow the link : https://github.com/apache/spark/pull/23576#issuecomment-524686985
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
