HeartSaVioR commented on PR #48570: URL: https://github.com/apache/spark/pull/48570#issuecomment-2436987898
@viirya Probably yes. We know the problem of PruneFilters which could prune the subtree, risk of losing nodes. This was problematic for losing CollectMetrics node, but for EventTimeWatermark, filter pushdown makes the filter with `false` be pushed down under EventTimeWatermark, hence when PruneFilters triggers, EventTimeWatermark isn't pruned out. EmptyRelationPropagation won't propagate empty relation with EventTimeWatermark, hence the node is not removed in this route. But it's definitely not 100% sure; it's hard to tell against 10s or even 100s of QO rules. I'm just saying I've dealt with several cases of streaming & QO issues and EventTimeWatermark seems to be unaffected from these cases. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
