Github user tdas commented on the issue: https://github.com/apache/spark/pull/21220 @brkyvz Answers to your questions. 1. We have already fixed those emptyDF optimizations. The optimization only quick in `df.isStreaming = false`, and emptyDFs generated by sources should be have isStreaming = true. That's for v1 sources. For v2 sources, the engine already takes care of that by making sure that `StreamingDataSourceV2Relation.isStreaming` is true where `StreamingDataSourceV2Relation` is the logical plan leaf inserted into the micro-batch logical plan irrespective of empty or not. 2. I will do both type of timeouts in flatMapGroupsWithState in later PR.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org