[ https://issues.apache.org/jira/browse/SPARK-43183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jungtaek Lim resolved SPARK-43183. ---------------------------------- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40845 [https://github.com/apache/spark/pull/40845] > Move update event on idleness in streaming query listener to separate > callback method > ------------------------------------------------------------------------------------- > > Key: SPARK-43183 > URL: https://issues.apache.org/jira/browse/SPARK-43183 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming > Affects Versions: 3.5.0 > Reporter: Jungtaek Lim > Assignee: Jungtaek Lim > Priority: Major > Fix For: 3.5.0 > > > People has been having a lot of confusions about update event on idleness; > it’s not only the matter of understanding but also comes up with various > types of complaints. For example, since we give the latest batch ID for > update event on idleness, if the listener implementation blindly performs > upsert based on batch ID, they are in risk to lose metrics. > This also complicates the logic because we have to memorize the execution for > the previous batch, which is arguably not necessary. > Because of this, we’d be better to move the idle event out of progress update > event and have separate callback method for this. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org