Dong Wang created SPARK-28781: --------------------------------- Summary: Unneccesary persist in PeriodicCheckpointer.update() Key: SPARK-28781 URL: https://issues.apache.org/jira/browse/SPARK-28781 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 2.4.3 Environment: Once update(newData) is called, newData is persisted. However, only when the checkpoint is handling, the cached data is used for the second time (do the checkpoint job). The other data which is not satisfied to the checkpoint condition is actually unnecessary to be persisted. The persistedQueue avoids too much unnecessary cached data, but it is best to avoid every unnecessary persist operation. Reporter: Dong Wang
* Once the update is called, newData is persisted. However, only when the checkpoint is handling, the persist data is used for the second time. The other data which is not satisfied to the checkpoint condition is unnecessary to be cached. The persistedQueue avoids too many unnecessary cached data, but it is best to avoid every unnecessary persist operation. -- This message was sent by Atlassian Jira (v8.3.2#803003) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org