himanshukandwal opened a new issue, #2957:
URL: https://github.com/apache/helix/issues/2957
### Describe the bug
When a Workflow is marked as deleted (TargetState.DELETE) via
`TaskDriver.delete` API, then in the workflow is deleted by the pipeline in the
next run. With deletion, the entries from ZK is deleted and the TaskDataCache
is also updated. However, the other entries are still present in ResourceConfig
(Base) cache. Usually, the next event which is in the pipeline is the
`ResourceConfigChange` Event, which takes care of the resourceConfig cache
update, but in case of a very busy cluster, other change event might be earlier
in the pipeline than `ResourceConfigChange` event. Now, the ResourceConfig
cache is updated `selectively` and not always so the ResourceConfig cache keeps
the deleted workflow entries, and when the Workflow (TaskDataCache) is
prepared, these (previously deleted) workflow entries comes back in again. This
causes, same workflows to be deleted multiple times. (Until we see the
`ResourceConfigChange` or `OnDemandRebalance` event).
TLDR: In Busy Cluster, the resourceConfig cache can take time to be
eventually consistent and this causes duplicate deletes of the same workflow.
Impact: Some customers delete and re-create workflow with same name and this
behavior causes the recently deleted workflow to be deleted again
(unexpectedly).
### To Reproduce
Since this happens in a busy cluster where a lot of events are happening
(indeterministically) so its not possible to reproduce this behavior. This
however is evident in the cluster logs.
### Expected behavior
Deletion should happen once only, and ResourceConfig cache should be updated
immediately to serve the next events in a reliable and deterministic manner.
### Additional context
Add any other context about the problem here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@helix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@helix.apache.org
For additional commands, e-mail: reviews-h...@helix.apache.org
