jiajunwang commented on a change in pull request #1076:
URL: https://github.com/apache/helix/pull/1076#discussion_r437743715
##########
File path:
helix-core/src/main/java/org/apache/helix/controller/stages/TaskGarbageCollectionStage.java
##########
@@ -23,34 +27,53 @@ public AsyncWorkerType getAsyncWorkerType() {
}
@Override
- public void execute(ClusterEvent event) {
+ public void process(ClusterEvent event) throws Exception {
WorkflowControllerDataProvider dataProvider =
event.getAttribute(AttributeName.ControllerDataProvider.name());
+ event.addAttribute(AttributeName.WORKFLOW_CONFIG_MAP.name(),
+ dataProvider.getWorkflowConfigMap());
Review comment:
So which way do you think would be better? And what is the additional
latency we are talking about here?
So what I proposed is that we finish the main rebalance calculation first.
During that computing, we know what workflow/job is no longer alive. Then we
record them in the TaskCache. Then at the end of the pipeline, pass that list
to the GC stage which will remove thing async. I think it would be helpful
because,
1. no additional calculation to check if an object needs to be purged, since
the rebalance pipeline will have to check for this anyway.
2. no parallel logic to determine if a job or workflow needs to be purged
separately. So less conflict, and fewer race conditions.
There might be TF logic prevents us from doing that, but please investigate
and we can talk if there are any blocks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
