jiajunwang commented on a change in pull request #1326:
URL: https://github.com/apache/helix/pull/1326#discussion_r486786354
##########
File path:
helix-core/src/main/java/org/apache/helix/controller/stages/ResourceComputationStage.java
##########
@@ -124,17 +205,15 @@ public void process(ClusterEvent event) throws Exception {
// don't overwrite ideal state settings
if (!resourceMap.containsKey(resourceName)) {
- addResource(resourceName, resourceMap);
- Resource resource = resourceMap.get(resourceName);
+ Resource resource = new Resource(resourceName);
resource.setStateModelDefRef(currentState.getStateModelDefRef());
resource.setStateModelFactoryName(currentState.getStateModelFactoryName());
resource.setBucketSize(currentState.getBucketSize());
resource.setBatchMessageMode(currentState.getBatchMessageMode());
- if (resource.getStateModelDefRef() == null && !isTaskCache
- || resource.getStateModelDefRef() != null && (
-
resource.getStateModelDefRef().equals(TaskConstants.STATE_MODEL_NAME) &&
isTaskCache
- ||
!resource.getStateModelDefRef().equals(TaskConstants.STATE_MODEL_NAME)
- && !isTaskCache)) {
+ if (!isTaskCache && !TaskConstants.STATE_MODEL_NAME
Review comment:
How about "isTaskCache ==
TaskConstants.STATE_MODEL_NAME.equals(resource.getStateModelDefRef())"
You can comment that we are trying to match for 2 conditions:
1. If resource state model def is null, then we only proceed if it is a
regular resource pipeline.
2. If resource state model def is not null, then we only proceed if the
pipeline type matches the resource state model type.
##########
File path:
helix-core/src/main/java/org/apache/helix/controller/stages/ResourceComputationStage.java
##########
@@ -98,6 +102,15 @@ public void process(ClusterEvent event) throws Exception {
}
}
Review comment:
Yeah, that's not too bad.
I was thinking if the RESOURCES attribute is still necessary. But might be.
For example, for the targeted jobs. So let's keep it for now.
##########
File path:
helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -519,6 +520,8 @@ protected void handleJobTimeout(JobContext jobCtx,
WorkflowContext workflowCtx,
_clusterStatusMonitor.updateJobCounters(jobCfg, TaskState.TIMED_OUT);
_rebalanceScheduler.removeScheduledRebalance(jobResource);
TaskUtil.cleanupJobIdealStateExtView(_manager.getHelixDataAccessor(),
jobResource);
+ // New pipeline trigger for workflow status update
+ RebalanceUtil.scheduleOnDemandPipeline(_manager.getClusterName(),0L,false);
Review comment:
Then, can we add scheduleOnDemandPipeline into
cleanupJobIdealStateExtView() ? Otherwise, cleanupJobIdealStateExtView is not a
complete call that triggers the expected rebalance, right?
It also helps to simplify code, IMO.
##########
File path:
helix-core/src/main/java/org/apache/helix/controller/stages/ResourceComputationStage.java
##########
@@ -185,4 +250,21 @@ private void addPartition(String partition, String
resourceName, Map<String, Res
resource.addPartition(partition);
}
+
+ private void addResourceConfigToResourceMap(String resourceName,
ResourceConfig resourceConfig,
+ ClusterConfig clusterConfig, Map<String, Resource> resourceMap,
+ Map<String, Resource> resourceToRebalance) {
+ Resource resource = new Resource(resourceName, clusterConfig,
resourceConfig);
+ resourceMap.put(resourceName, resource);
Review comment:
Not a must, but we did add a IS to ResourceConfig conversion method. It
is ResourceConfig.mergeIdealStateWithResourceConfig(). So in theory, you can
keep only this method by converting the IS into ResourceConfig, then use this
method to add to the resource map.
1. This helps to reduce duplicate code.
2. ResourceConfig is the one we are going to use in the near future. IS will
be transformed for controller output only. So that logic will be replaced
anyway.
Up to you if you want to do it now or in the future. If in the future,
please add a TODO here.
##########
File path:
helix-core/src/main/java/org/apache/helix/controller/stages/ResourceComputationStage.java
##########
@@ -185,4 +250,21 @@ private void addPartition(String partition, String
resourceName, Map<String, Res
resource.addPartition(partition);
}
+
+ private void addResourceConfigToResourceMap(String resourceName,
ResourceConfig resourceConfig,
+ ClusterConfig clusterConfig, Map<String, Resource> resourceMap,
+ Map<String, Resource> resourceToRebalance) {
+ Resource resource = new Resource(resourceName, clusterConfig,
resourceConfig);
+ resourceMap.put(resourceName, resource);
Review comment:
Sure, that also makes sense.
##########
File path:
helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -519,6 +520,8 @@ protected void handleJobTimeout(JobContext jobCtx,
WorkflowContext workflowCtx,
_clusterStatusMonitor.updateJobCounters(jobCfg, TaskState.TIMED_OUT);
_rebalanceScheduler.removeScheduledRebalance(jobResource);
TaskUtil.cleanupJobIdealStateExtView(_manager.getHelixDataAccessor(),
jobResource);
+ // New pipeline trigger for workflow status update
+ RebalanceUtil.scheduleOnDemandPipeline(_manager.getClusterName(),0L,false);
Review comment:
This is necessary. So no matter who does the cleanup later, he or she
won't forget to remove the scheduleOnDemandPipeline(). High-level, these 2
logics are bundled. And we shall not assume everyone remembers this
relationship by their memory or document. Organize the code to ensure it.
##########
File path:
helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -519,6 +520,8 @@ protected void handleJobTimeout(JobContext jobCtx,
WorkflowContext workflowCtx,
_clusterStatusMonitor.updateJobCounters(jobCfg, TaskState.TIMED_OUT);
_rebalanceScheduler.removeScheduledRebalance(jobResource);
TaskUtil.cleanupJobIdealStateExtView(_manager.getHelixDataAccessor(),
jobResource);
+ // New pipeline trigger for workflow status update
+ RebalanceUtil.scheduleOnDemandPipeline(_manager.getClusterName(),0L,false);
Review comment:
As we discussed, please add a TODO there for the future change. For now,
it is OK to be left here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
