NealSun96 commented on a change in pull request #1326:
URL: https://github.com/apache/helix/pull/1326#discussion_r487197712
##########
File path:
helix-core/src/main/java/org/apache/helix/controller/stages/ResourceComputationStage.java
##########
@@ -185,4 +250,21 @@ private void addPartition(String partition, String
resourceName, Map<String, Res
resource.addPartition(partition);
}
+
+ private void addResourceConfigToResourceMap(String resourceName,
ResourceConfig resourceConfig,
+ ClusterConfig clusterConfig, Map<String, Resource> resourceMap,
+ Map<String, Resource> resourceToRebalance) {
+ Resource resource = new Resource(resourceName, clusterConfig,
resourceConfig);
+ resourceMap.put(resourceName, resource);
Review comment:
That is interesting. However, the next step of Task Framework will lead
to the removal of this section of ResourceComputationStage - it no longer needs
to create Resources from WorkflowConfigs/JobConfigs. (Check here
https://github.com/apache/helix/wiki/Task-Framework-IdealState-Dependency-Removal-Progression)
I will put a TODO here in case the next step doesn't happen.
##########
File path:
helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -519,6 +520,8 @@ protected void handleJobTimeout(JobContext jobCtx,
WorkflowContext workflowCtx,
_clusterStatusMonitor.updateJobCounters(jobCfg, TaskState.TIMED_OUT);
_rebalanceScheduler.removeScheduledRebalance(jobResource);
TaskUtil.cleanupJobIdealStateExtView(_manager.getHelixDataAccessor(),
jobResource);
+ // New pipeline trigger for workflow status update
+ RebalanceUtil.scheduleOnDemandPipeline(_manager.getClusterName(),0L,false);
Review comment:
`cleanupJobIdealStateExtView()` is kept for backward-compatibility
purpose and is marked as deprecated now; it will be removed on a later phase. I
suppose we put the on-demand rebalance in `cleanupJobIdealStateExtView()` now,
and move it out when we delete `cleanupJobIdealStateExtView()`. Do you think
that's necessary @jiajunwang ? Seems not to be too much difference.
##########
File path:
helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -519,6 +520,8 @@ protected void handleJobTimeout(JobContext jobCtx,
WorkflowContext workflowCtx,
_clusterStatusMonitor.updateJobCounters(jobCfg, TaskState.TIMED_OUT);
_rebalanceScheduler.removeScheduledRebalance(jobResource);
TaskUtil.cleanupJobIdealStateExtView(_manager.getHelixDataAccessor(),
jobResource);
+ // New pipeline trigger for workflow status update
+ RebalanceUtil.scheduleOnDemandPipeline(_manager.getClusterName(),0L,false);
Review comment:
`cleanupJobIdealStateExtView()` is kept for backward-compatibility
purpose and is marked as deprecated now; it will be removed on a later phase
(it could trigger rebalance if the participant side is not updated, and it
wouldn't trigger rebalance if the participant side is updated). I suppose we
put the on-demand rebalance in `cleanupJobIdealStateExtView()` now, and move it
out when we delete `cleanupJobIdealStateExtView()`. Do you think that's
necessary @jiajunwang ? Seems not to be too much difference.
##########
File path:
helix-core/src/main/java/org/apache/helix/controller/stages/ResourceComputationStage.java
##########
@@ -124,17 +205,15 @@ public void process(ClusterEvent event) throws Exception {
// don't overwrite ideal state settings
if (!resourceMap.containsKey(resourceName)) {
- addResource(resourceName, resourceMap);
- Resource resource = resourceMap.get(resourceName);
+ Resource resource = new Resource(resourceName);
resource.setStateModelDefRef(currentState.getStateModelDefRef());
resource.setStateModelFactoryName(currentState.getStateModelFactoryName());
resource.setBucketSize(currentState.getBucketSize());
resource.setBatchMessageMode(currentState.getBatchMessageMode());
- if (resource.getStateModelDefRef() == null && !isTaskCache
- || resource.getStateModelDefRef() != null && (
-
resource.getStateModelDefRef().equals(TaskConstants.STATE_MODEL_NAME) &&
isTaskCache
- ||
!resource.getStateModelDefRef().equals(TaskConstants.STATE_MODEL_NAME)
- && !isTaskCache)) {
+ if (!isTaskCache && !TaskConstants.STATE_MODEL_NAME
Review comment:
That's very smart - updating.
##########
File path:
helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -519,6 +520,8 @@ protected void handleJobTimeout(JobContext jobCtx,
WorkflowContext workflowCtx,
_clusterStatusMonitor.updateJobCounters(jobCfg, TaskState.TIMED_OUT);
_rebalanceScheduler.removeScheduledRebalance(jobResource);
TaskUtil.cleanupJobIdealStateExtView(_manager.getHelixDataAccessor(),
jobResource);
+ // New pipeline trigger for workflow status update
+ RebalanceUtil.scheduleOnDemandPipeline(_manager.getClusterName(),0L,false);
Review comment:
`cleanupJobIdealStateExtView()` is kept for backward-compatibility
purpose and is marked as deprecated now; it will be removed on a later phase
(it could trigger rebalance if the participant side is not updated, and it
wouldn't trigger rebalance if the participant side is updated). I suppose we
can put the on-demand rebalance in `cleanupJobIdealStateExtView()` now, and
move it out when we delete `cleanupJobIdealStateExtView()`. Do you think that's
necessary @jiajunwang ? Seems not to be too much difference.
##########
File path:
helix-core/src/main/java/org/apache/helix/controller/stages/ResourceComputationStage.java
##########
@@ -185,4 +250,21 @@ private void addPartition(String partition, String
resourceName, Map<String, Res
resource.addPartition(partition);
}
+
+ private void addResourceConfigToResourceMap(String resourceName,
ResourceConfig resourceConfig,
+ ClusterConfig clusterConfig, Map<String, Resource> resourceMap,
+ Map<String, Resource> resourceToRebalance) {
+ Resource resource = new Resource(resourceName, clusterConfig,
resourceConfig);
+ resourceMap.put(resourceName, resource);
Review comment:
That is interesting. However, the next step of Task Framework will lead
to the removal of this section of ResourceComputationStage - it no longer needs
to create Resources from WorkflowConfigs/JobConfigs. (Check here
https://github.com/apache/helix/wiki/Task-Framework-IdealState-Dependency-Removal-Progression)
I will put a TODO here in case the next step doesn't happen.
##########
File path:
helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -519,6 +520,8 @@ protected void handleJobTimeout(JobContext jobCtx,
WorkflowContext workflowCtx,
_clusterStatusMonitor.updateJobCounters(jobCfg, TaskState.TIMED_OUT);
_rebalanceScheduler.removeScheduledRebalance(jobResource);
TaskUtil.cleanupJobIdealStateExtView(_manager.getHelixDataAccessor(),
jobResource);
+ // New pipeline trigger for workflow status update
+ RebalanceUtil.scheduleOnDemandPipeline(_manager.getClusterName(),0L,false);
Review comment:
`cleanupJobIdealStateExtView()` is kept for backward-compatibility
purpose and is marked as deprecated now; it will be removed on a later phase. I
suppose we put the on-demand rebalance in `cleanupJobIdealStateExtView()` now,
and move it out when we delete `cleanupJobIdealStateExtView()`. Do you think
that's necessary @jiajunwang ? Seems not to be too much difference.
##########
File path:
helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -519,6 +520,8 @@ protected void handleJobTimeout(JobContext jobCtx,
WorkflowContext workflowCtx,
_clusterStatusMonitor.updateJobCounters(jobCfg, TaskState.TIMED_OUT);
_rebalanceScheduler.removeScheduledRebalance(jobResource);
TaskUtil.cleanupJobIdealStateExtView(_manager.getHelixDataAccessor(),
jobResource);
+ // New pipeline trigger for workflow status update
+ RebalanceUtil.scheduleOnDemandPipeline(_manager.getClusterName(),0L,false);
Review comment:
`cleanupJobIdealStateExtView()` is kept for backward-compatibility
purpose and is marked as deprecated now; it will be removed on a later phase
(it could trigger rebalance if the participant side is not updated, and it
wouldn't trigger rebalance if the participant side is updated). I suppose we
put the on-demand rebalance in `cleanupJobIdealStateExtView()` now, and move it
out when we delete `cleanupJobIdealStateExtView()`. Do you think that's
necessary @jiajunwang ? Seems not to be too much difference.
##########
File path:
helix-core/src/main/java/org/apache/helix/controller/stages/ResourceComputationStage.java
##########
@@ -124,17 +205,15 @@ public void process(ClusterEvent event) throws Exception {
// don't overwrite ideal state settings
if (!resourceMap.containsKey(resourceName)) {
- addResource(resourceName, resourceMap);
- Resource resource = resourceMap.get(resourceName);
+ Resource resource = new Resource(resourceName);
resource.setStateModelDefRef(currentState.getStateModelDefRef());
resource.setStateModelFactoryName(currentState.getStateModelFactoryName());
resource.setBucketSize(currentState.getBucketSize());
resource.setBatchMessageMode(currentState.getBatchMessageMode());
- if (resource.getStateModelDefRef() == null && !isTaskCache
- || resource.getStateModelDefRef() != null && (
-
resource.getStateModelDefRef().equals(TaskConstants.STATE_MODEL_NAME) &&
isTaskCache
- ||
!resource.getStateModelDefRef().equals(TaskConstants.STATE_MODEL_NAME)
- && !isTaskCache)) {
+ if (!isTaskCache && !TaskConstants.STATE_MODEL_NAME
Review comment:
That's very smart - updating.
##########
File path:
helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -519,6 +520,8 @@ protected void handleJobTimeout(JobContext jobCtx,
WorkflowContext workflowCtx,
_clusterStatusMonitor.updateJobCounters(jobCfg, TaskState.TIMED_OUT);
_rebalanceScheduler.removeScheduledRebalance(jobResource);
TaskUtil.cleanupJobIdealStateExtView(_manager.getHelixDataAccessor(),
jobResource);
+ // New pipeline trigger for workflow status update
+ RebalanceUtil.scheduleOnDemandPipeline(_manager.getClusterName(),0L,false);
Review comment:
`cleanupJobIdealStateExtView()` is kept for backward-compatibility
purpose and is marked as deprecated now; it will be removed on a later phase
(it could trigger rebalance if the participant side is not updated, and it
wouldn't trigger rebalance if the participant side is updated). I suppose we
can put the on-demand rebalance in `cleanupJobIdealStateExtView()` now, and
move it out when we delete `cleanupJobIdealStateExtView()`. Do you think that's
necessary @jiajunwang ? Seems not to be too much difference.
##########
File path:
helix-core/src/main/java/org/apache/helix/controller/stages/ResourceComputationStage.java
##########
@@ -185,4 +250,21 @@ private void addPartition(String partition, String
resourceName, Map<String, Res
resource.addPartition(partition);
}
+
+ private void addResourceConfigToResourceMap(String resourceName,
ResourceConfig resourceConfig,
+ ClusterConfig clusterConfig, Map<String, Resource> resourceMap,
+ Map<String, Resource> resourceToRebalance) {
+ Resource resource = new Resource(resourceName, clusterConfig,
resourceConfig);
+ resourceMap.put(resourceName, resource);
Review comment:
That is interesting. However, the next step of Task Framework will lead
to the removal of this section of ResourceComputationStage - it no longer needs
to create Resources from WorkflowConfigs/JobConfigs. (Check here
https://github.com/apache/helix/wiki/Task-Framework-IdealState-Dependency-Removal-Progression)
I will put a TODO here in case the next step doesn't happen.
##########
File path:
helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -519,6 +520,8 @@ protected void handleJobTimeout(JobContext jobCtx,
WorkflowContext workflowCtx,
_clusterStatusMonitor.updateJobCounters(jobCfg, TaskState.TIMED_OUT);
_rebalanceScheduler.removeScheduledRebalance(jobResource);
TaskUtil.cleanupJobIdealStateExtView(_manager.getHelixDataAccessor(),
jobResource);
+ // New pipeline trigger for workflow status update
+ RebalanceUtil.scheduleOnDemandPipeline(_manager.getClusterName(),0L,false);
Review comment:
`cleanupJobIdealStateExtView()` is kept for backward-compatibility
purpose and is marked as deprecated now; it will be removed on a later phase. I
suppose we put the on-demand rebalance in `cleanupJobIdealStateExtView()` now,
and move it out when we delete `cleanupJobIdealStateExtView()`. Do you think
that's necessary @jiajunwang ? Seems not to be too much difference.
##########
File path:
helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -519,6 +520,8 @@ protected void handleJobTimeout(JobContext jobCtx,
WorkflowContext workflowCtx,
_clusterStatusMonitor.updateJobCounters(jobCfg, TaskState.TIMED_OUT);
_rebalanceScheduler.removeScheduledRebalance(jobResource);
TaskUtil.cleanupJobIdealStateExtView(_manager.getHelixDataAccessor(),
jobResource);
+ // New pipeline trigger for workflow status update
+ RebalanceUtil.scheduleOnDemandPipeline(_manager.getClusterName(),0L,false);
Review comment:
`cleanupJobIdealStateExtView()` is kept for backward-compatibility
purpose and is marked as deprecated now; it will be removed on a later phase
(it could trigger rebalance if the participant side is not updated, and it
wouldn't trigger rebalance if the participant side is updated). I suppose we
put the on-demand rebalance in `cleanupJobIdealStateExtView()` now, and move it
out when we delete `cleanupJobIdealStateExtView()`. Do you think that's
necessary @jiajunwang ? Seems not to be too much difference.
##########
File path:
helix-core/src/main/java/org/apache/helix/controller/stages/ResourceComputationStage.java
##########
@@ -124,17 +205,15 @@ public void process(ClusterEvent event) throws Exception {
// don't overwrite ideal state settings
if (!resourceMap.containsKey(resourceName)) {
- addResource(resourceName, resourceMap);
- Resource resource = resourceMap.get(resourceName);
+ Resource resource = new Resource(resourceName);
resource.setStateModelDefRef(currentState.getStateModelDefRef());
resource.setStateModelFactoryName(currentState.getStateModelFactoryName());
resource.setBucketSize(currentState.getBucketSize());
resource.setBatchMessageMode(currentState.getBatchMessageMode());
- if (resource.getStateModelDefRef() == null && !isTaskCache
- || resource.getStateModelDefRef() != null && (
-
resource.getStateModelDefRef().equals(TaskConstants.STATE_MODEL_NAME) &&
isTaskCache
- ||
!resource.getStateModelDefRef().equals(TaskConstants.STATE_MODEL_NAME)
- && !isTaskCache)) {
+ if (!isTaskCache && !TaskConstants.STATE_MODEL_NAME
Review comment:
That's very smart - updating.
##########
File path:
helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -519,6 +520,8 @@ protected void handleJobTimeout(JobContext jobCtx,
WorkflowContext workflowCtx,
_clusterStatusMonitor.updateJobCounters(jobCfg, TaskState.TIMED_OUT);
_rebalanceScheduler.removeScheduledRebalance(jobResource);
TaskUtil.cleanupJobIdealStateExtView(_manager.getHelixDataAccessor(),
jobResource);
+ // New pipeline trigger for workflow status update
+ RebalanceUtil.scheduleOnDemandPipeline(_manager.getClusterName(),0L,false);
Review comment:
`cleanupJobIdealStateExtView()` is kept for backward-compatibility
purpose and is marked as deprecated now; it will be removed on a later phase
(it could trigger rebalance if the participant side is not updated, and it
wouldn't trigger rebalance if the participant side is updated). I suppose we
can put the on-demand rebalance in `cleanupJobIdealStateExtView()` now, and
move it out when we delete `cleanupJobIdealStateExtView()`. Do you think that's
necessary @jiajunwang ? Seems not to be too much difference.
##########
File path:
helix-core/src/main/java/org/apache/helix/controller/stages/ResourceComputationStage.java
##########
@@ -185,4 +250,21 @@ private void addPartition(String partition, String
resourceName, Map<String, Res
resource.addPartition(partition);
}
+
+ private void addResourceConfigToResourceMap(String resourceName,
ResourceConfig resourceConfig,
+ ClusterConfig clusterConfig, Map<String, Resource> resourceMap,
+ Map<String, Resource> resourceToRebalance) {
+ Resource resource = new Resource(resourceName, clusterConfig,
resourceConfig);
+ resourceMap.put(resourceName, resource);
Review comment:
That is interesting. However, the next step of Task Framework will lead
to the removal of this section of ResourceComputationStage - it no longer needs
to create Resources from WorkflowConfigs/JobConfigs. (Check here
https://github.com/apache/helix/wiki/Task-Framework-IdealState-Dependency-Removal-Progression)
I will put a TODO here in case the next step doesn't happen.
##########
File path:
helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -519,6 +520,8 @@ protected void handleJobTimeout(JobContext jobCtx,
WorkflowContext workflowCtx,
_clusterStatusMonitor.updateJobCounters(jobCfg, TaskState.TIMED_OUT);
_rebalanceScheduler.removeScheduledRebalance(jobResource);
TaskUtil.cleanupJobIdealStateExtView(_manager.getHelixDataAccessor(),
jobResource);
+ // New pipeline trigger for workflow status update
+ RebalanceUtil.scheduleOnDemandPipeline(_manager.getClusterName(),0L,false);
Review comment:
`cleanupJobIdealStateExtView()` is kept for backward-compatibility
purpose and is marked as deprecated now; it will be removed on a later phase. I
suppose we put the on-demand rebalance in `cleanupJobIdealStateExtView()` now,
and move it out when we delete `cleanupJobIdealStateExtView()`. Do you think
that's necessary @jiajunwang ? Seems not to be too much difference.
##########
File path:
helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -519,6 +520,8 @@ protected void handleJobTimeout(JobContext jobCtx,
WorkflowContext workflowCtx,
_clusterStatusMonitor.updateJobCounters(jobCfg, TaskState.TIMED_OUT);
_rebalanceScheduler.removeScheduledRebalance(jobResource);
TaskUtil.cleanupJobIdealStateExtView(_manager.getHelixDataAccessor(),
jobResource);
+ // New pipeline trigger for workflow status update
+ RebalanceUtil.scheduleOnDemandPipeline(_manager.getClusterName(),0L,false);
Review comment:
`cleanupJobIdealStateExtView()` is kept for backward-compatibility
purpose and is marked as deprecated now; it will be removed on a later phase
(it could trigger rebalance if the participant side is not updated, and it
wouldn't trigger rebalance if the participant side is updated). I suppose we
put the on-demand rebalance in `cleanupJobIdealStateExtView()` now, and move it
out when we delete `cleanupJobIdealStateExtView()`. Do you think that's
necessary @jiajunwang ? Seems not to be too much difference.
##########
File path:
helix-core/src/main/java/org/apache/helix/controller/stages/ResourceComputationStage.java
##########
@@ -124,17 +205,15 @@ public void process(ClusterEvent event) throws Exception {
// don't overwrite ideal state settings
if (!resourceMap.containsKey(resourceName)) {
- addResource(resourceName, resourceMap);
- Resource resource = resourceMap.get(resourceName);
+ Resource resource = new Resource(resourceName);
resource.setStateModelDefRef(currentState.getStateModelDefRef());
resource.setStateModelFactoryName(currentState.getStateModelFactoryName());
resource.setBucketSize(currentState.getBucketSize());
resource.setBatchMessageMode(currentState.getBatchMessageMode());
- if (resource.getStateModelDefRef() == null && !isTaskCache
- || resource.getStateModelDefRef() != null && (
-
resource.getStateModelDefRef().equals(TaskConstants.STATE_MODEL_NAME) &&
isTaskCache
- ||
!resource.getStateModelDefRef().equals(TaskConstants.STATE_MODEL_NAME)
- && !isTaskCache)) {
+ if (!isTaskCache && !TaskConstants.STATE_MODEL_NAME
Review comment:
That's very smart - updating.
##########
File path:
helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -519,6 +520,8 @@ protected void handleJobTimeout(JobContext jobCtx,
WorkflowContext workflowCtx,
_clusterStatusMonitor.updateJobCounters(jobCfg, TaskState.TIMED_OUT);
_rebalanceScheduler.removeScheduledRebalance(jobResource);
TaskUtil.cleanupJobIdealStateExtView(_manager.getHelixDataAccessor(),
jobResource);
+ // New pipeline trigger for workflow status update
+ RebalanceUtil.scheduleOnDemandPipeline(_manager.getClusterName(),0L,false);
Review comment:
`cleanupJobIdealStateExtView()` is kept for backward-compatibility
purpose and is marked as deprecated now; it will be removed on a later phase
(it could trigger rebalance if the participant side is not updated, and it
wouldn't trigger rebalance if the participant side is updated). I suppose we
can put the on-demand rebalance in `cleanupJobIdealStateExtView()` now, and
move it out when we delete `cleanupJobIdealStateExtView()`. Do you think that's
necessary @jiajunwang ? Seems not to be too much difference.
##########
File path:
helix-core/src/main/java/org/apache/helix/controller/stages/ResourceComputationStage.java
##########
@@ -185,4 +250,21 @@ private void addPartition(String partition, String
resourceName, Map<String, Res
resource.addPartition(partition);
}
+
+ private void addResourceConfigToResourceMap(String resourceName,
ResourceConfig resourceConfig,
+ ClusterConfig clusterConfig, Map<String, Resource> resourceMap,
+ Map<String, Resource> resourceToRebalance) {
+ Resource resource = new Resource(resourceName, clusterConfig,
resourceConfig);
+ resourceMap.put(resourceName, resource);
Review comment:
That is interesting. However, the next step of Task Framework will lead
to the removal of this section of ResourceComputationStage - it no longer needs
to create Resources from WorkflowConfigs/JobConfigs. (Check here
https://github.com/apache/helix/wiki/Task-Framework-IdealState-Dependency-Removal-Progression)
I will put a TODO here in case the next step doesn't happen.
##########
File path:
helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -519,6 +520,8 @@ protected void handleJobTimeout(JobContext jobCtx,
WorkflowContext workflowCtx,
_clusterStatusMonitor.updateJobCounters(jobCfg, TaskState.TIMED_OUT);
_rebalanceScheduler.removeScheduledRebalance(jobResource);
TaskUtil.cleanupJobIdealStateExtView(_manager.getHelixDataAccessor(),
jobResource);
+ // New pipeline trigger for workflow status update
+ RebalanceUtil.scheduleOnDemandPipeline(_manager.getClusterName(),0L,false);
Review comment:
`cleanupJobIdealStateExtView()` is kept for backward-compatibility
purpose and is marked as deprecated now; it will be removed on a later phase. I
suppose we put the on-demand rebalance in `cleanupJobIdealStateExtView()` now,
and move it out when we delete `cleanupJobIdealStateExtView()`. Do you think
that's necessary @jiajunwang ? Seems not to be too much difference.
##########
File path:
helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -519,6 +520,8 @@ protected void handleJobTimeout(JobContext jobCtx,
WorkflowContext workflowCtx,
_clusterStatusMonitor.updateJobCounters(jobCfg, TaskState.TIMED_OUT);
_rebalanceScheduler.removeScheduledRebalance(jobResource);
TaskUtil.cleanupJobIdealStateExtView(_manager.getHelixDataAccessor(),
jobResource);
+ // New pipeline trigger for workflow status update
+ RebalanceUtil.scheduleOnDemandPipeline(_manager.getClusterName(),0L,false);
Review comment:
`cleanupJobIdealStateExtView()` is kept for backward-compatibility
purpose and is marked as deprecated now; it will be removed on a later phase
(it could trigger rebalance if the participant side is not updated, and it
wouldn't trigger rebalance if the participant side is updated). I suppose we
put the on-demand rebalance in `cleanupJobIdealStateExtView()` now, and move it
out when we delete `cleanupJobIdealStateExtView()`. Do you think that's
necessary @jiajunwang ? Seems not to be too much difference.
##########
File path:
helix-core/src/main/java/org/apache/helix/controller/stages/ResourceComputationStage.java
##########
@@ -124,17 +205,15 @@ public void process(ClusterEvent event) throws Exception {
// don't overwrite ideal state settings
if (!resourceMap.containsKey(resourceName)) {
- addResource(resourceName, resourceMap);
- Resource resource = resourceMap.get(resourceName);
+ Resource resource = new Resource(resourceName);
resource.setStateModelDefRef(currentState.getStateModelDefRef());
resource.setStateModelFactoryName(currentState.getStateModelFactoryName());
resource.setBucketSize(currentState.getBucketSize());
resource.setBatchMessageMode(currentState.getBatchMessageMode());
- if (resource.getStateModelDefRef() == null && !isTaskCache
- || resource.getStateModelDefRef() != null && (
-
resource.getStateModelDefRef().equals(TaskConstants.STATE_MODEL_NAME) &&
isTaskCache
- ||
!resource.getStateModelDefRef().equals(TaskConstants.STATE_MODEL_NAME)
- && !isTaskCache)) {
+ if (!isTaskCache && !TaskConstants.STATE_MODEL_NAME
Review comment:
That's very smart - updating.
##########
File path:
helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -519,6 +520,8 @@ protected void handleJobTimeout(JobContext jobCtx,
WorkflowContext workflowCtx,
_clusterStatusMonitor.updateJobCounters(jobCfg, TaskState.TIMED_OUT);
_rebalanceScheduler.removeScheduledRebalance(jobResource);
TaskUtil.cleanupJobIdealStateExtView(_manager.getHelixDataAccessor(),
jobResource);
+ // New pipeline trigger for workflow status update
+ RebalanceUtil.scheduleOnDemandPipeline(_manager.getClusterName(),0L,false);
Review comment:
`cleanupJobIdealStateExtView()` is kept for backward-compatibility
purpose and is marked as deprecated now; it will be removed on a later phase
(it could trigger rebalance if the participant side is not updated, and it
wouldn't trigger rebalance if the participant side is updated). I suppose we
can put the on-demand rebalance in `cleanupJobIdealStateExtView()` now, and
move it out when we delete `cleanupJobIdealStateExtView()`. Do you think that's
necessary @jiajunwang ? Seems not to be too much difference.
##########
File path:
helix-core/src/main/java/org/apache/helix/controller/stages/ResourceComputationStage.java
##########
@@ -185,4 +250,21 @@ private void addPartition(String partition, String
resourceName, Map<String, Res
resource.addPartition(partition);
}
+
+ private void addResourceConfigToResourceMap(String resourceName,
ResourceConfig resourceConfig,
+ ClusterConfig clusterConfig, Map<String, Resource> resourceMap,
+ Map<String, Resource> resourceToRebalance) {
+ Resource resource = new Resource(resourceName, clusterConfig,
resourceConfig);
+ resourceMap.put(resourceName, resource);
Review comment:
That is interesting. However, the next step of Task Framework will lead
to the removal of this section of ResourceComputationStage - it no longer needs
to create Resources from WorkflowConfigs/JobConfigs. (Check here
https://github.com/apache/helix/wiki/Task-Framework-IdealState-Dependency-Removal-Progression)
I will put a TODO here in case the next step doesn't happen.
##########
File path:
helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -519,6 +520,8 @@ protected void handleJobTimeout(JobContext jobCtx,
WorkflowContext workflowCtx,
_clusterStatusMonitor.updateJobCounters(jobCfg, TaskState.TIMED_OUT);
_rebalanceScheduler.removeScheduledRebalance(jobResource);
TaskUtil.cleanupJobIdealStateExtView(_manager.getHelixDataAccessor(),
jobResource);
+ // New pipeline trigger for workflow status update
+ RebalanceUtil.scheduleOnDemandPipeline(_manager.getClusterName(),0L,false);
Review comment:
`cleanupJobIdealStateExtView()` is kept for backward-compatibility
purpose and is marked as deprecated now; it will be removed on a later phase. I
suppose we put the on-demand rebalance in `cleanupJobIdealStateExtView()` now,
and move it out when we delete `cleanupJobIdealStateExtView()`. Do you think
that's necessary @jiajunwang ? Seems not to be too much difference.
##########
File path:
helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -519,6 +520,8 @@ protected void handleJobTimeout(JobContext jobCtx,
WorkflowContext workflowCtx,
_clusterStatusMonitor.updateJobCounters(jobCfg, TaskState.TIMED_OUT);
_rebalanceScheduler.removeScheduledRebalance(jobResource);
TaskUtil.cleanupJobIdealStateExtView(_manager.getHelixDataAccessor(),
jobResource);
+ // New pipeline trigger for workflow status update
+ RebalanceUtil.scheduleOnDemandPipeline(_manager.getClusterName(),0L,false);
Review comment:
`cleanupJobIdealStateExtView()` is kept for backward-compatibility
purpose and is marked as deprecated now; it will be removed on a later phase
(it could trigger rebalance if the participant side is not updated, and it
wouldn't trigger rebalance if the participant side is updated). I suppose we
put the on-demand rebalance in `cleanupJobIdealStateExtView()` now, and move it
out when we delete `cleanupJobIdealStateExtView()`. Do you think that's
necessary @jiajunwang ? Seems not to be too much difference.
##########
File path:
helix-core/src/main/java/org/apache/helix/controller/stages/ResourceComputationStage.java
##########
@@ -124,17 +205,15 @@ public void process(ClusterEvent event) throws Exception {
// don't overwrite ideal state settings
if (!resourceMap.containsKey(resourceName)) {
- addResource(resourceName, resourceMap);
- Resource resource = resourceMap.get(resourceName);
+ Resource resource = new Resource(resourceName);
resource.setStateModelDefRef(currentState.getStateModelDefRef());
resource.setStateModelFactoryName(currentState.getStateModelFactoryName());
resource.setBucketSize(currentState.getBucketSize());
resource.setBatchMessageMode(currentState.getBatchMessageMode());
- if (resource.getStateModelDefRef() == null && !isTaskCache
- || resource.getStateModelDefRef() != null && (
-
resource.getStateModelDefRef().equals(TaskConstants.STATE_MODEL_NAME) &&
isTaskCache
- ||
!resource.getStateModelDefRef().equals(TaskConstants.STATE_MODEL_NAME)
- && !isTaskCache)) {
+ if (!isTaskCache && !TaskConstants.STATE_MODEL_NAME
Review comment:
That's very smart - updating.
##########
File path:
helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -519,6 +520,8 @@ protected void handleJobTimeout(JobContext jobCtx,
WorkflowContext workflowCtx,
_clusterStatusMonitor.updateJobCounters(jobCfg, TaskState.TIMED_OUT);
_rebalanceScheduler.removeScheduledRebalance(jobResource);
TaskUtil.cleanupJobIdealStateExtView(_manager.getHelixDataAccessor(),
jobResource);
+ // New pipeline trigger for workflow status update
+ RebalanceUtil.scheduleOnDemandPipeline(_manager.getClusterName(),0L,false);
Review comment:
`cleanupJobIdealStateExtView()` is kept for backward-compatibility
purpose and is marked as deprecated now; it will be removed on a later phase
(it could trigger rebalance if the participant side is not updated, and it
wouldn't trigger rebalance if the participant side is updated). I suppose we
can put the on-demand rebalance in `cleanupJobIdealStateExtView()` now, and
move it out when we delete `cleanupJobIdealStateExtView()`. Do you think that's
necessary @jiajunwang ? Seems not to be too much difference.
##########
File path:
helix-core/src/main/java/org/apache/helix/controller/stages/ResourceComputationStage.java
##########
@@ -185,4 +250,21 @@ private void addPartition(String partition, String
resourceName, Map<String, Res
resource.addPartition(partition);
}
+
+ private void addResourceConfigToResourceMap(String resourceName,
ResourceConfig resourceConfig,
+ ClusterConfig clusterConfig, Map<String, Resource> resourceMap,
+ Map<String, Resource> resourceToRebalance) {
+ Resource resource = new Resource(resourceName, clusterConfig,
resourceConfig);
+ resourceMap.put(resourceName, resource);
Review comment:
That is interesting. However, the next step of Task Framework will lead
to the removal of this section of ResourceComputationStage - it no longer needs
to create Resources from WorkflowConfigs/JobConfigs. (Check here
https://github.com/apache/helix/wiki/Task-Framework-IdealState-Dependency-Removal-Progression)
I will put a TODO here in case the next step doesn't happen.
##########
File path:
helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -519,6 +520,8 @@ protected void handleJobTimeout(JobContext jobCtx,
WorkflowContext workflowCtx,
_clusterStatusMonitor.updateJobCounters(jobCfg, TaskState.TIMED_OUT);
_rebalanceScheduler.removeScheduledRebalance(jobResource);
TaskUtil.cleanupJobIdealStateExtView(_manager.getHelixDataAccessor(),
jobResource);
+ // New pipeline trigger for workflow status update
+ RebalanceUtil.scheduleOnDemandPipeline(_manager.getClusterName(),0L,false);
Review comment:
`cleanupJobIdealStateExtView()` is kept for backward-compatibility
purpose and is marked as deprecated now; it will be removed on a later phase. I
suppose we put the on-demand rebalance in `cleanupJobIdealStateExtView()` now,
and move it out when we delete `cleanupJobIdealStateExtView()`. Do you think
that's necessary @jiajunwang ? Seems not to be too much difference.
##########
File path:
helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -519,6 +520,8 @@ protected void handleJobTimeout(JobContext jobCtx,
WorkflowContext workflowCtx,
_clusterStatusMonitor.updateJobCounters(jobCfg, TaskState.TIMED_OUT);
_rebalanceScheduler.removeScheduledRebalance(jobResource);
TaskUtil.cleanupJobIdealStateExtView(_manager.getHelixDataAccessor(),
jobResource);
+ // New pipeline trigger for workflow status update
+ RebalanceUtil.scheduleOnDemandPipeline(_manager.getClusterName(),0L,false);
Review comment:
`cleanupJobIdealStateExtView()` is kept for backward-compatibility
purpose and is marked as deprecated now; it will be removed on a later phase
(it could trigger rebalance if the participant side is not updated, and it
wouldn't trigger rebalance if the participant side is updated). I suppose we
put the on-demand rebalance in `cleanupJobIdealStateExtView()` now, and move it
out when we delete `cleanupJobIdealStateExtView()`. Do you think that's
necessary @jiajunwang ? Seems not to be too much difference.
##########
File path:
helix-core/src/main/java/org/apache/helix/controller/stages/ResourceComputationStage.java
##########
@@ -124,17 +205,15 @@ public void process(ClusterEvent event) throws Exception {
// don't overwrite ideal state settings
if (!resourceMap.containsKey(resourceName)) {
- addResource(resourceName, resourceMap);
- Resource resource = resourceMap.get(resourceName);
+ Resource resource = new Resource(resourceName);
resource.setStateModelDefRef(currentState.getStateModelDefRef());
resource.setStateModelFactoryName(currentState.getStateModelFactoryName());
resource.setBucketSize(currentState.getBucketSize());
resource.setBatchMessageMode(currentState.getBatchMessageMode());
- if (resource.getStateModelDefRef() == null && !isTaskCache
- || resource.getStateModelDefRef() != null && (
-
resource.getStateModelDefRef().equals(TaskConstants.STATE_MODEL_NAME) &&
isTaskCache
- ||
!resource.getStateModelDefRef().equals(TaskConstants.STATE_MODEL_NAME)
- && !isTaskCache)) {
+ if (!isTaskCache && !TaskConstants.STATE_MODEL_NAME
Review comment:
That's very smart - updating.
##########
File path:
helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -519,6 +520,8 @@ protected void handleJobTimeout(JobContext jobCtx,
WorkflowContext workflowCtx,
_clusterStatusMonitor.updateJobCounters(jobCfg, TaskState.TIMED_OUT);
_rebalanceScheduler.removeScheduledRebalance(jobResource);
TaskUtil.cleanupJobIdealStateExtView(_manager.getHelixDataAccessor(),
jobResource);
+ // New pipeline trigger for workflow status update
+ RebalanceUtil.scheduleOnDemandPipeline(_manager.getClusterName(),0L,false);
Review comment:
`cleanupJobIdealStateExtView()` is kept for backward-compatibility
purpose and is marked as deprecated now; it will be removed on a later phase
(it could trigger rebalance if the participant side is not updated, and it
wouldn't trigger rebalance if the participant side is updated). I suppose we
can put the on-demand rebalance in `cleanupJobIdealStateExtView()` now, and
move it out when we delete `cleanupJobIdealStateExtView()`. Do you think that's
necessary @jiajunwang ? Seems not to be too much difference.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
