alirezazamani commented on a change in pull request #1422:
URL: https://github.com/apache/helix/pull/1422#discussion_r499724813
##########
File path:
helix-core/src/main/java/org/apache/helix/task/AbstractTaskDispatcher.java
##########
@@ -122,30 +122,37 @@ public void updatePreviousAssignedTasksStatus(
Set<Integer> donePartitions = new TreeSet<>();
for (int pId : pSet) {
final String pName = pName(jobResource, pId);
- TaskPartitionState currState =
updateJobContextAndGetTaskCurrentState(currStateOutput,
+ TaskPartitionState currState = getTaskCurrentState(currStateOutput,
jobResource, pId, pName, instance, jobCtx, jobTgtState);
- if (!instance.equals(jobCtx.getAssignedParticipant(pId))) {
- LOG.warn(
- "Instance {} does not match the assigned participant for pId {}
in the job context. Skipping task scheduling.",
- instance, pId);
- continue;
- }
-
// Check for pending state transitions on this (partition, instance).
If there is a pending
// state transition, we prioritize this pending state transition and
set the assignment from
// this pending state transition, essentially "waiting" until this
pending message clears
+ // If there is a pending message, we should not continue to update the
context because from
Review comment:
This is more toward the functionality of our scheduling. Without this
change, there is a whole in our scheduling decision. Before IS removal change,
we were relying on IS to remove the last CS from the participants. However,
removing IS revealed this issue. This PR basically handles these cases and
helps CS to be cleaned up after the job is being completed. Also, we think
about it logically, it makes sense to consider ST not complete when a pending
message has not been deleted from the ZK.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
