[
https://issues.apache.org/jira/browse/GOBBLIN-1782?focusedWorklogId=844450&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-844450
]
ASF GitHub Bot logged work on GOBBLIN-1782:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 08/Feb/23 20:02
Start Date: 08/Feb/23 20:02
Worklog Time Spent: 10m
Work Description: Will-Lo commented on code in PR #3639:
URL: https://github.com/apache/gobblin/pull/3639#discussion_r1100616376
##########
gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaJobStatusMonitor.java:
##########
@@ -251,10 +251,17 @@ static void
addJobStatusToStateStore(org.apache.gobblin.configuration.State jobS
int currentGeneration =
jobStatus.getPropAsInt(TimingEvent.FlowEventConstants.CURRENT_GENERATION_FIELD,
previousGeneration);
int previousAttempts =
previousJobStatus.getPropAsInt(TimingEvent.FlowEventConstants.CURRENT_ATTEMPTS_FIELD,
1);
int currentAttempts =
jobStatus.getPropAsInt(TimingEvent.FlowEventConstants.CURRENT_ATTEMPTS_FIELD,
previousAttempts);
+ // Verify if the current job status is flow status. If yes, we check
for its current execution status to be PENDING_RESUME (limiting to just resume
flow statuses)
+ // When the above two conditions satisfy, we NEED NOT check for the
out-of-order events since GaaS would manage the lifecycle of these events
+ // Hence, we update the merge state so that the flow can proceed with
its execution
+ if (jobName != null && jobGroup != null
+ && jobName.equals(JobStatusRetriever.NA_KEY) &&
jobGroup.equals(JobStatusRetriever.NA_KEY) &&
currentStatus.equals(ExecutionStatus.PENDING_RESUME.name())) {
Review Comment:
can we abstract this check into a function similar to how JobStatusRetriever
has this check?
##########
gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaJobStatusMonitor.java:
##########
@@ -251,10 +251,17 @@ static void
addJobStatusToStateStore(org.apache.gobblin.configuration.State jobS
int currentGeneration =
jobStatus.getPropAsInt(TimingEvent.FlowEventConstants.CURRENT_GENERATION_FIELD,
previousGeneration);
int previousAttempts =
previousJobStatus.getPropAsInt(TimingEvent.FlowEventConstants.CURRENT_ATTEMPTS_FIELD,
1);
int currentAttempts =
jobStatus.getPropAsInt(TimingEvent.FlowEventConstants.CURRENT_ATTEMPTS_FIELD,
previousAttempts);
+ // Verify if the current job status is flow status. If yes, we check
for its current execution status to be PENDING_RESUME (limiting to just resume
flow statuses)
+ // When the above two conditions satisfy, we NEED NOT check for the
out-of-order events since GaaS would manage the lifecycle of these events
+ // Hence, we update the merge state so that the flow can proceed with
its execution
+ if (jobName != null && jobGroup != null
+ && jobName.equals(JobStatusRetriever.NA_KEY) &&
jobGroup.equals(JobStatusRetriever.NA_KEY) &&
currentStatus.equals(ExecutionStatus.PENDING_RESUME.name())) {
Review Comment:
Also, since the side effect of this change is similar to the else case,
would it be simpler for us to append to the previous if statement and make an
initial check that the currentStatus != a flow status and pending resume?
Issue Time Tracking
-------------------
Worklog Id: (was: 844450)
Time Spent: 0.5h (was: 20m)
> Fix Merge State for Flow Pending Resume statuses
> ------------------------------------------------
>
> Key: GOBBLIN-1782
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1782
> Project: Apache Gobblin
> Issue Type: Improvement
> Reporter: Meeth Gala
> Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
