vsinghal85 commented on code in PR #4113:
URL: https://github.com/apache/gobblin/pull/4113#discussion_r2244505067
##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/fork/Fork.java:
##########
@@ -613,6 +629,11 @@ private boolean checkDataQuality(Optional<Object> schema)
TaskLevelPolicyCheckResults taskResults =
this.taskContext.getTaskLevelPolicyChecker(this.forkTaskState,
this.branches > 1 ? this.index : -1)
.executePolicies();
+ boolean allRequiredPoliciesPassed =
taskResults.getPolicyResults().entrySet().stream()
+ .filter(e -> e.getValue() == TaskLevelPolicy.Type.FAIL)
+ .allMatch(e -> e.getKey() == TaskLevelPolicy.Result.PASSED);
+ forkTaskState.setProp(ConfigurationKeys.TASK_LEVEL_POLICY_RESULT_KEY,
+ allRequiredPoliciesPassed ? DataQualityStatus.PASSED.name() :
DataQualityStatus.FAILED.name());
Review Comment:
This is eventually what above statement does, it filters out all the
required policies in the first step, and then checks whether all these required
policies had their data quality result passed or not, if any policy would fail,
it would result in overall failure of fork data quality.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
