Blazer-007 commented on code in PR #4127:
URL: https://github.com/apache/gobblin/pull/4127#discussion_r2262024460
##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/SafeDatasetCommit.java:
##########
@@ -94,13 +94,17 @@ public Void call()
metricContext = Instrumented.getMetricContext(datasetState,
SafeDatasetCommit.class);
finalizeDatasetStateBeforeCommit(this.datasetState);
- // evaluate data quality at the dataset commit level, only when commit
source is CommitActivityImpl
- if
(SafeDatasetCommit.COMMIT_SRC_COMMIT_ACTIVITY_IMPL.equals(this.datasetCommitSrc))
{
+ // evaluate data quality at the dataset commit level, only when commit
source is CommitActivityImpl and policies are applied
+ JobState jobState = this.jobContext.getJobState();
+ String policiesApplied =
jobState.getProperties().getProperty(ConfigurationKeys.TASK_LEVEL_POLICY_LIST,
StringUtils.EMPTY);
+ log.info("Policies applied: {}", policiesApplied);
+ boolean shouldEvaluateDataQuality = !policiesApplied.isEmpty();
+ if (shouldEvaluateDataQuality &&
SafeDatasetCommit.COMMIT_SRC_COMMIT_ACTIVITY_IMPL.equals(this.datasetCommitSrc))
{
Review Comment:
Shouldn't this check be part of `evaluateAndEmitDatasetQuality()` function
itself ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
