vsinghal85 commented on code in PR #4113:
URL: https://github.com/apache/gobblin/pull/4113#discussion_r2244838065
##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/SafeDatasetCommit.java:
##########
@@ -90,6 +94,14 @@ public Void call()
metricContext = Instrumented.getMetricContext(datasetState,
SafeDatasetCommit.class);
finalizeDatasetStateBeforeCommit(this.datasetState);
+ // evaluate data quality at the dataset commit level, only when commit
source is CommitActivityImpl
+
if(SafeDatasetCommit.COMMIT_SRC_COMMIT_ACTIVITY_IMPL.equals(this.datasetCommitSrc)){
+ log.info("Evaluating data quality for commit activity for dataset {}.",
this.datasetUrn);
+ evaluateAndEmitDatasetQuality();
+ } else {
+ log.warn("Skipping data quality evaluation for dataset {} as commit
source is {}", this.datasetUrn,
+ this.datasetCommitSrc);
Review Comment:
when SafeDatasetCommit is invoked via jobContext.commit(), it would go to
else, and as per current existing implementation, this is expected case,
changed the log to info.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
