[
https://issues.apache.org/jira/browse/GOBBLIN-1864?focusedWorklogId=873909&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-873909
]
ASF GitHub Bot logged work on GOBBLIN-1864:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 31/Jul/23 21:58
Start Date: 31/Jul/23 21:58
Worklog Time Spent: 10m
Work Description: jack-moseley commented on code in PR #3728:
URL: https://github.com/apache/gobblin/pull/3728#discussion_r1279924289
##########
gobblin-completeness/src/main/java/org/apache/gobblin/completeness/verifier/KafkaAuditCountVerifier.java:
##########
@@ -125,10 +126,13 @@ public Map<CompletenessType, Boolean>
calculateCompleteness(String datasetName,
countsByTier.forEach((x,y) -> log.info(String.format(" %s : %s ", x, y)));
Map<CompletenessType, Boolean> result = new HashMap<>();
- result.put(CompletenessType.ClassicCompleteness,
calculateCompleteness(datasetName, beginInMillis, endInMillis,
- CompletenessType.ClassicCompleteness, countsByTier) > threshold);
- result.put(CompletenessType.TotalCountCompleteness,
calculateCompleteness(datasetName, beginInMillis, endInMillis,
- CompletenessType.TotalCountCompleteness, countsByTier) > threshold);
+ Arrays.stream(CompletenessType.values()).forEach(type -> {
+ try {
+ result.put(type, calculateCompleteness(datasetName, beginInMillis,
endInMillis, type, countsByTier) > threshold);
+ } catch (IOException e) {
+ log.error("Failed to calculate completeness for type " + type, e);
Review Comment:
Yes good point, I missed that it would cause a null failure there. Instead
of always returning false, I changed the checks themselves to handle null
properly, that seems safer in case the map is incomplete for some reason.
`if
(!results.getOrDefault(KafkaAuditCountVerifier.CompletenessType.ClassicCompleteness,
false))`
Issue Time Tracking
-------------------
Worklog Id: (was: 873909)
Time Spent: 50m (was: 40m)
> Catch exceptions if specific watermark calculation fails
> --------------------------------------------------------
>
> Key: GOBBLIN-1864
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1864
> Project: Apache Gobblin
> Issue Type: Improvement
> Reporter: Jack Moseley
> Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
