[
https://issues.apache.org/jira/browse/GOBBLIN-1862?focusedWorklogId=874122&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-874122
]
ASF GitHub Bot logged work on GOBBLIN-1862:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 01/Aug/23 20:40
Start Date: 01/Aug/23 20:40
Worklog Time Spent: 10m
Work Description: ZihanLi58 commented on code in PR #3725:
URL: https://github.com/apache/gobblin/pull/3725#discussion_r1279923416
##########
gobblin-completeness/src/main/java/org/apache/gobblin/completeness/verifier/KafkaAuditCountVerifier.java:
##########
@@ -217,8 +217,10 @@ private static void validateTierCounts(String datasetName,
long beginInMillis, l
throw new IOException(String.format("Reference tier %s audit count
cannot be retrieved for dataset %s between %s and %s", refTier, datasetName,
beginInMillis, endInMillis));
}
long refCount = countsByTier.get(refTier);
- if(refCount <= 0) {
- throw new IOException(String.format("Reference tier %s count cannot be
less than or equal to zero", refTier));
+ if (refCount == 0) {
Review Comment:
Oh I saw it in the PR description, can you also add that as one comment in
the code saying if refTier is 0, we will always think data for that hour is
completed?
##########
gobblin-completeness/src/main/java/org/apache/gobblin/completeness/verifier/KafkaAuditCountVerifier.java:
##########
@@ -217,8 +217,10 @@ private static void validateTierCounts(String datasetName,
long beginInMillis, l
throw new IOException(String.format("Reference tier %s audit count
cannot be retrieved for dataset %s between %s and %s", refTier, datasetName,
beginInMillis, endInMillis));
}
long refCount = countsByTier.get(refTier);
- if(refCount <= 0) {
- throw new IOException(String.format("Reference tier %s count cannot be
less than or equal to zero", refTier));
+ if (refCount == 0) {
Review Comment:
what will happen if the src tier is not 0 but ref tier is 0?
Issue Time Tracking
-------------------
Worklog Id: (was: 874122)
Time Spent: 40m (was: 0.5h)
> Allow reference count to be 0 but no less
> -----------------------------------------
>
> Key: GOBBLIN-1862
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1862
> Project: Apache Gobblin
> Issue Type: Bug
> Reporter: Andy Jiang
> Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Due to possible issues upstream reporting incorrect reference counts for
> number of records, there could be times when the reference count reported to
> be 0.
> In the current case, if reference tier counter is reported to be 0, an
> exception is thrown, however is silently swallowed. Thus, causing the
> watermark to not move forward in this case or in the case of a quiet topic
> for hours where there are no records.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
