abhishekmjain commented on code in PR #4044:
URL: https://github.com/apache/gobblin/pull/4044#discussion_r1981876448
##########
gobblin-core/src/main/java/org/apache/gobblin/source/PartitionedFileSourceBase.java:
##########
@@ -367,7 +384,25 @@ private long getLowWaterMark(Iterable<WorkUnitState>
previousStates, String lowW
return lowWaterMarkValue + getRetriever().getWatermarkIncrementMs();
}
+ /** Returns the low watermark value based on lookback which is equal to
current time minus lookback time. */
+ private long getLowWaterMarkFromLookbackTime(String lookBackTime) {
+ try {
+ Duration lookBackDuration =
PartitionAwareFileRetrieverUtils.getLookbackTimeDuration(lookBackTime);
+ return new DateTime().minus(lookBackDuration).getMillis();
+ } catch (IOException e) {
+ Throwables.propagate(e);
Review Comment:
Should we propagate the exception if we want to have a fallback to 0?
Also, looks like Throwables.propagate is deprecated. Let's just log the
exception and move ahead here.
##########
gobblin-core/src/main/java/org/apache/gobblin/source/PartitionedFileSourceBase.java:
##########
@@ -367,7 +384,25 @@ private long getLowWaterMark(Iterable<WorkUnitState>
previousStates, String lowW
return lowWaterMarkValue + getRetriever().getWatermarkIncrementMs();
}
+ /** Returns the low watermark value based on lookback which is equal to
current time minus lookback time. */
+ private long getLowWaterMarkFromLookbackTime(String lookBackTime) {
+ try {
+ Duration lookBackDuration =
PartitionAwareFileRetrieverUtils.getLookbackTimeDuration(lookBackTime);
+ return new DateTime().minus(lookBackDuration).getMillis();
+ } catch (IOException e) {
+ Throwables.propagate(e);
Review Comment:
Should we propagate the exception if we want to have a fallback to 0?
Also, looks like Throwables.propagate is deprecated. Let's just log the
exception instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
