[
https://issues.apache.org/jira/browse/GOBBLIN-1985?focusedWorklogId=899482&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-899482
]
ASF GitHub Bot logged work on GOBBLIN-1985:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 12/Jan/24 19:49
Start Date: 12/Jan/24 19:49
Worklog Time Spent: 10m
Work Description: codecov-commenter commented on PR #3859:
URL: https://github.com/apache/gobblin/pull/3859#issuecomment-1889863905
##
[Codecov](https://app.codecov.io/gh/apache/gobblin/pull/3859?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base
[(`22248e4`)](https://app.codecov.io/gh/apache/gobblin/commit/22248e4f2b01859c7e833b99f6770a2d4bd15d6e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
47.56% compared to head
[(`892037b`)](https://app.codecov.io/gh/apache/gobblin/pull/3859?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
45.98%.
> Report is 1 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #3859 +/- ##
============================================
- Coverage 47.56% 45.98% -1.58%
+ Complexity 11068 2186 -8882
============================================
Files 2160 417 -1743
Lines 85563 18052 -67511
Branches 9509 2199 -7310
============================================
- Hits 40700 8302 -32398
+ Misses 41154 8865 -32289
+ Partials 3709 885 -2824
```
</details>
[:umbrella: View full report in Codecov by
Sentry](https://app.codecov.io/gh/apache/gobblin/pull/3859?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it
here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
Issue Time Tracking
-------------------
Worklog Id: (was: 899482)
Time Spent: 50m (was: 40m)
> Incorporate the mod time of enclosing dirs into the `SourceHadoopFsEndPoint`
> watermark calculations
> ---------------------------------------------------------------------------------------------------
>
> Key: GOBBLIN-1985
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1985
> Project: Apache Gobblin
> Issue Type: Bug
> Components: gobblin-core
> Reporter: Kip Kohn
> Assignee: Abhishek Tiwari
> Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> `SourceHadoopFsEndPoint.getWatermark` currently misses certain changes to the
> contents of the dirs it covers because it doesn't use the mod time of those
> enclosing dirs when calculating the watermark. despite those dirs not being
> among the files to copy, they must still participate in the watermark
> calculation because some compute engines, like spark, first write files to a
> temp subdir beneath the ultimate dest dir. (since those temp subdirs are not
> valid paths, they'll anyway be skipped.) once all executors have written
> their file, spark moves each from that temp subdir up to the enclosing
> ultimate dir location. such file movement DOES NOT update the mod time of
> the file itself, only that of its enclosing dir--hence the need to
> incorporate the enclosing dir's mod time, in order to observe such dir
> changes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)