[
https://issues.apache.org/jira/browse/GOBBLIN-1985?focusedWorklogId=899315&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-899315
]
ASF GitHub Bot logged work on GOBBLIN-1985:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 12/Jan/24 04:00
Start Date: 12/Jan/24 04:00
Worklog Time Spent: 10m
Work Description: phet opened a new pull request, #3859:
URL: https://github.com/apache/gobblin/pull/3859
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1985
### Description
- [ ] Here are some details about my PR, including screenshots (if
applicable):
`SourceHadoopFsEndPoint.getWatermark` currently misses certain changes to
the contents of the dirs it covers because it doesn't use the mod time of those
enclosing dirs when calculating the watermark. despite those dirs not being
among the files to copy, they must still participate in the watermark
calculation because some compute engines, like spark, first write files to a
temp subdir beneath the ultimate dest dir. (since those temp subdirs are not
valid paths, they'll anyway be skipped.) once all executors have written their
file, spark moves each from that temp subdir up to the enclosing ultimate dir
location. such file movement DOES NOT update the mod time of the file itself,
only that of its enclosing dir--hence the need to incorporate the enclosing
dir's mod time, in order to observe such dir changes.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
none
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
Issue Time Tracking
-------------------
Worklog Id: (was: 899315)
Remaining Estimate: 0h
Time Spent: 10m
> Incorporate the mod time of enclosing dirs into the `SourceHadoopFsEndPoint`
> watermark calculations
> ---------------------------------------------------------------------------------------------------
>
> Key: GOBBLIN-1985
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1985
> Project: Apache Gobblin
> Issue Type: Bug
> Components: gobblin-core
> Reporter: Kip Kohn
> Assignee: Abhishek Tiwari
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> `SourceHadoopFsEndPoint.getWatermark` currently misses certain changes to the
> contents of the dirs it covers because it doesn't use the mod time of those
> enclosing dirs when calculating the watermark. despite those dirs not being
> among the files to copy, they must still participate in the watermark
> calculation because some compute engines, like spark, first write files to a
> temp subdir beneath the ultimate dest dir. (since those temp subdirs are not
> valid paths, they'll anyway be skipped.) once all executors have written
> their file, spark moves each from that temp subdir up to the enclosing
> ultimate dir location. such file movement DOES NOT update the mod time of
> the file itself, only that of its enclosing dir--hence the need to
> incorporate the enclosing dir's mod time, in order to observe such dir
> changes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)