phet opened a new pull request, #3859:
URL: https://github.com/apache/gobblin/pull/3859

   Dear Gobblin maintainers,
   
   Please accept this PR. I understand that it will not be reviewed until I 
have checked off all the steps below!
   
   
   ### JIRA
   - [ ] My PR addresses the following [Gobblin 
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references 
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
       - https://issues.apache.org/jira/browse/GOBBLIN-1985
   
   
   ### Description
   - [ ] Here are some details about my PR, including screenshots (if 
applicable):
   
   `SourceHadoopFsEndPoint.getWatermark` currently misses certain changes to 
the contents of the dirs it covers because it doesn't use the mod time of those 
enclosing dirs when calculating the watermark.  despite those dirs not being 
among the files to copy, they must still participate in the watermark 
calculation because some compute engines, like spark, first write files to a 
temp subdir beneath the ultimate dest dir.  (since those temp subdirs are not 
valid paths, they'll anyway be skipped.)  once all executors have written their 
file, spark moves each from that temp subdir up to the enclosing ultimate dir 
location.  such file movement DOES NOT update the mod time of the file itself, 
only that of its enclosing dir--hence the need to incorporate the enclosing 
dir's mod time, in order to observe such dir changes.
   
   ### Tests
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   none
   
   ### Commits
   - [ ] My commits all reference JIRA issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
       1. Subject is separated from body by a blank line
       2. Subject is limited to 50 characters
       3. Subject does not end with a period
       4. Subject uses the imperative mood ("add", not "adding")
       5. Body wraps at 72 characters
       6. Body explains "what" and "why", not "how"
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to