abhijeetkushe edited a comment on issue #2850: URL: https://github.com/apache/hudi/issues/2850#issuecomment-824070964
@nsivabalan I am using the default source limit i.e 9223372036854775807 so this is not directly connected with my issue.But I wanted to talk about another related issue.I realized while going through [ hudi's checkpoint code](https://github.com/apache/hudi/blob/release-0.6.0/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/DFSPathSelector.java#L92 ) that there could possibly be a bug for S3Filesystems as LastModifiedTime is stored at the granularity of seconds and it is possible that multiple files to have the same LastModifiedTime but have landed at different millisecond so wanted to know whether something like this would be needed ? -> [aws-glue-job-bookmarking](https://stackoverflow.com/questions/51529192/aws-glue-job-bookmarking ) In my case the file that got skipped did not have a another file with the same timestamp ctct-tdp-p2-send-5-2021-04-06-12-30-43-66bc4ad4-36da-40e1-819f-cdd33b3ecd91 April 6, 2021, 08:35:49 (UTC-04:00) . I have opened a AWS support ticket to see whether there is S3 consistency issue as well I wanted to know what solutions would you recommend to handle these problems ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
