abhijeetkushe edited a comment on issue #2850: URL: https://github.com/apache/hudi/issues/2850#issuecomment-824070964
@nsivabalan I am using the default source limit i.e 9223372036854775807 so this is not directly connected with my issue.But I wanted to talk about another related issue.I realized while going through [ hudi's checkpoint code](https://github.com/apache/hudi/blob/release-0.6.0/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/DFSPathSelector.java#L92 ) that there is a possible bug so wanted to know whether something like this would be needed ? -> [aws-glue-job-bookmarking](https://stackoverflow.com/questions/51529192/aws-glue-job-bookmarking ) In my case the file that got skipped did not have a another file with the same timestamp ctct-tdp-p2-send-5-2021-04-06-12-30-43-66bc4ad4-36da-40e1-819f-cdd33b3ecd91 April 6, 2021, 08:35:49 (UTC-04:00) . I have opened a AWS support ticket to see whether there is S3 consistency issue as well I wanted to know what solutions would you recommend to address the checkpoint bug because if even if the consistency issue is addressed this issue still needs to be addressed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
