pratyakshsharma commented on issue #5020: URL: https://github.com/apache/hudi/issues/5020#issuecomment-1120364293
Just thinking out loud here. With respect to HUDI-3657, as I understand, the problem is for every commit, we have 3 files in requested, inflight and completed states. How about have a mechanism where after a certain threshold in terms of number of commits, we archive or simply delete only the requested and inflight state files and keep the completed state .commit file in the active timeline. Is there any purpose or use case that is being solved by keeping requested or inflight files as well in the active timeline? Ideally once the commit is completed, requested/inflight files should not be needed by our internal logic or by the user. Please correct me if I am wrong @danny0405 @nsivabalan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
