Aitozi commented on PR #7793: URL: https://github.com/apache/paimon/pull/7793#issuecomment-4442459617
Hi @leaves12138 , Thank you for your comment. It is certainly possible that the situation you described could happen.But currently, our check logic actually doesn't verify this type of situation. For example, taking our Writer-side Bucket Rescale detection as an instance: it actually stops detecting once the WriterContainer has been initialized. Therefore, the scenario you mentioned is similarly unavoidable at this stage. In a scenario where one job is performing a normal write while another job is performing an "insert overwrite rescale" the expected outcome is that the data will be overwritten. However, this is not considered a normal or intended write behavior. Based on this consideration, I believe adding a cache to the Commit side is a reasonable approach. Its primary purpose is to prevent the Commit side from repeatedly checking the Manifest file and result in additional cost. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
