tsreaper commented on pull request #24: URL: https://github.com/apache/flink-table-store/pull/24#issuecomment-1049439763
Hi @shenzhu ! Firstly thanks for your interest in contributing and also thanks @JingsongLi for the review. I understand that we're introducing this configuration to optimize merging as well as to decrease the write overhead. However I'd like to suggest a better method for this. Consider the merging process across snapshots. It's obvious that once a merged file exceeds the size limit it won't be merged again and will stay there forever. That is, the actual merging only happens at the tail of our manifest list. So why not just check the list from tail to head and once we hit a merged file we just exit? In this way we don't need this new configuration and we don't even need to iterate over the whole manifest list. What do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
