StephanEwen commented on issue #8751: [FLINK-11937][StateBackend]Resolve small file problem in RocksDB incremental checkpoint URL: https://github.com/apache/flink/pull/8751#issuecomment-507639922 I like the idea of fixing the small file problem a lot - I think it is very relevant in practice. Before looking at the code, we need a lot more information for a good review: - An extensive high-level description of the changes, including the changes to state handles, checkpoint metadata, etc. - Is this replacing the existing functionality or a new option that users activate? - What is the compatibility for existing users? - Can this be abstracted within the state backend without changes to the state handle system? If not, why and could it be done with a bit different approach? Something that is in the state backend is easier/less sensitive than major changes to the shared state registry, etc. Given this list, I am wondering if this is a FLIP-worthy change in the end. I think this also needs attention from @tzulitai for checking if anything about snapshot metadata is affected here, as well as @aljoscha @kl0u to check if this may affect savepoint / checkpoint properties etc.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
