StefanRRichter commented on issue #6537: [FLINK-10122] KafkaConsumer should use partitionable state over union state if partition discovery is not active URL: https://github.com/apache/flink/pull/6537#issuecomment-413787110 Thanks @tzulitai ! I was aware that this will break the behavior for partition discovery. However, the current implementation was already broken for user at large scale, as pointed out in the description. This PR was intended as a quick solution for this case. I think that we can have better non-breaking solutions in the future like splitting the source into two operators or a different state partitioning scheme. I think that we can close the PR and go for the long term solution in official releases. Nevertheless I think that we should cherry-pick two parts of this PR into releases, the hotfix to improve memory utilization and the option to remove operator state (or - even better - states in general).
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
