Stefan Richter created FLINK-10122:

             Summary: KafkaConsumer should use partitionable state over union 
state if partition discovery is not active
                 Key: FLINK-10122
             Project: Flink
          Issue Type: Improvement
          Components: Kafka Connector
            Reporter: Stefan Richter
            Assignee: Stefan Richter
             Fix For: 1.7.0

KafkaConsumer store its offsets state always as union state. I think this is 
only required in the case that partition discovery is active. For jobs with a 
very high parallelism, the union state can lead to prohibitively expensive 
deployments. For example, a job with 2000 source and a total of 10MB 
checkpointed union state offsets state would have to ship ~ 2000 x 10MB = 20GB 
of state. With partitionable state, it would have to ship ~10MB.

For now, I would suggest to go back to partitionable state in case that 
partition discovery is not active. In the long run, I have some ideas for more 
efficient partitioning schemes that would also work for active discovery.

This message was sent by Atlassian JIRA

Reply via email to