Does this mean I can use the same consumer group G1 for the newer version A'? And inspite of same consumer group, A' will receive messages from all partitions when its started from savepoint?
I am using Flink 1.2.1. Does the above plan require setting uid on the Kafka source in the job? Thanks, Moiz On Wed, Jul 19, 2017 at 1:06 PM, Tzu-Li (Gordon) Tai <tzuli...@apache.org> wrote: > Hi! > > The only occasions which the consumer group is used is: > 1. When committing offsets back to Kafka. Since Flink 1.3, this can be > disabled completely (both when checkpointing is enabled or disabled). See > [1] on details about that. > 2. When starting fresh (not starting from some savepoint), if you choose > to use GROUP_OFFSETS as the start position, then the consumer group would > also be used. If starting from a savepoint, then this is irrelevant. See > [2]. > > Note that it actually isn’t used in any critical paths for Flink’s > exactly-once processing guarantees, or partition to source parallel > instance assignments. > > So, the desired behavior in which you described is exactly the expected > behavior for the Flink Kafka Consumer. > > Cheers, > Gordon > > [1] https://ci.apache.org/projects/flink/flink-docs- > release-1.3/dev/connectors/kafka.html#kafka-consumers- > offset-committing-behaviour-configuration > [2] https://ci.apache.org/projects/flink/flink-docs- > release-1.3/dev/connectors/kafka.html#kafka-consumers- > start-position-configuration > > On 19 July 2017 at 3:23:01 PM, Moiz Jinia (moiz.ji...@gmail.com) wrote: > > Below is a plan for downtime-free upgrade of a Flink job. The downstream > consumer of the Flink job is duplicate proof. > > Scenario 1 - > 1. Start Flink job A with consumer group G1 (12 slot job) > 2. While job A is running, take a savepoint AS. > 3. Start newer version of Flink job A' from savepoint AS with consumer > group > *G1* (12 slot job again) > 4. Stop job A. > > Scenario 2 - > 1. Start Flink job A with consumer group G1 (12 slot job) > 2. While job A is running, take a savepoint AS. > 3. Start newer version of Flink job A' from savepoint AS with consumer > group > *G2* (12 slot job again) > 4. Stop job A > > Does it matter what consumer group job A' uses? The desired behavior is > that > during the window when both A and A' are running, all messages should go > to > both jobs. (And of course I want that job A' should start consuming from > the > offsets in the savepoint and not the earliest). > > > > > > > -- > View this message in context: http://apache-flink-user- > mailing-list-archive.2336050.n4.nabble.com/Does- > FlinkKafkaConsumer010-care-about-consumer-group-tp14323.html > Sent from the Apache Flink User Mailing List archive. mailing list archive > at Nabble.com. > >
