Hello Avi, Similar questions have been discussed previously:
https://issues.apache.org/jira/browse/KAFKA-3775 One concern of doing this, though is that if you want to do the change on-the-fly by rolling bounce (i.e. combining step 5 6 and 7) the instances to switch back the assignor class, then there will be a period of time while some instances are using the old assignor while some other are using the new one, hence undefined behavior. But since you are doing steps 5 6 7 consecutively I think it should be fine. BTW regarding your first email, it actually is OK to have consumers in the same group to be subscribing to different topics (e.g. consumerA subscribing to topic 1 2 3, consumerB subscribing to topic 4 5 6), and the topic partitions will then be assigned correspondingly if you are using the consumer's default assignor, however in streams partition assignor we did not do this yet. Guozhang On Thu, Dec 15, 2016 at 10:28 AM, Avi Flax <avi.f...@parkassist.com> wrote: > > > On Dec 15, 2016, at 11:33, Damian Guy <damian....@gmail.com> wrote: > > > > Technically you can, but not without writing some code. If you want to > use > > consumer groups then you would need to write a custom PartitionAssignor > and > > configure it in your Consumer Config, like so: > > consumerProps.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, > > YourPartitionAssignor.class.getName()); > > > > Alternatively you could use manual partition assignment to temporarily > work > > around it. > > Thanks Damian! Very helpful! > > As an alternative, would there be any adverse effects of changing the > config of a Kafka Streams app to temporarily (say, for 30 minutes) consume > from a smaller set of topics and then change it right back? > > To be clear, the workflow I have in mind is: > > 1. Stop app (gracefully) > 2. Change config to have the app consume from a subset of its current > topics > 3. Start app > 4. Wait ~30 minutes > 5. Stop app (gracefully) > 6. Revert config to the full set of topics > 6. Start app > > Would there be any adverse effects, like lost local state, lost offsets, > etc? > > It may be relevant that my app currently creates an instance of > KafkaStreams for each topic it consumes from. > > Thanks! > Avi > > ———— > Software Architect @ Park Assist > We’re hiring! http://tech.parkassist.com/jobs/ > -- -- Guozhang
