Re: New consumer - consumer group init
Thanks all for fast feedback! It's great news if that aspect is improved as well in new HLC. I will test and get back with any related findings. Kind regards, Stevo Slavic. On Mon, Jul 20, 2015 at 9:57 PM, Guozhang Wang wangg...@gmail.com wrote: Hi Stevo, I am still not very clear on your point yet, I guess I was trying to figure out under which circumstances would users prefer to re-set the group id at an existing consumer rather than creating a new instance. As Jason mentioned, since the new consumer is single threaded it should usually be cheap to construct. Guozhang On Mon, Jul 20, 2015 at 11:06 AM, Jason Gustafson ja...@confluent.io wrote: Hey Stevo, The new consumer doesn't have any threads of its own, so I think construction should be fairly cheap. -Jason On Sun, Jul 19, 2015 at 2:13 PM, Stevo Slavić ssla...@gmail.com wrote: Hello Guozhang, It would be enough if consumer group could, besides at construction time, be set once only after construction. Have to retest, but high level consumer in 0.8.1 used to be very heavy weight object (lots of threads started, and it would block and take time to construct it). It's understandable, considering all of the high level features it has, and since it's supposed to be long living object. What would improve with this change is that construction penalty could be paid upfront, while topic subscription and joining consumer group ensemble could be done on first use, so that first use does not have to suffer from both init and subscription penalties. It would be nice also if consumer group just as subscription could be changed later even, so multiple times throughout lifetime of high level consumer instance, to avoid constructing new consumer when instance purpose changes. After looking more into the HLC API, thought maybe this is not needed, since there is public void subscribe(TopicPartition... partitions) which does not use consumer group management, but problem is that there is no matching explicit commit where one could pass consumer group parameter as well, to label for which consumer group should offset(s) be committed. Seems like new HLC has split personality. Maybe (at least) two APIs could have been provided instead of one with such differing behaviors. Kind regards, Stevo Slavic. On Sun, Jul 19, 2015 at 12:01 AM, Guozhang Wang wangg...@gmail.com wrote: Hi Stevo, Hmm this is interesting, do you have any use cases in mind that need dynamic group changing? Guozhang On Fri, Jul 17, 2015 at 11:13 PM, Stevo Slavić ssla...@gmail.com wrote: Hello Apache Kafka community, In new KafkaConsumer API on trunk, it seems it's only possible to define consumer group id at construction time of KafkaConsumer, through property with group.id key. Would it make sense and be possible to support setting/changing consumer group id after construction, but before it's actually used for the first time, similar to how subscription is supported through public void subscribe(String... topics)? Maybe this can be done through additional method public void subscribe(String consumerGroupId, String... topics) which would first set provided consumer group id in coordinator and then call public void subscribe(String... topics). Kind regards, Stevo Slavic. -- -- Guozhang -- -- Guozhang
Re: New consumer - consumer group init
Hey Stevo, The new consumer doesn't have any threads of its own, so I think construction should be fairly cheap. -Jason On Sun, Jul 19, 2015 at 2:13 PM, Stevo Slavić ssla...@gmail.com wrote: Hello Guozhang, It would be enough if consumer group could, besides at construction time, be set once only after construction. Have to retest, but high level consumer in 0.8.1 used to be very heavy weight object (lots of threads started, and it would block and take time to construct it). It's understandable, considering all of the high level features it has, and since it's supposed to be long living object. What would improve with this change is that construction penalty could be paid upfront, while topic subscription and joining consumer group ensemble could be done on first use, so that first use does not have to suffer from both init and subscription penalties. It would be nice also if consumer group just as subscription could be changed later even, so multiple times throughout lifetime of high level consumer instance, to avoid constructing new consumer when instance purpose changes. After looking more into the HLC API, thought maybe this is not needed, since there is public void subscribe(TopicPartition... partitions) which does not use consumer group management, but problem is that there is no matching explicit commit where one could pass consumer group parameter as well, to label for which consumer group should offset(s) be committed. Seems like new HLC has split personality. Maybe (at least) two APIs could have been provided instead of one with such differing behaviors. Kind regards, Stevo Slavic. On Sun, Jul 19, 2015 at 12:01 AM, Guozhang Wang wangg...@gmail.com wrote: Hi Stevo, Hmm this is interesting, do you have any use cases in mind that need dynamic group changing? Guozhang On Fri, Jul 17, 2015 at 11:13 PM, Stevo Slavić ssla...@gmail.com wrote: Hello Apache Kafka community, In new KafkaConsumer API on trunk, it seems it's only possible to define consumer group id at construction time of KafkaConsumer, through property with group.id key. Would it make sense and be possible to support setting/changing consumer group id after construction, but before it's actually used for the first time, similar to how subscription is supported through public void subscribe(String... topics)? Maybe this can be done through additional method public void subscribe(String consumerGroupId, String... topics) which would first set provided consumer group id in coordinator and then call public void subscribe(String... topics). Kind regards, Stevo Slavic. -- -- Guozhang
Re: New consumer - consumer group init
Hi Stevo, I am still not very clear on your point yet, I guess I was trying to figure out under which circumstances would users prefer to re-set the group id at an existing consumer rather than creating a new instance. As Jason mentioned, since the new consumer is single threaded it should usually be cheap to construct. Guozhang On Mon, Jul 20, 2015 at 11:06 AM, Jason Gustafson ja...@confluent.io wrote: Hey Stevo, The new consumer doesn't have any threads of its own, so I think construction should be fairly cheap. -Jason On Sun, Jul 19, 2015 at 2:13 PM, Stevo Slavić ssla...@gmail.com wrote: Hello Guozhang, It would be enough if consumer group could, besides at construction time, be set once only after construction. Have to retest, but high level consumer in 0.8.1 used to be very heavy weight object (lots of threads started, and it would block and take time to construct it). It's understandable, considering all of the high level features it has, and since it's supposed to be long living object. What would improve with this change is that construction penalty could be paid upfront, while topic subscription and joining consumer group ensemble could be done on first use, so that first use does not have to suffer from both init and subscription penalties. It would be nice also if consumer group just as subscription could be changed later even, so multiple times throughout lifetime of high level consumer instance, to avoid constructing new consumer when instance purpose changes. After looking more into the HLC API, thought maybe this is not needed, since there is public void subscribe(TopicPartition... partitions) which does not use consumer group management, but problem is that there is no matching explicit commit where one could pass consumer group parameter as well, to label for which consumer group should offset(s) be committed. Seems like new HLC has split personality. Maybe (at least) two APIs could have been provided instead of one with such differing behaviors. Kind regards, Stevo Slavic. On Sun, Jul 19, 2015 at 12:01 AM, Guozhang Wang wangg...@gmail.com wrote: Hi Stevo, Hmm this is interesting, do you have any use cases in mind that need dynamic group changing? Guozhang On Fri, Jul 17, 2015 at 11:13 PM, Stevo Slavić ssla...@gmail.com wrote: Hello Apache Kafka community, In new KafkaConsumer API on trunk, it seems it's only possible to define consumer group id at construction time of KafkaConsumer, through property with group.id key. Would it make sense and be possible to support setting/changing consumer group id after construction, but before it's actually used for the first time, similar to how subscription is supported through public void subscribe(String... topics)? Maybe this can be done through additional method public void subscribe(String consumerGroupId, String... topics) which would first set provided consumer group id in coordinator and then call public void subscribe(String... topics). Kind regards, Stevo Slavic. -- -- Guozhang -- -- Guozhang
Re: New consumer - consumer group init
Hello Guozhang, It would be enough if consumer group could, besides at construction time, be set once only after construction. Have to retest, but high level consumer in 0.8.1 used to be very heavy weight object (lots of threads started, and it would block and take time to construct it). It's understandable, considering all of the high level features it has, and since it's supposed to be long living object. What would improve with this change is that construction penalty could be paid upfront, while topic subscription and joining consumer group ensemble could be done on first use, so that first use does not have to suffer from both init and subscription penalties. It would be nice also if consumer group just as subscription could be changed later even, so multiple times throughout lifetime of high level consumer instance, to avoid constructing new consumer when instance purpose changes. After looking more into the HLC API, thought maybe this is not needed, since there is public void subscribe(TopicPartition... partitions) which does not use consumer group management, but problem is that there is no matching explicit commit where one could pass consumer group parameter as well, to label for which consumer group should offset(s) be committed. Seems like new HLC has split personality. Maybe (at least) two APIs could have been provided instead of one with such differing behaviors. Kind regards, Stevo Slavic. On Sun, Jul 19, 2015 at 12:01 AM, Guozhang Wang wangg...@gmail.com wrote: Hi Stevo, Hmm this is interesting, do you have any use cases in mind that need dynamic group changing? Guozhang On Fri, Jul 17, 2015 at 11:13 PM, Stevo Slavić ssla...@gmail.com wrote: Hello Apache Kafka community, In new KafkaConsumer API on trunk, it seems it's only possible to define consumer group id at construction time of KafkaConsumer, through property with group.id key. Would it make sense and be possible to support setting/changing consumer group id after construction, but before it's actually used for the first time, similar to how subscription is supported through public void subscribe(String... topics)? Maybe this can be done through additional method public void subscribe(String consumerGroupId, String... topics) which would first set provided consumer group id in coordinator and then call public void subscribe(String... topics). Kind regards, Stevo Slavic. -- -- Guozhang
New consumer - consumer group init
Hello Apache Kafka community, In new KafkaConsumer API on trunk, it seems it's only possible to define consumer group id at construction time of KafkaConsumer, through property with group.id key. Would it make sense and be possible to support setting/changing consumer group id after construction, but before it's actually used for the first time, similar to how subscription is supported through public void subscribe(String... topics)? Maybe this can be done through additional method public void subscribe(String consumerGroupId, String... topics) which would first set provided consumer group id in coordinator and then call public void subscribe(String... topics). Kind regards, Stevo Slavic.
