[jira] [Updated] (KAFKA-6758) Default "" consumer group tracks committed offsets, but is otherwise not a real group
[
https://issues.apache.org/jira/browse/KAFKA-6758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David van Geest updated KAFKA-6758:
---
Description:
*To reproduce:*
* Use the default config for `group.id` of "" (the empty string)
* Use the default config for `enable.auto.commit` of `true`
* Use manually assigned partitions with `assign`
*Actual (unexpected) behaviour:*
Consumer offsets are stored for the "" group. Example:
{{~ $ /opt/kafka/kafka_2.11-0.11.0.2/bin/kafka-consumer-groups.sh
--bootstrap-server localhost:9092 --describe --group ""}}
{{Note: This will only show information about consumers that use the Java
consumer API (non-ZooKeeper-based consumers).}}
{{Consumer group '' has no active members.}}
{{TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID}}
{{my_topic 54 7859593 7865082 5489 - - -}}
{{my_topic 5 14252813 14266419 13606 - - -}}
{{my_topic 39 19099099 19122441 23342 - - -}}
{{my_topic 43 16434573 16449180 14607 - - -.}}
However, the "" is not a real group. It doesn't show up with:
{{~ $ /opt/kafka/kafka_2.11-0.11.0.2/bin/kafka-consumer-groups.sh
--bootstrap-server localhost:9092 --list}}
You also can't do dynamic partition assignment with it - if you try to
`subscribe` when using the default "" group ID, you get:
{{AbstractCoordinator: Attempt to join group failed due to fatal error: The
configured groupId is invalid}}
*Better behaviours:*
(any of these would be preferable, in my opinion)
* Don't commit offsets with the "" group, and log a warning telling the user
that `enable.auto.commit = true` is meaningless in this situation. This is what
I would have expected.
* Don't have a default `group.id`. Some of my reading indicates that the new
consumer basically needs a `group.id` to function. If so, force users to choose
a group ID so that they're more aware of what will happen.
* Have a default `group.id` of `default`, and make it a real consumer group.
That is, it shows up in lists of groups, it has dynamic partitioning, etc.
As a user, when I don't set `group.id` I expect that I'm not using consumer
groups. This is confirmed to me by listing the consumer groups on the broker
and not seeing anything. Therefore, I expect that there will be no offset
tracking in Kafka.
In my specific application, I was wanting `auto.offset.reset` to kick in so
that a failed consumer would start at the `latest` offset. However, it started
at this unexpectedly stored offset instead.
was:
*To reproduce:*
* Use the default config for `group.id` of "" (the empty string)
* Use the default config for `enable.auto.commit` of `true`
* Use manually assigned partitions with `assign`
*Actual (unexpected) behaviour:*
Consumer offsets are stored for the "" group. Example:
{{~ $ /opt/kafka/kafka_2.11-0.11.0.2/bin/kafka-consumer-groups.sh
--bootstrap-server localhost:9092 --describe --group ""}}
{{Note: This will only show information about consumers that use the Java
consumer API (non-ZooKeeper-based consumers).Consumer group '' has no active
members.TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST
CLIENT-ID}}
{{my_topic 54 7859593 7865082 5489 - - -}}
{{my_topic 5 14252813 14266419 13606 - - -}}
{{my_topic 39 19099099 19122441 23342 - - -}}
{{my_topic 43 16434573 16449180 14607 - - -.}}
However, the "" is not a real group. It doesn't show up with:
{{~ $ /opt/kafka/kafka_2.11-0.11.0.2/bin/kafka-consumer-groups.sh
--bootstrap-server localhost:9092 --list}}
You also can't do dynamic partition assignment with it - if you try to
`subscribe` when using the default "" group ID, you get:
{{AbstractCoordinator: Attempt to join group failed due to fatal error: The
configured groupId is invalid}}
*Better behaviours:*
(any of these would be preferable, in my opinion)
* Don't commit offsets with the "" group, and log a warning telling the user
that `enable.auto.commit = true` is meaningless in this situation. This is what
I would have expected.
* Don't have a default `group.id`. Some of my reading indicates that the new
consumer basically needs a `group.id` to function. If so, force users to choose
a group ID so that they're more aware of what will happen.
* Have a default `group.id` of `default`, and make it a real consumer group.
That is, it shows up in lists of groups, it has dynamic partitioning, etc.
As a user, when I don't set `group.id` I expect that I'm not using consumer
groups. This is confirmed to me by listing the consumer groups on the broker
and not seeing anything. Therefore, I expect that there will be no offset
tracking in Kafka.
In my specific application, I was wanting `auto.offset.reset` to kick in so
that a failed consumer would start at the `latest` offset. However, it started
at this unexpectedly stored offset instead.
> Default "" consumer group tracks committed offsets, but is otherwise
[jira] [Updated] (KAFKA-6758) Default "" consumer group tracks committed offsets, but is otherwise not a real group
[
https://issues.apache.org/jira/browse/KAFKA-6758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David van Geest updated KAFKA-6758:
---
Description:
*To reproduce:*
* Use the default config for `group.id` of "" (the empty string)
* Use the default config for `enable.auto.commit` of `true`
* Use manually assigned partitions with `assign`
*Actual (unexpected) behaviour:*
Consumer offsets are stored for the "" group. Example:
{{~ $ /opt/kafka/kafka_2.11-0.11.0.2/bin/kafka-consumer-groups.sh
--bootstrap-server localhost:9092 --describe --group ""}}
{{Note: This will only show information about consumers that use the Java
consumer API (non-ZooKeeper-based consumers).Consumer group '' has no active
members.TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST
CLIENT-ID}}
{{my_topic 54 7859593 7865082 5489 - - -}}
{{my_topic 5 14252813 14266419 13606 - - -}}
{{my_topic 39 19099099 19122441 23342 - - -}}
{{my_topic 43 16434573 16449180 14607 - - -.}}
However, the "" is not a real group. It doesn't show up with:
{{~ $ /opt/kafka/kafka_2.11-0.11.0.2/bin/kafka-consumer-groups.sh
--bootstrap-server localhost:9092 --list}}
You also can't do dynamic partition assignment with it - if you try to
`subscribe` when using the default "" group ID, you get:
{{AbstractCoordinator: Attempt to join group failed due to fatal error: The
configured groupId is invalid}}
*Better behaviours:*
(any of these would be preferable, in my opinion)
* Don't commit offsets with the "" group, and log a warning telling the user
that `enable.auto.commit = true` is meaningless in this situation. This is what
I would have expected.
* Don't have a default `group.id`. Some of my reading indicates that the new
consumer basically needs a `group.id` to function. If so, force users to choose
a group ID so that they're more aware of what will happen.
* Have a default `group.id` of `default`, and make it a real consumer group.
That is, it shows up in lists of groups, it has dynamic partitioning, etc.
As a user, when I don't set `group.id` I expect that I'm not using consumer
groups. This is confirmed to me by listing the consumer groups on the broker
and not seeing anything. Therefore, I expect that there will be no offset
tracking in Kafka.
In my specific application, I was wanting `auto.offset.reset` to kick in so
that a failed consumer would start at the `latest` offset. However, it started
at this unexpectedly stored offset instead.
was:
*To reproduce:*
* Use the default config for `group.id` of "" (the empty string)
* Use the default config for `enable.auto.commit` of `true`
* Use manually assigned partitions with `assign`
*Actual (unexpected) behaviour:*
Consumer offsets are stored for the "" group. Example:
{{~ $ /opt/kafka/kafka_2.11-0.11.0.2/bin/kafka-consumer-groups.sh
--bootstrap-server localhost:9092 --describe --group ""}}
{{Note: This will only show information about consumers that use the Java
consumer API (non-ZooKeeper-based consumers).Consumer group '' has no active
members.TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST
CLIENT-ID}}
{{my_topic 54 7859593 7865082 5489 - - -}}
{{my_topic 5 14252813 14266419 13606 - - -}}
{{my_topic 39 19099099 19122441 23342 - - -}}
{{my_topic 43 16434573 16449180 14607 - - -.}}
However, the "" is not a real group. It doesn't show up with:
{{~ $ /opt/kafka/kafka_2.11-0.11.0.2/bin/kafka-consumer-groups.sh
--bootstrap-server localhost:9092 --list}}
You also can't do dynamic partition assignment with it - if you try to
`subscribe` when using the default "" group ID, you get:
{{AbstractCoordinator: Attempt to join group failed due to fatal error: The
configured groupId is invalid}}
*Better behaviours:*
(any of these would be preferable, in my opinion)
* Don't commit offsets with the "" group, and log a warning telling the user
that `enable.auto.commit = true` is meaningless in this situation. This is what
I would have expected.
* Don't have a default `group.id`. Some of my reading indicates that the new
consumer basically needs a `group.id` to function. If so, force users to choose
a group ID so that they're more aware of what will happen.
* Have a default `group.id` of `default`, and make it a real consumer group.
That is, it shows up in lists of groups, it has dynamic partitioning, etc.
As a user, when I don't set `group.id` I expect that I'm not using consumer
groups. Therefore, I expect that there will be no offset tracking in Kafka.
In my specific application, I was wanting `auto.offset.reset` to kick in so
that a failed consumer would start at the `latest` offset. However, it started
at this unexpectedly stored offset instead.
> Default "" consumer group tracks committed offsets, but is otherwise not a
> real group
> -
>
>
