hubcio opened a new issue, #4447:
URL: https://github.com/apache/iggy/issues/4447
## Problem
- You cannot see consumer lag (the messages that a group did not consume
yet). `consumer-group get` shows members and partitions, but no offsets.
`consumer-offset get` shows one partition per call.
- The CLI cannot delete a consumer offset. The SDK
(`core/common/src/traits/consumer_offset_client.rs:65`) and the binary protocol
(code 122) support it. The HTTP API deletes only regular consumer offsets
(`core/server/src/http/handlers.rs:1530`).
- The CLI cannot set a consumer group offset. Over TCP, QUIC and WebSocket,
`consumer-offset set -k consumer-group` fails with error 5009, because the CLI
is not a member of the group (`core/server/src/namespace.rs:280`):
```
Consumer group member with client ID: 0 does not own partition: 0 at the
current generation (rebalance in progress).
```
No rebalance runs, because the group has no members. Binary errors carry
only the code, so the client fills the fields with zeros (#3735). Over HTTP,
the same command writes a regular consumer offset instead (#4408). `-k
external-group` works, but Iggy consumer groups never read external group
offsets (#4392).
## Proposal
- Add a partitions table to `consumer-group get`: partition, member, current
offset, stored offset and lag. A client does not need to be a member to read a
group offset. The stored offset is the last consumed offset, so the lag is the
current offset minus the stored offset. If no offset is stored, the lag is the
message count of the partition.
- Add `consumer-offset delete`, with the same arguments as `consumer-offset
get`.
- Server: if the group has no members, accept an offset write or delete for
it from a non-member. If the group has members, keep the current fence. The
Kafka admin API has the same rule: it resets the offsets of an empty group only.
- Add `consumer-group reset <STREAM_ID> <TOPIC_ID> <GROUP_ID>` with
`--to-earliest`, `--to-latest`, `--to-offset <N>` or `--to-timestamp <TIME>`.
Print the plan first, and write it only with `--execute`.
- `--to-offset N` stores `N - 1`, because the next poll starts after the
stored offset. `--to-earliest` and `--to-offset 0` delete the offset.
## Notes
- Agree on the server rule before coding. The lag view does not depend on
it. Until the rule exists, `consumer-offset delete` works only for regular
consumers and external groups.
- The membership check and the offset write are not atomic, so a consumer
that joins during a reset can overwrite it. Before a reset, stop all consumers
of the group and wait for `consumer_group.session_timeout`.
- An offset write needs only the poll permission. With the new rule, any
user who can poll the topic can reset an idle group.
- A stored group offset holds back retention. If a reset deletes it,
retention can remove messages that the group did not read. The help text must
say this.
- Over HTTP, the group kind is lost (#4408), so the lag view needs TCP, QUIC
or WebSocket until #4279 is done. Offset reads come from the local replica, so
the lag can be stale (#4409).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
