gmr opened a new issue, #4279:
URL: https://github.com/apache/iggy/issues/4279
### Description
The HTTP API cannot read a consumer group's stored offset. `GET
/streams/{stream_id}/topics/{topic_id}/consumer-offsets` always resolves a
regular consumer, because `Consumer.kind` is `#[serde(skip)]`
(`core/common/src/types/consumer/consumer_kind.rs:34`). A `kind` query
parameter is ignored, and `consumer_offset_wire_request` always sends kind
`Consumer` (`core/server/src/http/wire.rs:128`, and see the comment at `:146`).
The server already supports this read. `resolve_consumer_offset_request`
(`core/server/src/dispatch/partition.rs:1103`) handles `KIND_CONSUMER_GROUP`,
and any client can read a group offset without being a member. Only the TCP
path can reach it (for example, `iggy consumer-offset get <group> <stream>
<topic> 0 --kind consumer-group`).
`GET .../consumer-groups/{group_id}` returns members and their partitions,
but no offsets.
This matters for monitoring. To compute consumer lag (partition
`current_offset` minus the group's stored offset) from an HTTP-only tool such
as a dashboard, a health check, or an admin UI, you must add a TCP client only
for this read. In #4273, a stuck group showed healthy connector status for
days. Group lag over HTTP would have shown the problem at once.
### Affected area / component
Server (HTTP API); common (`GetConsumerOffset` / `Consumer` HTTP types); SDK
HTTP client.
### Proposed solution
Either of these, or both:
1. Accept a `kind` query parameter on `GET .../consumer-offsets` (`consumer`
by default, or `consumer_group`) and pass it through to the wire request. For
example: `GET
/streams/s/topics/t/consumer-offsets?consumer_id=my-group&kind=consumer_group&partition_id=0`.
2. Include a per-partition `stored_offset` in the `GET
.../consumer-groups/{group_id}` response, so that one call gives the members,
the assignments and the group's progress.
Option 1 keeps the existing default, so current HTTP clients see no change.
### Alternatives considered
- Use the TCP protocol or the CLI for this one read. This works, but an
HTTP-only client must then keep a second transport and credentials.
- Compute lag from connector runtime metrics
(`iggy_connector_messages_consumed_total`). This covers only connector
consumers, and it cannot show a group's committed position.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
