bartoszkobylinski opened a new issue, #3748:
URL: https://github.com/apache/iggy/issues/3748
**TL;DR** — Leaving `poll_interval` unset on `consumer_group()` makes the
consumer re-poll as fast as the broker can answer. On a 1-vCPU host with an
essentially idle topic that measured **~30% of the core** (iggy-server 17.3% +
consumer 12.6%) and **~34,700 context switches/sec**; passing
`poll_interval=250ms` took both processes below 0.3% with no functional change.
This is my omission rather than an SDK bug — I'm filing it because two of my
codebases made the identical omission independently, and because the
consequence isn't discoverable from Python. Please re-triage as
docs/enhancement if that fits better.
### Bug description
My worker created its consumer without passing `poll_interval`.
`consume_messages(callback, shutdown_event)` takes no interval either, so
`consumer_group()` is the only place it can be set. With it unset the SDK
leaves it `None`, and the consumer polls continuously.
The cost, on a topic receiving a handful of messages per day:
| | unset | `poll_interval=250ms` |
|---|---|---|
| iggy-server | 17.3 % | < 0.3 % |
| consumer process | 12.6 % | < 0.3 % |
| context switches/sec | ~34,700 | ~2,300–3,900 |
| PSI `cpu some avg10` | 70.2 | 1.6 |
Stopping just the broker and the consumer took the whole machine to 100%
idle, so this loop was effectively everything the host was doing. The server's
own CPU tracks the client's poll rate — it drops below 0.3% while a consumer is
still attached and polling at 250ms.
Three things made it hard to notice:
1. In `core/sdk/src/clients/consumer_builder.rs`, `polling_retry_interval`
is given a default of `IggyDuration::ONE_SECOND`, while `polling_interval` is a
bare `Option<IggyDuration>` with no default — seeing the first handled for you
makes the second look handled too.
2. The Python stub types it `poll_interval: Optional[datetime.timedelta] =
None`, and the complete docstring for `consumer_group` is *"Creates a new
consumer group consumer. Returns the consumer or a PyRuntimeError on failure."*
— a Python user cannot learn what omitting it does without reading the Rust
source.
3. The only `poll_interval` example in the docs is
`.poll_interval(IggyDuration::from_str("1ms")?)` — 1000 polls/sec — presented
with no note that it's a throughput setting for dedicated hardware.
That two separate projects of mine, written months apart, both omitted it is
what makes me think it's the shape of the API rather than one careless call
site.
### Suggested direction
Either a small non-zero default for `polling_interval`, or (cheaper) one
sentence in the Python `consumer_group` docstring plus a non-`1ms` example in
the docs.
### One question, not a complaint
Does Iggy support or plan **server-side long-polling** — the broker holding
a poll open until a message arrives or a timeout expires (Kafka's
`fetch.max.wait.ms`, SQS long polling)? That makes an idle consumer nearly free
regardless of what the client sets, and would remove this category at the
source rather than documenting around it.
### Affected area / component
Python SDK (`foreign/python`), consumer configuration;
`core/sdk/src/clients/consumer_builder.rs` defaults; docs (high-level SDK page).
### Deployment
Single `iggy-server` binary under systemd, TCP transport on loopback, one
stream / one topic / one partition, one consumer group.
### Versions
Server `iggy-server` 0.8.0, Python SDK `apache-iggy` 0.8.0 (PyPI).
### Hardware / environment
Proxmox LXC container, **1 vCPU**, 4 GiB RAM, no swap, ~20 other services
co-resident. Local disk.
### Sample code
Before (the omission):
```python
consumer = await client.consumer_group(
group, stream, topic,
polling_strategy=PollingStrategy.Next(),
auto_commit=AutoCommit.After(AutoCommitAfter.ConsumingEachMessage()),
)
await consumer.consume_messages(on_message, shutdown)
```
After:
```python
consumer = await client.consumer_group(
group, stream, topic,
polling_strategy=PollingStrategy.Next(),
auto_commit=AutoCommit.After(AutoCommitAfter.ConsumingEachMessage()),
poll_interval=timedelta(milliseconds=250),
)
```
### Reproduction
1. Run `iggy-server` and one Python consumer built as above, on a box where
you can see per-process CPU clearly (a single core makes it obvious).
2. Publish nothing — leave the topic idle.
3. Observe `iggy-server` and the consumer process in `top`, and the
context-switch rate in `vmstat 2`.
4. Add `poll_interval=timedelta(milliseconds=250)` and repeat.
On Linux, `cat /proc/pressure/cpu` is the clearest single signal — it went
from `some avg10=70.2` to `some avg10=1.6` across that change.
### Note on framing
I want to be straightforward that this may be working as intended: Iggy is a
throughput broker, and "set the interval, that's what it's for" is a fair
answer. If that's the call, I'd suggest keeping only the documentation part —
the Python docstring never mentions the consequence, and the sole published
example is `1ms`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
