jiengup opened a new pull request, #3613:
URL: https://github.com/apache/iggy/pull/3613
Add user headers to SendMessage and ReceiveMessage, expose origin timestamp,
and fix Docker test infrastructure.
## Which issue does this PR address?
<!--
We generally require a GitHub issue for all bug fixes and enhancements. Link
it with GitHub syntax, keep the line that applies and delete the other:
- `Closes #123` auto-closes the issue when this PR merges (full fix).
- `Relates to #123` links without closing (partial or related work).
-->
Closes #3601 #3612
## Rationale
<!--
Why is this change needed? If the issue explains it well, a one-liner is
fine.
-->
The Python SDK could send and receive message payloads, but it did not
expose the typed user headers already carried by the underlying Rust
`IggyMessage`. This left Python behind the Rust, Node, Go, Java, and C# SDKs
for message metadata.
## What changed?
<!--
2-4 sentences. Problem first (before), then solution (after).
GOOD:
"Messages were unavailable when background message_saver committed the
journal and started async disk I/O before completion. Polling during
this window found neither journal nor disk data.
The fix freezes journal batches in the in-flight buffer before async
persist."
GOOD:
"When many small messages accumulate in the journal, the flush passes
thousands of IO vectors to writev(), exceeding IOV_MAX (1024 on Linux)."
BAD:
- Walls of text
- "This PR adds..." (we can see the diff)
-->
Python messages can now attach common user headers via plain Python values
and read them back from received messages. `SendMessage` accepts `user_headers`
and an optional custom `id`, while `ReceiveMessage` exposes `user_headers()`
and `origin_timestamp()`.
The binding maps Python values to existing Rust header types: `str` to
`String`, `bytes` to `Raw`, `bool` to `Bool`, `int` to `Int64`, and `float` to
`Float64`. Receiving maps all integer header kinds back to Python `int`, float
kinds to `float`, and returns `None` when no user headers are present.
### Minor Fix
The Python test compose setup was adjusted to start the server with fresh
default root credentials, bind HTTP/TCP/QUIC addresses explicitly, and use
`iggy ping` for the healthcheck instead of the HTTP stats endpoint.
## Discussion Notes
Implemented choices from the issue discussion:
- Python uses a plain `dict[str, str | bytes | bool | int | float]` API for
the common case, rather than exposing HeaderKey/HeaderValue wrapper classes.
- Python `int` sends as `Int64`; values outside the signed 64-bit range
raise `ValueError`.
- `ReceiveMessage.user_headers()` returns `None` for no headers.
- Non-string header keys are rejected with `ValueError` in Python.
- Header decode errors from known-but-invalid or unknown semantic kinds
surface as `ValueError`.
- `SendMessage(id=...)` is included in the same binding update.
- `origin_timestamp()` is exposed on received messages.
Worth discussing before merge:
- Go supports explicit `HeaderEntry` values with `HeaderKey{Kind, Value}`
and `HeaderValue{Kind, Value}`; Python currently only supports string keys. Do
we need to support any other kind `HeaderKey` in Python SDK for now?
- Rust supports the full signed, unsigned, 128-bit, and Float32/Float64 kind
surface. Python sends only `Int64` and `Float64` from plain values.
- Message `id` parity could be expanded later: Node accepts number, bigint,
and UUID string, while this Python change accepts only Python `int` mapped to
`u128`. Do we need them?
## Local Execution
- Passed: `cargo fmt --check --manifest-path foreign/python/Cargo.toml`
- Passed: `cargo check --manifest-path foreign/python/Cargo.toml`
- Passed: `cargo test --manifest-path foreign/python/Cargo.toml`
- Passed: `uv run --extra dev ruff check tests/test_message_operations.py
tests/test_consumer_group.py`
- Passed: `.venv/bin/python -m pytest
tests/test_message_operations.py::TestMessageOperations::test_invalid_user_headers_are_rejected
-q`
## AI Usage
Codex was used to inspect the existing Python, Rust, Node, and Go SDK
behavior, implement the Python binding changes, add tests. All the modification
was reviewed carefully by the human.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
