richardcocks opened a new issue, #4066:
URL: https://github.com/apache/iggy/issues/4066
I've not tagged this a bug, because current behaviour is as-designed, but
the current state of play is that client SDKs have a feature to generate
messageId values rather than passing 0 and leaving it up to the server.
The message Id is 16 opaque random bytes. It doesn't appear to be used for
idempotency, it doesn't get keyed off, and it doesn't appear to need to be
secret.
As far as I can tell, it does not therefore need to be crypto-secure, and
indeed clients are free to pass sequential IDs if they wish to do so.
The default behaviour however is the current generation schemes:
| client | trigger | generator | RNG | form → 16 bytes | overhead |
|---|---|---|---|---|---|
| **Rust** | `id == 0` | `Uuid::new_v4().to_u128_le()`
(`core/common/src/utils/random_id.rs:22`) | `getrandom` (OS) | native `u128` |
~none (no alloc, no string) |
| **C#** | `Header.Id == 0` (`Iggy_SDK/Contracts/Tcp/TcpContracts.cs:350`) |
`Guid.NewGuid().ToUInt128()` | OS RNG | `UInt128` reinterpret (struct) | ~none
(value type, no alloc) |
| **Go** | `Header.Id == zero` (`foreign/go/internal/command/message.go:98`)
| `uuid.NewRandom()` (google/uuid) | `crypto/rand` (OS) | fills a `[16]byte`
directly | low (one small value) |
| **Java** | id `signum == 0` (`serde/BytesSerializer.java:342`) |
`UUID.randomUUID()` | shared `SecureRandom` (contended) | `UUID` obj →
`byte[16]` copy | higher: crypto RNG + object + `byte[]` alloc |
| **Node** | id absent/zero (`wire/message/message.utils.ts:127`) |
`uuidv4()` (uuidv7 pkg) | `getRandomValues` (OS), inside | string →
`BigInt('0x…')` → `u128ToBuf` | worst: ~2 strings + a BigInt per msg |
Node is by far the worst on my ( admittedly very artificial ) benchmark, my
throughput was ~1.8M/sec on go/rust/csharp, ~1M/sec on Java and just 0.183M/sec
on Node, with a large part of the different attributable to message Id
generation, with a 2.3x throughput improvement by switching to sequential IDs.
If it's confirmed that message Ids do not need to be crypto-secure, then
there are much better options for message IDs. Twitter developed Snowflake ID
for multi-node collision resistant monotonic ID generation, and I'd recommend
switching to that.
Another alternative to consider would be UUIDv7, which generates monotonic
but still unique IDs, but snowflake has the advantage of removing the need for
random generation at all which puts it on a par with sequential IDs for
performance.
Sticking to entirely random IDs, performance improvements could be made
across the board by switching from CSPRNGs to regular random sources. In the
case of Node, the difference seems especially dramatic, with the default
xoroshift128 being close to a sequential counter in performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
