mattp5657 opened a new issue, #4386:
URL: https://github.com/apache/iggy/issues/4386
### Description
`field_mappings` silently loses or misroutes data in three pre-existing
cases. None of the 9 config types that carry `field_mappings` validate it.
All three are edge cases. A bad config (two mappings onto one target, or
two proto fields sharing a JSON name) is the likelier trigger. Payloads
that already contain the target key or duplicate keys are more rare.
1. **Rename collision drops a value.** Two keys rename onto one target,
or a rename lands on a key already in the payload. One value is
dropped. Which one survives is unspecified once simd_json switches to
its hash-backed map (over 32 keys).
2. **Duplicate keys in incoming JSON.** simd_json keeps every duplicate
when parsing (`{"a":1,"a":2}` parses with `len() == 2`), so a naive
runtime collision check (`insert(..).is_some()`) fails on payloads
whose mapped keys never collide. Real producers rarely emit
duplicate keys.
3. **Proto encoder reverse lookup is ambiguous.** `ProtoStreamEncoder`
maps JSON name to proto name with `.find()` over a `HashMap`. If two
mappings share a JSON name, the proto field chosen depends on
per-process hash order, so a restart can change the wire field.
### Affected area / component
Connectors
### Proposed solution
Decide the policy first, then apply it across all 9 config types:
- Config load: reject duplicate JSON names in `ProtoEncoderConfig` (3).
For many-to-one targets (1), decide whether aliasing two source names
onto one key is legitimate, since a config check alone cannot tell
aliasing from a real collision.
- Runtime: decide how duplicate-keyed JSON (2) is handled before adding
any per-message collision check.
### Alternatives considered
- Sort candidates in the proto encoder for a deterministic winner.
- Warn on collision at runtime. This runs per message, so it needs
once, rate-limited or metric-based reporting rather than a plain `warn!`.
- Enable simd_json's `value-no-dup-keys` to reject duplicates at parse time.
### Contribution
- [ ] I'm willing to submit a pull request to implement this feature
### Good first issue
- [x] I think this could be a good first issue for a new contributor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
