stinger1206 opened a new issue, #114:
URL: https://github.com/apache/doris-kafka-connector/issues/114
# [Issue] Writer selection by string prefix + MAX offset aggregation can
lose/stall records
## Summary
Two related defects in `DorisDefaultSinkService` (26.0.0, 26.1.0, master):
### 1. String-prefix writer selection aggregates unrelated topic-partitions
Writers are keyed `topic_partition[_table]` and selected with
`entry.getKey().startsWith(topic + "_" + partition)`. Because one key can be
a
string-prefix of another, this matches the wrong writers:
- `topic_test` partition 1 (`topic_test_1...`) also matches partition 10's
writers
(`topic_test_10...`);
- topic `X` partition 1 collides with topic `X_1` partition 0.
`getOffset()` then aggregates unrelated partitions — reporting a buffered
writer's offset as
another partition's committed offset (stalled or wrong commits), and
`commit()` commits
another partition's writers.
### 2. MAX offset aggregation loses buffered records (multi-table routing)
With `record.tablename.field`, one topic-partition routes to multiple
tables, one writer
each. `getOffset()` returned `max(writer.getOffset())` — safe only for the
most advanced
writer. Records still buffered in a lagging writer are below that offset;
after the offset
is committed and the process crashes, they are never re-consumed.
```text
offset 100 -> table B (still buffered in writer B)
offset 101 -> table A (flushed + committed in writer A)
getOffset() = MAX = 102 -> committed -> crash -> record 100 for table
B is lost
```
## Impact
- Defect 1: any deployment whose topic/partition names can prefix-collide
(partition 1 vs
10 of the same topic is enough). Wrong offsets for both `getOffset` and
`commit`.
- Defect 2: deployments using `record.tablename.field`. Silent record loss
on crash.
- A naive MAX->MIN replacement is not a fix: an idle table writer would
stall the
topic-partition's committed offset indefinitely.
## Suggested fix
1. Select writers by exact (topic, partition) identity — e.g. a structured
writer key with
`equals`/`hashCode` — not string-prefix matching.
2. Report a contiguous safe watermark per topic-partition:
- if any writer still buffers records: the minimum first-buffered offset
across writers;
- else: the max committed offset across writers.
`DorisSinkTask.preCommit` calls `commit()` (durable flush/2PC, throws on
failure) before
`getOffset()`, so "empty buffer + committed offset" at getOffset time means
durably loaded;
see the PR for the full lifecycle analysis.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]