Bharath Reddy Gunapati created FLINK-41219:
----------------------------------------------
Summary: Add an idempotency key header per HTTP sink request entry
Key: FLINK-41219
URL: https://issues.apache.org/jira/browse/FLINK-41219
Project: Flink
Issue Type: Sub-task
Components: Connectors / HTTP
Reporter: Bharath Reddy Gunapati
The HTTP sink is at-least-once: a retried request (FLINK-40277), a batch resent
after a partial failure (FLINK-40278) or a replay after a failover can deliver
the same record twice. Many targets can deduplicate when every request carries
a stable idempotency key (for example an {{Idempotency-Key}} header), but the
sink gives users no way to produce one that survives retries and restarts.
h3. Proposed scope
* Option to enable an idempotency key per request entry, with the header name
configurable (default {{Idempotency-Key}}).
* The key is generated once when the entry is created and stored with the entry
in writer state, so it is identical on every retry and after recovery from a
checkpoint.
* Optional derivation from a record field for Table API users who already have
a natural key.
* In batch submission mode, document how keys are sent (one header per request
is not possible for many entries), for example as a per-entry field in the JSON
array or a combined header, decided during design.
h3. Acceptance criteria
* The same entry resent after a retry or a restore carries the same key.
* Off by default; existing jobs are unchanged.
* Documented next to the batch retry caveat as the recommended way to avoid
duplicate writes.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
