Bharath Reddy Gunapati created FLINK-41218:
----------------------------------------------
Summary: Add per-record result processing for HTTP sink batch
responses
Key: FLINK-41218
URL: https://issues.apache.org/jira/browse/FLINK-41218
Project: Flink
Issue Type: Sub-task
Components: Connectors / HTTP
Reporter: Bharath Reddy Gunapati
In batch submission mode one HTTP request carries many request entries and gets
one status code, so the sink retries or fails the whole batch (FLINK-40278
documents this). Targets with bulk APIs report a result per item in the
response body, for example Elasticsearch-style bulk endpoints or webhooks that
validate each item. For those the sink currently resends entries the target
already accepted, which produces duplicate writes, and fails entries that would
have succeeded on their own.
h3. Proposed scope
* A pluggable response processor interface for the sink that receives the HTTP
response and the request entries of that request, and classifies each entry as
success, retryable, fatal or ignored.
* A default implementation for JSON bodies that maps results to entries by
index or by an id field, configurable via table and DataStream options.
* When no processor is configured, the status code keeps deciding for the whole
request, so existing jobs are unchanged.
* Metrics from FLINK-40281 and the retry layer from FLINK-40277 work per entry.
h3. Acceptance criteria
* A batch whose response marks one entry as failed retries only that entry; the
others complete.
* A processor that throws or returns no result for an entry falls back to the
status-code classification for that entry.
* Documented in the "Retries and handling errors (Sink)" section with an
example response and configuration.
* Unit tests for the default processor and for the retry path with partial
results; an end-to-end test with an embedded server.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
