surafel58 opened a new issue, #11911: URL: https://github.com/apache/seatunnel/issues/11911
### Search before asking - [X] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement. ### Description `PrometheusWriter.flush()` issues a single-shot HTTP remote-write POST and throws on any non-204 response or transport error. Every flush trigger calls this same method: the `batch_size` threshold in `write()`, the Zeta engine timer flush (`registerFlushAction`), the checkpoint flush (`prepareCommit()`, added in #11827), and the final flush in `close()`. Because there is no retry, a transient Prometheus / remote-write outage fails whichever operation triggered the flush. For the checkpoint path this means a brief blip can fail a checkpoint, and on Flink/Spark repeated checkpoint failures can eventually fail the job (subject to the restart strategy). The engine restart strategy does provide recovery, but a small bounded retry with backoff inside `flush()` would absorb short transient failures before escalating to a checkpoint/job failure. This was raised in the review of #11827 (@SEZ9). It is intentionally scoped as a follow-up because the single-shot behavior is pre-existing (it predates #11827, which only added `prepareCommit()` as one more caller), and a retry should apply uniformly to all flush paths rather than only the checkpoint path. **Proposal:** add a bounded retry with backoff around the remote-write POST in `PrometheusWriter.flush()` so all flush callers tolerate short transient failures. Retries should be bounded (small max attempt count + capped backoff) and still throw after exhaustion, so a genuine outage still fails loudly rather than blocking indefinitely. Consider reusing an existing retry helper if one is available in the codebase, and making the attempt count / backoff configurable via the sink options. ### Usage Scenario A streaming job writes metrics to a remote-write endpoint that has a brief transient error (for example a rolling restart of the receiver). Today the in-flight flush fails immediately; on the checkpoint path this fails the checkpoint. With a bounded retry, the flush retries a few times with backoff and usually succeeds, avoiding an unnecessary checkpoint/job failure, while a sustained outage still fails after the retries are exhausted. ### Related issues - #11827 (added the checkpoint flush; retry raised in review as a follow-up) - #11816 (the checkpoint-flush issue #11827 closes) ### Are you willing to submit a PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
