surafel58 opened a new issue, #11816: URL: https://github.com/apache/seatunnel/issues/11816
### Search before asking - [X] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement. ### Description The Prometheus sink buffers records and flushes them in three cases: when `batch_size` is reached, when the Zeta engine delivers a timer flush signal (`Context#registerFlushAction`), and in `close()`. It does not flush on checkpoint. On Spark and Flink the engine does not implement `registerFlushAction` (it keeps the interface's no-op default), so there is no periodic timer flush on those engines. Because the writer also has no checkpoint flush, buffered records on Spark/Flink only reach Prometheus when `batch_size` is hit or when the job closes. For a long-running streaming job whose incoming rate is below `batch_size`, buffered points can sit in memory for a long time before they are sent. All five sibling sinks that use the same FlushSignal mechanism already flush their buffer in `prepareCommit()`, so on Spark/Flink they still flush at every checkpoint: | Connector | Flushes buffer in `prepareCommit()` | |---|---| | Doris | yes | | ClickHouse | yes | | Elasticsearch | yes | | StarRocks | yes | | MongoDB | yes | | **Prometheus** | **no** | Prometheus is the only sink in this group without a `prepareCommit` flush. **Proposal:** add a `prepareCommit()` override to `PrometheusWriter` that flushes the buffer and returns `Optional.empty()`, matching the sibling connectors. This bounds the buffered window to one checkpoint interval on every engine, not just Zeta, without adding a connector-owned thread or any engine-level change. The Prometheus doc and the `incompatible-changes` note should be updated so the Spark/Flink limitation reflects the smaller remaining gap (no sub-checkpoint timer flush) instead of no periodic flush at all. This was flagged during review of #11778 as a broader follow-up. That review weighed documenting the limitation against an engine-level change to add `registerFlushAction` support to Spark/Flink. The connector-level `prepareCommit` path that the five sibling sinks already use was not raised at the time, so the limitation was documented instead. This issue proposes that connector-level path. ### Usage Scenario A streaming job writes Prometheus metrics on Spark or Flink with a `batch_size` larger than the incoming rate. Today the buffered points are not sent until `batch_size` is reached or the job stops, so they are not visible in Prometheus for a long time. With a checkpoint flush, the buffer is sent at each checkpoint, so points become visible within one checkpoint interval. ### Related issues - #11773 (original FlushSignal migration issue) - #11778 (migration PR where this follow-up was flagged in review) ### Are you willing to submit a PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
