disha-chhabra-0002 opened a new issue, #3835: URL: https://github.com/apache/iggy/issues/3835
### Description Currently, the Iggy Iceberg sink connector only supports count-based/batch flushing (e.g., flushing after N messages).In low-throughput scenarios or topics with sporadic traffic, messages can sit buffered in memory for an extended period before reaching the batch size threshold. This introduces high end-to-end data latency and risks potential data loss if the connector crashes before reaching the message limit.Adding time-based flushing (e.g., flushing every X seconds/minutes regardless of message count) ensures predictable data latency in Apache Iceberg tables. ### Affected area / component _No response_ ### Proposed solution Introduce a configurable time interval parameter (e.g., flush_interval_ms or max_commit_interval_secs) in the Iceberg sink configuration. Behavior: The connector should flush buffered records to Iceberg whenever either condition is met: The maximum batch size (batch_size) is reached. The elapsed time since the last commit exceeds flush_interval_ms. ### Alternatives considered _No response_ ### Contribution - [x] I'm willing to submit a pull request to implement this feature ### Good first issue - [ ] I think this could be a good first issue for a new contributor -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
