jbonofre opened a new pull request, #41503: URL: https://github.com/apache/beam/pull/41503
Similar to the recent JmsIO fixes (#40605, #41013), `MqttCheckpointMark` previously stored pending `Mqtt3Publish` messages only in a `transient` list: 1. **Restored checkpoints lost pending messages**: When a runner serializes and restores `MqttCheckpointMark` (or recreates the reader), `readObject` resets `messages` to an empty list, so `finalizeCheckpoint()` silently acknowledges nothing and messages are redelivered. 2. **Unfinalized earlier checkpoints & premature disconnect**: Checkpoints that are taken without being finalized (e.g., dropped callbacks or progress snapshots) are never acknowledged by subsequent checkpoints or on `createReader(..., checkpointMark)` resume, and `UnboundedMqttReader.close()` immediately disconnects the MQTT client before in-flight checkpoints finalize. ## Changes - Give each reader a per-JVM weak-valued `PendingCheckpoints` registry (`readerId` + monotonic `sequence` on `MqttCheckpointMark`, with `serialVersionUID` pinned for backward compatibility). - `finalizeCheckpoint()` resolves the live reader's `PendingCheckpoints` (even from a restored copy) and acknowledges this checkpoint together with all earlier unfinalized checkpoints of the same reader in order. - `UnboundedMqttSource.createReader(options, checkpointMark)` finalizes the checkpoint it resumes from when its originating reader is still live in the same JVM. - `UnboundedMqttReader.close()` waits for in-flight checkpoints to finalize (up to a 60s timeout, closing immediately once drained) before disconnecting the MQTT client, abandoning any remaining pending checkpoints on timeout. - Added unit tests in `MqttIOTest` covering restored checkpoint finalization, catching up earlier checkpoints, empty checkpoints, and resuming via `createReader`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
