jbonofre opened a new pull request, #41503:
URL: https://github.com/apache/beam/pull/41503

   Similar to the recent JmsIO fixes (#40605, #41013), `MqttCheckpointMark` 
previously stored pending `Mqtt3Publish` messages only in a `transient` list:
   
   1. **Restored checkpoints lost pending messages**: When a runner serializes 
and restores `MqttCheckpointMark` (or recreates the reader), `readObject` 
resets `messages` to an empty list, so `finalizeCheckpoint()` silently 
acknowledges nothing and messages are redelivered.
   2. **Unfinalized earlier checkpoints & premature disconnect**: Checkpoints 
that are taken without being finalized (e.g., dropped callbacks or progress 
snapshots) are never acknowledged by subsequent checkpoints or on 
`createReader(..., checkpointMark)` resume, and `UnboundedMqttReader.close()` 
immediately disconnects the MQTT client before in-flight checkpoints finalize.
   
   ## Changes
   - Give each reader a per-JVM weak-valued `PendingCheckpoints` registry 
(`readerId` + monotonic `sequence` on `MqttCheckpointMark`, with 
`serialVersionUID` pinned for backward compatibility).
   - `finalizeCheckpoint()` resolves the live reader's `PendingCheckpoints` 
(even from a restored copy) and acknowledges this checkpoint together with all 
earlier unfinalized checkpoints of the same reader in order.
   - `UnboundedMqttSource.createReader(options, checkpointMark)` finalizes the 
checkpoint it resumes from when its originating reader is still live in the 
same JVM.
   - `UnboundedMqttReader.close()` waits for in-flight checkpoints to finalize 
(up to a 60s timeout, closing immediately once drained) before disconnecting 
the MQTT client, abandoning any remaining pending checkpoints on timeout.
   - Added unit tests in `MqttIOTest` covering restored checkpoint 
finalization, catching up earlier checkpoints, empty checkpoints, and resuming 
via `createReader`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to