jbonofre opened a new pull request, #41510:
URL: https://github.com/apache/beam/pull/41510

   ## Summary
   Fixes two checkpoint lifecycle issues in `KafkaIO`'s `UnboundedSource` 
implementation (analogous to the `JmsIO` fixes):
   
   1. **Avoid NPE and preserve live reader lookup across Avro checkpoint 
serialization (`KafkaCheckpointMark` / `KafkaUnboundedReader`)**:
      - In the no-arg Avro constructor `KafkaCheckpointMark()`, `reader` was 
left `null` (triggering an NPE whenever `finalizeCheckpoint()` or 
`getOffsetLimit()` was called on a deserialized `KafkaCheckpointMark`). 
Initialize `this.reader = Optional.empty()` in the no-arg constructor.
      - Register live `KafkaUnboundedReader` instances in a weak-valued 
`Cache<String, KafkaUnboundedReader<?, ?>>` keyed by a per-reader `readerId` 
when `getCheckpointMark()` creates tracked checkpoints, serialize `readerId` in 
`KafkaCheckpointMark`, and unregister in `KafkaUnboundedReader.close()`. 
Restored `KafkaCheckpointMark` instances can still trigger async finalization 
on the live reader before closing, and safely no-op once the reader is closed.
   2. **Finalize previous checkpoint when resuming a reader 
(`KafkaUnboundedSource`)**:
      - In `KafkaUnboundedSource.createReader(PipelineOptions, 
KafkaCheckpointMark)`, call `checkpointMark.finalizeCheckpoint()` when 
`checkpointMark != null` so a previously unfinalized checkpoint is committed 
before creating a new reader.
   
   ## Testing
   - Added 
`KafkaUnboundedReaderIdlePartitionTest#testRestoredCheckpointCommitsOffsetsAndResumeDoesNotThrow`
 covering serialized/restored checkpoint finalization on a live reader, 
finalizing on `createReader(..., checkpointMark)`, and post-close 
`finalizeCheckpoint()` safety.
   - Ran `./gradlew :sdks:java:io:kafka:spotlessApply 
:sdks:java:io:kafka:spotlessCheck :sdks:java:io:kafka:checkstyleMain 
:sdks:java:io:kafka:checkstyleTest :sdks:java:io:kafka:spotbugsMain 
:sdks:java:io:kafka:test --tests "*KafkaUnboundedReader*" --tests 
"*KafkaIOTest*"`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to