github-actions[bot] commented on code in PR #41510:
URL: https://github.com/apache/beam/pull/41510#discussion_r4645209824


##########
sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReaderIdlePartitionTest.java:
##########
@@ -143,6 +144,60 @@ public void testIdlePartitionsNotCommittedRepeatedly() 
throws Exception {
     reader.close();
   }
 
+  @Test
+  public void testRestoredCheckpointCommitsOffsetsAndResumeDoesNotThrow() 
throws Exception {
+    int numElements = 50;
+    int numPartitions = 4;
+    List<String> topics = ImmutableList.of("test_topic");
+
+    TrackingConsumerFactory consumerFactory =
+        new TrackingConsumerFactory(topics, numPartitions, numElements);
+
+    UnboundedSource<KafkaRecord<Integer, Long>, KafkaCheckpointMark> source =
+        KafkaIO.<Integer, Long>read()
+            .withBootstrapServers("test_server")
+            .withTopics(topics)
+            .withConsumerFactoryFn(consumerFactory)
+            .withKeyDeserializer(IntegerDeserializer.class)
+            .withValueDeserializer(LongDeserializer.class)
+            .withConsumerConfigUpdates(
+                ImmutableMap.of(ConsumerConfig.GROUP_ID_CONFIG, 
"test_group_id"))
+            .commitOffsetsInFinalize()
+            .makeSource()
+            .split(1, PipelineOptionsFactory.create())
+            .get(0);
+
+    UnboundedReader<KafkaRecord<Integer, Long>> reader = 
source.createReader(null, null);
+    assertTrue(reader.start());
+    for (int i = 0; i < 10; i++) {
+      assertTrue(reader.advance());
+    }
+

Review Comment:
   **🟡 Medium**: Consider also adding a wire-compatibility assertion (similar 
to `testDecodeCheckpointMarkWithoutId` in `JmsLocalTest`) that decodes 
`KafkaCheckpointMark` bytes encoded with the pre-existing Avro schema 
(containing only `partitions`) and calls `finalizeCheckpoint()` to guard 
against regressions during streaming pipeline upgrades.



##########
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaCheckpointMark.java:
##########
@@ -37,18 +39,44 @@
 public class KafkaCheckpointMark implements UnboundedSource.CheckpointMark {
   private static final long OFFSET_DEDUP_PARTITIONS_PER_SPLIT = 1;
 
+  private static final Cache<String, KafkaUnboundedReader<?, ?>> LIVE_READERS =
+      CacheBuilder.newBuilder().weakValues().build();
+
   private List<PartitionMark> partitions;
+  private String readerId = "";

Review Comment:
   **🔴 Critical**: Adding a non-`@AvroIgnore` field `readerId` to 
`KafkaCheckpointMark` breaks backward and update compatibility for streaming 
pipelines:
   
   1. `KafkaUnboundedSource.getCheckpointMarkCoder()` uses 
`AvroCoder.of(KafkaCheckpointMark.class)`, which uses Avro `ReflectData` to 
generate the schema and encodes raw binary bytes without an embedded writer 
schema (`ReflectDatumReader` uses the current reflected schema as both writer 
and reader schema).
   2. When a pipeline is updated or restored from a savepoint/snapshot created 
by an earlier Beam version, `AvroCoder.decode()` attempts to read the trailing 
`readerId` string after `partitions` and throws `java.io.EOFException`.
   3. Changing the reflected Avro schema also changes `AvroCoder`'s serialized 
schema definition, which fails coder compatibility checks on runners such as 
Dataflow (`--update`).
   
   Because each `KafkaUnboundedReader` handles a disjoint split of 
topic-partitions and `PartitionMark` already serializes `topic` and 
`partition`, you can derive the lookup key directly from `partitions` (e.g., 
mapping `(topic, partition)` pairs) without adding `readerId` to the Avro 
schema, and use `LIVE_READERS.asMap().remove(key, reader)` in 
`unregisterLiveReader` so closing an old reader cannot evict a newly created 
reader for the same split.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to