github-actions[bot] commented on code in PR #41510:
URL: https://github.com/apache/beam/pull/41510#discussion_r4645209824
##########
sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReaderIdlePartitionTest.java:
##########
@@ -143,6 +144,60 @@ public void testIdlePartitionsNotCommittedRepeatedly()
throws Exception {
reader.close();
}
+ @Test
+ public void testRestoredCheckpointCommitsOffsetsAndResumeDoesNotThrow()
throws Exception {
+ int numElements = 50;
+ int numPartitions = 4;
+ List<String> topics = ImmutableList.of("test_topic");
+
+ TrackingConsumerFactory consumerFactory =
+ new TrackingConsumerFactory(topics, numPartitions, numElements);
+
+ UnboundedSource<KafkaRecord<Integer, Long>, KafkaCheckpointMark> source =
+ KafkaIO.<Integer, Long>read()
+ .withBootstrapServers("test_server")
+ .withTopics(topics)
+ .withConsumerFactoryFn(consumerFactory)
+ .withKeyDeserializer(IntegerDeserializer.class)
+ .withValueDeserializer(LongDeserializer.class)
+ .withConsumerConfigUpdates(
+ ImmutableMap.of(ConsumerConfig.GROUP_ID_CONFIG,
"test_group_id"))
+ .commitOffsetsInFinalize()
+ .makeSource()
+ .split(1, PipelineOptionsFactory.create())
+ .get(0);
+
+ UnboundedReader<KafkaRecord<Integer, Long>> reader =
source.createReader(null, null);
+ assertTrue(reader.start());
+ for (int i = 0; i < 10; i++) {
+ assertTrue(reader.advance());
+ }
+
Review Comment:
**🟡 Medium**: Consider also adding a wire-compatibility assertion (similar
to `testDecodeCheckpointMarkWithoutId` in `JmsLocalTest`) that decodes
`KafkaCheckpointMark` bytes encoded with the pre-existing Avro schema
(containing only `partitions`) and calls `finalizeCheckpoint()` to guard
against regressions during streaming pipeline upgrades.
##########
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaCheckpointMark.java:
##########
@@ -37,18 +39,44 @@
public class KafkaCheckpointMark implements UnboundedSource.CheckpointMark {
private static final long OFFSET_DEDUP_PARTITIONS_PER_SPLIT = 1;
+ private static final Cache<String, KafkaUnboundedReader<?, ?>> LIVE_READERS =
+ CacheBuilder.newBuilder().weakValues().build();
+
private List<PartitionMark> partitions;
+ private String readerId = "";
Review Comment:
**🔴 Critical**: Adding a non-`@AvroIgnore` field `readerId` to
`KafkaCheckpointMark` breaks backward and update compatibility for streaming
pipelines:
1. `KafkaUnboundedSource.getCheckpointMarkCoder()` uses
`AvroCoder.of(KafkaCheckpointMark.class)`, which uses Avro `ReflectData` to
generate the schema and encodes raw binary bytes without an embedded writer
schema (`ReflectDatumReader` uses the current reflected schema as both writer
and reader schema).
2. When a pipeline is updated or restored from a savepoint/snapshot created
by an earlier Beam version, `AvroCoder.decode()` attempts to read the trailing
`readerId` string after `partitions` and throws `java.io.EOFException`.
3. Changing the reflected Avro schema also changes `AvroCoder`'s serialized
schema definition, which fails coder compatibility checks on runners such as
Dataflow (`--update`).
Because each `KafkaUnboundedReader` handles a disjoint split of
topic-partitions and `PartitionMark` already serializes `topic` and
`partition`, you can derive the lookup key directly from `partitions` (e.g.,
mapping `(topic, partition)` pairs) without adding `readerId` to the Avro
schema, and use `LIVE_READERS.asMap().remove(key, reader)` in
`unregisterLiveReader` so closing an old reader cannot evict a newly created
reader for the same split.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]