dcasado commented on issue #31313:
URL: https://github.com/apache/beam/issues/31313#issuecomment-2216976681
Hi, we are seeing the same behavior on our pipeline.
```
Logs from a Task Manager
2024-07-08 14:29:13,867 WARN org.apache.beam.sdk.coders.SerializableCoder
[] - Can't verify serialized elements of type
KinesisReaderCheckpoint have well defined equals method. This may produce
incorrect results on some PipelineRunner implementations
2024-07-08 14:29:13,867 INFO org.apache.beam.sdk.io.kinesis.KinesisSource
[] - Creating new reader using [Checkpoint AFTER_SEQUENCE_NUMBER
for stream <streamName>, shard shardId-000000002111:
49653574367537750625153914864609254626539224692753990642]
2024-07-08 14:29:13,945 INFO org.apache.beam.sdk.io.kinesis.KinesisReader
[] - Starting reader using [Checkpoint AFTER_SEQUENCE_NUMBER for
stream <streamName>, shard shardId-000000002111:
49653574367537750625153914864609254626539224692753990642]
2024-07-08 14:29:14,122 INFO
org.apache.beam.sdk.io.kinesis.ShardReadersPool [] - Starting to
read <streamName> stream from [shardId-000000002111] shards
2024-07-08 14:29:14,772 INFO
org.apache.beam.runners.flink.translation.wrappers.streaming.io.source.FlinkSourceReaderBase
[] - Adding splits [FlinkSourceSplit{splitIndex=2,
beamSource=org.apache.beam.sdk.io.kinesis.KinesisSource@a09dd60,
splitState.isNull=true, checkpointMark=null}]
2024-07-08 14:29:14,773 INFO
org.apache.beam.runners.flink.translation.wrappers.streaming.io.source.FlinkSourceReaderBase
[] - Received NoMoreSplits signal from enumerator.
2024-07-08 14:29:14,775 INFO org.apache.beam.sdk.io.kinesis.KinesisSource
[] - Creating new reader using [Checkpoint LATEST for stream
<streamName>, shard shardId-000000002111: null]
2024-07-08 14:29:14,778 INFO org.apache.beam.sdk.io.kinesis.KinesisReader
[] - Starting reader using [Checkpoint LATEST for stream
<streamName>, shard shardId-000000002111: null]
2024-07-08 14:29:14,830 INFO
org.apache.beam.sdk.io.kinesis.ShardReadersPool [] - Starting to
read <streamName> stream from [shardId-000000002111] shards
```
In our case we are using:
- FlinkRunner 1.17.2
- Beam 2.56.0
- KinesisIO (the deprecated one from `beam-sdks-java-io-kinesis`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
