npawar opened a new issue #7555:
URL: https://github.com/apache/pinot/issues/7555
If the realtime servers are restarted, the KinesisConsumer should pick up
reading from the start offset in the realtime segment metadata. But the
shardIterator ignores the "startingSequenceNumber" if the iterator type is
LATEST.
```
private String getShardIterator(String shardId, String sequenceNumber) {
GetShardIteratorRequest.Builder requestBuilder =
GetShardIteratorRequest.builder().streamName(_streamTopicName).shardId(shardId)
.shardIteratorType(_shardIteratorType);
if (sequenceNumber != null &&
(_shardIteratorType.equals(ShardIteratorType.AT_SEQUENCE_NUMBER) ||
_shardIteratorType
.equals(ShardIteratorType.AFTER_SEQUENCE_NUMBER))) {
requestBuilder = requestBuilder.startingSequenceNumber(sequenceNumber);
}
return
_kinesisClient.getShardIterator(requestBuilder.build()).shardIterator();
}
```
This is incorrect and will cause missed events.
In general as well, the LATEST mode might not be behaving correctly. The
Kinesis consumer should always read from provided startOffset, regardless of
shardIteratorType.
I also noticed that in KinesisConsumer the "smallest/largest" setting from
streamConfig might be getting ignored, and we're always reading from smallest.
@KKcorps FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
