shanthoosh commented on a change in pull request #918: SAMZA-2094: Implement
the StartpointVisitor for the KafkaSystemConsumer.
URL: https://github.com/apache/samza/pull/918#discussion_r261468553
##########
File path:
samza-core/src/main/scala/org/apache/samza/system/SystemConsumers.scala
##########
@@ -208,7 +223,11 @@ class SystemConsumers (
if (startpoint != null) {
consumer.register(systemStreamPartition, startpoint)
} else {
- consumer.register(systemStreamPartition, offset)
+ val existingOffset = sspToRegisteredOffsets.get(systemStreamPartition)
+ val systemAdmin =
systemAdmins.getSystemAdmin(systemStreamPartition.getSystem)
+ if (existingOffset == null ||
systemAdmin.offsetComparator(existingOffset, offset) > 0) {
+ sspToRegisteredOffsets.put(systemStreamPartition, offset)
Review comment:
1. This would happen in the broadcast stream scenario. This
older-offset-comparator check is done by some of the SystemConsumer
implementation and not in the others. I'm moving it to a common layer like
`SystemConsumers` above to ensure functional correctness. More context
[here](https://github.com/apache/samza/pull/918#discussion_r257523873).
2. Currently this odd behavior is what all the system consumer
implementations that support broadcast streams do.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
