[GitHub] atoomula merged pull request #912: SEP-19 : Refactoring sideInputs from SamzaContainer to ContainerStorageManager
atoomula merged pull request #912: SEP-19 : Refactoring sideInputs from SamzaContainer to ContainerStorageManager URL: https://github.com/apache/samza/pull/912 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] shanthoosh commented on a change in pull request #918: SAMZA-2094: Implement the StartpointVisitor for the KafkaSystemConsumer.
shanthoosh commented on a change in pull request #918: SAMZA-2094: Implement
the StartpointVisitor for the KafkaSystemConsumer.
URL: https://github.com/apache/samza/pull/918#discussion_r261470344
##
File path:
samza-kafka/src/main/java/org/apache/samza/system/kafka/KafkaSystemConsumer.java
##
@@ -330,6 +320,73 @@ public String getSystemName() {
return systemName;
}
+ @VisibleForTesting
+ static class KafkaStartpointRegistrationHandler implements StartpointVisitor
{
+
+private final Consumer kafkaConsumer;
+
+KafkaStartpointRegistrationHandler(Consumer kafkaConsumer) {
+ this.kafkaConsumer = kafkaConsumer;
+}
+
+@Override
+public void visit(SystemStreamPartition systemStreamPartition,
StartpointSpecific startpointSpecific) {
+ TopicPartition topicPartition = toTopicPartition(systemStreamPartition);
+ long offsetInStartpoint =
Long.parseLong(startpointSpecific.getSpecificOffset());
+ LOG.info("Updating the consumer fetch offsets of topic partition: {} to
{}.", topicPartition, offsetInStartpoint);
+
+ // KafkaConsumer is not thread-safe.
+ synchronized (kafkaConsumer) {
+kafkaConsumer.seek(topicPartition, offsetInStartpoint);
+ }
+}
+
+@Override
+public void visit(SystemStreamPartition systemStreamPartition,
StartpointTimestamp startpointTimestamp) {
+ Long timestampInStartpoint = startpointTimestamp.getTimestampOffset();
+ TopicPartition topicPartition = toTopicPartition(systemStreamPartition);
+ Map topicPartitionsToTimeStamps =
ImmutableMap.of(topicPartition, timestampInStartpoint);
+
+ // Look up the offset by timestamp.
+ LOG.info("Looking up the offsets of the topic partition: {} by
timestamp: {}.", topicPartition, timestampInStartpoint);
+ Map topicPartitionToOffsetTimestamps
= kafkaConsumer.offsetsForTimes(topicPartitionsToTimeStamps);
Review comment:
Fixed. Handled the null-case.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] shanthoosh commented on a change in pull request #918: SAMZA-2094: Implement the StartpointVisitor for the KafkaSystemConsumer.
shanthoosh commented on a change in pull request #918: SAMZA-2094: Implement
the StartpointVisitor for the KafkaSystemConsumer.
URL: https://github.com/apache/samza/pull/918#discussion_r261470344
##
File path:
samza-kafka/src/main/java/org/apache/samza/system/kafka/KafkaSystemConsumer.java
##
@@ -330,6 +320,73 @@ public String getSystemName() {
return systemName;
}
+ @VisibleForTesting
+ static class KafkaStartpointRegistrationHandler implements StartpointVisitor
{
+
+private final Consumer kafkaConsumer;
+
+KafkaStartpointRegistrationHandler(Consumer kafkaConsumer) {
+ this.kafkaConsumer = kafkaConsumer;
+}
+
+@Override
+public void visit(SystemStreamPartition systemStreamPartition,
StartpointSpecific startpointSpecific) {
+ TopicPartition topicPartition = toTopicPartition(systemStreamPartition);
+ long offsetInStartpoint =
Long.parseLong(startpointSpecific.getSpecificOffset());
+ LOG.info("Updating the consumer fetch offsets of topic partition: {} to
{}.", topicPartition, offsetInStartpoint);
+
+ // KafkaConsumer is not thread-safe.
+ synchronized (kafkaConsumer) {
+kafkaConsumer.seek(topicPartition, offsetInStartpoint);
+ }
+}
+
+@Override
+public void visit(SystemStreamPartition systemStreamPartition,
StartpointTimestamp startpointTimestamp) {
+ Long timestampInStartpoint = startpointTimestamp.getTimestampOffset();
+ TopicPartition topicPartition = toTopicPartition(systemStreamPartition);
+ Map topicPartitionsToTimeStamps =
ImmutableMap.of(topicPartition, timestampInStartpoint);
+
+ // Look up the offset by timestamp.
+ LOG.info("Looking up the offsets of the topic partition: {} by
timestamp: {}.", topicPartition, timestampInStartpoint);
+ Map topicPartitionToOffsetTimestamps
= kafkaConsumer.offsetsForTimes(topicPartitionsToTimeStamps);
Review comment:
Will handle the null-case.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
