lianetm commented on code in PR #21457:
URL: https://github.com/apache/kafka/pull/21457#discussion_r2827234894
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/OffsetFetcher.java:
##########
@@ -136,11 +137,16 @@ public Map<TopicPartition, OffsetAndTimestamp>
offsetsForTimes(Map<TopicPartitio
private ListOffsetResult fetchOffsetsByTimes(Map<TopicPartition, Long>
timestampsToSearch,
Timer timer,
- boolean requireTimestamps) {
+ boolean requireTimestamps,
+ boolean
shouldUpdatePartitionEndOffsets) {
Review Comment:
this name is very confusing because it clearly says it will update end
offsets (first thing that comes to mind is an actual change to positions, not a
flag).
Would it help if we rename to mention it's to update a flag (maybe
updatePartitionEndOffsetsFlag), or at least a description of the param?
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/SubscriptionState.java:
##########
@@ -659,14 +659,25 @@ public synchronized Long
partitionEndOffset(TopicPartition tp, IsolationLevel is
public synchronized void requestPartitionEndOffset(TopicPartition tp) {
TopicPartitionState topicPartitionState = assignedState(tp);
- topicPartitionState.requestEndOffset();
+ topicPartitionState.setRequestEndOffset(true);
}
public synchronized boolean partitionEndOffsetRequested(TopicPartition tp)
{
TopicPartitionState topicPartitionState = assignedState(tp);
return topicPartitionState.endOffsetRequested();
}
+ public synchronized boolean
maybeClearPartitionEndOffsetRequested(TopicPartition tp) {
+ TopicPartitionState topicPartitionState = assignedStateOrNull(tp);
+
+ if (topicPartitionState != null &&
topicPartitionState.endOffsetRequested()) {
+ topicPartitionState.setRequestEndOffset(false);
+ return true;
+ } else {
+ return false;
Review Comment:
ok to leave it if you think it makes troubleshooting easier
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/OffsetFetcher.java:
##########
@@ -185,23 +197,48 @@ public void onFailure(RuntimeException e) {
}
public Map<TopicPartition, Long>
beginningOffsets(Collection<TopicPartition> partitions, Timer timer) {
- return beginningOrEndOffset(partitions,
ListOffsetsRequest.EARLIEST_TIMESTAMP, timer);
+ return beginningOrEndOffset(partitions,
ListOffsetsRequest.EARLIEST_TIMESTAMP, timer, false);
}
public Map<TopicPartition, Long> endOffsets(Collection<TopicPartition>
partitions, Timer timer) {
- return beginningOrEndOffset(partitions,
ListOffsetsRequest.LATEST_TIMESTAMP, timer);
+ return beginningOrEndOffset(partitions,
ListOffsetsRequest.LATEST_TIMESTAMP, timer, false);
+ }
+
+ public OptionalLong currentLag(TopicPartition topicPartition) {
+ final Long lag = subscriptions.partitionLag(topicPartition,
isolationLevel);
+
+ // if the log end offset is not known and hence cannot return lag and
there is
+ // no in-flight list offset requested yet,
+ // issue a list offset request for that partition so that next time
+ // we may get the answer; we do not need to wait for the return value
+ // since we would not try to poll the network client synchronously
+ if (lag == null) {
+ if (subscriptions.partitionEndOffset(topicPartition,
isolationLevel) == null) {
Review Comment:
aren't we missing the check here to ensure there is no request in-flight?
We should also ensure we have a test for this: 2 consecutive calls to
currentLag at this level, first one should generate a request, no response,
second call should not generate the request. I would expect such test should be
failing now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
