bbejeck commented on code in PR #23086:
URL: https://github.com/apache/kafka/pull/23086#discussion_r3754399696
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StoreChangelogReader.java:
##########
@@ -199,13 +200,119 @@ long calculateRestoreTime(final long restoreEndTimeNs) {
private static final long DEFAULT_OFFSET_UPDATE_MS =
Duration.ofMinutes(5L).toMillis();
+ // Windows the probe reads back from each partition's end. Not 1: under
EOS the last offset is
+ // usually a transaction control record, which is never delivered to a
consumer, so probing
+ // there is a guaranteed empty poll. The first window answers the large
majority; only the
+ // partitions it cannot answer for pay to widen.
+ private static final long[] PROBE_WINDOWS = {128L, 512L, 2048L};
Review Comment:
You may have had this before but consider adding a shallower rung:
`PROBE_WINDOWS = {8L, 128L, 512L, 2048L}`. The probe fetches the entire
window, but we only need one record from it — so starting smaller cuts traffic.
I know this is a reversal of a comment I had earlier but some local testing
convinced me otherwise.
UPDATE: I've been thinking about this and I'm not sure - if the record with
the correct ending timestamp is in this first rung, then we don't pull nearly
the same amount of data - but on the other hand if it's not we're increasing
the load on the broker, so I'm leaning on keeping this as is to keep broker
load to a minimum
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
