jadami10 commented on code in PR #11345:
URL: https://github.com/apache/pinot/pull/11345#discussion_r1294541715
##########
pinot-server/src/main/java/org/apache/pinot/server/starter/helix/FreshnessBasedConsumptionStatusChecker.java:
##########
@@ -76,15 +90,36 @@ protected boolean isSegmentCaughtUp(String segmentName,
LLRealtimeSegmentDataMan
StreamPartitionMsgOffset currentOffset =
rtSegmentDataManager.getCurrentOffset();
StreamPartitionMsgOffset latestStreamOffset =
rtSegmentDataManager.fetchLatestStreamOffset(5000);
if (isOffsetCaughtUp(currentOffset, latestStreamOffset)) {
- _logger.info("Segment {} with freshness {}ms has not caught up within
min freshness {}."
+ _logger.info("Segment {} with freshness {}ms has not caught up within
min freshness {}. "
+ "But the current ingested offset is equal to the latest
available offset {}.", segmentName, freshnessMs,
_minFreshnessMs, currentOffset);
return true;
}
+ // If the earliest available kafka offset is later than the current
offset, then for the purposes of
+ // this check, we are caught up. This can happen if the segment has not
consumed data for some time,
+ // and that data has since been retentioned.
+ StreamPartitionMsgOffset earliestStreamOffset =
rtSegmentDataManager.fetchEarliestStreamOffset(5000);
Review Comment:
that's fair. and even if it doesn't for some reason, the idleness check
would catch it for us. I left the earliest offset for logging purposes.
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/IdleTimer.java:
##########
@@ -0,0 +1,85 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pinot.core.data.manager.realtime;
+
+/**
+ * The IdleTimer is responsible for keeping track of 2 different idle times:
+ * - The stream idle time which resets every time we remake the stream
consumer.
+ * This depends on the user configured "idle.timeout.millis" stream config.
+ * - the total idle time which only resets when we consume something.
+ *
+ * This is not a running timer. It only advances as long as we keep calling
"markIdle".
Review Comment:
I thought about this a lot, so tell me if my logic doesn't make sense.
- the user can request the idle time for the stream or the idle time for
consumption
- meaning you have to be able to answer `now - streamIdleStart` or now -
consumeIdleStart`
- so you have to track either the start time of both, or 1 start time + the
difference between both start times
Tracking both start times felt like easier logic to reason about. I had a
version of the code where we tracked the difference, and it was really
confusing.
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/IdleTimer.java:
##########
@@ -0,0 +1,85 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pinot.core.data.manager.realtime;
+
+/**
+ * The IdleTimer is responsible for keeping track of 2 different idle times:
+ * - The stream idle time which resets every time we remake the stream
consumer.
+ * This depends on the user configured "idle.timeout.millis" stream config.
+ * - the total idle time which only resets when we consume something.
+ *
+ * This is not a running timer. It only advances as long as we keep calling
"markIdle".
+ * This also makes it sightly inaccurate as we don't start counting idle time
until
+ * we've been idle for the first time. This is fine for our use case as we are
using this
+ * in a fast moving consumeLoop with iterations taking on the order of
milliseconds to seconds.
+ */
+public class IdleTimer {
+
+ private long _streamIdleStartTimeMs = 0;
Review Comment:
done
##########
pinot-server/src/main/java/org/apache/pinot/server/starter/helix/FreshnessBasedConsumptionStatusChecker.java:
##########
@@ -76,15 +90,36 @@ protected boolean isSegmentCaughtUp(String segmentName,
LLRealtimeSegmentDataMan
StreamPartitionMsgOffset currentOffset =
rtSegmentDataManager.getCurrentOffset();
StreamPartitionMsgOffset latestStreamOffset =
rtSegmentDataManager.fetchLatestStreamOffset(5000);
if (isOffsetCaughtUp(currentOffset, latestStreamOffset)) {
- _logger.info("Segment {} with freshness {}ms has not caught up within
min freshness {}."
+ _logger.info("Segment {} with freshness {}ms has not caught up within
min freshness {}. "
Review Comment:
Currently, not unless you change the code yourself. The initial version of
this checker effectively said `if it's a LongMessageOffset, check that
currentOffset <= latestOffset -1`. But there was contention about having that
special case in here. I worked around it internally by updating our ingestion
plugin to do the -1 whenever someone requested the latest offset criteria.
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/IdleTimer.java:
##########
@@ -0,0 +1,85 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pinot.core.data.manager.realtime;
+
+/**
+ * The IdleTimer is responsible for keeping track of 2 different idle times:
+ * - The stream idle time which resets every time we remake the stream
consumer.
Review Comment:
oof, bad bug on my part. When we recreate the stream consumer, we're
supposed to `markStreamNotIdle`. I see in our logs we're basically resetting
the stream consumer every time.
One is supposed to track how long the stream has been idle and reset every
time we recreate the consumer. The other continues to run even if the stream is
reset until we consume data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
