anshul98ks123 commented on code in PR #20137:
URL: https://github.com/apache/pinot/pull/20137#discussion_r4586225885
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/writer/StatelessRealtimeSegmentWriter.java:
##########
@@ -272,6 +273,13 @@ public void run() {
_consumer.start(_startOffset);
_logger.info("Created new consumer thread {} for {}", _consumerThread,
this);
_currentOffset = _startOffset;
+ // Decode retry state of the current message, and the ongoing decode
outage as in RealtimeSegmentDataManager
+ int numDecodeRetries = 0;
+ long decodeRetryBackoffMs = 0;
+ boolean inDecodeOutage = false;
Review Comment:
Moved into `DecodeRetryTracker` (pinot-segment-local) in f2647eb06b. It
holds the outage window, retries, backoff and reporting cadence, and both the
consumer and this writer use it, so the writer now also reminds once a minute.
Segment errors stay consumer-only, since the writer has none.
##########
pinot-spi/src/main/java/org/apache/pinot/spi/stream/StreamConfigProperties.java:
##########
@@ -127,6 +127,17 @@ private StreamConfigProperties() {
/// Config used to indicate whether the topic is a temporary topic for
offset auto reset backfilling
public static final String BACKFILL_TOPIC =
"realtime.segment.isBackfillTopic";
+ /// Cap, in milliseconds, of the backoff before a message whose decoder
threw [RetriableDecodeException] is fetched
+ /// again. The backoff starts at
[StreamConfig#INITIAL_DECODE_RETRY_BACKOFF_MILLIS] and doubles up to the cap.
+ /// Defaults to [StreamConfig#DEFAULT_DECODE_RETRY_MAX_BACKOFF_MILLIS] (10
seconds).
+ public static final String DECODE_RETRY_MAX_BACKOFF_MILLIS =
"decodeRetryMaxBackoffMillis";
+
+ /// How long, in milliseconds, retriable decode failures are retried during
an outage of a decoder dependency. An
+ /// outage lasts while a message is being retried or retriable failures keep
happening less than this apart; once it
+ /// has lasted this long, its failures are handled like any other decode
failure. No limit by default; `0` disables
+ /// retries.
+ public static final String DECODE_RETRY_MAX_DURATION_MILLIS =
"decodeRetryMaxDurationMillis";
Review Comment:
Done in f2647eb06b: the doc now says the limit is per segment, not a hard
cap, and that a new segment, pause, restart or rebalance starts a fresh window.
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java:
##########
@@ -285,6 +287,15 @@ public void deleteSegmentFile() {
private volatile long _numBytesDropped = 0;
private volatile int _consecutiveErrorCount = 0;
private volatile boolean _firstDecodeErrorLogged = false;
+ // Decode retry state, only accessed by the consumer thread
Review Comment:
Done in f2647eb06b: the field comment now says the Helix thread uses it when
catching up in `CONSUMING_TO_ONLINE`, after the consumer thread has stopped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]