raghavyadav01 commented on code in PR #20023:
URL: https://github.com/apache/pinot/pull/20023#discussion_r4546311279


##########
pinot-core/src/main/java/org/apache/pinot/core/transport/InstanceRequestHandler.java:
##########
@@ -106,6 +126,15 @@ public InstanceRequestHandler(String instanceName, 
PinotConfiguration config, Qu
     } else {
       _executionContexts = null;
     }
+    _requestDeadlineClockSkewMs = 
config.getProperty(Server.CONFIG_OF_REQUEST_DEADLINE_CLOCK_SKEW_MS,
+        Server.DEFAULT_REQUEST_DEADLINE_CLOCK_SKEW_MS);
+    if (_requestDeadlineClockSkewMs >= 0) {
+      LOGGER.info("Requests arriving more than {}ms after the broker deadline 
are not executed",
+          _requestDeadlineClockSkewMs);
+    } else {
+      LOGGER.info("Requests arriving after the broker deadline are executed 
(check disabled)");
+    }
+    _queriesExpiredOnArrivalMeter = 
_serverMetrics.getMeteredValue(ServerMeter.QUERIES_EXPIRED_ON_ARRIVAL);

Review Comment:
   Do we need to cache this meter? 
`addMeteredGlobalValue(ServerMeter.QUERIES_EXPIRED_ON_ARRIVAL, 1)` should be 
enough.



##########
pinot-core/src/main/java/org/apache/pinot/core/transport/InstanceRequestHandler.java:
##########
@@ -141,6 +170,18 @@ protected void channelRead0(ChannelHandlerContext ctx, 
ByteBuf msg) {
       sendPingResponse(ctx, instanceRequest.getRequestId());
       return;
     }
+    try {
+      if (rejectIfExpired(instanceRequest, ctx, queryArrivalTimeMs)) {
+        return;
+      }
+    } catch (Exception e) {

Review Comment:
   Can we just ignore an invalid deadline and execute as before? This extra 
try/catch and error path seems unnecessary.



##########
pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java:
##########
@@ -1839,6 +1850,24 @@ public enum SortedSelectionMergeMode {
     public static final String DEFAULT_SERVER_QUERY_REGEX_CLASS = "JAVA_UTIL";
     public static final String CONFIG_OF_ENABLE_QUERY_CANCELLATION = 
"pinot.server.enable.query.cancellation";
     public static final boolean DEFAULT_ENABLE_QUERY_CANCELLATION = true;
+    /// Allowance for clock skew between a broker and this server when judging 
whether a request arrived after the
+    /// broker's deadline for it 
([Broker.Request.QueryOptionKey#REQUEST_DEADLINE_MS]). Such a request is 
answered
+    /// with a timeout error instead of being executed, but only when it 
arrives more than this many milliseconds
+    /// after the deadline. The broker stamps the deadline a few statements 
before it starts its own wait for the
+    /// servers, and network or channel-lock delays come after that wait has 
started, so with synchronized clocks a
+    /// request is rejected only once the broker has stopped waiting for it 
for longer than this allowance; neither a
+    /// slow hop nor a short `timeoutMs` changes that, since the query's 
timeout is not part of the comparison. The one
+    /// exposure is a server clock running ahead of the broker's by more than 
this allowance: it rejects requests that
+    /// arrive within the excess of their deadline, i.e. every request of a 
query whose remaining timeout is shorter
+    /// than the excess. A server clock behind the broker's only makes the 
check more lenient. The default, 10
+    /// seconds, is far above the skew between hosts that synchronize their 
clocks and far below the minutes a backlog
+    /// on a stalled connection can be old. Rejections show up as 
`QuerySchedulingTimeoutError` responses (error code
+    /// 240, the code of a scheduler queue timeout as well) whose message 
starts with `Expired on arrival:` and
+    /// carries the arrival time and the deadline, and in the server-wide 
`QUERIES_EXPIRED_ON_ARRIVAL` meter. A
+    /// negative value disables the check.
+    public static final String CONFIG_OF_REQUEST_DEADLINE_CLOCK_SKEW_MS =

Review Comment:
   Can we trim this javadoc (and the one on `REQUEST_DEADLINE_MS`) to 1-2 lines 
and keep the reasoning in the PR description?



##########
pinot-core/src/main/java/org/apache/pinot/core/transport/InstanceRequestHandler.java:
##########
@@ -183,6 +224,74 @@ private void sendPingResponse(ChannelHandlerContext ctx, 
long requestId) {
     }
   }
 
+  /// Answers a request that reached this server after the broker's deadline 
for it with a timeout error, without
+  /// executing it, and returns `true`; returns `false` when the request is to 
be executed as usual.
+  ///
+  /// The broker sends its deadline as the `requestDeadlineMs` query option, 
so a request from an old broker (no
+  /// option) is always executed. The deadline is compared against this 
server's clock, which is why it has to be more
+  /// than [Server#CONFIG_OF_REQUEST_DEADLINE_CLOCK_SKEW_MS] in the past 
before the request is dropped. Without this
+  /// check a request that spent minutes in the socket buffers of a stalled 
connection, and is delivered when the
+  /// connection recovers, is executed with a full timeout budget measured 
from its arrival, for a broker that gave up
+  /// on it long ago. The check reads the option straight from the 
deserialized request, before the query is compiled,
+  /// so that such a backlog costs no more than its deserialization on the 
event loop. The response is still sent: it
+  /// costs a few bytes and tells the broker side that the channel is alive, 
even though the broker no longer has a
+  /// query waiting for it. A malformed option propagates to the caller, which 
answers with the parse error under the
+  /// right request id.
+  private boolean rejectIfExpired(InstanceRequest instanceRequest, 
ChannelHandlerContext ctx,
+      long queryArrivalTimeMs) {
+    if (_requestDeadlineClockSkewMs < 0) {
+      return false;
+    }
+    Map<String, String> queryOptions = getQueryOptions(instanceRequest);
+    if (queryOptions == null) {
+      return false;
+    }
+    Long requestDeadlineMs = 
QueryOptionsUtils.getRequestDeadlineMs(queryOptions);
+    if (requestDeadlineMs == null) {
+      return false;
+    }
+    // Subtract rather than add, so that an absurdly far deadline still runs 
instead of overflowing into a rejection.
+    long lateByMs = queryArrivalTimeMs - requestDeadlineMs;
+    if (lateByMs <= _requestDeadlineClockSkewMs) {

Review Comment:
   Should the server also cap the execution deadline at `min(arrival + 
timeoutMs, deadline + skew)` instead of only rejecting? A request arriving just 
inside the allowance still gets a full budget.



##########
pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java:
##########
@@ -850,6 +850,17 @@ public static class Request {
 
       public static class QueryOptionKey {
         public static final String TIMEOUT_MS = "timeoutMs";
+        /// Broker-internal: the epoch time in milliseconds at which the 
broker stops waiting for this request, i.e.
+        /// the time it set [#TIMEOUT_MS] plus the remaining time it will wait 
(for a hybrid table the longer of the
+        /// two table types' remaining times). Set on every single-stage 
server request; a user-supplied value is
+        /// overwritten. A request that reaches the server's Netty query 
transport (`InstanceRequestHandler`) after
+        /// this time plus the server's clock-skew allowance
+        /// 
([CommonConstants.Server#CONFIG_OF_REQUEST_DEADLINE_CLOCK_SKEW_MS]) is answered 
with a timeout error
+        /// instead of being executed: a request can sit for minutes in the 
socket buffers of a stalled
+        /// broker-to-server connection, and once the connection recovers the 
server would otherwise run it with a
+        /// full timeout budget measured from its arrival, long after the 
broker gave up on it. Absent or non-positive
+        /// means no such check, which is what an old broker gets. The gRPC 
and multi-stage paths do not check it.
+        public static final String REQUEST_DEADLINE_MS = "requestDeadlineMs";

Review Comment:
   Why not add an optional `deadlineMs` field to `InstanceRequest`, like `ping` 
in #20022? Old servers ignore it, and we avoid a user-visible query option that 
gets silently overwritten.



##########
pinot-core/src/main/java/org/apache/pinot/core/transport/InstanceRequestHandler.java:
##########
@@ -183,6 +224,74 @@ private void sendPingResponse(ChannelHandlerContext ctx, 
long requestId) {
     }
   }
 
+  /// Answers a request that reached this server after the broker's deadline 
for it with a timeout error, without
+  /// executing it, and returns `true`; returns `false` when the request is to 
be executed as usual.
+  ///
+  /// The broker sends its deadline as the `requestDeadlineMs` query option, 
so a request from an old broker (no
+  /// option) is always executed. The deadline is compared against this 
server's clock, which is why it has to be more
+  /// than [Server#CONFIG_OF_REQUEST_DEADLINE_CLOCK_SKEW_MS] in the past 
before the request is dropped. Without this
+  /// check a request that spent minutes in the socket buffers of a stalled 
connection, and is delivered when the
+  /// connection recovers, is executed with a full timeout budget measured 
from its arrival, for a broker that gave up
+  /// on it long ago. The check reads the option straight from the 
deserialized request, before the query is compiled,
+  /// so that such a backlog costs no more than its deserialization on the 
event loop. The response is still sent: it
+  /// costs a few bytes and tells the broker side that the channel is alive, 
even though the broker no longer has a
+  /// query waiting for it. A malformed option propagates to the caller, which 
answers with the parse error under the
+  /// right request id.
+  private boolean rejectIfExpired(InstanceRequest instanceRequest, 
ChannelHandlerContext ctx,
+      long queryArrivalTimeMs) {
+    if (_requestDeadlineClockSkewMs < 0) {
+      return false;
+    }
+    Map<String, String> queryOptions = getQueryOptions(instanceRequest);
+    if (queryOptions == null) {
+      return false;
+    }
+    Long requestDeadlineMs = 
QueryOptionsUtils.getRequestDeadlineMs(queryOptions);
+    if (requestDeadlineMs == null) {
+      return false;
+    }
+    // Subtract rather than add, so that an absurdly far deadline still runs 
instead of overflowing into a rejection.
+    long lateByMs = queryArrivalTimeMs - requestDeadlineMs;
+    if (lateByMs <= _requestDeadlineClockSkewMs) {
+      return false;
+    }
+    long requestId = instanceRequest.getRequestId();
+    String tableNameWithType = getTableNameWithType(instanceRequest);
+    
_serverMetrics.addMeteredGlobalValue(ServerMeter.QUERIES_EXPIRED_ON_ARRIVAL, 1, 
_queriesExpiredOnArrivalMeter);
+    logExpiredRequest(instanceRequest, tableNameWithType, queryArrivalTimeMs, 
requestDeadlineMs, lateByMs);
+    try {
+      DataTable dataTable = DataTableBuilderFactory.getEmptyDataTable();
+      dataTable.getMetadata().put(MetadataKey.REQUEST_ID.getName(), 
Long.toString(requestId));
+      dataTable.addException(QueryErrorCode.QUERY_SCHEDULING_TIMEOUT,
+          String.format("%s request reached %s at %d, %dms after the broker 
deadline %d, and was not executed",
+              EXPIRED_ON_ARRIVAL_MESSAGE_PREFIX, _instanceName, 
queryArrivalTimeMs, lateByMs, requestDeadlineMs));
+      sendResponse(ctx, requestId, tableNameWithType, queryArrivalTimeMs, 
dataTable.toDataBuffer());
+    } catch (Exception e) {
+      LOGGER.error("Exception while sending the expired response for request: 
{}", requestId, e);
+    }
+    return true;
+  }
+
+  /// Logs at most one line per [#EXPIRED_REQUEST_LOG_INTERVAL_MS] of 
wall-clock time, carrying the number of expired
+  /// requests since the previous line and the details of the latest one. The 
handler is shared by every channel,
+  /// hence the atomics. The count is approximate: an increment that races the 
winning CAS lands in this line or the
+  /// next, never nowhere; the meter is exact.
+  private void logExpiredRequest(InstanceRequest instanceRequest, @Nullable 
String tableNameWithType,

Review Comment:
   Is the meter enough here? Could we drop the rate-limited logging (two 
atomics + CAS), or just log at debug?



##########
pinot-core/src/main/java/org/apache/pinot/core/transport/ServerChannels.java:
##########
@@ -329,6 +329,19 @@ public UnwritableChannelException(String message) {
     }
   }
 
+  /// The channel to one server, with the lock that serializes the writes into 
it.
+  ///
+  /// A request is handed to Netty with `writeAndFlush`. Once the flush has 
run on the event loop, which happens
+  /// right after the write task, the message sits in the channel's outbound 
buffer until the kernel accepts it, and
+  /// Netty offers no way to take a single message back out of that buffer: 
`ChannelOutboundBuffer` only discards an
+  /// entry whose promise was cancelled before the flush marked it 
uncancellable. So when the connection stalls, the
+  /// requests queued on it are delivered as soon as it recovers, including 
those whose query has long timed out at
+  /// the broker. Pulling them back here is not possible; what keeps them 
harmless is the server, which answers a
+  /// request that arrives after the broker's deadline (the 
`requestDeadlineMs` query option) with a timeout error
+  /// instead of executing it. On the broker side, the request write timeout 
([RequestWriteTimeoutHandler]) and the
+  /// write-buffer high watermark 
([NettyConfig#WRITE_BUFFER_HIGH_WATER_MARK_BYTES]) bound how much can queue up 
on a
+  /// stalled channel, and closing the channel, which that write timeout does, 
is the only way to drop what is
+  /// already queued.
   @ThreadSafe
   class ServerChannel {

Review Comment:
   This comment doesn't relate to code changed in this PR. Can we drop it or 
move it to the PR description?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to