vvivekiyer opened a new pull request, #13104:
URL: https://github.com/apache/pinot/pull/13104
Contains 2 fixes
**1. Adaptive Server Selection - race condition:**
A race condition between jetty threads and netty threads can result in
setting negative values for numInFlightRequests for servers. This can result in
that particular server being overloaded when compared to other.
The race condition is explained below
Let's say a query is routed to 2 servers S1 and S2. Say the query has a
timeout of 1s. The race condition timeline is as follows:
T1: Query is routed to S1 and S2. The ADSS stats will look as follows:
`S1 Stats = {
numInFlightRequests = 1
}
S2 Stats = {
numInFlightRequests = 1
}
`
T2: S1 responds with a results (dataTable). The ADSS stats [will be updated
](https://github.com/apache/pinot/blob/70bfd4185ca18eb64658c5a35813e3578f7c843d/pinot-core/src/main/java/org/apache/pinot/core/transport/AsyncQueryResponse.java#L158)
look as follows. Note that this update is by the netty thread that receives
the response.
`S1 Stats = {
numInFlightRequests = 0
}
S2 Stats = {
numInFlightRequests = 1
}
`
T3: Let's say the query timed out. The jetty thread will update the ADSS
stats for S2 as per
[code](https://github.com/apache/pinot/blob/70bfd4185ca18eb64658c5a35813e3578f7c843d/pinot-core/src/main/java/org/apache/pinot/core/transport/AsyncQueryResponse.java#L97)
`S1 Stats = {
numInFlightRequests = 0
}
S2 Stats = {
numInFlightRequests = 0
}
`
T4: Before the [jetty thread removes the QueryResponse object
](https://github.com/apache/pinot/blob/70bfd4185ca18eb64658c5a35813e3578f7c843d/pinot-core/src/main/java/org/apache/pinot/core/transport/AsyncQueryResponse.java#L102)for
the request, the server S2 could respond and the corresponding netty thread
would update the ADSS stats incorrectly to look as follows
`S1 Stats = {
numInFlightRequests = 0
}
S2 Stats = {
numInFlightRequests = -1
}
`
**2. Updates client error list to add a few more exceptions.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
