yashmayya opened a new pull request, #19854:
URL: https://github.com/apache/pinot/pull/19854
## Problem
When a server terminates a multi-stage query, a stage on that server that
has not started yet sends nothing to its receivers. For example, the resource
accountant terminates a query to protect the heap. The broker then waits until
the query timeout and reports a timeout instead of the kill.
| Server terminates the query | Master | This PR |
|---|---|---|
| Before it schedules the stages | `250` "Timed out on stage 0 waiting for
data from child stage 1", after the full timeout | `245` with the kill message,
in less than 1 s |
| After it schedules the op chains, before they start | `250`, after the
full timeout | `245` with the kill message, in less than 1 s |
Two paths drop the op chain:
1. `OpChainSchedulerService` scheduled the op chain with
`ExecutorService.submit()`. The context-aware executor registers the returned
`Future` with the `QueryExecutionContext`, and `terminate()` cancels it. A
`FutureTask` cancelled before it starts never runs its task, so the
`FutureCallback` that cancels and closes the op chain never runs. The server
logs nothing.
2. `register()` rejected the op chain of a terminated query. `QueryRunner`
then closed the op chain, but `GrpcSendingMailbox.close()` sends nothing when
no stream is open yet. The error op chain from
`OpChainConverter.sendEarlyError` also went through `register()`, and the same
check rejected it ("Failed to propagate stage error via OpChainConverter").
On the same path, a pipeline breaker cancelled before it ran looked like it
collected no rows. Also, `sendEarlyError` replaced the error code with
`QUERY_EXECUTION`, so a forwarded kill lost its
`SERVER_RESOURCE_LIMIT_EXCEEDED` code.
## Change
- `register()` no longer rejects the op chain of a terminated query,
including a query cancelled through the cancelled-query cache. It cancels the
op chain with the termination error on the calling thread, without the
executor. This sends the error code and message to the receivers.
- The scheduler submits a small `OpChainTask` with `execute()` and registers
it with the query context. Cancelling it interrupts the op chain only while it
runs. A queued op chain still runs, fails before it does any work, and the
existing failure callback sends the termination error.
- `PipelineBreakerOperator.cancel()` keeps the cancel reason as its error
block.
- `OpChainConverter.sendEarlyError` keeps the error code of the error block.
- The Javadoc of `QueryThreadContext.contextAwareExecutorService` now says
that only `submit()` registers tasks for cancellation.
There is no change to configuration, SPI or wire format.
## Tests
- `QueryTerminatedBeforeStageStartsTest` (new) runs two servers and a broker
reducer. One server terminates the query:
- before it schedules the stages, for a group-by query and for a
dynamic-broadcast join (a pipeline breaker);
- after it schedules the op chains, but before they start.
`GatedExecutorServicePlugin`, a test executor, holds the op chains until the
test terminates the query.
The broker must report `SERVER_RESOURCE_LIMIT_EXCEEDED` with the kill
message within 10 s. The query timeout is 30 s.
- `OpChainSchedulerServiceTest`: an op chain that waits in the executor
queue when the query is terminated, op chains registered after a termination
and after a cancel, and no interrupt that leaks to the next task of the thread.
- `OpChainConverterDispatcherTest`: `sendEarlyError` keeps the error code.
All the new and changed tests fail on master. The `pinot-query-runtime`
tests, `QueryKillingIntegrationTest`, `WindowResourceAccountingTest` and
`MultiStageEngineIntegrationTest` pass.
Not in this PR: the executor can still drop an op chain that it accepted.
`HardLimitExecutor` checks the limit again when the task starts, and a task can
time out in the heap throttling queue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
