xiangfu0 opened a new pull request, #18709: URL: https://github.com/apache/pinot/pull/18709
## Summary - Add a broker-local drain manager plus `POST /drain` and `GET /drain/status` admin APIs. - Reject new broker query admissions during drain with `BrokerShuttingDown` while waiting for already accepted queries to finish. - Mark the broker `shutdownInProgress=true`, remove it from `brokerResource`, fail health checks while draining, and restore stale drain state on startup. ## User manual To gracefully drain the broker and stop it after accepted queries finish: ```bash curl -X POST "http://<broker-host>:<admin-port>/drain?timeoutMs=-1&shutdown=true" ``` `timeoutMs=-1` uses `pinot.broker.delayShutdownTimeMs`. Use `shutdown=false` when an external orchestrator should terminate the process after drain completes: ```bash curl -X POST "http://<broker-host>:<admin-port>/drain?timeoutMs=30000&shutdown=false" curl "http://<broker-host>:<admin-port>/drain/status" ``` While draining, the broker health check returns 503 and new valid queries sent to this broker fail with `BrokerShuttingDown` (`212`), so clients should retry another broker. ## Sample query ```sql SELECT COUNT(*) FROM myTable; ``` If this query is sent to a draining broker, the broker returns `BrokerShuttingDown` instead of accepting it for execution. ## Tests - `./mvnw -pl pinot-broker -am -Dtest=BrokerDrainManagerTest,BrokerRequestHandlerDelegateTest -Dsurefire.failIfNoSpecifiedTests=false test` - `./mvnw spotless:apply -pl pinot-broker,pinot-spi,pinot-controller` - `./mvnw checkstyle:check -pl pinot-broker,pinot-spi,pinot-controller` - `./mvnw license:format -pl pinot-broker,pinot-spi,pinot-controller` - `./mvnw license:check -pl pinot-broker,pinot-spi,pinot-controller` - `git diff --check upstream/master...HEAD` ## Review notes - Self-review found and fixed one shutdown lifecycle issue: `shutdown=true` now delays the asynchronous stop callback briefly so the drain API response is not raced by admin server shutdown. - No remaining self-review findings. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
