priyen opened a new pull request, #13249: URL: https://github.com/apache/pinot/pull/13249
One of the potential solutions to https://github.com/apache/pinot/issues/12477 was a broker api that returns table freshness info. This PR: - adds a new broker api `/debug/tableFreshness/{table}?timeoutMs=1000` that returns the minimum ingestion lag timestamp reported by servers across all the consuming segments. Underneath, the broker uses a new server api, `/tables/{table}/consumingSegmentsFreshnessInfo` that takes in a list of segments and returns the ingestion lag timestamp for the consuming segments among them - the broker was chosen because it is a component designed to be highly available, fast, & the information needed is right there: the routing map. - the api will route to 1 of the replica's based on whatever config is set, just as it does for any normal query Example usage from broker: ``` ❯ curl -i 'http://localhost:8000/debug/tableFreshness/airlineStats_REALTIME?timeoutMs=1000' HTTP/1.1 200 OK Content-Type: application/json Content-Length: 30 {"timestamp-ms":1716915412399}% ``` Example server usage: ``` ❯ curl -X POST -H "Content-Type: application/json" \ -d '["airlineStats__0__0__20240528T1641Z", "airlineStats__4__0__20240528T1641Z"]' \ http://localhost:7500/tables/airlineStats/consumingSegmentsFreshnessInfo {"airlineStats__4__0__20240528T1641Z":1716915549395,"airlineStats__0__0__20240528T1641Z":1716915552629}% ``` Possible improvements in future PRs: - collect this data periodically via a "FreshnessManager" of sorts -------- Instructions: 1. The PR has to be tagged with at least one of the following labels (*): 1. `feature` 2. `bugfix` 3. `performance` 4. `ui` 5. `backward-incompat` 6. `release-notes` (**) 2. Remove these instructions before publishing the PR. (*) Other labels to consider: - `testing` - `dependencies` - `docker` - `kubernetes` - `observability` - `security` - `code-style` - `extension-point` - `refactor` - `cleanup` (**) Use `release-notes` label for scenarios like: - New configuration options - Deprecation of configurations - Signature changes to public methods/interfaces - New plugins added or old plugins removed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
