Jackie-Jiang opened a new pull request, #19859: URL: https://github.com/apache/pinot/pull/19859
## Summary Follow-up to apache/pinot#19756. The remote access stats are now collected per query execution through `QueryThreadContext`, so every source of remote reads can report them, and the reads that bypass the buffers fetched for the query are reported on their own. - New `RemoteAccessStats` in `pinot-spi` accumulates the remote accesses and the bytes they read, with the direct remote accesses (reads that went to the remote storage tier directly instead of through the fetched buffers) counted both in the totals and on their own. `FetchContext` exposes its accumulator through `getRemoteAccessStats()` instead of the three forwarding methods added in apache/pinot#19756, which is not in any release yet. - `QueryThreadContext` carries the `RemoteAccessStats` of the execution the thread serves: a top-level `open` creates one, a task of a context-aware executor inherits the one of the submitting thread, and `ServerQueryExecutorV1Impl.execute` installs a fresh one per execution, because the two executions of a hybrid table are spawned from the same opchain context and must not share it. The static `addRemoteAccesses` / `addDirectRemoteAccesses` report into it from any thread of the execution and are no-ops without a context, e.g. when loading a segment. - Every owner of a released fetch reports it there: `InstanceResponseOperator.releaseAll`, and now also `BloomFilterSegmentPruner`, whose bloom filter fetches during pruning were not counted before. `ServerQueryExecutorV1Impl` is the only writer of the response metadata keys. - New query stats `numDirectRemoteAccesses` / `directRemoteAccessBytes`: `DataTable` metadata keys 47 and 48, `BrokerResponse` fields, `LeafOperator` and `BrokerResponseNativeV2` stat keys, and the time series carry-through, plumbed like the first pair. Each pair of metadata keys is emitted only when its count is not zero, and the new response fields are left out of the JSON when zero. A direct remote access is a round trip to the remote storage tier for a single read, orders of magnitude slower than a read through the fetched buffers, so a non-zero `numDirectRemoteAccesses` points at reads the fetch plan of the query did not cover. ## Compatibility - Older brokers ignore the new metadata keys, and both `StatKey` enums are appended last so the `StatMap` wire ordinals of existing keys do not move. - `FetchContext#addRemoteAccesses`, `getNumRemoteAccesses` and `getRemoteAccessBytes` are replaced by `getRemoteAccessStats()`; they were added in apache/pinot#19756 and have not been released. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
