snleee commented on PR #8911: URL: https://github.com/apache/pinot/pull/8911#issuecomment-1163303596
@mcvsubbu I discussed this with @PrachiKhobragade yesterday. 1. The server-side time pruner is covered if that's your concern. We compute the segment's min/max timestamp before we run the segment pruner. I think that this is covering most Pinot cases. (100% of use cases at LNKD fall into this bucket) 2. If the broker-side time pruner is used. This can be a concern. We have 2 ways to solve this: ``` 1. We check the query object and check if there's a time predicate. If there is, we don't emit a metric. 2. We check query metadata for all-consuming segments even if the broker prunes those. 3. Create a shared atomic long value on the server and cache the most recent freshness metrics and use that number when reporting. ``` All approaches that I can think of will add more CPU cycles to the performance-critical query path. I don't think that it's worth making the above changes to achieve the complete correctness of the freshness metric. IMO, we probably need to re-think our freshness metrics (computing the freshness metrics on the query time looks not the best approach). I would prefer that the metrics should be emitted periodically in the background, not interfering with the query execution path but this is more like the long-term change. I would prefer to add the comment that the metric would not behave as expected if the broker-side time-based pruner is being used and address the issue in the long-term re-do for the freshness metric. Please let me know if you have a better approach to this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
