xiangfu0 commented on PR #16556: URL: https://github.com/apache/pinot/pull/16556#issuecomment-3691544895
> > This PR introduces a query option: `ignoreMissingSegments` to ignore server-side missing segment errors and an optional broker dynamic config (`pinot.broker.query.ignore.missing.segments`) to set it automatically. > > Key changes: > > > > * New QueryOptionKey: ignoreMissingSegments > > * New Broker config: pinot.broker.query.ignore.missing.segments (default false) > > * SSE: broker sets option and filters SERVER_SEGMENT_MISSING in reduce > > * MSE: broker sets option and skips adding missing-segment exceptions > > * Server: respects option to not populate SERVER_SEGMENT_MISSING > > * Tests: QueryOptionsUtils parsing, BrokerReduceService behavior, server query exceptions test. > > > > This alleviates transient failures when a segment is deleted but the routing table is not yet updated. > > @Jackie-Jiang , do you know how transient these failures typically are? how did you learn/what motivated this PR? For large table with many segments(especially table with many kafka partitions and high retention frequency), the routing table rebuild may take longer time like couple minutes. Then queries to those servers will fail find segments. E.g. `pinot.server.instance.table.deleted.segments.cache.ttl.minutes` is default to 2minutes, but the caveat is it won't prevent server restart and local cache got cleaned. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
