gianm commented on issue #12829: URL: https://github.com/apache/druid/issues/12829#issuecomment-1200495524
Would be great. A few thoughts: 1. I think we'd want the option to be settable in query context. If we do this we'll also get a server-wide setting for free, since you can set server-wide defaults for query context parameters. 2. I think we'd want the option to apply to published, unavailable, _non-realtime_ segments too. So that suggests the Broker should keep a list of published segments and verify that they are all available. This would also cover the handoff case, since segments being handed-off are published. 3. The case of segments that haven't been handed off yet is tricky: the list of segments that _should_ exist aren't currently registered anywhere central that the Broker would be able to get at. There's the pending segments table, but that also includes segments that _shouldn't_ exist (because they can be abandoned). Perhaps a solution here would involve ensuring the pending segments table only includes segments that should exist (i.e., delete abandoned records immediately). 4. We'd need to make sure we correctly handle the case of time-chunk replacement. In this case, while the new set of segments isn't yet fully available, we still want to ensure we recognize the old set as valid. We shouldn't throw errors just because the new set isn't pushed out yet. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
