peferron commented on issue #6993: [Proposal] Dynamic prioritization and laning URL: https://github.com/apache/incubator-druid/issues/6993#issuecomment-461159982 > I was thinking a period threshold is likely to align with how people often set up historical tiers: a 'hot' tier for the latest 30 days and a 'cold' tier for older data. Got it. It makes sense if the goal is to be an alternative to tiering in certain situations, as you mentioned in your proposal. Some of the confusion, I think, is coming from the fact that the proposal matches "light" with "high priority", and "heavy" with "low priority", but then goes on to propose a period threshold that's intuitively not the best way to achieve that: heavy queries hitting recent data will remain high priority, while light queries hitting older data will be adjusted down to low priority. That's why the comments went on to suggest thresholds that are intuitively better correlated, such as duration or segment count (and we could keep going until we end up applying ML to it). Let me ask you this: if there was an omniscient function available in the broker that could exactly predict how light or heavy a query is going to be, would you remove the period threshold, or leave it in as an optional configuration option? If you remove it, then this proposal is indeed about classifying light vs heavy queries. If you leave it in, then this proposal is rather about letting users dynamically adjust priorities based on abitrary criteria, with light or heavy just being one of them; and it's OK to sometimes classify a heavy query as high-priority and vice-versa. I hope this doesn't sound like nitpicking—it would be helpful (at least to me) to clear this up. BTW, my suggestion of a duration threshold was only as a review comment—in our case, we fully control the query priority and don't have any need for dynamic prioritization, so laning is the only useful feature for us here. My selfish interest would even be to ship laning first and push dynamic prioritization to a follow-up PR. 😛
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
