gianm commented on issue #6993: [Proposal] Dynamic prioritization and laning URL: https://github.com/apache/incubator-druid/issues/6993#issuecomment-461180795 > Let me ask you this: if there was an omniscient function available in the broker that could exactly predict how light or heavy a query is going to be, would you remove the period threshold, or leave it in as an optional configuration option? It depends on how omniscient the function is :) Depending on how omniscient the function is, I still think there'd be value in overriding its decisions. Imagine you have a 'hot' and 'cold' tier and you really do want to apply a priority penalty to anything that'd hit the 'cold' tier, even if it looks like a cheap query. The reason might be because you have set up the cold tier to be storage-dense, and even seemingly-cheap queries might take a long time to run there due to needing to swap data in and out of disk. And you want those to go in the 'heavy' lane even if they look cheap. > If you leave it in, then this proposal is rather about letting users dynamically adjust priorities based on arbitrary criteria, with light or heavy just being one of them; and it's OK to sometimes classify a heavy query as high-priority and vice-versa. I think this is the best way to think about what I'm proposing. The main motivation for dynamic prioritization is to differentiate heavy vs light queries, but I don't think that's going to be the only way people will want to do it. Maybe people will want to adjust priorities for specific datasources, or queries from specific users, or whatever. > BTW, my suggestion of a duration threshold was only as a review comment—in our case, we fully control the query priority and don't have any need for dynamic prioritization, so laning is the only useful feature for us here. My selfish interest would even be to ship laning first and push dynamic prioritization to a follow-up PR. 😛 That's a good point, the two features don't need to be done at the same time, and probably shouldn't be. (Although there is value in designing them together to make sure they align with each other.)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
