gianm commented on issue #12262: URL: https://github.com/apache/druid/issues/12262#issuecomment-1055084660
@samarthjain Thanks for your comments. I totally agree about the importance of fault tolerance and good resource management. It'll be important as queries can get more complex and longer-running. > Resource fairness For resource management, I was thinking that for really big clusters, tiering (i.e. isolation of different workloads onto different servers) is going to be important. That way you can avoid sharing caches, etc between your interactive and your noninteractive workloads. I was thinking that in a dynamic cloudy sort of environment, you'd want the servers running interactive queries to run persistently, whereas you'd want the servers running multi-stage queries to be spun up on demand. In smaller or less dynamic clusters, where you're likely to run mixed workloads on the same servers, I agree we'll need to improve the current resource management system to make that work better. I have some fuzzy ideas about this but it really deserves to be fleshed out into its own proposal. I think it'll need to involve some degree of dynamism, like observing how much resource queries are actually using and adjusting their priorities appropriately. > Scaling and decoupling of shuffle servers I'd like to provide a smooth experience for both small clusters and large ones. I was thinking that for small clusters, or even single servers, you'd want the same servers doing all kinds of functionality, to keep management simple. For larger clusters I was thinking you'd want more isolation and independent scaling. I think we can make this possible with a server type that can handle a wide array of functionality, coupled with the ability to configure specific instances to only do one specific thing if you want independence. > A UI to show the query DAG and plan would be good to have. Yes!! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
