clintropolis commented on a change in pull request #7629: Add basic tuning guide, getting started page, updated clustering docs URL: https://github.com/apache/incubator-druid/pull/7629#discussion_r284055868
########## File path: docs/content/tutorials/cluster.md ########## @@ -22,56 +22,107 @@ title: "Clustering" ~ under the License. --> -# Clustering +# Setting up a Clustered Deployment Apache Druid (incubating) is designed to be deployed as a scalable, fault-tolerant cluster. In this document, we'll set up a simple cluster and discuss how it can be further configured to meet your needs. This simple cluster will feature: - - A single Master server to host the Coordinator and Overlord processes + - Two Master servers to host the Coordinator and Overlord processes - Scalable, fault-tolerant Data servers running Historical and MiddleManager processes - - Query servers, hosting Druid Broker processes + - A Query server, hosting the Druid Broker and Router processes -In production, we recommend deploying multiple Master servers with Coordinator and Overlord processes in a fault-tolerant configuration as well. +In production, we recommend deploying multiple Master servers with Coordinator and Overlord processes in a fault-tolerant configuration. ## Select hardware -### Master Server +### Fresh Deployment -The Coordinator and Overlord processes can be co-located on a single server that is responsible for handling the metadata and coordination needs of your cluster. -The equivalent of an AWS [m3.xlarge](https://aws.amazon.com/ec2/instance-types/#M3) is sufficient for most clusters. This -hardware offers: +If you do not have an existing Druid cluster, and wish to start running Druid in a clustered deployment, this guide provides an example clustered deployment with pre-made configurations. -- 4 vCPUs -- 15 GB RAM -- 80 GB SSD storage +#### Master Server -### Data Server +The Coordinator and Overlord processes are responsible for handling the metadata and coordination needs of your cluster. They can be colocated together on the same server. -Historicals and MiddleManagers can be colocated on a single server to handle the actual data in your cluster. These servers benefit greatly from CPU, RAM, -and SSDs. The equivalent of an AWS [r3.2xlarge](https://aws.amazon.com/ec2/instance-types/#r3) is a -good starting point. This hardware offers: +In this example, we will be deploying the equivalent of two AWS [m5.2xlarge](https://aws.amazon.com/ec2/instance-types/m5/) instances for fault-tolerance. +This hardware offers: - 8 vCPUs -- 61 GB RAM -- 160 GB SSD storage +- 31 GB RAM -### Query Server +Example Master server configurations that have been sized for this hardware can be found under `conf/druid/cluster/master`. + +#### Data Server + +Historicals and MiddleManagers can be colocated on the same server to handle the actual data in your cluster. These servers benefit greatly from CPU, RAM, +and SSDs. + +In this example, we will be deploying the equivalent of two AWS [i3.4xlarge](https://aws.amazon.com/ec2/instance-types/i3/) instances. + +This hardware offers: + +- 16 vCPUs +- 122 GB RAM +- 2 * 1.9TB SSD storage + +Example Data server configurations that have been sized for this hardware can be found under `conf/druid/cluster/data`. + +#### Query Server Druid Brokers accept queries and farm them out to the rest of the cluster. They also optionally maintain an -in-memory query cache. These servers benefit greatly from CPU and RAM, and can also be deployed on -the equivalent of an AWS [r3.2xlarge](https://aws.amazon.com/ec2/instance-types/#r3). This hardware -offers: +in-memory query cache. These servers benefit greatly from CPU and RAM. + +In this example, we will be deploying the equivalent of one AWS [m5.2xlarge](https://aws.amazon.com/ec2/instance-types/m5/) instance. +This hardware offers: - 8 vCPUs -- 61 GB RAM -- 160 GB SSD storage +- 31 GB RAM You can consider co-locating any open source UIs or query libraries on the same server that the Broker is running on. -Very large clusters should consider selecting larger servers. +Example Query server configurations that have been sized for this hardware can be found under `conf/druid/cluster/query`. + +#### Other Hardware Sizes + +The example cluster above is chosen as a single example out of many possible ways to size a Druid cluster. + +You can choose smaller/larger hardware or less/more servers for your specific needs and constraints. Review comment: I think it might also be worth mentioning here that advanced use cases can also choose not to co-locate services, to scale out the workload? It doesn't need to be very detailed, it just isn't incompatible with the following part that directs the reader to the basic tuning guide which describes how all the processes use resources independently, so it seems like there is no reason to leave it out since this section is about making choices. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
