gianm commented on a change in pull request #6916: Add master/data/query server concepts to docs/packaging URL: https://github.com/apache/incubator-druid/pull/6916#discussion_r251411280
########## File path: docs/content/design/index.md ########## @@ -93,47 +93,100 @@ Druid has a multi-process, distributed architecture that is designed to be cloud Druid process type can be configured and scaled independently, giving you maximum flexibility over your cluster. This design also provides enhanced fault tolerance: an outage of one component will not immediately affect other components. -Druid's process types are: +## Server Types -* [**Historical**](../design/historical.html) processes are the workhorses that handle storage and querying on "historical" data +A Druid cluster is organized into 3 server types: + +### Master server + +A master server manages data ingestion and storage: it is responsible for starting new ingestion jobs and coordinating availability of data on the "Data servers" described below. + +Within a master server, functionality is split between two processes, the coordinator and overlord. + +#### Coordinator process + +[**Coordinator**](../design/coordinator.html) processes watch over the Historical processes on the Data servers. They are responsible for assigning segments to specific servers, and for ensuring segments are well-balanced across Historicals. + +#### Overlord process + +[**Overlord**](../design/overlord.html) processes watch over the MiddleManager processes on the Data servers and are the controllers of data ingestion into Druid. They are responsible for assigning ingestion tasks to MiddleManagers and for coordinating segment publishing. + +### Query server + +A query server provides the endpoints that users and client applications interact with, routing queries to data servers or other query servers (and optionally proxied master server requests as well). + +Within a query server, functionality is split between two processes, the broker and router. + +#### Broker process + +[**Broker**](../design/broker.html) processes receive queries from external clients and forward those queries to Data servers. When Brokers receive results from those subqueries, they merge those results and return them to the +caller. End users typically query Brokers rather than querying Historicals or MiddleManagers processes on Data servers directly. + +#### Router process (optional) + +[**Router**](../development/router.html) processes are _optional_ processes that provide a unified API gateway in front of Druid Brokers, +Overlords, and Coordinators. They are optional since you can also simply contact the Druid Brokers, Overlords, and +Coordinators directly. + +### Data server + +A data server executes ingestion jobs and stores queryable data. + +Within a data server, functionality is split between two processes, the historical and middle manager. + +#### Historical process + +[**Historical**](../design/historical.html) processes are the workhorses that handle storage and querying on "historical" data (including any streaming data that has been in the system long enough to be committed). Historical processes download segments from deep storage and respond to queries about these segments. They don't accept writes. -* [**MiddleManager**](../design/middlemanager.html) processes handle ingestion of new data into the cluster. They are responsible + +#### Middle Manager process + +[**MiddleManager**](../design/middlemanager.html) processes handle ingestion of new data into the cluster. They are responsible for reading from external data sources and publishing new Druid segments. -* [**Broker**](../design/broker.html) processes receive queries from external clients and forward those queries to Historicals and -MiddleManagers. When Brokers receive results from those subqueries, they merge those results and return them to the -caller. End users typically query Brokers rather than querying Historicals or MiddleManagers directly. -* [**Coordinator**](../design/coordinator.html) processes watch over the Historical processes. They are responsible for assigning -segments to specific servers, and for ensuring segments are well-balanced across Historicals. -* [**Overlord**](../design/overlord.html) processes watch over the MiddleManager processes and are the controllers of data ingestion -into Druid. They are responsible for assigning ingestion tasks to MiddleManagers and for coordinating segment -publishing. -* [**Router**](../development/router.html) processes are _optional_ processes that provide a unified API gateway in front of Druid Brokers, -Overlords, and Coordinators. They are optional since you can also simply contact the Druid Brokers, Overlords, and -Coordinators directly. -Druid processes can be deployed individually (one per physical server, virtual server, or container) or can be colocated -on shared servers. One common colocation plan is a three-type plan: +### Colocation + +While Druid processes are typically organized and colocated based on the server types described above, it is possible to deploy the processes separately. -1. "Data" servers run Historical and MiddleManager processes. -2. "Query" servers run Broker and (optionally) Router processes. -3. "Master" servers run Coordinator and Overlord processes. They may run ZooKeeper as well. +Please see [process colocation](../operations/process-colocation.html) for details. -In addition to these process types, Druid also has three external dependencies. These are intended to be able to +### External dependencies + +In addition to these server and process types, Druid also has three external dependencies. These are intended to be able to Review comment: > In addition to its built-in process types Might be more clear. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
