Team, We have done our POC on DRUID cluster and are pretty much impressed by its performance. We had deployed it from Horton works stack and so are limited to use 0.10 version, but we want to take full advantage of the latest version 0.16
We are planning to deploy a Apache Druid cluster with the latest available version, but need some guidance. We have 10 nodes with below configurations: Cores (CPU): 80 Disk: 3984.7GB Memory: 503.36GB We also want to install ZK on 3 servers. The below is how we are planning to distribute, please check and suggest any changes. S. No. Components 1 Co-ord, overlord, zk1, broker1 2 Co-ord, overlord, zk2, broker2 3 zk3, Superset UI, Grafana , router (optional), mySQL server 4 MM, Historical 5 MM, Historical 6 MM, Historical 7 MM, Historical 8 MM, Historical 9 MM, Historical 10 MM, Historical We want to use Mysql for both druid and superset UI. Also, as per https://druid.apache.org/docs/latest/operations/basic-cluster-tuning.html We are collocating historical and Middle manager so will split nodes resources by 2 Implies each of it will get 40 cores and 250 GB Memory. HISTORICAL Heap : 0.5GB * number of CPU cores = 0.5 * 40 = 20 GB < 24 GB, so we will have heap set to 20 GB Considering total size of all loaded lookups will make it maximum of 24 GB. druid.processing.numThreads = 40 -1 = 39 druid.processing.buffer.sizeBytes= 500 MB druid.processing.numMergeBuffers (a 1:4 ratio of merge buffers to processing threads) implies 9 Direct Memory Sizing direct memory usage= (druid.processing.numThreads + druid.processing.numMergeBuffers + 1) * druid.processing.buffer.sizeBytes = 39+9+1=49*500 MB ~ 24 GB These are few memory settings which we are planning, can you please review our plan to distribute druid and supporting services and also the memory settings and suggest any changes. Please let me know if you need anything else. Thanks & Regards Lakshman Pervatoju Enterprise Data Services O: 404.269.2224 | C: 205.515.7866 CTECH A05-166D 6305 Peachtree Dunwoody Rd. Atlanta, GA 30328<webextlink://6305%20Peachtree%20Dunwoody%20Rd.%20Atlanta,%20GA%2030328> [cid:image001.png@01D480D8.ED44B020]
