Hi there with regard to "> Can you please suggest me how to calculate this!"
No I can't thats why I asked, getting idea only way you going to figure this out is by building 6 servers, with say 12-16 vCPU's assigned and then run test/volume load and see what Prometheus shows you from a CPU/memory Utilisation. The 90Mb/s, how did you calculate that? G On Wed, Feb 19, 2020 at 7:29 AM Gowtham S <gowtham.co....@gmail.com> wrote: > Thanks, George and Alexandre for the reply. > > George, to answer your question > > > How did you get to 90MB/s - this is our expected throughput. > > > For the RAM size and CPU count? > > > Can you please suggest me how to calculate this! > > > > With regards, > Gowtham S, MCA > PH: 9597000673 > > > On Wed, 19 Feb 2020 at 09:27, George <george...@gmail.com> wrote: > > > Asking as I also need to do some of this math. > > > > How did you get to 90MB/s, the RAM size and CPU count ? > > You missing some dedicated SSD's for the ZK's > > Assume you doing licensing for 6 brokers... > > I'd suggest maybe rather go with 5 x 1TB drives, you provisioning 24TB > for > > a 23.4 TB sizing, cutting it very close. > > What industry are you in, with 3 ZK's you can only handle one failure, > > assuming this infrastructure is going to be very critical, you already > > planning 6 machines, rather look at using 5 ZK's. > > > > G > > > > > > Total Broker machine size = Message size per second * Retention > > period * > > > > Replication Factor > > > > = 90 MB/Sec * 86760 * 3 > > > > = 23425200 MB > > > > = 23.4252 Tb > > > > > > > > *Machine Configuration* > > > > 6 Brokers with 3 ZK > > > > > > > > *Kafka (per machine)* > > > > Disk Space - 2 * 2TB > > > > RAM - 128 GB > > > > CPU - 40 core > > > > > > On Tue, Feb 18, 2020 at 10:15 PM Alexandre Dupriez < > > alexandre.dupr...@gmail.com> wrote: > > > > > Hello Gowtham, > > > > > > You need to include the size of offset and time index files in your > > > calculations, plus potentially transaction indexes. > > > If you use default values, that means 10 MB each for every log > > > segment, which default size is itself 1 GB. > > > > > > Alexandre > > > > > > Le lun. 27 janv. 2020 à 08:51, Gowtham S <gowtham.co....@gmail.com> a > > > écrit : > > > > > > > > Hi, > > > > > > > > Please help/guide me to identify the size of the cluster. > > > > > > > > Thanks and Regards, > > > > Gowtham S. > > > > > > > > > > > > On Fri, 24 Jan 2020 at 14:31, Gowtham S <gowtham.co....@gmail.com> > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > We are in the process of deploying Kafka in our service. We need to > > > decide > > > > > the machine capacity plan, we arrived at the below formulae for > > > deriving > > > > > total machine capacity. > > > > > > > > > > Total Broker machine size = Message size per second * Retention > > > period * > > > > > Replication Factor > > > > > > > > > > Am I need to consider the topic, index files in the calculation? > > Please > > > > > help/guide me if i missing any param required in the formulae. > > > > > > > > > > Index file calculation (Reference > > > > > <https://issues.apache.org/jira/browse/KAFKA-3300>) > > > > > > > > > > Currently, the initial/max size of offset index file is configured > by > > > log.index.max.bytes. > > > > > This will be the offset index file size for the active log segment > > > until it > > > > > rolls out. > > > > > > > > > > Theoretically, we can calculate the upper bound of offset index > size > > > using > > > > > the following formula: > > > > > > > > > > log.segment.bytes / index.interval.bytes * 8 > > > > > > > > > > With the default setting the bytes needed for an offset index size > is > > > 1GB > > > > > / 4K * 8 = 2MB. And the default log.index.max.bytes is 10MB. > > > > > > > > > > > > > > > Retention Period = Retention period + ( > > log.retention.check.interval.ms > > > + > > > > > log.segment.delete.delay.ms ) / 1000 > > > > > = 86400 + (30000 + 6000)/1000 > > > > > = 86400 + 360 > > > > > = 86760 Seconds > > > > > > > > > > Total Broker machine size = Message size per second * Retention > > > period * > > > > > Replication Factor > > > > > = 90 MB/Sec * 86760 * 3 > > > > > = 23425200 MB > > > > > = 23.4252 Tb > > > > > > > > > > *Machine Configuration* > > > > > 6 Brokers with 3 ZK > > > > > > > > > > *Kafka (per machine)* > > > > > Disk Space - 2 * 2TB > > > > > RAM - 128 GB > > > > > CPU - 40 core > > > > > > > > > > Thanks and Regards > > > > > Gowtham.S > > > > > > > > > > > > > > -- > > You have the obligation to inform one honestly of the risk, and as a > person > > you are committed to educate yourself to the total risk in any activity! > > > > Once informed & totally aware of the risk, > > every fool has the right to kill or injure themselves as they see fit! > > > -- You have the obligation to inform one honestly of the risk, and as a person you are committed to educate yourself to the total risk in any activity! Once informed & totally aware of the risk, every fool has the right to kill or injure themselves as they see fit!
