Re: Kafka disk recommendation for production cluster
I meant because there is no actual Raid Controller for running Raid on cloud, it is basically Software Based Raid [1]. [1] https://en.wikipedia.org/wiki/RAID#Software-based On Thu, Jul 12, 2018 at 6:29 PM M. Manna wrote: > From all cases I know so far (including RDBMS storage solution), RAID 10 > has been the choice. Also, the replication you are mentioning is the s/w > replication nothing to do with RAID 0 setup. > > > > On 11 July 2018 at 23:59, Ali Nazemian wrote: > > > Thanks. As this proposal is not available for the version of Kafka that > we > > are using, we should go with Raid option, but the question is how is it > > acceptable to use Software Raid0 on a cloud infrastructure? > > > > On Thu, 12 Jul. 2018, 08:02 M. Manna, wrote: > > > > > Dong Lin's KIP - > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > 112%3A+Handle+disk+failure+for+JBOD > > > > > > Should give you some ideas. > > > > > > On 11 July 2018 at 14:31, Ali Nazemian wrote: > > > > > > > Hi All, > > > > > > > > I was wondering what the disk recommendation is for Kafka cluster? Is > > it > > > > acceptable to use RAID0 in the case that replication is 3? We are > > running > > > > on a cloud infrastructure and disk failure is addressed at another > > level, > > > > so the chance of single disk failure would be very low. Besides, our > > > > version of Kafka does not fully support JBOD as even a single disk > > > failure > > > > can stop a Kafka broker, so we will not get the actual benefit of > using > > > > JBOD anyway. However, I am not quite sure how software raid acts in > > this > > > > situation as there is no option to use HW Raid on cloud. > > > > > > > > Regards, > > > > Ali > > > > > > > > > > -- A.Nazemian
Re: Kafka disk recommendation for production cluster
>From all cases I know so far (including RDBMS storage solution), RAID 10 has been the choice. Also, the replication you are mentioning is the s/w replication nothing to do with RAID 0 setup. On 11 July 2018 at 23:59, Ali Nazemian wrote: > Thanks. As this proposal is not available for the version of Kafka that we > are using, we should go with Raid option, but the question is how is it > acceptable to use Software Raid0 on a cloud infrastructure? > > On Thu, 12 Jul. 2018, 08:02 M. Manna, wrote: > > > Dong Lin's KIP - > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 112%3A+Handle+disk+failure+for+JBOD > > > > Should give you some ideas. > > > > On 11 July 2018 at 14:31, Ali Nazemian wrote: > > > > > Hi All, > > > > > > I was wondering what the disk recommendation is for Kafka cluster? Is > it > > > acceptable to use RAID0 in the case that replication is 3? We are > running > > > on a cloud infrastructure and disk failure is addressed at another > level, > > > so the chance of single disk failure would be very low. Besides, our > > > version of Kafka does not fully support JBOD as even a single disk > > failure > > > can stop a Kafka broker, so we will not get the actual benefit of using > > > JBOD anyway. However, I am not quite sure how software raid acts in > this > > > situation as there is no option to use HW Raid on cloud. > > > > > > Regards, > > > Ali > > > > > >
Re: Kafka disk recommendation for production cluster
Dong Lin's KIP - https://cwiki.apache.org/confluence/display/KAFKA/KIP-112%3A+Handle+disk+failure+for+JBOD Should give you some ideas. On 11 July 2018 at 14:31, Ali Nazemian wrote: > Hi All, > > I was wondering what the disk recommendation is for Kafka cluster? Is it > acceptable to use RAID0 in the case that replication is 3? We are running > on a cloud infrastructure and disk failure is addressed at another level, > so the chance of single disk failure would be very low. Besides, our > version of Kafka does not fully support JBOD as even a single disk failure > can stop a Kafka broker, so we will not get the actual benefit of using > JBOD anyway. However, I am not quite sure how software raid acts in this > situation as there is no option to use HW Raid on cloud. > > Regards, > Ali >
Kafka disk recommendation for production cluster
Hi All, I was wondering what the disk recommendation is for Kafka cluster? Is it acceptable to use RAID0 in the case that replication is 3? We are running on a cloud infrastructure and disk failure is addressed at another level, so the chance of single disk failure would be very low. Besides, our version of Kafka does not fully support JBOD as even a single disk failure can stop a Kafka broker, so we will not get the actual benefit of using JBOD anyway. However, I am not quite sure how software raid acts in this situation as there is no option to use HW Raid on cloud. Regards, Ali
