> > 1100+ Kafka brokers, over 31,000 topics
am i reading this right, as this seems like an unhealthy ratio. a new broker for every 30 topics? so i must be misinterpreting the numbers... On Wed, Nov 28, 2018 at 10:18 PM Dominic Kim <style9...@gmail.com> wrote: > Dear Rodric. > > We`ve just started benchmarking of the scheduler with a variety of > scenarios. > I will come with more results. > > If you ask that because of scalability of Kafka, I have considered Kafka as > a very scalable solution. > > Linkedin announced that they are utilizing a huge Kafka cluster. > > https://events.static.linuxfound.org/sites/events/files/slides/Kafka%20At%20Scale.pdf > > As per the document, they are utilizing 1100+ Kafka brokers, over 31,000 > topics, 350,000+ partitions. > > And as per my understanding, only a limited number of brokers participate > in message delivery at a time for a topic. > Since partitions are the unit of parallelism, if we have 5 partitions in a > topic, only maximum 5 brokers can deliver the message at the same time. > (Actually, consumers fetch the messages instead brokers deliver them, but > anyway.) > > Since it's not required for all brokers to participate in all message > processing, when we need to scale out Kafka, we can just add more brokers. > > For example, if we have 5 topics with 5 partitions each in 5 brokers. > There will be 1 partition for each topic in each broker. As more and more > topics are created, there will be more partitions in one broker. > Let's say the maximum number of partitions that one broker properly handles > in time is 10, then if there are 10 topics, message processing will start > becoming slow. > Then we can just add more brokers. If we add 1 broker, it can handle 10 > partitions and it is equal to 2 topics in this example. > So whenever 2 new topics are created we can add a broker so that we can > keep the number of partitions that one broker holds same. > If we can keep the number of partitions in one broker at the same level, we > can guarantee the performance of Kafka as one broker will only participate > in the limited number of partition processing. > > In this scene, metadata sharing for partitions might be considered as a > bottleneck. > But once partitions are assigned to brokers, it does not change over > time(if we don't explicitly rebalance them). > Then consumers can just statically read the location of partitions and > fetch data from target brokers. > > Partitions are evenly distributed among brokers and each consumer will > access to a few of them only at a time, we can achieve massive scalability > with Kafka. > > Since this is based on my understanding about Kafka rather than real > experience, if I observe any issues during benchmarking I will share them > as well. > > > > Thanks > Regards > Dominic > > > > > > > 2018년 11월 29일 (목) 오전 9:54, Rodric Rabbah <rod...@gmail.com>님이 작성: > > > Thanks for sharing Dominic. I'll need to read this slowly to understand > it. > > It does look like a bit of the proposal you've discussed in the past > with 1 > > topic per action name (per namespace). Have you seen this scale in your > > production to thousands of topics? > > > > -r > > > > On Wed, Nov 28, 2018 at 5:59 AM Dominic Kim <style9...@gmail.com> wrote: > > > > > Dear whiskers. > > > > > > I`ve just submitted the proposal for the next version of Autonomous > > > Container Scheduler. > > > > > > > > > https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduler+v2 > > > > > > We(Naver), already implemented the scheduler based on that proposal and > > > it's almost finished. > > > It's not all-round scheduler but we could observe many performance > > benefits > > > in a few cases. > > > > > > We(Naver) are willing to contribute it and preparing for the > > contribution. > > > I hope the proposal would help you guys understand the implementation. > > > > > > To contribute the scheduler, it is required to contribute SPI for > > > ContainerPool and ContainerProxy first. > > > I will start working on this. > > > And I will describe the details about each component in each subpage. > > > Currently, they are blank. > > > > > > Please be noted again that the scheduler is not for all cases. > > > But it would be useful in many cases until new implementation based on > > the > > > future architecture of OW would be landed. > > > > > > > > > https://cwiki.apache.org/confluence/display/OPENWHISK/OpenWhisk+future+architecture?src=contextnavpagetreemode > > > > > > Any comments and feedback would be very grateful. > > > > > > Thanks > > > Regards > > > Dominic > > > > > >
