Going to pile on here, and also say take that blog post with a grain of salt :)
Look at your data size, and your desired degree of parallelism for consumers, and guide to that. Yes, 1000 partitions takes a while to create. But unless your data size is something like 1 TB per day or more, you probably don't need that. -Todd On Sat, Oct 26, 2019, 1:19 PM Edward Capriolo <edlinuxg...@gmail.com> wrote: > On Saturday, October 26, 2019, M. Manna <manme...@gmail.com> wrote: > > > You should also check out Becket Qin’s presentation on producer > performance > > tuning on YouTube. Both these items should give you all positives and > > negatives of having many/less portions. > > > > Thanks, > > > > On Sat, 26 Oct 2019 at 09:19, Manasvi Gupta <fatbab...@gmail.com> wrote: > > > > > > > > https://www.confluent.io/blog/how-choose-number-topics- > > partitions-kafka-cluster > > > > > > This might be useful for you. > > > > > > On Sat 26 Oct, 2019, 12:40 PM Jungyeol Lee, <jungy...@skelterlabs.com> > > > wrote: > > > > > > > I'm running a kafka cluster running only one broker with GCP > > > n1-standard-2 > > > > instance. > > > > I configured the number of partitions to 1000, but it is suffering > from > > > > running a broker on startup. It seems that it needs much more time > for > > > > topic creation. So, what I did is just set the smaller number of > > > partitions > > > > which is now 32. Everything works okay for now. Is this normal > behavior > > > or > > > > performance? Or, are there any guidelines on setting the number of > > > > partitions? > > > > > > > > Best, > > > > -- > > > > > > > > > > > To many partitions is too many files and folders and file systems and > longer failovers. > > > -- > Sorry this was sent from mobile. Will do less grammar and spell check than > usual. >
