Jay Kreps has commented on this before: https://issues.apache.org/jira/browse/KAFKA-1670?focusedCommentId=14161185&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14161185
Basically, you can always have more segment files. Having too large of segment files will significantly slow down the opening of files which is done whenever a broker comes online or has to recover. On Thu, May 14, 2015 at 3:10 AM, Mayuresh Gharat <gharatmayures...@gmail.com > wrote: > I suppose it is way log management works in kafka. > I am not sure the exact reason for this. Also the index files that are > constructed have a mapping of relative offset to the base offset of log > file to the real offset. The key value in index file is of the form > <Int,Int>. > > > Thanks, > > Mayuresh > > > On Wed, May 13, 2015 at 5:57 PM, Lance Laursen < > llaur...@rubiconproject.com> > wrote: > > > Hey folks, > > > > Any update on this? > > > > On Thu, Apr 30, 2015 at 5:34 PM, Lance Laursen < > > llaur...@rubiconproject.com> > > wrote: > > > > > Hey all, > > > > > > I am attempting to create a topic which uses 8GB log segment sizes, > like > > > so: > > > ./kafka-topics.sh --zookeeper localhost:2181 --create --topic > > perftest6p2r > > > --partitions 6 --replication-factor 2 --config max.message.bytes=655360 > > > --config segment.bytes=8589934592 > > > > > > And am getting the following error: > > > Error while executing topic command For input string: "8589934592" > > > java.lang.NumberFormatException: For input string: "8589934592" > > > at > > > > > > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > > > at java.lang.Integer.parseInt(Integer.java:583) > > > ... > > > ... > > > > > > Upon further testing with --alter topic, it would appear that > > > segment.bytes will not accept a value higher than 2,147,483,647, which > is > > > the upper limit for a signed 32bit int. This then restricts log segment > > > size to an upper limit of ~2GB. > > > > > > We run Kafka on hard drive dense machines, each with 10gbit uplinks. We > > > can set ulimits higher in order to deal with all the open file handles > > > (since Kafka keeps all log segment file handles open), but it would be > > > preferable to minimize this number, as well as minimize the amount of > log > > > segment rollover experienced at high traffic (ie: a rollover every 1-2 > > > seconds or so when saturating 10gbe). > > > > > > Is there a reason (performance or otherwise) that a 32 bit integer is > > used > > > rather than something larger? > > > > > > Thanks, > > > -Lance > > > > > > > > > > > > -- > > > > [image: elogo.png] > > > > Leading the Automation of Advertising > > > > LANCE LAURSEN | Systems Architect > > > > ••• (M) 310.903.0546 > > > > 12181 BLUFF CREEK DRIVE, 4TH FLOOR, PLAYA VISTA, CA 90094 > > > > RUBICONPROJECT.COM <http://www.rubiconproject.com/> | @RUBICONPROJECT > > <http://twitter.com/rubiconproject> > > > > > > -- > -Regards, > Mayuresh R. Gharat > (862) 250-7125 >
