Here it is ! https://issues.apache.org/jira/browse/KAFKA-6828
Thank you Enrico 2018-04-24 20:53 GMT+02:00 Ismael Juma <ism...@juma.me.uk>: > A JIRA ticket would be appreciated. :) > > Ismael > > On Sat, Apr 21, 2018 at 12:51 AM, Enrico Olivelli <eolive...@gmail.com> > wrote: > > > Il sab 21 apr 2018, 06:29 Ismael Juma <ism...@juma.me.uk> ha scritto: > > > > > Hi Enrico, > > > > > > It is a real problem because it causes indexes to take a lot more disk > > > space upfront. The sparsity is an important if people over partition, > for > > > example. > > > > > > > Got it. > > In production I saw no issue, maybe due to much availability of disk > space. > > > > Should I file a JIRA or you will do? > > Enrico > > > > > > > Ismael > > > > > > On Fri, Apr 20, 2018 at 12:41 PM, Enrico Olivelli <eolive...@gmail.com > > > > > wrote: > > > > > > > Il ven 20 apr 2018, 20:24 Ismael Juma <ism...@juma.me.uk> ha > scritto: > > > > > > > > > Hi Enrico, > > > > > > > > > > Coincidentally, I saw your message to nio-dev and followed up > there. > > > > > > > > > > > > > I think this is not a 'real' problem, you will notice the difference > > only > > > > if you have a lot of empty topics/partitions. > > > > In fact when you simply upgrade to jdk9/10 immediately you are > charged > > > with > > > > 10MB of disk space for each partition. > > > > In production I did not suffer this change because usually you do not > > > have > > > > empty partitions. > > > > In my test environments, where I had thousands of test empty > > partitions, > > > > disks filled up immediately and without any reason, it took time to > > > > understand the real cause. Broker will crash without much 'log' as > disk > > > is > > > > out of space. > > > > > > > > Maybe it would be useful to add some notice about this potential > > problem > > > > during the upgrade of the jdk. > > > > > > > > Hope that helps > > > > Enrico > > > > > > > > > > > > > Ismael > > > > > > > > > > On Fri, Apr 20, 2018 at 8:18 AM, Enrico Olivelli < > > eolive...@gmail.com> > > > > > wrote: > > > > > > > > > > > It is a deliberate change in JDK code > > > > > > > > > > > > Just for reference see this discussion on nio-dev list on > OpenJDK > > > > > > > > > http://mail.openjdk.java.net/pipermail/nio-dev/2018-April/005008.html > > > > > > > > > > > > > > > > > > see > > > > > > https://bugs.openjdk.java.net/browse/JDK-8168628 > > > > > > > > > > > > Cheers > > > > > > Enrico > > > > > > > > > > > > > > > > > > > > > > > > 2018-03-05 14:29 GMT+01:00 Enrico Olivelli <eolive...@gmail.com > >: > > > > > > > > > > > > > Workaround: > > > > > > > as these brokers are only for test environments I have set very > > > small > > > > > > > values for index file size, which affects pre-allocation > > > > > > > segment.index.bytes=65536 > > > > > > > log.index.size.max.bytes=65536 > > > > > > > > > > > > > > If anyone has some thought it will be very appreciated > > > > > > > Cheers > > > > > > > > > > > > > > Enrico > > > > > > > > > > > > > > > > > > > > > 2018-03-05 13:21 GMT+01:00 Enrico Olivelli < > eolive...@gmail.com > > >: > > > > > > > > > > > > > >> The only fact I have found is that with Java8 Kafka is > creating > > > > > "SPARSE" > > > > > > >> files and with Java9 this is not true anymore > > > > > > >> > > > > > > >> Enrico > > > > > > >> > > > > > > >> 2018-03-05 12:44 GMT+01:00 Enrico Olivelli < > eolive...@gmail.com > > >: > > > > > > >> > > > > > > >>> Hi, > > > > > > >>> This is a very strage case. I have a Kafka broker (part of a > > > > cluster > > > > > of > > > > > > >>> 3 brokers) which cannot start upgrading Java from Oracle JDK8 > > to > > > > > > Oracle JDK > > > > > > >>> 9.0.4. > > > > > > >>> > > > > > > >>> There are a lot of .index and .timeindex files taking 10MB, > > they > > > > are > > > > > > for > > > > > > >>> empty partiions. > > > > > > >>> > > > > > > >>> Running with Java 9 the server seems to rebuild these files > and > > > > each > > > > > > >>> file takes "really" 10MB. > > > > > > >>> The sum of all the files (calculated using du -sh) is 22GB > and > > > the > > > > > > >>> broker crashes during startup, disk becomes full and no log > > more > > > is > > > > > > >>> written. (I can send an extraction of the logs, but the tell > > only > > > > > > about > > > > > > >>> 'rebuilding index', the same as on Java 8) > > > > > > >>> > > > > > > >>> Reverting the same broker to Java 8 and removing the index > > files, > > > > the > > > > > > >>> broker rebuilds such files, each files take 10MB, but the > full > > > sum > > > > of > > > > > > sizes > > > > > > >>> (calculated using du -sh) is 38 MB ! > > > > > > >>> > > > > > > >>> I am running this broker on CentosOS 7 on EXT4 FS. > > > > > > >>> > > > > > > >>> I have upgraded the broker to latest and greatest Kafka 1.0.0 > > > (from > > > > > > >>> 0.10.2) without any success. > > > > > > >>> > > > > > > >>> All of the other testing clusters on CentOS7 (same SO > settings) > > > did > > > > > not > > > > > > >>> have any problem. > > > > > > >>> > > > > > > >>> The broker is given 4GB or RAM and 4G of Max Direct Memory > Size > > > (on > > > > > > >>> Java8 it works fine with 1GB and default max direct memory > > size) > > > > > > >>> > > > > > > >>> Which are the relevant configuration options ? > > > > > > >>> > > > > > > >>> Thoughts ? > > > > > > >>> > > > > > > >>> Thank you > > > > > > >>> > > > > > > >>> Enrico > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > -- Enrico Olivelli > > > > > > > > > -- > > > > > > -- Enrico Olivelli > > >
