Hello Milind, and thank you for your answer. We just found the solution to our problem. Apparently, some developers were sending bad messages with no key, so the log cleaner couldn’t wipe them properly and crashed at startup. The topics were set in compact mode. We have set the compact.policy to delete instead and after the cleaning, the problem disappeared, and we were back to a normal IO operation.
I consider this as a serious bug, as we’re running on SSDs and several weeks or less of this 100% usage can easily break this kind of hardware. Guillaume ______________________________________________ FENOLLAR Guillaume (SLN-EXT) Prestataire pour DSI/ESI Société Le Nickel - SLN Courriel : ext-gfenol...@eramet-sln.nc Site internet : www.sln.nc Page Facebook : LeNickel.SLN ______________________________________________ -----Message d'origine----- De : Milind Vaidya [mailto:kava...@gmail.com] Envoyé : mardi 13 juin 2017 05:17 À : users@kafka.apache.org Objet : Re: Kafka 0.10.1 cluster using 100% disk usage (reads) This sound exactly similar to what I experienced in the similar scenario. Can you please take a look at the File System time stamp of the actual log files on one of the broker hosts ? For me when I restarted the new brokers with version 0.10.0 it changes to current ts. Meaning if I have set 48 hrs of retention window, the file which is about to get deleted is stamped with latest ts so it will be scheduled to be deleted 48 hrs later. More times you restart the broker it will push deletion operation further into future. So this will lead to additional data retained on the disk. I posted the same question to which I was told it should be fixed in 0.10.1 but I observed same behaviour. Apparently the change from 0.8 to 0.10 is that, the file ts of the log chunk is not more taken into consideration but the time stamp with in the message as set by new protocol is used and it is backward compatible, meaning 0.8 logs files will be treated for their actual file TS. But does not look like in practical. Here are solutions I used. 1. Reduce the retention period if you can so the disk will still hold the logs till they expire in near future after which things become smooth 2. Use log.retention.bytes option for brokers to limit size of the logs retained. My request is that this should be clearly added to documentation of upgrade. Hope this helps. On Sun, Jun 11, 2017 at 7:46 PM, ext-gfenol...@eramet-sln.nc < ext-gfenol...@eramet-sln.nc> wrote: > Hello, > > Since we upgraded from Kafka 0.8 to Kafka0.10.1, our brokers are using > 100% of our disk capacity (100% util in iostat), with from 100MB/s to > 1000MB/s constant read stats. > > It’s been half a week now, and usage is not decreasing. Note that we > didn’t experience that before the upgrade. > > > > There’s nothing particularly helpful in the logs (except the fact that > we have corrupted index files at startup that kafka recreated > corrupted again, but it was already present in Kafka 0.8). > > It leads to a fairly high amount of CPU iowait, in all our environments. > > Even weirder thing is that the usage is not exactly the same on all > brokers. In first one, we have 100MB/s, 200MB/s for the 2nd one, and > 1000MB/s for the 3rd one, all of them having the same datastore, same > disk speed. > > > > With htop, I can see that only one Java thread is doing that, but I’m > not sure how to gather much info about it. > > > > Can you help ? > > Thanks, > > > > Guillaume > > > > > > *Guillaume FENOLLAR* > > Prestataire pour DSI/ESI > > Société Le Nickel - SLN > > Courriel : ext-gfenol...@eramet-sln.nc > > Site internet : www.sln.nc > > Page Facebook : LeNickel.SLN > > > > > > > ------------------------------ > CONFIDENTIALITE > L'information contenue dans ce courrier électronique et ses pièces > jointes est confidentielle, et est établie à l'intention exclusive de > ses destinataires. Dans le cas où ce message ne vous serait pas > destiné, nous vous remercions de bien vouloir en aviser immédiatement > l'émetteur et de procéder à sa suppression. Toutes copies, diffusions > ou accès non autorisés à ce message sont interdits à toutes personnes, > autre que le(s) destinataire(s). Un courrier électronique est > susceptible d’altération ou de falsification et peut entrainer des > pertes et/ou la destruction de données. Le Groupe ERAMET et/ou ses > filiales déclinent toute responsabilité en la matière. En conséquence > ce courrier électronique ainsi que ses pièces jointes sont utilisés à votre > propre risque. > > CONFIDENTIALITY > The information contained in this e-mail and any accompanying > documents is confidential or otherwise protected from disclosure. If > you are not the intended recipient, please immediately alert the > sender by reply e-mail and delete this message and any attachments. > Any copy, dissemination or unauthorized access of the contents of this > message by anyone other than the intended recipient is strictly > prohibited. E-mails may be susceptible to falsification or alteration > and cause data corruption and/or loss of data. ERAMET and/or any of > its subsidiaries decline any liability resulting from the consequences > thereof. Therefore, this e-mail and any attachments are used at your own risk. > ________________________________ CONFIDENTIALITE L'information contenue dans ce courrier électronique et ses pièces jointes est confidentielle, et est établie à l'intention exclusive de ses destinataires. Dans le cas où ce message ne vous serait pas destiné, nous vous remercions de bien vouloir en aviser immédiatement l'émetteur et de procéder à sa suppression. Toutes copies, diffusions ou accès non autorisés à ce message sont interdits à toutes personnes, autre que le(s) destinataire(s). Un courrier électronique est susceptible d’altération ou de falsification et peut entrainer des pertes et/ou la destruction de données. Le Groupe ERAMET et/ou ses filiales déclinent toute responsabilité en la matière. En conséquence ce courrier électronique ainsi que ses pièces jointes sont utilisés à votre propre risque. CONFIDENTIALITY The information contained in this e-mail and any accompanying documents is confidential or otherwise protected from disclosure. If you are not the intended recipient, please immediately alert the sender by reply e-mail and delete this message and any attachments. Any copy, dissemination or unauthorized access of the contents of this message by anyone other than the intended recipient is strictly prohibited. E-mails may be susceptible to falsification or alteration and cause data corruption and/or loss of data. ERAMET and/or any of its subsidiaries decline any liability resulting from the consequences thereof. Therefore, this e-mail and any attachments are used at your own risk.
