Re: [DISCUSS] KIP-785 Automatic storage formatting

2021-12-13 Thread Colin McCabe
Hi Igor, Disk failures have always required manual intervention in Kafka. For example, if you are using a RAID array and a disk goes bad, you will need to remove the disk, insert a new one, and start the RAID rebuild process. Kafka can't help with this process since we don't operate at this

Re: [DISCUSS] KIP-785 Automatic storage formatting

2021-12-09 Thread Igor Soarez
Hi Colin, Thank you for your kind and thoughtful reply. Thank also you for clarifying why it is important to distinguish between disk problems and first boot for a log directory. I completely agree that loosing all metadata is a very serious issue and we should strive to make that as least

Re: [DISCUSS] KIP-785 Automatic storage formatting

2021-12-02 Thread Colin McCabe
Hi Igor, It is common for databases, filesystems, and other similar programs to require a formatting step before they are used. For example, postgres requires you to run initdb. Linux requires you to run mkfs before using a filesystem. Windows requires you to run "format c:/", or something

Re: [DISCUSS] KIP-785 Automatic storage formatting

2021-11-29 Thread Igor Soarez
Hi all, Bumping this thread as it’s been a while. Looking forward to any kind of feedback, pease take a look. I created a short PR with a possible implementation - https://github.com/apache/kafka/pull/11549 -- Igor > On 18 Oct 2021, at 15:11, Igor Soarez wrote: > > Hi all, > > I'd like

[DISCUSS] KIP-785 Automatic storage formatting

2021-10-18 Thread Igor Soarez
Hi all, I'd like to propose that we simplify the operation of KRaft servers a bit by removing the requirement to run kafka-storage.sh for new storage directories. Please take a look at the KIP and provide your feedback: