Re: Help with log.dirs please

Thanks Peter. Chris > On 23 Sep 2022, at 6:05 pm, Peter Bukowinski wrote: > > Great to hear, Chris! > > I would recommend not using zookeeper addresses in any of your kafka commands > as that has been deprecated for some time. Switch to using > `--bootstrap-server`. > > — > Peter > >> On

Re: Help with log.dirs please

Great to hear, Chris! I would recommend not using zookeeper addresses in any of your kafka commands as that has been deprecated for some time. Switch to using `--bootstrap-server`. — Peter > On Sep 22, 2022, at 1:08 AM, Chris Peart wrote: > > Hi Peter, > > > > Thanks for getting back to me

Re: Help with log.dirs please

Hi Peter, Thanks for getting back to me on this, i have done the following checks as suggested: Kafka version: 2.8.1 I can connect to all 4 brokers on port 9092 from my client system I can see this on 3 of the 4 brokers have the following message: DEBUG [Controller id=X] Broker Y has been

Re: Help with log.dirs please

Thanks Peter, I think we are using version 2.8.1 will verify this tomorrow. I can remember seeing the election message too in the server logs and will also confirm this in the morning too. I will answer the rest of the questions tomorrow. Many Thanks Chris > On 21 Sep 2022, at 10:24 pm, P

Re: Help with log.dirs please

Hmmm. Let’s start with some low level troubleshooting. What kafka version are you using? Can reach the ip:port of all broker listener addresses? I like to use `nc -vz ip port` to validate connectivity from my kafka client. Check the controller.log on all brokers. All but one of them should say

Re: Help with log.dirs please

Hi Peter, Kafka brokers are staying up and show when I query zookeeper brokers/ids Is the any other files that require deletion as it doesn’t make sense I have 0 brokers, when trying to create a topic. I get no errors when listing topics, but obviously don’t see any topics as I cannot create an

Re: Help with log.dirs please

Hi Chris, Are you sure kafka is starting up (and remains up) successfully? From the error, it seems like none of the brokers are online. — Peter > On Sep 21, 2022, at 4:02 AM, Chris Peart wrote: > > > > Hi Peter, > > I have made the changes i suggested: > > Stopping Kafka > > Deleting a

Re: Help with log.dirs please

Hi Peter, I have made the changes i suggested: Stopping Kafka Deleting all files and folders in /Kafka/data Changing the zookeeper setting to point to a different path in the zookeeper cluster Start Kafka I see the usual files in /kafka/data/ meta.properties and the offset files in all

Re: Help with log.dirs please

Thanks Peter, I’ll give this a go tomorrow and let you know how I get on. Many Thanks, Chris > On 20 Sep 2022, at 9:32 pm, Peter Bukowinski wrote: > > Hi Chris, > > If the configs are correct and the permissions on all the /data/X/kafka/data > directories are correct, then kafka should use

Re: Help with log.dirs please

Hi Chris, If the configs are correct and the permissions on all the /data/X/kafka/data directories are correct, then kafka should use all of the log dirs when creating topics. Remember that kafka will not automatically move any existing topic data when the cluster configs change. I’d test by cr

Re: Help with log.dirs please

Thinking about this, as this is not in production it might be easier just reset everything. Would it be something like: Stopping Kafka Deleting all files and folders in /Kafka/data Changing the zookeeper setting to point to a different path in the zookeeper cluster Start Kafka Some help on

Re: Help with log.dirs please

Hi Peter, I have checked the logs on all 4 brokers and could only see /data/1/data/kafka being used, log.dirs config in the logs showed all the disks but no errors. I managed to get the other 5 disks working by adding the path /data/[2-6]/data/kafka and setting the owner as kafka and resta

Re: Help with log.dirs please

Thanks Peter, I’ll check the logs next week and let you know my findings. Many Thanks Chris > On 16 Sep 2022, at 7:45 am, Peter Bukowinski wrote: > > The next thing I’d check is the broker logs. The parameters read from the > config should appear in the logs when kafka starts up. Search the

Re: Help with log.dirs please

The next thing I’d check is the broker logs. The parameters read from the config should appear in the logs when kafka starts up. Search the logs for ‘log.dirs’ and ensure the correct configs are loaded. -- Peter > On Sep 15, 2022, at 11:10 PM, Chris Peart wrote: > > Hi Peter, > > Thanks for

Re: Help with log.dirs please

Hi Peter, Thanks for your response, we have the following configuration: Partition count=4 Replication factor=3 All four brokers have topics-partitions in /data/1/kafka/data and are receiving data. Each server has 6 x 2TB disks for kaka data. Many Thanks, Chris > On 16 Sep 2022, at 1:56 am

Re: Help with log.dirs please

Hi Chris, Can you share the partition count and replication factor of your partitions? Also, do all four brokers contain topic-partition directories in /data/1/kafka/data or just a single broker? Depending on your topic config, it may be entirely normal that his has happened. — Peter Bukowins

Help with log.dirs please

Hi All, I have a 4 node kafka cluster running version 2.8.1, we have started pushing data to the cluster but can only see one disk being used. We had 6 disk configured as non-raid and 1 partition per disk, we have the following in fstab: /dev/mapper/VolGroup01-data/data/1xfsno