Re: Migration from zookeeper to kraft not working

2023-05-18 Thread David Arthur 
Elena,

Did you provision a KRaft controller quorum before restarting the brokers?

If you don't mind, could you create a JIRA and attach the config files used
for the brokers before/after the migration along with the controller
configs? Please include the sequence of steps you took in the JIRA as well.

Here is our JIRA project:
https://issues.apache.org/jira/projects/KAFKA/issues, and general info on
filing issues
https://cwiki.apache.org/confluence/display/KAFKA/Reporting+Issues+in+Apache+Kafka

Thanks!
David



On Tue, May 16, 2023 at 2:54 AM Elena Batranu
 wrote:

> Hello! I have a problem with my kafka configuration (kafka 3.4). I'm
> trying to migrate from zookeeper to kraft. I have 3 brokers, on one of them
> was also the zookeeper. I want to restart my brokers one by one, without
> having downtime. I started with putting the configuration with also kraft
> and zookeeper, to do the migration gradually. In this step my nodes are up,
> but i have the following error in the logs from kraft.
> [2023-05-16 06:35:19,485] DEBUG [BrokerToControllerChannelManager broker=0
> name=quorum]: No controller provided, retrying after backoff
> (kafka.server.BrokerToControllerRequestThread)[2023-05-16 06:35:19,585]
> DEBUG [BrokerToControllerChannelManager broker=0 name=quorum]: Controller
> isn't cached, looking for local metadata changes
> (kafka.server.BrokerToControllerRequestThread)[2023-05-16 06:35:19,586]
> DEBUG [BrokerToControllerChannelManager broker=0 name=quorum]: No
> controller provided, retrying after backoff
> (kafka.server.BrokerToControllerRequestThread)[2023-05-16 06:35:19,624]
> INFO [RaftManager nodeId=0] Node 3002 disconnected.
> (org.apache.kafka.clients.NetworkClient)[2023-05-16 06:35:19,624] WARN
> [RaftManager nodeId=0] Connection to node 3002 (/192.168.25.172:9093)
> could not be established. Broker may not be available.
> (org.apache.kafka.clients.NetworkClient)[2023-05-16 06:35:19,642] INFO
> [RaftManager nodeId=0] Node 3001 disconnected.
> (org.apache.kafka.clients.NetworkClient)[2023-05-16 06:35:19,642] WARN
> [RaftManager nodeId=0] Connection to node 3001 (/192.168.25.232:9093)
> could not be established. Broker may not be available.
> (org.apache.kafka.clients.NetworkClient)[2023-05-16 06:35:19,643] INFO
> [RaftManager nodeId=0] Node 3000 disconnected.
> (org.apache.kafka.clients.NetworkClient)[2023-05-16 06:35:19,643] WARN
> [RaftManager nodeId=0] Connection to node 3000 (/192.168.25.146:9093)
> could not be established. Broker may not be available.
> (org.apache.kafka.clients.NetworkClient)
> I configured the controller on each broker, the file looks like this:
> # Licensed to the Apache Software Foundation (ASF) under one or more#
> contributor license agreements.  See the NOTICE file distributed with# this
> work for additional information regarding copyright ownership.# The ASF
> licenses this file to You under the Apache License, Version 2.0# (the
> "License"); you may not use this file except in compliance with# the
> License.  You may obtain a copy of the License at##
> http://www.apache.org/licenses/LICENSE-2.0## Unless required by
> applicable law or agreed to in writing, software# distributed under the
> License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR
> CONDITIONS OF ANY KIND, either express or implied.# See the License for the
> specific language governing permissions and# limitations under the License.
> ## This configuration file is intended for use in KRaft mode, where#
> Apache ZooKeeper is not present.  See config/kraft/README.md for details.#
> # Server Basics #
> # The role of this server. Setting this puts us in KRaft
> modeprocess.roles=controller
> # The node id associated with this instance's rolesnode.id=3000# The
> connect string for the controller
> quorum#controller.quorum.voters=3000@localhost
> :9093controller.quorum.voters=3000@192.168.25.146:9093,
> 3001@192.168.25.232:9093,
> 3002@192.168.25.172:9093#
> 
> Socket Server Settings #
> # The address the socket server listens on.# Note that only the controller
> listeners are allowed here when `process.roles=controller`, and this
> listener should be consistent with `controller.quorum.voters` value.#
>  FORMAT:# listeners = listener_name://host_name:port#   EXAMPLE:#
>  listeners = PLAINTEXT://your.host.name:9092listeners=CONTROLLER://:9093
> # A comma-separated list of the names of the listeners used by the
> controller.# This is required if running in KRaft
> mode.controller.listener.names=CONTROLLER
> # Maps listener names to security protocols, the default is for them to be
> the same. See the config documentation for more
> details#listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
> # The number

Migration from zookeeper to kraft not working

2023-05-16 Thread Elena Batranu 
Hello! I have a problem with my kafka configuration (kafka 3.4). I'm trying to 
migrate from zookeeper to kraft. I have 3 brokers, on one of them was also the 
zookeeper. I want to restart my brokers one by one, without having downtime. I 
started with putting the configuration with also kraft and zookeeper, to do the 
migration gradually. In this step my nodes are up, but i have the following 
error in the logs from kraft.
[2023-05-16 06:35:19,485] DEBUG [BrokerToControllerChannelManager broker=0 
name=quorum]: No controller provided, retrying after backoff 
(kafka.server.BrokerToControllerRequestThread)[2023-05-16 06:35:19,585] DEBUG 
[BrokerToControllerChannelManager broker=0 name=quorum]: Controller isn't 
cached, looking for local metadata changes 
(kafka.server.BrokerToControllerRequestThread)[2023-05-16 06:35:19,586] DEBUG 
[BrokerToControllerChannelManager broker=0 name=quorum]: No controller 
provided, retrying after backoff 
(kafka.server.BrokerToControllerRequestThread)[2023-05-16 06:35:19,624] INFO 
[RaftManager nodeId=0] Node 3002 disconnected. 
(org.apache.kafka.clients.NetworkClient)[2023-05-16 06:35:19,624] WARN 
[RaftManager nodeId=0] Connection to node 3002 (/192.168.25.172:9093) could not 
be established. Broker may not be available. 
(org.apache.kafka.clients.NetworkClient)[2023-05-16 06:35:19,642] INFO 
[RaftManager nodeId=0] Node 3001 disconnected. 
(org.apache.kafka.clients.NetworkClient)[2023-05-16 06:35:19,642] WARN 
[RaftManager nodeId=0] Connection to node 3001 (/192.168.25.232:9093) could not 
be established. Broker may not be available. 
(org.apache.kafka.clients.NetworkClient)[2023-05-16 06:35:19,643] INFO 
[RaftManager nodeId=0] Node 3000 disconnected. 
(org.apache.kafka.clients.NetworkClient)[2023-05-16 06:35:19,643] WARN 
[RaftManager nodeId=0] Connection to node 3000 (/192.168.25.146:9093) could not 
be established. Broker may not be available. 
(org.apache.kafka.clients.NetworkClient)
I configured the controller on each broker, the file looks like this:
# Licensed to the Apache Software Foundation (ASF) under one or more# 
contributor license agreements.  See the NOTICE file distributed with# this 
work for additional information regarding copyright ownership.# The ASF 
licenses this file to You under the Apache License, Version 2.0# (the 
"License"); you may not use this file except in compliance with# the License.  
You may obtain a copy of the License at##    
http://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law 
or agreed to in writing, software# distributed under the License is distributed 
on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 
express or implied.# See the License for the specific language governing 
permissions and# limitations under the License.
## This configuration file is intended for use in KRaft mode, where# Apache 
ZooKeeper is not present.  See config/kraft/README.md for details.#
# Server Basics #
# The role of this server. Setting this puts us in KRaft 
modeprocess.roles=controller
# The node id associated with this instance's rolesnode.id=3000# The connect 
string for the controller 
quorum#controller.quorum.voters=3000@localhost:9093controller.quorum.voters=3000@192.168.25.146:9093,3001@192.168.25.232:9093,3002@192.168.25.172:9093#
 Socket Server Settings #
# The address the socket server listens on.# Note that only the controller 
listeners are allowed here when `process.roles=controller`, and this listener 
should be consistent with `controller.quorum.voters` value.#   FORMAT:#     
listeners = listener_name://host_name:port#   EXAMPLE:#     listeners = 
PLAINTEXT://your.host.name:9092listeners=CONTROLLER://:9093
# A comma-separated list of the names of the listeners used by the controller.# 
This is required if running in KRaft mode.controller.listener.names=CONTROLLER
# Maps listener names to security protocols, the default is for them to be the 
same. See the config documentation for more 
details#listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
# The number of threads that the server uses for receiving requests from the 
network and sending responses to the networknum.network.threads=3
# The number of threads that the server uses for processing requests, which may 
include disk I/Onum.io.threads=8
# The send buffer (SO_SNDBUF) used by the socket 
serversocket.send.buffer.bytes=102400
# The receive buffer (SO_RCVBUF) used by the socket 
serversocket.receive.buffer.bytes=102400
# The maximum size of a request that the socket server will accept (protection 
against OOM)socket.request.max.bytes=104857600

# Log Basics #
# A comma separated list of directories under which to store log 
fileslog.dirs=/data/kraft-controller-logs
# The default number of log partitions