Hi all, we are deploying Onap Casablanca release by following this guide:
https://docs.onap.org/en/casablanca/submodules/oom.git/docs/oom_quickstart_guide.html After we run the installation command ( helm deploy dev ...), helm returns that all components have been deployed. But when we run kubectl get pods -n onap, about 30 pods aren't in the running state, in particular: onap dev-dmaap-message-router-fb9f4bc7d-grhtt 0/1 Init:0/1 5 57m onap dev-dmaap-message-router-kafka-8c558bfc-l7854 0/1 CrashLoopBackOff 15 57m onap dev-dmaap-message-router-zookeeper-bbd595c46-hdvhw 1/1 Running 0 57m onap dev-sdnc-sdnc-dmaap-listener-84bffc54-sgntp 0/1 Init:0/1 1 2h The message-router's pod kafka has the following log error: root@sb4-rancher:~# kubectl logs -n onap dev-dmaap-message-router-kafka-5fbc897f48-hsxlp [2019-02-04 16:06:06,166] INFO Initiating client connection, connectString=message-router-zookeeper:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@4313f5bc<mailto:watcher=org.I0Itec.zkclient.ZkClient@4313f5bc> (org.apache.zookeeper.ZooKeeper) [2019-02-04 16:06:06,199] INFO Waiting for keeper state SyncConnected (org.I0Itec.zkclient.ZkClient) [2019-02-04 16:06:12,200] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread) waiting for kafka to be ready [2019-02-04 16:06:16,218] INFO Opening socket connection to server 10.42.8.38/10.42.8.38:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [2019-02-04 16:06:16,240] INFO Socket connection established to 10.42.8.38/10.42.8.38:2181, initiating session (org.apache.zookeeper.ClientCnxn) [2019-02-04 16:06:16,256] INFO Session establishment complete on server 10.42.8.38/10.42.8.38:2181, sessionid = 0x168b8e2e4280005, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn) [2019-02-04 16:06:16,263] INFO Session: 0x168b8e2e4280005 closed (org.apache.zookeeper.ZooKeeper) [2019-02-04 16:06:16,266] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server 'message-router-zookeeper:2181' with timeout of 6000 ms at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1233) at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:157) at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:131) at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:103) at kafka.utils.ZkUtils$.apply(ZkUtils.scala:85) at kafka.server.KafkaServer.initZk(KafkaServer.scala:341) at kafka.server.KafkaServer.startup(KafkaServer.scala:191) at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38) at kafka.Kafka$.main(Kafka.scala:65) at kafka.Kafka.main(Kafka.scala) [2019-02-04 16:06:16,266] INFO EventThread shut down for session: 0x168b8e2e4280005 (org.apache.zookeeper.ClientCnxn) [2019-02-04 16:06:16,271] INFO shutting down (kafka.server.KafkaServer) [2019-02-04 16:06:16,285] INFO shut down completed (kafka.server.KafkaServer) [2019-02-04 16:06:16,286] FATAL Exiting Kafka. (kafka.server.KafkaServerStartable) [2019-02-04 16:06:16,291] INFO shutting down (kafka.server.KafkaServer) while zookeeper log is : 2019-02-04 16:07:47,856 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /10.42.0.1:49336 2019-02-04 16:07:47,856 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203) at java.lang.Thread.run(Thread.java:745) 2019-02-04 16:07:47,857 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008] - Closed socket connection for client /10.42.0.1:49336 (no session established for client) 2019-02-04 16:07:48,902 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /10.42.0.1:49360 2019-02-04 16:07:48,903 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exceptionEndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket At org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) At org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203) at java.lang.Thread.run(Thread.java:745) 2019-02-04 16:07:48,903 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008] - Closed socket connection for client /10.42.0.1:49360 (no session established for client) and zookeeper is listening on 0.0.0.0:2181 that I believe it is correct. The iptables of each k8s node has the following rule for kafka: -A KUBE-SERVICES -d 10.43.140.245/32 -p tcp -m comment --comment "onap/message-router-kafka:message-router-kafka has no endpoints" -m tcp --dport 9092 -j REJECT --reject-with icmp-port-unreachable We have also tried to change the zookeeper.connection.timeout params into oom/kubernetes/dmaap/charts/message-router/resources/config/dmaap/MsgRtrApi.properties increasing it from 6000 ms to 60000 ms. We have reinstalled the dmaap component but zookeeper does not update this config value and so we have the same issue "Unable to connect to zookeeper server 'message-router-zookeeper:2181' with timeout of 6000 ms" Did I change the wrong configuration file for zoookeeper? Has anybody else experienced the same issue and was able to overcome it? Any help will be greatly appreciated Thanks in advance Best regards, Marco Signorelli Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione, Grazie. This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised. If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail, Thanks. Rispetta l'ambiente. Non stampare questa mail se non รจ necessario. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#15342): https://lists.onap.org/g/onap-discuss/message/15342 Mute This Topic: https://lists.onap.org/mt/29665079/21656 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
