If everything is up except dmaap then you can just delete the dev-dmaap helm and its config/artifacts.
https://wiki.onap.org/pages/viewpage.action?pageId=29787124 might have some pointers on how to cleanup a specific project like dev-dmaap But a 5 to 6 second timeout for dns inside kubernetes is abnormal so you need to find out why that is occurring. root@dev-robot-robot-d576f8448-d4m5l:/# time nslookup message-router.onap Server: 10.43.0.10 Address: 10.43.0.10#53 Non-authoritative answer: Name: message-router.onap.svc.cluster.local Address: 10.43.225.105 real 0m0.062s user 0m0.052s sys 0m0.004s Brian From: OBRIEN, FRANK MICHAEL Sent: Tuesday, February 05, 2019 2:41 PM To: [email protected]; FREEMAN, BRIAN D <[email protected]>; Signorelli Marco <[email protected]> Subject: RE: [onap-discuss] [Casablanca] Kafka's pod in crashloobackoff As Brian mentions – a full make build and purge will be required if you are experiencing leftover config/artifacts – as currently some pv’s are out of bounds and will require manual cleaning outside of the namespace delete – this includes wiping dockerdata-nfs as some config jobs will not rerun. It is also important to allow some pods to fully complete before attempting to use the system – as healthcheck does not necessarily verify DB functionality – just 200 readiness. The DNS service routing in k8s should not need to be modified out-of-band. Curious as to your deployment model – if running on multiple VMs make sure the ::1/0 and 0.0.0.0/0 open CIDR security groups are set – or step back and run everything co-located to verify your k8s cluster on a single VM. Also verify you are running the Rancher bootstrapped version of kubernetes that most of us run as the RI Verify the OS – Ubuntu 16 is usually ok, but RHEL 7.6 will require extra network and firewall config Verify everything is purged via https://wiki.onap.org/display/DW/Cloud+Native+Deployment#CloudNativeDeployment-RemoveaDeployment<https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.onap.org_display_DW_Cloud-2BNative-2BDeployment-23CloudNativeDeployment-2DRemoveaDeployment&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=ih19h8HuZMWR3wxkyNDOLDd3YjeQwsL_hjbhHeET9vY&s=-i0ELHRjWNRq-kohwqNnuanJqMWZmBU7yZGSmJ1n09Y&e=> kubectl delete namespace onap sudo helm delete --purge onap kubectl delete pv --all kubectl delete pvc --all kubectl delete secrets --all kubectl delete clusterrolebinding --all sudo rm -rf /dockerdata-nfs/onap-<pod> then rebuild https://git.onap.org/logging-analytics/tree/deploy/cd.sh#n189<https://urldefense.proofpoint.com/v2/url?u=https-3A__git.onap.org_logging-2Danalytics_tree_deploy_cd.sh-23n189&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=ih19h8HuZMWR3wxkyNDOLDd3YjeQwsL_hjbhHeET9vY&s=-SiCHv9DqtsBtwkkq2qR3e3Oj7JTLttYbxqOeW_TZP8&e=> cd oom/kubernetes/ sudo make clean sudo make all sudo make $ENVIRON deploy (use integrations’ cloud override yaml as well) one ---set at a time if you like empty first sudo helm deploy onap local/onap --namespace $ENVIRON -f $DISABLE_CHARTS_YAML –verbose dmaap and the rest in sequence sudo helm deploy onap local/onap --namespace $ENVIRON -f $DISABLE_CHARTS_YAML -f $DEV0_YAML $APPENDABLE_ENABLED_FLAGS --verbose DMAAP usually comes up in 3 min except for the occasional issue with DR in master 17:00:19 onap onap-dmaap-dbc-pg-0 1/1 Running 0 3h 10.42.229.159 a-ld0.fkpj53f4vfsulhfnozmmhljcya.cx.internal.cloudapp.net <none> 17:00:19 onap onap-dmaap-dbc-pg-1 1/1 Running 0 3h 10.42.37.56 a-ld0.fkpj53f4vfsulhfnozmmhljcya.cx.internal.cloudapp.net <none> 17:00:19 onap onap-dmaap-dbc-pgpool-d865b8fdd-62rtj 1/1 Running 0 3h 10.42.195.122 a-ld0.fkpj53f4vfsulhfnozmmhljcya.cx.internal.cloudapp.net <none> 17:00:19 onap onap-dmaap-dbc-pgpool-d865b8fdd-zn8x9 1/1 Running 0 3h 10.42.172.243 a-ld0.fkpj53f4vfsulhfnozmmhljcya.cx.internal.cloudapp.net <none> 17:00:19 onap onap-dmaap-dmaap-bus-controller-847cc8f99b-d62cb 1/1 Running 0 3h 10.42.56.233 a-ld0.fkpj53f4vfsulhfnozmmhljcya.cx.internal.cloudapp.net <none> 17:00:19 onap onap-dmaap-dmaap-dr-db-576f7968b8-rdbjx 1/1 Running 1 3h 10.42.189.181 a-ld0.fkpj53f4vfsulhfnozmmhljcya.cx.internal.cloudapp.net <none> 17:00:19 onap onap-dmaap-dmaap-dr-node-7647f9d6d8-qgvxp 0/1 Init:0/1 21 3h 10.42.134.128 a-ld0.fkpj53f4vfsulhfnozmmhljcya.cx.internal.cloudapp.net <none> 17:00:19 onap onap-dmaap-dmaap-dr-prov-f4d84869f-r7w7s 0/1 CrashLoopBackOff 45 3h 10.42.93.142 a-ld0.fkpj53f4vfsulhfnozmmhljcya.cx.internal.cloudapp.net <none> 17:00:19 onap onap-dmaap-message-router-0 1/1 Running 0 3h 10.42.5.86 a-ld0.fkpj53f4vfsulhfnozmmhljcya.cx.internal.cloudapp.net <none> 17:00:19 onap onap-dmaap-message-router-kafka-0 1/1 Running 1 3h 10.42.210.69 a-ld0.fkpj53f4vfsulhfnozmmhljcya.cx.internal.cloudapp.net <none> 17:00:19 onap onap-dmaap-message-router-kafka-1 1/1 Running 2 3h 10.42.80.212 a-ld0.fkpj53f4vfsulhfnozmmhljcya.cx.internal.cloudapp.net <none> 17:00:19 onap onap-dmaap-message-router-kafka-2 1/1 Running 1 3h 10.42.131.2 a-ld0.fkpj53f4vfsulhfnozmmhljcya.cx.internal.cloudapp.net <none> 17:00:19 onap onap-dmaap-message-router-zookeeper-0 1/1 Running 0 3h 10.42.133.199 a-ld0.fkpj53f4vfsulhfnozmmhljcya.cx.internal.cloudapp.net <none> 17:00:19 onap onap-dmaap-message-router-zookeeper-1 1/1 Running 0 3h 10.42.55.225 a-ld0.fkpj53f4vfsulhfnozmmhljcya.cx.internal.cloudapp.net <none> 17:00:19 onap onap-dmaap-message-router-zookeeper-2 1/1 Running 0 3h 10.42.45.19 a-ld0.fkpj53f4vfsulhfnozmmhljcya.cx.internal.cloudapp.net <none> There are some times per component that I have prototyped in the following https://wiki.onap.org/display/DW/Cloud+Native+Deployment#CloudNativeDeployment-DeploymentIntegritybasedonPodDependencies<https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.onap.org_display_DW_Cloud-2BNative-2BDeployment-23CloudNativeDeployment-2DDeploymentIntegritybasedonPodDependencies&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=ih19h8HuZMWR3wxkyNDOLDd3YjeQwsL_hjbhHeET9vY&s=94elrTMiudQWDrub41FeZKF7jBLfDoCj8f--9L6E7eg&e=> https://git.onap.org/logging-analytics/tree/deploy/cd.sh#n228<https://urldefense.proofpoint.com/v2/url?u=https-3A__git.onap.org_logging-2Danalytics_tree_deploy_cd.sh-23n228&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=ih19h8HuZMWR3wxkyNDOLDd3YjeQwsL_hjbhHeET9vY&s=O7JumSnX08eEDaGd__SDpGiJV6Vfosy4RtxnRiGV4U8&e=> DEPLOY_ORDER_POD_NAME_ARRAY=('consul msb dmaap dcaegen2 aaf robot aai esr multicloud oof so sdc sdnc vid policy portal log vfc uui vnfsdk appc clamp cli pomba vvp contrib sniro-emulator') DEPLOY_NUMBER_PODS_DESIRED_ARRAY=(4 5 11 11 13 1 15 2 6 17 10 12 11 2 8 6 3 18 2 5 5 5 1 11 11 3 1) Timing example is below (of course timing gets longer as more components use cpu/network/hd resources – and half of these can be done in parallel) Note that some management of problem containers is required, and not every container participates in healthcheck and dependency checking = so healthcheck is actually a <100% check of deployment status, do we wait for the whole cluster or a particular quorum for 3/5/7 ReplicaSets. http://jenkins.onap.info/job/oom-cd-master2-aws/124/consoleFull<https://urldefense.proofpoint.com/v2/url?u=http-3A__jenkins.onap.info_job_oom-2Dcd-2Dmaster2-2Daws_124_consoleFull&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=ih19h8HuZMWR3wxkyNDOLDd3YjeQwsL_hjbhHeET9vY&s=5Wnjtv1fJ4bubw_5a-gpmvp0n_bFHUN_vUUC1UOcLeU&e=> Minutes: component 1: consul 2: msb 3: dmaap 2: dcaegen2 (part 1) 32-70: dcaegen2 (part 2 cloudify) 4: aaf 4: robot 5: aai 2: esr 2: multicloud 20+: oof (wait for 15/16 is optional) 12: so 20+: sdc (20m timeout waiting on 7/11) 9: sdnc 2: vid 9: policy 12: portal 12: log 8: vfc 5: uui 10: vnfsdk 12: appc 8: clamp 3: cli 8: pomba 8: vvp 6: contrib 6: sniro-emulator /michael From: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> On Behalf Of Brian Sent: Tuesday, February 5, 2019 1:01 PM To: Signorelli Marco <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Subject: Re: [onap-discuss] [Casablanca] Kafka's pod in crashloobackoff Are you doing the make dmapp , make onap and then re-deploying dmaap (helm delete dev-dmaap –purge then helm deploy onap -f integration-override.yaml etc) ? The other method is to use the rancher GUI (port 8080) and modify the POD or Deployment yaml that contains that parameter. 5 to 6 seconds to resolve DNS is going to create a lot of other problems for you. Brian From: Signorelli Marco <[email protected]<mailto:[email protected]>> Sent: Tuesday, February 05, 2019 11:15 AM To: FREEMAN, BRIAN D <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Subject: Re: [onap-discuss] [Casablanca] Kafka's pod in crashloobackoff Hi Brian, the iptables rule is setted into all k8s nodes VM. We have tried to delete it, but after some seconds it is restored. We cannot change them. About the timeout, we have run the command "nslookup message-router-zookeeper" from others containers (i.e. SO container) and it takes about 5-6 seconds to perform the kube-dns query resolution. For this reason, we think that increasing the zookeeper connection timeoiut would solve this issue. But, where is the zookeeper config file to set the zookeeper.conmnect.timeout from 6000 to 60000 ? We have tried to change the file oom/kubernetes/dmaap/charts/message-router/resources/config/dmaap/MsgRtrApi.properties, but the tiomeout is always setted to 6000. Is is the right place? This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.amdocs.com_about_email-2Dterms-2Dof-2Dservice&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=ih19h8HuZMWR3wxkyNDOLDd3YjeQwsL_hjbhHeET9vY&s=4_4n7hvosgyhXKKrogjOLQTevFdfv7fIcJHG1PHkEn8&e=> -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#15355): https://lists.onap.org/g/onap-discuss/message/15355 Mute This Topic: https://lists.onap.org/mt/29665079/21656 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
