Try a POST to make sure you can write to message router. I doubt its connectivity.
If you are on master branch - try ./ete-k8s.sh onap healthmr to test a write/read to a test topic. (do it twice since the first time it creates a test topic and kafka doesnt forward the message till both the publisher and the subscriber have connected) Brian From: [email protected] <[email protected]> On Behalf Of Calamita Agostino Sent: Friday, March 01, 2019 4:30 AM To: [email protected] Subject: R: [onap-discuss] dmaap-message-router NodePort not recheable I tried to execute a wget command from sdc-be POD to message-router REST API and I see that dmaap-message-router is reacheable from sdc-be. This is the result: # kubectl exec -it dev-sdc-sdc-be-656bd64b9b-jh57x -n onap -- /bin/bash bash-4.4# wget "http://message-router:3904/topics";<https://urldefense.proofpoint.com/v2/url?u=http-3A__message-2Drouter-3A3904_topics-2522&d=DwQFAw&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=IEjWup2PaphGE0Khg3vrGEzEaMk8Vw36gzDqfaj5Hy0&s=E3zU3bhx_qFtVdCemSvrjg1neB63nEeWRcEhZCGNqTY&e=> Connecting to message-router:3904 (10.43.1.20:3904) topics 100% |*******************************| 131 0:00:00 ETA bash-4.4# cat topics {"topics": [ "__consumer_offsets", "champRawEvents", "SDC-DISTR-NOTIF-TOPIC-AUTO", "org.onap.dmaap.mr.PNF_READY" ]}bash-4.4# But audit.log of sdc-be, after "Distribution Service" action from Portal , says: 2019-03-01T08:32:07.986Z [qtp215145189-323354] INFO o.o.sdc.be.filters.BeServletFilter ResponseCode=500 InstanceUUID=null RequestId=d2f65e19-b07b-4266-8be2-f170aba42fb1 AlertSeverity=0 ElapsedTime=3 EndTimestamp=2019-03-01 08:32:07.986Z PartnerName=op0001 auditOn=true ServerFQDN=dev-sdc-sdc-be-656bd64b9b-jh57x StatusCode=ERROR TargetEntity=Distribution Engine is DOWN CustomField1=POST: http://sdc-be.onap:8080/sdc2/rest/v1/catalog/services/02e0c5a4-be65-4d09-9f1e-49a2dab0f865/distribution/PROD/activate <https://urldefense.proofpoint.com/v2/url?u=http-3A__sdc-2Dbe.onap-3A8080_sdc2_rest_v1_catalog_services_02e0c5a4-2Dbe65-2D4d09-2D9f1e-2D49a2dab0f865_distribution_PROD_activate&d=DwQFAw&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=IEjWup2PaphGE0Khg3vrGEzEaMk8Vw36gzDqfaj5Hy0&s=BmxuPjSo0lkXbgjGURLiUvhFZsKQ3N21_D7s1H861Mk&e=> timer=3 CustomField2=500 AuditBeginTimestamp=2019-03-01 08:32:07.983Z RemoteHost=10.42.194.84 ErrorCategory=ERROR ServerIPAddress=10.42.179.134 ServiceName=/v1/catalog/services/02e0c5a4-be65-4d09-9f1e-49a2dab0f865/distribution/PROD/activate ServiceInstanceId=null ClassName=org.openecomp.sdc.be.filters.BeServletFilter ResponseDescription=Internal Server Error ErrorCode=500 null In the same log file I found a lot of messages like this one: 2019-03-01T09:21:31.850Z [qtp215145189-399996] INFO o.o.sdc.be.filters.BeServletFilter AuditMessage=ACTION = "HttpAuthentication" URL = "v1/registerForDistribution" USER = "aai" AUTH_STATUS = "AUTH_SUCCESS" REALM = "ASDC" ResponseCode=500 InstanceUUID=aai-ml RequestId=7f01a5b2-ee38-42c9-b7a4-330f020a4134 AlertSeverity=0 ElapsedTime=169 EndTimestamp=2019-03-01 09:21:31.850Z PartnerName=Apache-HttpClient/4.5.6 (Java/1.8.0_171) auditOn=true ServerFQDN=dev-sdc-sdc-be-656bd64b9b-jh57x StatusCode=ERROR TargetEntity=registerInDistributionEngine CustomField1=POST: https://sdc-be.onap:8443/sdc/v1/registerForDistribution <https://urldefense.proofpoint.com/v2/url?u=https-3A__sdc-2Dbe.onap-3A8443_sdc_v1_registerForDistribution&d=DwQFAw&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=IEjWup2PaphGE0Khg3vrGEzEaMk8Vw36gzDqfaj5Hy0&s=bqsPBZIFYLIFhg5N8GOvmw1y4bhzY7y3C1rgEdE263M&e=> timer=169 CustomField2=500 AuditBeginTimestamp=2019-03-01 09:21:31.681Z RemoteHost=10.42.209.109 ErrorCategory=ERROR ServerIPAddress=10.42.179.134 ServiceName=/v1/registerForDistribution ServiceInstanceId=null ClassName=org.openecomp.sdc.be.filters.BeServletFilter ResponseDescription=Internal Server Error ErrorCode=500 ACTION = "HttpAuthentication" URL = "v1/registerForDistribution" USER = "aai" AUTH_STATUS = "AUTH_SUCCESS" REALM = "ASDC" Thanks. Da: [email protected]<mailto:[email protected]> [mailto:[email protected]] Per conto di Calamita Agostino Inviato: giovedì 28 febbraio 2019 16:13 A: [email protected]<mailto:[email protected]> Oggetto: [onap-discuss] dmaap-message-router NodePort not recheable Hi all, I have an issue releted to connectivity from sdc-be pod and dmaap-message-router. My installation is Casablanca 3.0.0 on 7 kubernetes VM cluster. All dmaap pods are up and running: dev-dmaap-dbc-pg-0 1/1 Running 0 1d 10.42.173.158 onapkm5 <none> dev-dmaap-dbc-pg-1 1/1 Running 0 1d 10.42.188.140 onapkm2 <none> dev-dmaap-dbc-pgpool-7b748d5894-mr2m9 1/1 Running 0 1d 10.42.237.193 onapkm3 <none> dev-dmaap-dbc-pgpool-7b748d5894-n6dks 1/1 Running 0 1d 10.42.192.244 onapkm2 <none> dev-dmaap-dmaap-bus-controller-6757c4c86-8rq5p 1/1 Running 0 1d 10.42.185.132 onapkm1 <none> dev-dmaap-dmaap-dr-db-bb4c67cfd-tm7td 1/1 Running 0 1d 10.42.152.59 onapkm1 <none> dev-dmaap-dmaap-dr-node-66c8749959-tpdtf 1/1 Running 0 1d 10.42.216.13 onapkm2 <none> dev-dmaap-dmaap-dr-prov-5c766b8d69-qzqn2 1/1 Running 0 1d 10.42.115.247 onapkm6 <none> dev-dmaap-message-router-fb9f4bc7d-5z52j 1/1 Running 0 6h 10.42.138.31 onapkm3 <none> dev-dmaap-message-router-kafka-5fbc897f48-4bpb6 1/1 Running 0 1d 10.42.78.141 onapkm4 <none> dev-dmaap-message-router-zookeeper-557954854-8d6p9 1/1 Running 0 1d 10.42.169.205 onapkm1 <none> but when I try to distribute a service, from SDC Portal, I got "Internal Server Error". SDC-BE log file traces: 2019-02-28T08:50:35.318Z [qtp215145189-159837] INFO o.o.sdc.be.filters.BeServletFilter ResponseCode=500 InstanceUUID=null RequestId=dab0fd50-b06e-4a65-b4a8-7d7edeae3e01 AlertSeverity=0 ElapsedTime=99 EndTimestamp=2019-02-28 08:50:35.318Z PartnerName=op0001 auditOn=true ServerFQDN=dev-sdc-sdc-be-656bd64b9b-jh57x StatusCode=ERROR TargetEntity=Distribution Engine is DOWN CustomField1=POST: http://sdc-be.onap:8080/sdc2/rest/v1/catalog/services/02e0c5a4-be65-4d09-9f1e-49a2dab0f865/distribution/PROD/activate<https://urldefense.proofpoint.com/v2/url?u=http-3A__sdc-2Dbe.onap-3A8080_sdc2_rest_v1_catalog_services_02e0c5a4-2Dbe65-2D4d09-2D9f1e-2D49a2dab0f865_distribution_PROD_activate&d=DwMFAw&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=IEjWup2PaphGE0Khg3vrGEzEaMk8Vw36gzDqfaj5Hy0&s=BmxuPjSo0lkXbgjGURLiUvhFZsKQ3N21_D7s1H861Mk&e=> timer=99 CustomField2=500 AuditBeginTimestamp=2019-02-28 08:50:35.219Z RemoteHost=10.42.194.84 ErrorCategory=ERROR ServerIPAddress=10.42.179.134 ServiceName=/v1/catalog/services/02e0c5a4-be65-4d09-9f1e-49a2dab0f865/distribution/PROD/activate ServiceInstanceId=null ClassName=org.openecomp.sdc.be.filters.BeServletFilter ResponseDescription=Internal Server Error ErrorCode=500 null Also SDC healthcheck reports that U-EB Cluster is DOWN. Inside SDC-BE POD, I tried to make a traceroute to "message-router-zookeeper" and to "message-router". This is the result ( the first is OK, the second one NOT OK ): bash-4.4# traceroute message-router-zookeeper traceroute to message-router-zookeeper (10.42.169.205), 30 hops max, 46 byte packets 1 10.42.7.46 (10.42.7.46) 0.213 ms 0.005 ms 0.005 ms 2 10.42.190.179 (10.42.190.179) 0.194 ms 0.145 ms 0.135 ms 3 10.42.169.205 (10.42.169.205) 0.461 ms 0.160 ms 0.134 ms bash-4.4# traceroute message-router traceroute to message-router (10.43.1.20), 30 hops max, 46 byte packets 1 10.42.0.1 (10.42.0.1) 0.009 ms 0.005 ms 0.005 ms 2 itpat1ng505.palermo.italtel.it (138.132.168.173) 0.344 ms 2.211 ms 1.910 ms <-- 138.132.168.X is VM public network 3 138.132.169.2 (138.132.169.2) 5.063 ms 3.859 ms 3.934 ms 4 * * * 5 * * * 6 * * * traceroute to message-router-kafka (10.43.148.154), 30 hops max, 46 byte packets 1 10.42.0.1 (10.42.0.1) 0.006 ms 0.005 ms 0.004 ms 2 itpat1ng505.palermo.italtel.it (138.132.168.173) 0.391 ms 0.337 ms 0.314 ms 3 138.132.169.2 (138.132.169.2) 0.803 ms 0.748 ms 0.807 ms 4 * * * 5 * * * 6 * * * It seems that I cannot reach NodePort or ClusterIP inside a POD. This is routing table inside POD: bash-4.4# netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 0.0.0.0 10.42.0.1 0.0.0.0 UG 0 0 0 eth0 10.42.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 What can I check on Kubernetes Cluster ? Thanks. Agostino. Internet Email Confidentiality Footer ** La presente comunicazione, con le informazioni in essa contenute e ogni documento o file allegato, e' rivolta unicamente alla/e persona/e cui e' indirizzata ed alle altre da questa autorizzata/e a riceverla. Se non siete i destinatari/autorizzati siete avvisati che qualsiasi azione, copia, comunicazione, divulgazione o simili basate sul contenuto di tali informazioni e' vietata e potrebbe essere contro la legge vigente (ad es. art. 616 C.P., D.Lgs n. 196/2003 Codice Privacy, Regolamento Europeo n. 679/2016/GDPR). Se avete ricevuto questa comunicazione per errore, vi preghiamo di darne immediata notizia al mittente e di distruggere il messaggio originale e ogni file allegato senza farne copia alcuna o riprodurne in alcun modo il contenuto. Al link seguente e' disponibile l'informativa Privacy: http://www.italtel.com/it/about/privacy/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.italtel.com_it_about_privacy_&d=DwMFAw&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=IEjWup2PaphGE0Khg3vrGEzEaMk8Vw36gzDqfaj5Hy0&s=AukdcpBODjjV9MdRBLbIlT9TPeryMvWd8zHSlfRqmK4&e=> ** This e-mail and its attachments are intended for the addressee(s) only and are confidential and/or may contain legally privileged information. If you have received this message by mistake or are not one of the addressees above, you may take no action based on it, and you may not copy or show it to anyone; please reply to this e-mail and point out the error which has occurred. Click here to read your privacy notice: http://www.italtel.com/it/about/privacy/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.italtel.com_it_about_privacy_&d=DwMFAw&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=IEjWup2PaphGE0Khg3vrGEzEaMk8Vw36gzDqfaj5Hy0&s=AukdcpBODjjV9MdRBLbIlT9TPeryMvWd8zHSlfRqmK4&e=> Internet Email Confidentiality Footer ** La presente comunicazione, con le informazioni in essa contenute e ogni documento o file allegato, e' rivolta unicamente alla/e persona/e cui e' indirizzata ed alle altre da questa autorizzata/e a riceverla. Se non siete i destinatari/autorizzati siete avvisati che qualsiasi azione, copia, comunicazione, divulgazione o simili basate sul contenuto di tali informazioni e' vietata e potrebbe essere contro la legge vigente (ad es. art. 616 C.P., D.Lgs n. 196/2003 Codice Privacy, Regolamento Europeo n. 679/2016/GDPR). Se avete ricevuto questa comunicazione per errore, vi preghiamo di darne immediata notizia al mittente e di distruggere il messaggio originale e ogni file allegato senza farne copia alcuna o riprodurne in alcun modo il contenuto. Al link seguente e' disponibile l'informativa Privacy: http://www.italtel.com/it/about/privacy/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.italtel.com_it_about_privacy_&d=DwQFAw&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=IEjWup2PaphGE0Khg3vrGEzEaMk8Vw36gzDqfaj5Hy0&s=AukdcpBODjjV9MdRBLbIlT9TPeryMvWd8zHSlfRqmK4&e=> ** This e-mail and its attachments are intended for the addressee(s) only and are confidential and/or may contain legally privileged information. If you have received this message by mistake or are not one of the addressees above, you may take no action based on it, and you may not copy or show it to anyone; please reply to this e-mail and point out the error which has occurred. Click here to read your privacy notice: http://www.italtel.com/it/about/privacy/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.italtel.com_it_about_privacy_&d=DwQFAw&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=IEjWup2PaphGE0Khg3vrGEzEaMk8Vw36gzDqfaj5Hy0&s=AukdcpBODjjV9MdRBLbIlT9TPeryMvWd8zHSlfRqmK4&e=> -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#15823): https://lists.onap.org/g/onap-discuss/message/15823 Mute This Topic: https://lists.onap.org/mt/30175085/21656 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
