It looks that problem was caused by wrong initialized zookeeper (git clone messageservice repo failure)
based on what Sunil wrote in DMAAP-1144 <https://jira.onap.org/browse/DMAAP-1144> this is required by SDC team, I think removing this dependency might bring some stability improvements. Is it planned already ? Anyone from SDC to comment ?? Thanks, Michal From: Michal Ptacek [mailto:[email protected]] Sent: Sunday, April 28, 2019 6:54 PM To: '[email protected]' <[email protected]>; '[email protected]' <[email protected]>; 'PLATANIA, MARCO (MARCO)' <[email protected]>; '[email protected]' <[email protected]>; '[email protected]' <[email protected]> Cc: 'Seshu m' <[email protected]>; 'huangxiangyu' <[email protected]>; 'Chenchuanyu' <[email protected]> Subject: [onap-discuss] basic service distribution failing for 3.0.2 ?? (subject changed) Hello again, This or similar problem is mentioned in recently updated tickets DMAAP-157, DMAAP-1007 … unfortunatelly I can’t derive fix from them. It looks that SDC-DISTR-NOTIF-TOPIC-AUTO don’t have owner so producer can’t subscribe to it ? Was this 3.0.2 release really passing integration tests ? The only test report I see is for 3.0.1 and couple of months old. https://wiki.onap.org/display/DW/Casablanca+Maintenance+Release+Integration+Testing+Status (Unfortunatelly 3.0.1 is also useless these days due to expired certificates) [root@tomas-infra helm_charts]# curl -X GET http://tomas-node0:30227/topics/ {"topics": [ "POA-RULE-VALIDATION", "AAI-EVENT", "unauthenticated.VES_MEASUREMENT_OUTPUT", "POA-AUDIT-RESULT", "__consumer_offsets", "org.onap.dmaap.mr.PNF_READY", "unauthenticated.SEC_HEARTBEAT_OUTPUT", "org.onap.dmaap.mr.PNF_REGISTRATION", "PDPD-CONFIGURATION", "SDC-DISTR-NOTIF-TOPIC-AUTO", "POA-AUDIT-INIT" ]}[root@tomas-infra helm_charts] curl -X GET http://tomas-node0:30227/topics/SDC-DISTR-NOTIF-TOPIC-AUTO/ { "owner": "", "readerAcl": { "enabled": true, "users": [] }, "name": "SDC-DISTR-NOTIF-TOPIC-AUTO", "description": "ASDC distribution notification topic", "writerAcl": { "enabled": true, "users": [] } }[root@tomas-infra helm_charts]#curl -X PUT http://tomas-node0:30227/topics/SDC-DISTR-NOTIF-TOPIC-AUTO/producers/iPIxkpAMI8qTcQj8 org.apache.cxf.interceptor.Fault at org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:162) at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:128) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:192) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:103) at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59) at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96) at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) at org.apache.camel.component.cxf.cxfbean.CxfBeanDestination.process(CxfBeanDestination.java:83) at org.apache.camel.impl.ProcessorEndpoint.onExchange(ProcessorEndpoint.java:103) at org.apache.camel.impl.ProcessorEndpoint$1.process(ProcessorEndpoint.java:71) at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61) at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:148) at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:97) at org.apache.camel.http.common.CamelServlet.doService(CamelServlet.java:208) at org.apache.camel.http.common.CamelServlet.service(CamelServlet.java:78) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655) at ajsc.filter.PassthruFilter.doFilter(PassthruFilter.java:26) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at com.att.ajsc.csi.writeablerequestfilter.WriteableRequestFilter.doFilter(WriteableRequestFilter.java:41) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.onap.dmaap.util.DMaaPAuthFilter.doFilter(DMaaPAuthFilter.java:82) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:531) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NullPointerException at com.att.nsa.security.NsaAclUtils.updateAcl(NsaAclUtils.java:74) at org.onap.dmaap.dmf.mr.beans.DMaaPKafkaMetaBroker$KafkaTopic.updateAcl(DMaaPKafkaMetaBroker.java:446) at org.onap.dmaap.dmf.mr.beans.DMaaPKafkaMetaBroker$KafkaTopic.permitWritesFromUser(DMaaPKafkaMetaBroker.java:422) at org.onap.dmaap.dmf.mr.service.impl.TopicServiceImpl.permitPublisherForTopic(TopicServiceImpl.java:529) at org.onap.dmaap.service.TopicRestService.permitPublisherForTopic(TopicRestService.java:478) at sun.reflect.GeneratedMethodAccessor131.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179) at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) ... 60 more Please advise, Michal From: Michal Ptacek [mailto:[email protected]] Sent: Friday, April 26, 2019 5:01 PM To: '[email protected]' <[email protected] <mailto:[email protected]> >; '[email protected]' <[email protected] <mailto:[email protected]> >; 'PLATANIA, MARCO (MARCO)' <[email protected] <mailto:[email protected]> > Cc: 'Seshu m' <[email protected] <mailto:[email protected]> >; 'huangxiangyu' <[email protected] <mailto:[email protected]> >; 'Chenchuanyu' <[email protected] <mailto:[email protected]> > Subject: RE: [onap-discuss] Error code POL5000 : exception on distributing service Hi Guys, I have the same problem and even recreating message-router component doesn’t work for me (tried several times) What i found so far is that SDC is in "healthCheckComponent": "DE", "healthCheckStatus": "DOWN", "description": "U-EB cluster is not available" Because it’s failing to register to topic: 2019-04-26T14:08:03.926Z||pool-78-thread-1|registerToTopic||registerToTopic||ERROR|500|PUT http://message-router.onap:3904/topics/SDC-DISTR-NOTIF-TOPIC-AUTO/producers/iPIxkpAMI8qTcQj8 (as i PIxkpAMI8qTcQj8) ...| 2019-04-26T14:08:03.970Z||pool-76-thread-1|registerToTopic||registerToTopic||ERROR|500| --> HTTP/1.1 500 Server Error| 2019-04-26T14:08:03.972Z||pool-76-thread-1|registerToTopic||registerToTopic||ERROR|500|Error occured during access to U-EB Server. Operation: register to topic as producer| Topic is available in message-router But message-router logs are displaying just following INFO messages: ""2019-04-26 14:27:09,761 [qtp1061804750-94] INFO org.onap.dmaap.service.TopicRestService - Granting write access to producer [iPIxkpAMI8qTcQj8] for topic SDC-DISTR-NOTIF-TOPIC-AUTO ""2019-04-26 14:27:09,765 [qtp1061804750-94] INFO org.onap.dmaap.dmf.mr.service.impl.TopicServiceImpl - Granting write access to producer [iPIxkpAMI8qTcQj8] for topic SDC-DISTR-NOTIF-TOPIC-AUTO ""2019-04-26 14:27:09,782 [qtp1061804750-96] INFO org.onap.dmaap.dmf.mr.security.impl.DMaaPOriginalUebAuthenticator - AUTH-LOG(10.42.207.81): No such API key iPIxkpAMI8qTcQj8 ""2019-04-26 14:27:09,783 [qtp1061804750-96] INFO org.onap.dmaap.dmf.mr.security.impl.DMaaPOriginalUebAuthenticator - AUTH-LOG(10.42.207.81): No such API key iPIxkpAMI8qTcQj8 ""2019-04-26 14:27:09,788 [qtp1061804750-94] INFO org.onap.dmaap.dmf.mr.security.impl.DMaaPOriginalUebAuthenticator - AUTH-LOG(10.42.207.81): No such API key iPIxkpAMI8qTcQj8 ""2019-04-26 14:27:09,790 [qtp1061804750-94] INFO org.onap.dmaap.dmf.mr.security.impl.DMaaPOriginalUebAuthenticator - AUTH-LOG(10.42.207.81): No such API key iPIxkpAMI8qTcQj8 ""2019-04-26 14:27:09,919 [qtp1061804750-12] INFO org.onap.dmaap.util.DMaaPAuthFilter - inside servlet filter Cambria Auth Headers checking before doing other Authentication It looks like some AAF related authentication problem ….. Any hint really appreciated, Thanks, Michal PS: I am testing 3.0.2 release (latest casablanca) Please note: for sdc-be to come-up properly I need to change dir ownership from root to jetty user, otherwise it was complaining with ERROR in ch.qos.logback.core.rolling.RollingFileAppender[METRICS_ROLLING] - Failed to create parent directories for [/var/lib/jetty/logs/SDC/SDC-BE/metrics.log] From: [email protected] <mailto:[email protected]> [mailto:[email protected]] On Behalf Of subhash kumar singh Sent: Tuesday, April 23, 2019 9:07 AM To: PLATANIA, MARCO (MARCO) <[email protected] <mailto:[email protected]> >; [email protected] <mailto:[email protected]> Cc: Seshu m <[email protected] <mailto:[email protected]> >; huangxiangyu <[email protected] <mailto:[email protected]> >; Chenchuanyu <[email protected] <mailto:[email protected]> > Subject: Re: [onap-discuss] Error code POL5000 : exception on distributing service Hello Marco, Thank you for the response. It worked for me :) -- Regards, Subhash Kumar Singh From: PLATANIA, MARCO (MARCO) [mailto:[email protected]] Sent: Monday, April 22, 2019 8:49 PM To: [email protected] <mailto:[email protected]> ; Subhash Kumar Singh Cc: Seshu m; huangxiangyu; Chenchuanyu Subject: Re: [onap-discuss] Error code POL5000 : exception on distributing service DMaaP components have to come up in a specific order: Zookeeper, Kafka, and finally message router. Delete DMaaP files in NFS when you rebuild it (/dockerdata-nfs/dev-dmaap/* or /dockerdata-nfs/dev-message-router/*, I can’t remember…) Look also at the SDC backend, that kind of error typically appears when SDC-BE is not fully up. Marco From: <[email protected] <mailto:[email protected]> > on behalf of subhash kumar singh <[email protected] <mailto:[email protected]> > Reply-To: "[email protected] <mailto:[email protected]> " <[email protected] <mailto:[email protected]> >, "[email protected] <mailto:[email protected]> " <[email protected] <mailto:[email protected]> > Date: Monday, April 22, 2019 at 8:43 AM To: "[email protected] <mailto:[email protected]> " <[email protected] <mailto:[email protected]> > Cc: Seshu m <[email protected] <mailto:[email protected]> >, huangxiangyu <[email protected] <mailto:[email protected]> >, Chenchuanyu <[email protected] <mailto:[email protected]> > Subject: [onap-discuss] Error code POL5000 : exception on distributing service Hello SDC Team, Could you please help me to understand reason for receiving Error POL 5000 on distributing service model. Following message I receive on distributing service from portal Error code : POL5000 Status code: 500 Internal Server Error. Please try again later. Similar exception I can see in SO logs also. Following are logs from SO: 019-04-22T12:29:44.856Z|trace-#| org.onap.sdc.impl.DistributionClientImpl - DistributionClient - init [27/4398] 2019-04-22T12:29:44.857Z|trace-#| org.onap.sdc.impl.DistributionClientImpl - get ueb cluster server list from component(configuration file) 2019-04-22T12:29:44.865Z|trace-#| org.onap.sdc.http.SdcConnectorClient - about to perform getServerList. requestId= 2accf8f7-f355-42ef-8455-c9779a5476f0 url= /sdc/v1/artifactTypes 2019-04-22T12:29:44.865Z|trace-#| org.onap.sdc.http.HttpAsdcClient - url to send https://sdc-be.onap:8443/sdc/v1/artifactTypes <https://urldefense.proofpoint.com/v2/url?u=https-3A__sdc-2Dbe.onap-3A8443_sdc_v1_artifactTypes&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=KgFIQiUJzSC0gUhJaQxg8eC3w16GC3sKgWIcs4iIee0&m=T0n9HHqWnh7JjJ5Vozwufa4erXiC-bh0OLGSGrTbrxs&s=JtwMgZyEPqPnL-AWA-ff3G-Qil_6bNrWS0Z_G9kVmaY&e=> 2019-04-22T12:29:45.103Z|trace-#| org.onap.sdc.http.HttpAsdcClient - GET Response Status 200 2019-04-22T12:29:45.104Z|trace-#| org.onap.sdc.impl.DistributionClientImpl - Artifact types: [HEAT, HEAT_ARTIFACT, HEAT_ENV, HEAT_NESTED, HEAT_NET, HEAT_VOL, OTHER, TOSCA_CSAR, VF_MODULES_METADATA] were v alidated with ASDC server 2019-04-22T12:29:45.104Z|trace-#| org.onap.sdc.impl.DistributionClientImpl - create keys 2019-04-22T12:29:45.104Z|trace-#| com.att.nsa.apiClient.http.HttpClient - POST http://message-router.onap:3904/apiKeys/create <https://urldefense.proofpoint.com/v2/url?u=http-3A__message-2Drouter.onap-3A3904_apiKeys_create&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=KgFIQiUJzSC0gUhJaQxg8eC3w16GC3sKgWIcs4iIee0&m=T0n9HHqWnh7JjJ5Vozwufa4erXiC-bh0OLGSGrTbrxs&s=tH1jmlJINve_3dki0tVyvWXyHB5-n4v9_SIOxHhcD7A&e=> will send credentials over a clear channel. 2019-04-22T12:29:45.210Z|trace-#| org.onap.sdc.http.SdcConnectorClient - about to perform registerAsdcTopics. requestId= 39614524-48f0-4ca3-8407-f7be5eeb10bd url= /sdc/v1/registerForDistribution 2019-04-22T12:29:45.381Z|trace-#| org.onap.sdc.http.SdcConnectorClient - status from ASDC is org.onap.sdc.http.HttpAsdcResponse@1f2d8e1a <mailto:org.onap.sdc.http.HttpAsdcResponse@1f2d8e1a> 2019-04-22T12:29:45.381Z|trace-#| org.onap.sdc.http.SdcConnectorClient - DistributionClientResultImpl [responseStatus=ASDC_SERVER_PROBLEM, responseMessage=ASDC server problem] 2019-04-22T12:29:45.381Z|trace-#| org.onap.sdc.http.SdcConnectorClient - error from ASDC is: { "requestError": { "policyException": { "messageId": "POL5000", "text": "Error: Internal Server Error. Please try again later.", "variables": [] } } } I tried to reinstall dmaap but it did not solve the problem. As part of my investigation I tried to find related logs (e.g DistributionEngineClusterHealth.java) but I am not sure where I can see these logs and how to enable trace logging. -- Regards, Subhash Kumar Singh FNOSS, ONAP Huawei Technologies India Pvt. Ltd. Survey No. 37, Next to EPIP Area, Kundalahalli, Whitefield Bengaluru-560066, Karnataka Tel: + 91-80-49160700 Ext 70992 II Mob: 8050101106 Email: [email protected] <mailto:[email protected]> _____ This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it! -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#16791): https://lists.onap.org/g/onap-discuss/message/16791 Mute This Topic: https://lists.onap.org/mt/31373892/21656 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
