Hi, we are aware of this and working to fix it: https://jira.onap.org/browse/AAI-2766
If you find a fix and can contribute, we welcome it 😊 Thanks, -jimmy From: "[email protected]" <[email protected]> Date: Thursday, March 26, 2020 at 3:19 PM To: "FORSYTH, JAMES" <[email protected]>, "[email protected]" <[email protected]> Subject: Re: [onap-discuss] #aai ALERT: Expiring AAI->AAF certificate in master & elalto!!! Hi Jimmy, After applying the patch, we are getting issues with AAI-Data-Router (Dublin). It goes on crashloopbackoff with following logs: 2020-03-26T18:11:49.900Z||main|data-router||org.onap.aai.datarouter.Application||INFO|Starting Application v1.4.0-SNAPSHOT on onap-aai-aai-data-router-779b69965f-wcd4v with PID 22 (/opt/app/data-router/data-router.jar started by root in /opt/app/data-router) 2020-03-26T18:11:49.901Z||main|data-router||org.onap.aai.datarouter.Application||INFO|No active profile set, falling back to default profiles: default 2020-03-26T18:11:51.902Z||background-preinit|data-router||org.hibernate.validator.internal.util.Version||INFO|HV000001: Hibernate Validator 5.3.6.Final 2020-03-26T18:12:01.769Z||main|data-router||org.springframework.boot.context.properties.ConfigurationPropertiesBindingPostProcessor||WARN|Multiple PropertySourcesPlaceholderConfigurer beans registered [propertySourcesPlaceholderConfigurer, org.springframework.context.support.PropertySourcesPlaceholderConfigurer#0], falling back to Environment 2020-03-26T18:12:03.620Z||main|data-router||org.eclipse.jetty.server.Server||INFO|jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_171-8u171-b11-2~14.04-b11 2020-03-26T18:12:03.729Z||main|data-router||org.eclipse.jetty.server.handler.ContextHandler.application||INFO|Initializing Spring embedded WebApplicationContext 2020-03-26T18:12:06.812Z||main|data-router||org.eclipse.jetty.server.session||INFO|DefaultSessionIdManager workerName=node0 2020-03-26T18:12:06.813Z||main|data-router||org.eclipse.jetty.server.session||INFO|No SessionScavenger set, using defaults 2020-03-26T18:12:06.817Z||main|data-router||org.eclipse.jetty.server.session||INFO|node0 Scavenging every 600000ms 2020-03-26T18:12:06.855Z||main|data-router||org.eclipse.jetty.server.handler.ContextHandler||INFO|Started o.s.b.c.e.j.JettyEmbeddedWebAppContext@c88a337{application,/,[file:///tmp/jetty-docbase.7292443127022132619.9502/],AVAILABLE} 2020-03-26T18:12:06.856Z||main|data-router||org.eclipse.jetty.server.Server||INFO|Started @21950ms 2020-03-26T18:12:23.727Z||main|data-router||org.onap.aai.schema.OxmModelLoader||INFO|DR0014I|DR0014I Successfully loaded schema: v11| 2020-03-26T18:12:23.785Z||main|data-router||org.onap.aai.schema.OxmModelLoader||INFO|DR0014I|DR0014I Successfully loaded schema: v12| 2020-03-26T18:12:23.792Z||main|data-router||org.onap.aai.schema.OxmModelLoader||INFO|DR0014I|DR0014I Successfully loaded schema: v13| 2020-03-26T18:12:23.798Z||main|data-router||org.onap.aai.schema.OxmModelLoader||INFO|DR0014I|DR0014I Successfully loaded schema: v14| 2020-03-26T18:12:23.803Z||main|data-router||org.onap.aai.schema.OxmModelLoader||INFO|DR0014I|DR0014I Successfully loaded schema: v15| 2020-03-26T18:12:23.809Z||main|data-router||org.onap.aai.schema.OxmModelLoader||INFO|DR0014I|DR0014I Successfully loaded schema: v16| 2020-03-26T18:12:23.813Z||main|data-router||org.onap.aai.datarouter.policy.EntityEventPolicy||INFO|EEP0005I|EEP0005I Found OXM model: v12| 2020-03-26T18:12:23.813Z||main|data-router||org.onap.aai.datarouter.policy.EntityEventPolicy||INFO|EEP0005I|EEP0005I Found OXM model: v11| 2020-03-26T18:12:23.813Z||main|data-router||org.onap.aai.datarouter.policy.EntityEventPolicy||INFO|EEP0005I|EEP0005I Found OXM model: v14| 2020-03-26T18:12:23.813Z||main|data-router||org.onap.aai.datarouter.policy.EntityEventPolicy||INFO|EEP0005I|EEP0005I Found OXM model: v13| 2020-03-26T18:12:23.813Z||main|data-router||org.onap.aai.datarouter.policy.EntityEventPolicy||INFO|EEP0005I|EEP0005I Found OXM model: v16| 2020-03-26T18:12:23.813Z||main|data-router||org.onap.aai.datarouter.policy.EntityEventPolicy||INFO|EEP0005I|EEP0005I Found OXM model: v15| 2020-03-26T18:12:23.901Z||main|data-router||AAIRESTClient||INFO|AC0001I|AC0001I PUT request at url = https://aai-search-data.onap:9509/services/search-data-service/v1/search/indexes/entity-search-index/<https://urldefense.proofpoint.com/v2/url?u=https-3A__aai-2Dsearch-2Ddata.onap-3A9509_services_search-2Ddata-2Dservice_v1_search_indexes_entity-2Dsearch-2Dindex_&d=DwQFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=Oej6QUk5p2KdqNEWySpOHA&m=tayX6YcnyezlcNdqxDQkTACMQ6VX_PQ2jQHa1jN4wgc&s=yCq_NNIzZ9YHqhLigZFIp34IDmRMx9netMxbac0f6hc&e=>| 2020-03-26T18:13:26.258Z||main|data-router||AAIRESTClient||ERROR|AC2002E|AC2002E Error during PUT operation to endpoint at url = https://aai-search-data.onap:9509/services/search-data-service/v1/search/indexes/entity-search-index/<https://urldefense.proofpoint.com/v2/url?u=https-3A__aai-2Dsearch-2Ddata.onap-3A9509_services_search-2Ddata-2Dservice_v1_search_indexes_entity-2Dsearch-2Dindex_&d=DwQFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=Oej6QUk5p2KdqNEWySpOHA&m=tayX6YcnyezlcNdqxDQkTACMQ6VX_PQ2jQHa1jN4wgc&s=yCq_NNIzZ9YHqhLigZFIp34IDmRMx9netMxbac0f6hc&e=> with error = java.net.SocketTimeoutException: Read timed out| 2020-03-26T18:13:26.265Z||main|data-router||AAIRESTClient||INFO|AC0003I|AC0003I PUT request operation time = 62374 ms for link = https://aai-search-data.onap:9509/services/search-data-service/v1/search/indexes/entity-search-index/<https://urldefense.proofpoint.com/v2/url?u=https-3A__aai-2Dsearch-2Ddata.onap-3A9509_services_search-2Ddata-2Dservice_v1_search_indexes_entity-2Dsearch-2Dindex_&d=DwQFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=Oej6QUk5p2KdqNEWySpOHA&m=tayX6YcnyezlcNdqxDQkTACMQ6VX_PQ2jQHa1jN4wgc&s=yCq_NNIzZ9YHqhLigZFIp34IDmRMx9netMxbac0f6hc&e=>| 2020-03-26T18:13:26.265Z||main|data-router||AAIRESTClient||INFO|AC0004I|AC0004I request at url = https://aai-search-data.onap:9509/services/search-data-service/v1/search/indexes/entity-search-index/<https://urldefense.proofpoint.com/v2/url?u=https-3A__aai-2Dsearch-2Ddata.onap-3A9509_services_search-2Ddata-2Dservice_v1_search_indexes_entity-2Dsearch-2Dindex_&d=DwQFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=Oej6QUk5p2KdqNEWySpOHA&m=tayX6YcnyezlcNdqxDQkTACMQ6VX_PQ2jQHa1jN4wgc&s=yCq_NNIzZ9YHqhLigZFIp34IDmRMx9netMxbac0f6hc&e=> resulted in http response: 500 Internal Server Error| 2020-03-26T18:13:26.272Z||main|data-router||org.onap.aai.datarouter.policy.EntityEventPolicy||ERROR|EEP012E|EEP012E Failed to create Search index entity-search-index due to: Error during GET operation to AAI with message = java.net.SocketTimeoutException: Read timed out | 2020-03-26T18:13:26.273Z||main|data-router||AAIRESTClient||INFO|AC0001I|AC0001I PUT request at url = https://aai-search-data.onap:9509/services/search-data-service/v1/search/indexes/topography-search-index/<https://urldefense.proofpoint.com/v2/url?u=https-3A__aai-2Dsearch-2Ddata.onap-3A9509_services_search-2Ddata-2Dservice_v1_search_indexes_topography-2Dsearch-2Dindex_&d=DwQFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=Oej6QUk5p2KdqNEWySpOHA&m=tayX6YcnyezlcNdqxDQkTACMQ6VX_PQ2jQHa1jN4wgc&s=PLUh_wfeQ8Tzd3ZmWAZE0uVe7rLqLh2OwZ2jhtWjT0s&e=>| 2020-03-26T18:14:26.348Z||main|data-router||AAIRESTClient||ERROR|AC2002E|AC2002E Error during PUT operation to endpoint at url = https://aai-search-data.onap:9509/services/search-data-service/v1/search/indexes/topography-search-index/<https://urldefense.proofpoint.com/v2/url?u=https-3A__aai-2Dsearch-2Ddata.onap-3A9509_services_search-2Ddata-2Dservice_v1_search_indexes_topography-2Dsearch-2Dindex_&d=DwQFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=Oej6QUk5p2KdqNEWySpOHA&m=tayX6YcnyezlcNdqxDQkTACMQ6VX_PQ2jQHa1jN4wgc&s=PLUh_wfeQ8Tzd3ZmWAZE0uVe7rLqLh2OwZ2jhtWjT0s&e=> with error = java.net.SocketTimeoutException: Read timed out| 2020-03-26T18:14:26.349Z||main|data-router||AAIRESTClient||INFO|AC0003I|AC0003I PUT request operation time = 60076 ms for link = https://aai-search-data.onap:9509/services/search-data-service/v1/search/indexes/topography-search-index/<https://urldefense.proofpoint.com/v2/url?u=https-3A__aai-2Dsearch-2Ddata.onap-3A9509_services_search-2Ddata-2Dservice_v1_search_indexes_topography-2Dsearch-2Dindex_&d=DwQFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=Oej6QUk5p2KdqNEWySpOHA&m=tayX6YcnyezlcNdqxDQkTACMQ6VX_PQ2jQHa1jN4wgc&s=PLUh_wfeQ8Tzd3ZmWAZE0uVe7rLqLh2OwZ2jhtWjT0s&e=>| 2020-03-26T18:14:26.349Z||main|data-router||AAIRESTClient||INFO|AC0004I|AC0004I request at url = https://aai-search-data.onap:9509/services/search-data-service/v1/search/indexes/topography-search-index/<https://urldefense.proofpoint.com/v2/url?u=https-3A__aai-2Dsearch-2Ddata.onap-3A9509_services_search-2Ddata-2Dservice_v1_search_indexes_topography-2Dsearch-2Dindex_&d=DwQFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=Oej6QUk5p2KdqNEWySpOHA&m=tayX6YcnyezlcNdqxDQkTACMQ6VX_PQ2jQHa1jN4wgc&s=PLUh_wfeQ8Tzd3ZmWAZE0uVe7rLqLh2OwZ2jhtWjT0s&e=> resulted in http response: 500 Internal Server Error| 2020-03-26T18:14:26.349Z||main|data-router||org.onap.aai.datarouter.policy.EntityEventPolicy||ERROR|EEP012E|EEP012E Failed to create Search index topography-search-index due to: Error during GET operation to AAI with message = java.net.SocketTimeoutException: Read timed out | 2020-03-26T18:14:26.349Z||main|data-router||org.onap.aai.datarouter.policy.EntityEventPolicy||INFO|EEP0007I|EEP0007I Entity Event Policy component started.| 2020-03-26T18:15:08.698Z||main|data-router||org.eclipse.jetty.server.handler.ContextHandler.application||INFO|Initializing Spring FrameworkServlet 'dispatcherServlet' 2020-03-26T18:15:09.010Z||main|data-router||org.eclipse.jetty.util.ssl.SslContextFactory||INFO|x509=X509@33a7331(dmaap-mr (intermediateca_1),h=[message-router, mr.api.simpledemo.onap.org, message-router.onap, dmaap-mr, dmaap-mr.onap],w=[]) for SslContextFactory@1e1a0406[provider=null,keyStore=file:///opt/app/data-router/config/auth/tomcat_keystore,trustStore=null] 2020-03-26T18:15:09.010Z||main|data-router||org.eclipse.jetty.util.ssl.SslContextFactory||INFO|x509=X509@7aef21bb(tomcat,h=[],w=[]) for SslContextFactory@1e1a0406[provider=null,keyStore=file:///opt/app/data-router/config/auth/tomcat_keystore,trustStore=null] 2020-03-26T18:15:09.090Z||main|data-router||org.eclipse.jetty.util.ssl.SslContextFactory.config||WARN|No Client EndPointIdentificationAlgorithm configured for SslContextFactory@1e1a0406[provider=null,keyStore=file:///opt/app/data-router/config/auth/tomcat_keystore,trustStore=null] 2020-03-26T18:15:09.491Z||main|data-router||org.eclipse.jetty.server.AbstractConnector||INFO|Started ServerConnector@12aba8be{SSL,[ssl, http/1.1]}{0.0.0.0:9502} 2020-03-26T18:15:09.510Z||main|data-router||org.onap.aai.datarouter.Application||INFO|Started Application in 202.473 seconds (JVM running for 204.604) 2020-03-26T18:15:12.005Z||pool-3-thread-1|data-router||org.apache.cxf.phase.PhaseInterceptorChain||WARN|Interceptor for {https://message-router.onap:3905/events/AAI-EVENT/datarouter/datarouter}WebClient<https://urldefense.proofpoint.com/v2/url?u=https-3A__message-2Drouter.onap-3A3905_events_AAI-2DEVENT_datarouter_datarouter-257DWebClient&d=DwQFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=Oej6QUk5p2KdqNEWySpOHA&m=tayX6YcnyezlcNdqxDQkTACMQ6VX_PQ2jQHa1jN4wgc&s=I42gyFe12u5iKh-mFwr7eOONblzmFq55d1-HEkNznqE&e=> has thrown exception, unwinding now org.apache.cxf.interceptor.Fault: Could not send Message. at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:67) at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:691) at org.apache.cxf.jaxrs.client.WebClient.doChainedInvocation(WebClient.java:1050) at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:897) at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:866) at org.apache.cxf.jaxrs.client.WebClient.invoke(WebClient.java:431) at org.apache.cxf.jaxrs.client.SyncInvokerImpl.method(SyncInvokerImpl.java:135) at org.apache.cxf.jaxrs.client.SyncInvokerImpl.method(SyncInvokerImpl.java:130) at org.apache.cxf.jaxrs.client.SyncInvokerImpl.get(SyncInvokerImpl.java:50) at org.apache.cxf.jaxrs.client.spec.InvocationBuilderImpl.get(InvocationBuilderImpl.java:88) at com.att.nsa.mr.client.impl.DmaapClientUtil.getResponsewtCambriaAuth(DmaapClientUtil.java:53) at com.att.nsa.mr.client.impl.MRBaseClient.getAuth(MRBaseClient.java:255) at com.att.nsa.mr.client.impl.MRConsumerImpl.fetch(MRConsumerImpl.java:206) at com.att.nsa.mr.client.impl.MRConsumerImpl.fetch(MRConsumerImpl.java:100) at org.onap.aai.event.client.DMaaPEventConsumer.consumeAndCommit(DMaaPEventConsumer.java:156) at org.onap.aai.event.EventBusConsumer.poll(EventBusConsumer.java:79) at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:174) at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:101) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: javax.net.ssl.SSLException: SSLException invoking https://message-router.onap:3905/events/AAI-EVENT/datarouter/datarouter?timeout=1000&limit=100<https://urldefense.proofpoint.com/v2/url?u=https-3A__message-2Drouter.onap-3A3905_events_AAI-2DEVENT_datarouter_datarouter-3Ftimeout-3D1000-26limit-3D100&d=DwQFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=Oej6QUk5p2KdqNEWySpOHA&m=tayX6YcnyezlcNdqxDQkTACMQ6VX_PQ2jQHa1jN4wgc&s=-m8J6QKZqjBMfDsYXMd_gkOHGVYMuL4GyM7tmSGh428&e=>: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1387) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1371) at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56) at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:658) at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:63) ... 25 common frames omitted Do you have any ideas about this? -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#20393): https://lists.onap.org/g/onap-discuss/message/20393 Mute This Topic: https://lists.onap.org/mt/72549806/21656 Mute #aai: https://lists.onap.org/mk?hashtag=aai&subid=2740164 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
