Hi,
I invested today in researching this problem. It occurs on non-Rancher
environments.
I realized that aai-cassandra chart, specially service is not well. There was a
problem with label app, it didn’t match the statefulSet.
This caused a strange behavior on Rancher, where serviceName defined in the
stateful set was routable. But on other k8s deployments it wasn’t.
Note the annotation that is needed for successful start of Cassandra:
service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
Also note that the publishNotReadyAddresses field that is supposed to replace
this annotation is not working
(https://github.com/kubernetes/kubernetes/issues/58662).
Had to update resources and traversal configuration.
Finally I managed to make it work in both Rancher and my kubeadm multimode
environment (versions 1.7.16 and 1.10.1).
Some Cassandra instances sometimes need a couple of restarts before they manage
to start successfully, but it resolves quickly.
If there is some Cassandra expert from aai team that would like to improve the
Cassandra start process, you are most welcome.
The fix is in a pending review: https://gerrit.onap.org/r/47323
Runcher:
root@borislav-new-rancher:~/onap/master/aai/oom/kubernetes# kgpa -w
NAMESPACE NAME READY STATUS
RESTARTS AGE
kube-system heapster-76b8cd7b5-l9sdn 1/1 Running
1 28d
kube-system kube-dns-5d7b4487c9-j5ctf 3/3 Running
15 28d
kube-system kubernetes-dashboard-f9577fffd-9sqfc 1/1 Running
1 28d
kube-system monitoring-grafana-997796fcf-krq9l 1/1 Running
1 28d
kube-system monitoring-influxdb-56fdcd96b-tvtm2 1/1 Running
1 28d
kube-system tiller-deploy-54bcc55dd5-sxcl2 1/1 Running
4 28d
onap aaa-aai-85684bcf79-ddrj8 1/1 Running
0 5m
onap aaa-aai-babel-7ff8f4d4-hlfqs 2/2 Running
0 5m
onap aaa-aai-cassandra-0 1/1 Running
2 5m
onap aaa-aai-cassandra-1 1/1 Running
0 3m
onap aaa-aai-cassandra-2 1/1 Running
0 3m
onap aaa-aai-champ-6857876bff-dl72c 0/1 Running
0 5m
onap aaa-aai-data-router-5b48bc6f78-n4wgb 1/1 Running
0 5m
onap aaa-aai-elasticsearch-ccd96645c-jrfqn 1/1 Running
0 5m
onap aaa-aai-gizmo-6966c665c5-9wjwf 2/2 Running
0 5m
onap aaa-aai-hbase-57f86cc86c-8q995 1/1 Running
0 5m
onap aaa-aai-modelloader-7b5fc6f474-462t7 2/2 Running
0 5m
onap aaa-aai-resources-7f9ddc68dc-shkcv 2/2 Running
0 5m
onap aaa-aai-search-data-cd976775f-fnstt 2/2 Running
0 5m
onap aaa-aai-sparky-be-6d8978bdbb-8dzzq 2/2 Running
0 5m
onap aaa-aai-traversal-6c58f8d698-6pchw 2/2 Running
0 5m
onap aaa-aai-traversal-update-query-data-bpfcw 0/1
Completed 0 5m
onap aaa-dbc-pg-0 1/1 Running
0 5m
onap aaa-dbc-pg-1 1/1 Running
0 3m
onap aaa-dmaap-bus-controller-595dfd5779-f68rc 1/1 Running
0 5m
onap aaa-message-router-5c694bbdcc-vxnzd 1/1 Running
0 5m
onap aaa-message-router-kafka-d6689b986-ncrtq 1/1 Running
0 5m
onap aaa-message-router-zookeeper-85f48f46b6-n2z9p 1/1 Running
0 5m
Kubeadm:
onap aaa-aai-7bf94cd486-982zj
1/1 Running 0 20m 10.244.6.18
kubernetes-node6-borislavto-13099-05-13-18.novalocal
onap aaa-aai-babel-6485c7944b-55ghh
2/2 Running 0 20m 10.244.6.17
kubernetes-node6-borislavto-13099-05-13-18.novalocal
onap aaa-aai-cassandra-0
1/1 Running 0 20m 10.244.6.19
kubernetes-node6-borislavto-13099-05-13-18.novalocal
onap aaa-aai-cassandra-1
1/1 Running 0 19m 10.244.3.15
kubernetes-node2-borislavto-13099-05-13-18.novalocal
onap aaa-aai-cassandra-2
1/1 Running 0 19m 10.244.1.14
kubernetes-node5-borislavto-13099-05-13-18.novalocal
onap aaa-aai-champ-77f7468b45-xj4vh
0/1 Running 0 20m 10.244.4.13
kubernetes-node1-borislavto-13099-05-13-18.novalocal
onap aaa-aai-data-router-7989ff6fff-99c92
1/1 Running 0 20m 10.244.3.13
kubernetes-node2-borislavto-13099-05-13-18.novalocal
onap aaa-aai-elasticsearch-5f7447dcb9-mndvt
1/1 Running 0 20m 10.244.5.16
kubernetes-node3-borislavto-13099-05-13-18.novalocal
onap aaa-aai-gizmo-9b5887fc7-g7tgq
2/2 Running 0 20m 10.244.7.13
kubernetes-node7-borislavto-13099-05-13-18.novalocal
onap aaa-aai-hbase-6fd66cc9f4-ldjcs
1/1 Running 0 20m 10.244.4.14
kubernetes-node1-borislavto-13099-05-13-18.novalocal
onap aaa-aai-modelloader-69cbcf4cfd-8mrsz
2/2 Running 0 20m 10.244.2.15
kubernetes-node4-borislavto-13099-05-13-18.novalocal
onap aaa-aai-resources-6f7c69cf75-vnsrh
2/2 Running 0 20m 10.244.2.16
kubernetes-node4-borislavto-13099-05-13-18.novalocal
onap aaa-aai-search-data-5df78c666f-scmvq
2/2 Running 0 20m 10.244.1.13
kubernetes-node5-borislavto-13099-05-13-18.novalocal
onap aaa-aai-sparky-be-6f7cb4858-skmnh
2/2 Running 0 20m 10.244.3.14
kubernetes-node2-borislavto-13099-05-13-18.novalocal
onap aaa-aai-traversal-f9b85cf97-gv7dn
2/2 Running 0 20m 10.244.4.15
kubernetes-node1-borislavto-13099-05-13-18.novalocal
onap aaa-aai-traversal-update-query-data-lnxt8
0/1 Completed 0 20m 10.244.5.17
kubernetes-node3-borislavto-13099-05-13-18.novalocal
onap aaa-dbc-pg-0
1/1 Running 0 14m 10.244.6.20
kubernetes-node6-borislavto-13099-05-13-18.novalocal
onap aaa-dbc-pg-1
1/1 Running 0 13m 10.244.4.17
kubernetes-node1-borislavto-13099-05-13-18.novalocal
onap aaa-dmaap-bus-controller-8545ffdfb7-4g2f9
1/1 Running 0 14m 10.244.5.18
kubernetes-node3-borislavto-13099-05-13-18.novalocal
onap aaa-message-router-57ddbd6ff7-wd6zb
1/1 Running 0 14m 10.244.7.14
kubernetes-node7-borislavto-13099-05-13-18.novalocal
onap aaa-message-router-kafka-7dfddbcdbb-cl8mv
1/1 Running 0 14m 10.244.4.16
kubernetes-node1-borislavto-13099-05-13-18.novalocal
onap aaa-message-router-zookeeper-598c6c6996-mq2gn
1/1 Running 0 14m 10.244.3.16
kubernetes-node2-borislavto-13099-05-13-18.novalocal
Thanks,
Borislav Glozman
O:+972.9.776.1988
M:+972.52.2835726
[amdocs-a]
Amdocs a Platinum member of
ONAP<https://www.amdocs.com/open-network/nfv-powered-by-onap>
From: HU, JUN NICOLAS [mailto:[email protected]]
Sent: Friday, May 11, 2018 6:27 PM
To: Roger Maitland <[email protected]>; Borislav Glozman
<[email protected]>; Daniel Silverthorn
<[email protected]>; [email protected]; Steven Blimkie
<[email protected]>; FORSYTH, JAMES <[email protected]>; Mandeep Khinda
<[email protected]>
Subject: RE: [onap-discuss] [oom] [aai] Resources fails to start
Hi Roger,
Pulled latest OOM code this morning. I still got same error in kubeadm
environment.
The aai-resource try to connect to Cassandra via hostname. But got time out.
Here is log from aai-resource container.
Caused by: com.netflix.astyanax.connectionpool.exceptions.PoolTimeoutException:
PoolTimeoutException: [host=onap-aai-cassandra-2(onap-aai-cassandra-2):9160,
latency=30001(30001), attempts=3]Timed out waiting for connection
So I checked the hostname, onap-aai-cassandra-1, realized that that hostname is
unresolvable in kubeadm environment.
aaiadmin@onap-aai-babel-8c6784976-96ph2:/$ ping onap-aai-cassandra-1
ping: unknown host onap-aai-cassandra-1
I also deployed same code in a Rancher cluster, that works. I checked the
hostname, in rancher that is reachable.
aaiadmin@onap-aai-babel-8c6784976-256g6:/$ ping onap-aai-cassandra-0
PING onap-aai-cassandra-0.rancher.internal (10.42.245.107) 56(84) bytes of data.
64 bytes from 10.42.245.107: icmp_seq=1 ttl=62 time=0.558 ms
64 bytes from 10.42.245.107: icmp_seq=2 ttl=62 time=0.526 ms
Do you have any ideal no that? I compared kubedns log between 2 environments,
seems similar.
Here is some reference may relate to this issue
https://github.com/kubernetes/kubernetes/issues/45779<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_kubernetes_kubernetes_issues_45779&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=HfOWhvzCtvUWjtduPm5ExHNTAphre2Lm4Rrfcw1fghI&s=dohYfIx845zs5L0UQZsaWWeaS3-RUF4VjkiC4bvJ3k0&e=>
https://github.com/kubernetes/kubernetes/issues/39197<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_kubernetes_kubernetes_issues_39197&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=HfOWhvzCtvUWjtduPm5ExHNTAphre2Lm4Rrfcw1fghI&s=zEi_K3fHMgeQFqLuAGsJxQmS7sjHjiC-EIhvnobuvOc&e=>
Borislav, Daniel
How you set the environment. And could you verify the connection to the pod
hostname.
Thanks,
Nicolas
From: Roger Maitland [mailto:[email protected]]
Sent: Thursday, May 10, 2018 4:36 PM
To: HU, JUN NICOLAS <[email protected]<mailto:[email protected]>>; Borislav Glozman
<[email protected]<mailto:[email protected]>>; SILVERTHORN,
DANIEL <[email protected]<mailto:[email protected]>>;
[email protected]<mailto:[email protected]>; BLIMKIE,
STEVEN <[email protected]<mailto:[email protected]>>; FORSYTH,
JAMES <[email protected]<mailto:[email protected]>>; Mandeep Khinda
<[email protected]<mailto:[email protected]>>
Subject: Re: [onap-discuss] [oom] [aai] Resources fails to start
Be aware that a following fix went in yesterday
(https://gerrit.onap.org/r/#/c/45941/<https://urldefense.proofpoint.com/v2/url?u=https-3A__gerrit.onap.org_r_-23_c_45941_&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=HfOWhvzCtvUWjtduPm5ExHNTAphre2Lm4Rrfcw1fghI&s=nvHpi3p-BEb8BMtZT91WWMF5WXt-TUHFS5SSdzlWqmI&e=>).
With a fresh pull do you still see the problem?
Thanks,
Roger
From:
<[email protected]<mailto:[email protected]>>
on behalf of "HU, JUN NICOLAS" <[email protected]<mailto:[email protected]>>
Date: Thursday, May 10, 2018 at 3:37 PM
To: Borislav Glozman
<[email protected]<mailto:[email protected]>>, Daniel
Silverthorn
<[email protected]<mailto:[email protected]>>,
"[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>, Steven
Blimkie <[email protected]<mailto:[email protected]>>,
"FORSYTH, JAMES" <[email protected]<mailto:[email protected]>>, Mandeep Khinda
<[email protected]<mailto:[email protected]>>
Subject: Re: [onap-discuss] [oom] [aai] Resources fails to start
Hi Borislav,
I have the same issue in my Kubeadm Cluster.
I created a JIRA ticket for that.
https://jira.onap.org/browse/OOM-1067<https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.onap.org_browse_OOM-2D1067&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=HfOWhvzCtvUWjtduPm5ExHNTAphre2Lm4Rrfcw1fghI&s=gfgtDWEz2NC_jCLZ72Y8cXp86pzLa3-dWU2Cue_5Om0&e=>
I believe that the reason why we got this issue is the POD cannot resolve each
other by expected hostnames.
The aai-resource try to use cassandra default hostnames, which is the Cassandra
pod name, to connect Cassandra. However, aai-resource pod cannot resolve that.
I logged in to the container (aai-resource is not running, I use aai-babel to
test), try to ping Cassandra hostnames.
aaiadmin@onap-aai-babel-8c6784976-5544q:/$ ping onap-aai-cassandra-0
ping: unknown host onap-aai-cassandra-0
But I can ping it by pod ip.
onap-aai-cassandra-0 1/1 Running 0
11m 192.168.183.72 kubeadm-test
aaiadmin@onap-aai-babel-8c6784976-5544q:/$ ping 192.168.183.72
PING 192.168.183.72 (192.168.183.72) 56(84) bytes of data.
64 bytes from 192.168.183.72: icmp_seq=1 ttl=63 time=0.097 ms
64 bytes from 192.168.183.72: icmp_seq=2 ttl=63 time=0.113 ms
64 bytes from 192.168.183.72: icmp_seq=3 ttl=63 time=0.058 ms
Here is some reference may relate to this issue
https://github.com/kubernetes/kubernetes/issues/45779<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_kubernetes_kubernetes_issues_45779&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=HfOWhvzCtvUWjtduPm5ExHNTAphre2Lm4Rrfcw1fghI&s=dohYfIx845zs5L0UQZsaWWeaS3-RUF4VjkiC4bvJ3k0&e=>
https://github.com/kubernetes/kubernetes/issues/39197<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_kubernetes_kubernetes_issues_39197&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=HfOWhvzCtvUWjtduPm5ExHNTAphre2Lm4Rrfcw1fghI&s=zEi_K3fHMgeQFqLuAGsJxQmS7sjHjiC-EIhvnobuvOc&e=>
Thanks,
Nicolas
From:
[email protected]<mailto:[email protected]>
[mailto:[email protected]] On Behalf Of Borislav Glozman
Sent: Thursday, May 10, 2018 12:15 PM
To: SILVERTHORN, DANIEL
<[email protected]<mailto:[email protected]>>;
[email protected]<mailto:[email protected]>; BLIMKIE,
STEVEN <[email protected]<mailto:[email protected]>>; FORSYTH,
JAMES <[email protected]<mailto:[email protected]>>
Subject: Re: [onap-discuss] [oom] [aai] Resources fails to start
Hi,
Same problem here.
Did anyone manage to resolve this?
Thanks,
Borislav Glozman
O:+972.9.776.1988
M:+972.52.2835726
[amdocs-a]
Amdocs a Platinum member of
ONAP<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.amdocs.com_open-2Dnetwork_nfv-2Dpowered-2Dby-2Donap&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=_GZeCv6HIMYVjl_nEIJoeljq51QP1yPfxMr9qada6N8&s=esi3MN_9IbBmi4PVHc282U-nZXyug6Mc4mh4VE-i5UY&e=>
From:
[email protected]<mailto:[email protected]>
[mailto:[email protected]] On Behalf Of Daniel Silverthorn
Sent: Monday, April 23, 2018 5:40 PM
To: [email protected]<mailto:[email protected]>
Subject: [onap-discuss] [oom] [aai] Resources fails to start
Using the latest oom as of Friday, I am able to start aai on a single node k8s
cluster, but when trying it on a multimode cluster, resources fails to start up
with the following error:
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
at
org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:594)
Caused by: java.lang.ExceptionInInitializerError
at org.onap.aai.dbmap.AAIGraph.getInstance(AAIGraph.java:98)
at org.onap.aai.dbgen.GenTester.main(GenTester.java:118)
... 8 more
Caused by: java.lang.RuntimeException: Failed to instantiate graphs
at org.onap.aai.dbmap.AAIGraph.<init>(AAIGraph.java:83)
at org.onap.aai.dbmap.AAIGraph.<init>(AAIGraph.java:55)
at org.onap.aai.dbmap.AAIGraph$Helper.<clinit>(AAIGraph.java:88)
... 10 more
Caused by: java.lang.IllegalArgumentException: Could not instantiate
implementation:
org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager
at
org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:69)
at
org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:477)
at
org.janusgraph.diskstorage.Backend.getStorageManager(Backend.java:409)
at
org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1376)
at
org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:164)
at
org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:133)
at
org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:113)
at org.onap.aai.dbmap.AAIGraph.loadGraph(AAIGraph.java:110)
at org.onap.aai.dbmap.AAIGraph.<init>(AAIGraph.java:80)
... 12 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
... 20 more
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary
failure in storage backend
at
org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager.ensureKeyspaceExists(AstyanaxStoreManager.java:619)
at
org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager.<init>(AstyanaxStoreManager.java:314)
... 25 more
Caused by: com.netflix.astyanax.connectionpool.exceptions.PoolTimeoutException:
PoolTimeoutException: [host=aai-aai-cassandra-1(aai-aai-cassandra-1):9160,
latency=30001(30001), attempts=3]Timed out waiting for connection
at
com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.waitForConnection(SimpleHostConnectionPool.java:231)
at
com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.borrowConnection(SimpleHostConnectionPool.java:198)
at
com.netflix.astyanax.connectionpool.impl.RoundRobinExecuteWithFailover.borrowConnection(RoundRobinExecuteWithFailover.java:84)
at
com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:117)
at
com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:352)
at
com.netflix.astyanax.thrift.ThriftClusterImpl.executeSchemaChangeOperation(ThriftClusterImpl.java:146)
at
com.netflix.astyanax.thrift.ThriftClusterImpl.internalCreateKeyspace(ThriftClusterImpl.java:321)
at
com.netflix.astyanax.thrift.ThriftClusterImpl.addKeyspace(ThriftClusterImpl.java:294)
at
org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager.ensureKeyspaceExists(AstyanaxStoreManager.java:614)
I have tried modifying the Cassandra deployment to only start one instance of
Cassandra and got the same result.
Has anyone seen something like this?
Thanks,
Daniel Silverthorn
Software Developer
Network R&D
613-595-5259 x55259 (office)
[amdocs-2017-brand-mark-rgb]
Read the latest on
Amdocs.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.amdocs.com_&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=_GZeCv6HIMYVjl_nEIJoeljq51QP1yPfxMr9qada6N8&s=HRvdXbAePkL-jAlt2FnxcyR1XRLDyVfqA_Ed0PyaLaE&e=>
and the Amdocs blog
network<https://urldefense.proofpoint.com/v2/url?u=http-3A__blogs.amdocs.com_&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=_GZeCv6HIMYVjl_nEIJoeljq51QP1yPfxMr9qada6N8&s=ogcb2gu-cvqvX0KXoqa7VfH9kl-6jwdyIO1_G3MhUsg&e=>
– and follow us on
Facebook<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.facebook.com_Amdocs&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=_GZeCv6HIMYVjl_nEIJoeljq51QP1yPfxMr9qada6N8&s=0iOaqB0sLnyUIEmgMsK9czrlCuZXugjFy3rfMXSmxRA&e=>,
Twitter<https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_Amdocs&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=_GZeCv6HIMYVjl_nEIJoeljq51QP1yPfxMr9qada6N8&s=yBXd2TPeTMO_VBFH-AFvKHi9co_w5Fo0vJBzIAIpMk8&e=>,
LinkedIn<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.linkedin.com_company_amdocs&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=_GZeCv6HIMYVjl_nEIJoeljq51QP1yPfxMr9qada6N8&s=6lc3xV1Ota0CsP7YV4lsiSV8UgEvWWJJ3i-_6O7j0Ec&e=>
and
YouTube<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.youtube.com_amdocs&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=_GZeCv6HIMYVjl_nEIJoeljq51QP1yPfxMr9qada6N8&s=93ZqSsAs1YtUR_f3OfqLli-Pjfv-HBCsMuTOF7a_qZ4&e=>.
This message and the information contained herein is proprietary and
confidential and subject to the Amdocs policy statement,
you may review at
https://www.amdocs.com/about/email-disclaimer<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.amdocs.com_about_email-2Ddisclaimer&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=_GZeCv6HIMYVjl_nEIJoeljq51QP1yPfxMr9qada6N8&s=hy4pjtsN3rtfEwg_oh0rbt8xidlWCp6PN5D68a2ChBM&e=>
This message and the information contained herein is proprietary and
confidential and subject to the Amdocs policy statement,
you may review at
https://www.amdocs.com/about/email-disclaimer<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.amdocs.com_about_email-2Ddisclaimer&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=_GZeCv6HIMYVjl_nEIJoeljq51QP1yPfxMr9qada6N8&s=hy4pjtsN3rtfEwg_oh0rbt8xidlWCp6PN5D68a2ChBM&e=>
This message and the information contained herein is proprietary and
confidential and subject to the Amdocs policy statement,
you may review at
https://www.amdocs.com/about/email-disclaimer<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.amdocs.com_about_email-2Ddisclaimer&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=HfOWhvzCtvUWjtduPm5ExHNTAphre2Lm4Rrfcw1fghI&s=i8aTqZIt2eLgxNTbusL7MLblS14MCXjwSmz8KIR7vOY&e=>
This message and the information contained herein is proprietary and
confidential and subject to the Amdocs policy statement,
you may review at https://www.amdocs.com/about/email-disclaimer
<https://www.amdocs.com/about/email-disclaimer>
_______________________________________________
onap-discuss mailing list
[email protected]
https://lists.onap.org/mailman/listinfo/onap-discuss
