Viswa, Good questions. This discussion is just for the kubernetes side of deploying onap (not the DCAE heatbridge in amsterdam or running VNFs on openstack – those are outside the context of the kubernetes deployment for now).
Currently we deploy VNFs to VMs on openstack – we do not have containerized VNF’s yet – so the memory requirements are separate. A simple use case like the vFW will consume 12G for example outside of where we run K8S. For the memory increase you are referring to these questions from Beka? https://wiki.onap.org/display/DW/ONAP+on+Kubernetes?focusedCommentId=25439660&refresh=1520778643640#comment-25439660 https://wiki.onap.org/display/DW/questions/25439159/onap-memory-leak I have not looked into the full cause but SDNC has been having issues since a refactor in mid Jan – I put an automated restart of the SDNC pod part way through the CD startup when all dependencies are up. https://gerrit.onap.org/r/#/c/32653/ There is a comment on the ueb listener – this is in the amsterdam branch and not master but may be related. We can do a full analysis of the startup and idle behavior of the containers together. I have been noticing the memory footprint enlarging for most of 2018 – Usually I shutdown vfc to be able to keep room for heaps. As to the cause – we are speculating unless we have a baseline from a tool like New Relic and track the sizes of our java, python and db based images. Any one or all of these could be demanding more runtime memory based on code changes across onap. You would need to track all the projects. In kubernetes you can get a snapshot of each container – keep these periodically and compare this. Ideally we run ram profiles as part of our CI/CD pipeline in the future. Thank you /michael From: Kumar Skand Priya, Viswanath V [mailto:[email protected]] Sent: Saturday, March 10, 2018 09:01 To: Michael O'Brien <[email protected]> Cc: [email protected] Subject: Re: [E] [onap-discuss] [OOM] Heads up: ONAP Kubernetes master has crossed the 64G VM barrier Hi Michael, Does this includes the memory occupied due to deployed VNF & NS as well? If so, what are the requirements for running just the ONAP ( for both R1&R2) ? And more particularly, what's eating more space compared to R2 & R1 ( assuming both are plan ONAP installation i.e without DCAE & VNFs) ? Few days back, someone else from community have highlighted about the possibility of memory leaks in ONAP code, which might account to increase memory consumption. Do you have any thoughts on the same? BR, Viswa [http://ss7.vzw.com/is/image/VerizonWireless/vz-sig-verizon?$defaultscale$]<http://www.verizon.com> Viswanath Kumar Skand Priya Architect Verizon India ( VDSI ) On Sat, Mar 10, 2018 at 2:12 AM, Michael O'Brien <[email protected]<mailto:[email protected]>> wrote: Team, ONAP Beijing is currently crossing the 64G boundary as of a couple weeks ago. If you run the system on a 128G VM then heaps will expand past 64G within 24 hours. If you stay on 64G (which you can) – reduce the optional pods or you will be getting OOME’s Use the ongoing POC JIRA as a guide – we need a full set of deploytime/runtime dependency trees to be able to know what to shutdown. https://jira.onap.org/browse/OOM-511<https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.onap.org_browse_OOM-2D511&d=DwMFAg&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=9F3pNUkzjE-2v1eTClkRVakDRN8GH7Bm-wt1lWkxoUyyDORTqf5MxNO_GrMBs0gZ&m=pUxXmcE2onWJWwHHrm7haZBwTfyT7z1hmkqJADf5-fI&s=CZKKOZ8gmBwWlnaQo4GEIHhNZ97f4rTBXS5s2yqk8Qg&e=> The recommended VM size (1 or a cluster) is now 80 to 128G – for Beijing (without the upcoming DCAE port) For Amsterdam the OOM side still fits in a 64G vm (you can shutdown vCPE/vVOLTE required pods like vfc) – heatbridge works there to DCAE which bring you up to 150G when the full heat side is up. https://wiki.onap.org/display/DW/ONAP+on+Kubernetes#ONAPonKubernetes-HardwareRequirements<https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.onap.org_display_DW_ONAP-2Bon-2BKubernetes-23ONAPonKubernetes-2DHardwareRequirements&d=DwMFAg&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=9F3pNUkzjE-2v1eTClkRVakDRN8GH7Bm-wt1lWkxoUyyDORTqf5MxNO_GrMBs0gZ&m=pUxXmcE2onWJWwHHrm7haZBwTfyT7z1hmkqJADf5-fI&s=P_NCcKmQWPfXRD2Ro7nCjsFF6ylM_vSPDzotXCHqos4&e=> ONAP startup now reaches a peak of 60 cores so the more vCores you have the less CPU bound you will be. Thank you /michael This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at https://www.amdocs.com/about/email-disclaimer<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.amdocs.com_about_email-2Ddisclaimer&d=DwMFAg&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=9F3pNUkzjE-2v1eTClkRVakDRN8GH7Bm-wt1lWkxoUyyDORTqf5MxNO_GrMBs0gZ&m=pUxXmcE2onWJWwHHrm7haZBwTfyT7z1hmkqJADf5-fI&s=Jr00SleRePZFL_-zBRl0v_ZiCpyUkGBI_cSrRsm2wOI&e=> _______________________________________________ onap-discuss mailing list [email protected]<mailto:[email protected]> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.onap.org_mailman_listinfo_onap-2Ddiscuss&d=DwICAg&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=9F3pNUkzjE-2v1eTClkRVakDRN8GH7Bm-wt1lWkxoUyyDORTqf5MxNO_GrMBs0gZ&m=pUxXmcE2onWJWwHHrm7haZBwTfyT7z1hmkqJADf5-fI&s=INv35wxpki9d5173aITT0lBcfPtccU4rwAyl066qOyE&e= This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at https://www.amdocs.com/about/email-disclaimer <https://www.amdocs.com/about/email-disclaimer>
_______________________________________________ onap-discuss mailing list [email protected] https://lists.onap.org/mailman/listinfo/onap-discuss
