Sirisha, Hi, that is a common OS based issue for some new VM’s depending on your undercloud – this is the vm.max_map_count of 64k instead of 256Kb limitation - not 256Gb on the OS. This occurs for all elasticsearch containers whether AAI, CLAMP, LOG, SDC…. Run the following before doing a helm install – and all the ES containers including the one in AAI will come up fine.
sudo sysctl -w vm.max_map_count=262144 There is documentation on this. First noticed this in Rancher 1.6.11 – fixed via https://jira.onap.org/browse/LOG-334 (OOM-431). https://wiki.onap.org/display/DW/Cloud+Native+Deployment#CloudNativeDeployment-vm.max_map_count64to256kblimit And I run this command by default in the installation script to bootstrap Helm/Kubernetes/Docker via the Rancher install in (there are 4 other installers that do the same in ONAP) Not in https://git.onap.org/logging-analytics/tree/deploy/rancher/oom_rancher_setup.sh but in the OOM CD deployment script just before doing a helm install https://git.onap.org/logging-analytics/tree/deploy/cd.sh#n49 # fix virtual memory for onap-log:elasticsearch under Rancher 1.6.11 - OOM-431 sudo sysctl -w vm.max_map_count=262144 thank you /michael From: [email protected] <[email protected]> On Behalf Of Gopigiri, Sirisha via Lists.Onap.Org Sent: Wednesday, October 17, 2018 9:56 AM To: [email protected]; Viswanath V Kumar Skand Priya <[email protected]> Subject: [onap-discuss] [AAI] [OOM] AAI Elastic search pod not coming up in master branch Hi All, I am installing ONAP using OOM master branch with AAI enabled, in a two node kubernetes cluster with each node having 64GB RAM. But the AAI elastic search pod is expecting 256GB RAM and is stuck in crashloopback state. Please find the logs below. So, to install OOM should the kubernetes node have 256 GB, or am I missing something. Kindly let me know. [2018-10-17T13:20:19,547][INFO ][o.e.n.Node ] [ES_ONAP] initializing ... [2018-10-17T13:20:19,739][INFO ][o.e.e.NodeEnvironment ] [ES_ONAP] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/vda1)]], net usable_space [81.1gb], net total_space [96.8gb], types [ext4] [2018-10-17T13:20:19,740][INFO ][o.e.e.NodeEnvironment ] [ES_ONAP] heap size [989.8mb], compressed ordinary object pointers [true] [2018-10-17T13:20:19,743][INFO ][o.e.n.Node ] [ES_ONAP] node name [ES_ONAP], node ID [BL4G5DFCS9G-wcrQS64tUQ] [2018-10-17T13:20:19,744][INFO ][o.e.n.Node ] [ES_ONAP] version[6.1.2], pid[1], build[5b1fea5/2018-01-10T02:35:59.208Z], OS[Linux/4.4.0-135-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_151/25.151-b12] [2018-10-17T13:20:19,744][INFO ][o.e.n.Node ] [ES_ONAP] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config] [2018-10-17T13:20:23,681][INFO ][o.e.p.PluginsService ] [ES_ONAP] loaded module [aggs-matrix-stats] [2018-10-17T13:20:23,682][INFO ][o.e.p.PluginsService ] [ES_ONAP] loaded module [analysis-common] [2018-10-17T13:20:23,682][INFO ][o.e.p.PluginsService ] [ES_ONAP] loaded module [ingest-common] [2018-10-17T13:20:23,683][INFO ][o.e.p.PluginsService ] [ES_ONAP] loaded module [lang-expression] [2018-10-17T13:20:23,683][INFO ][o.e.p.PluginsService ] [ES_ONAP] loaded module [lang-mustache] [2018-10-17T13:20:23,683][INFO ][o.e.p.PluginsService ] [ES_ONAP] loaded module [lang-painless] [2018-10-17T13:20:23,683][INFO ][o.e.p.PluginsService ] [ES_ONAP] loaded module [mapper-extras] [2018-10-17T13:20:23,684][INFO ][o.e.p.PluginsService ] [ES_ONAP] loaded module [parent-join] [2018-10-17T13:20:23,684][INFO ][o.e.p.PluginsService ] [ES_ONAP] loaded module [percolator] [2018-10-17T13:20:23,685][INFO ][o.e.p.PluginsService ] [ES_ONAP] loaded module [reindex] [2018-10-17T13:20:23,685][INFO ][o.e.p.PluginsService ] [ES_ONAP] loaded module [repository-url] [2018-10-17T13:20:23,685][INFO ][o.e.p.PluginsService ] [ES_ONAP] loaded module [transport-netty4] [2018-10-17T13:20:23,685][INFO ][o.e.p.PluginsService ] [ES_ONAP] loaded module [tribe] [2018-10-17T13:20:23,687][INFO ][o.e.p.PluginsService ] [ES_ONAP] loaded plugin [ingest-geoip] [2018-10-17T13:20:23,687][INFO ][o.e.p.PluginsService ] [ES_ONAP] loaded plugin [ingest-user-agent] [2018-10-17T13:20:23,687][INFO ][o.e.p.PluginsService ] [ES_ONAP] loaded plugin [x-pack] [2018-10-17T13:20:28,981][INFO ][o.e.d.DiscoveryModule ] [ES_ONAP] using discovery type [zen] [2018-10-17T13:20:30,228][INFO ][o.e.n.Node ] [ES_ONAP] initialized [2018-10-17T13:20:30,229][INFO ][o.e.n.Node ] [ES_ONAP] starting ... [2018-10-17T13:20:30,520][INFO ][o.e.t.TransportService ] [ES_ONAP] publish_address {127.0.0.1:8443<http://127.0.0.1:8443>}, bound_addresses {[::]:8443} [2018-10-17T13:20:30,557][INFO ][o.e.b.BootstrapChecks ] [ES_ONAP] bound or publishing to a non-loopback address, enforcing bootstrap checks ERROR: [1] bootstrap checks failed [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144] [2018-10-17T13:20:30,574][INFO ][o.e.n.Node ] [ES_ONAP] stopping ... [2018-10-17T13:20:30,729][INFO ][o.e.n.Node ] [ES_ONAP] stopped [2018-10-17T13:20:30,729][INFO ][o.e.n.Node ] [ES_ONAP] closing ... [2018-10-17T13:20:30,755][INFO ][o.e.n.Node ] [ES_ONAP] closed Thank you in advance! Best Regards Sirisha Gopigiri “Amdocs’ email platform is based on a third-party, worldwide, cloud-based system. Any emails sent to Amdocs will be processed and stored using such system and are accessible by third party providers of such system on a limited basis. Your sending of emails to Amdocs evidences your consent to the use of such system and such processing, storing and access”. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13106): https://lists.onap.org/g/onap-discuss/message/13106 Mute This Topic: https://lists.onap.org/mt/27380106/21656 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
