Hi Rohit
This is not a production environment. This is a test environment run
on minimum resources for my CD/CI cloud release process, so you have to
take this with a pitch of salt:
My environment is Ubuntu 18.04
Ram is 1.5GB + swap
DB is running on the same host.
However, just in case, note that:
With 4.14, 150 cloud environments were built and tested and this problem did
not arise.
with 4.15 the problem popped on the first environment.
I can see the changes made from 4.14 to the JVM options and those are:
-XX:+UseParallelGC
-XX:MaxGCPauseMillis=500
That could explain the difference in behaviour, but better tu remain suspicious.
I am going to raise memory and continue testing and see if the problem
persists.
Is memory profiling done as part of ACS release/testing?
Regards,
Rafael
On Wed, 2021-01-20 09:06 AM, Rohit Yadav <[email protected]> wrote:
> Can you share your host memory, distro (CentOS or Ubuntu and which version)
> and describe your installation (hypervisor, storage, any special plugin etc
> you may be using). Are you running DB and usage server on the same host? The
> java process requires at least 2G RAM[1] for itself.
>
>
> [1]
> https://github.com/apache/cloudstack/blob/master/packaging/systemd/cloudstack-management.default#L18
>
>
> Regards.
>
> ________________________________
> From: [email protected] " target="_blank"><[email protected]>
> Sent: Wednesday, January 20, 2021 13:13
> To: [email protected] "
> target="_blank"><[email protected]>
> Cc: [email protected] " target="_blank"><[email protected]>
> Subject: 4.15 Manager out of memory
>
> Hi!
>
> Congrats for the 4.15 release! UI makes a great difference.
>
> I started launching deployment tests with 4.15 and I just noticed that the
> manager is running out of memory were 4.14 did not.
>
> This is in a test environment where the manager host resources are
> constrained yet 100% functional in 4.14.
>
> Anybody experiencing this?
>
> ● cloudstack-management.service - CloudStack Management Server
> Loaded: loaded (/lib/systemd/system/cloudstack-management.service;
> enabled; vendor preset: enabled)
> Drop-In: /etc/systemd/system/cloudstack-management.service.d
> └─mesh-dep.conf, mysql-dep.conf
> Active: failed (Result: exit-code) since Tue 2021-01-19 11:02:44 EST; 15h
> ago
> Process: 6641 ExecStart=/usr/bin/java $JAVA_DEBUG $JAVA_OPTS -cp $CLASSPATH
> $BOOTSTRAP_CLASS (code=exited, status=1/FAILURE)
> Main PID: 6641 (code=exited, status=1/FAILURE)
>
> Jan 19 11:02:40 manager java[6641]: INFO [c.c.c.ConsoleProxyManagerImpl]
> (AgentTaskPool-8:ctx-71ff758f) (logid:58f404cb) Console p
> Jan 19 11:02:40 manager java[6641]: WARN [c.c.a.m.AgentManagerImpl]
> (AgentConnectTaskPool-2683:ctx-abc795fe) (logid:c4046d5d) Reso
> Jan 19 11:02:44 manager java[6641]: OpenJDK 64-Bit Server VM warning: INFO:
> os::commit_memory(0x000000009ea80000, 26738688, 0) fail
> Jan 19 11:02:44 manager java[6641]: #
> Jan 19 11:02:44 manager java[6641]: # There is insufficient memory for the
> Java Runtime Environment to continue.
> Jan 19 11:02:44 manager java[6641]: # Native memory allocation (mmap) failed
> to map 26738688 bytes for committing reserved memory.
> Jan 19 11:02:44 manager java[6641]: # An error report file with more
> information is saved as:
> Jan 19 11:02:44 manager java[6641]: #
> /var/log/cloudstack/management/cloudstack-management.err
> Jan 19 11:02:44 manager systemd[1]: cloudstack-management.service: Main
> process exited, code=exited, status=1/FAILURE
> Jan 19 11:02:44 manager systemd[1]: cloudstack-management.service: Failed
> with result 'exit-code'.
>
>
> [email protected]
> http://www.shapeblue.com
> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
> @shapeblue
>
>
>
>
