[ https://issues.apache.org/jira/browse/CLOUDSTACK-6016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Balineni reassigned CLOUDSTACK-6016: ------------------------------------------- Assignee: Suresh Balineni > MS: MS out of memory java.lang.OutOfMemoryError: PermGen space > -------------------------------------------------------------- > > Key: CLOUDSTACK-6016 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-6016 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Contrail, Management Server, XenServer > Affects Versions: 4.2.1 > Environment: Master with Contrail and Xen. > Reporter: Parth Jagirdar > Assignee: Suresh Balineni > Priority: Critical > Fix For: 4.2.1 > > Attachments: management-server_attachment.log.bz2 > > > Detailed logs are attached .. > +----+---------+--------------------------------------+-------------------+------+----------------+--------+------------+---------------+----------------+----------------+-----------+--------------------------------------------------------+---------------------+---------+-------------+--------+-----------------------+---------+------------+---------+---------------+ > | id | name | uuid | pool_type | > port | data_center_id | pod_id | cluster_id | used_bytes | capacity_bytes > | host_address | user_info | path > | created | removed | update_time | status | > storage_provider_name | scope | hypervisor | managed | capacity_iops | > +----+---------+--------------------------------------+-------------------+------+----------------+--------+------------+---------------+----------------+----------------+-----------+--------------------------------------------------------+---------------------+---------+-------------+--------+-----------------------+---------+------------+---------+---------------+ > | 16 | primary | 15027b8a-bb16-3f01-b5af-e980a0060f25 | NetworkFilesystem | > 2049 | 1 | 1 | 1 | 4029850058752 | 11810778316800 > | 10.223.110.232 | NULL | > /export/home/parth/contrail/cloudstack_248_196/primary | 2014-01-31 20:53:34 > | NULL | NULL | Up | DefaultPrimary | CLUSTER | NULL > | 0 | NULL | > +----+---------+--------------------------------------+-------------------+------+----------------+--------+------------+---------------+----------------+----------------+-----------+--------------------------------------------------------+---------------------+---------+-------------+--------+-----------------------+---------+------------+---------+---------------+ > 1 row in set (0.01 sec) > mysql> select * from storage_pool_view; > +----+--------------------------------------+---------+--------+--------------------------------------------------------+-------------------+----------------+---------------------+---------+----------------+---------------+---------+------------+-----------------------+------------+--------------------------------------+----------------+--------------+----------------+--------------------------------------+------------------+------------------+--------+--------------------------------------+-----------+------+--------------------+------------------------+--------+----------+------------+----------------+ > | id | uuid | name | status | path > | pool_type | host_address > | created | removed | capacity_bytes | capacity_iops | scope | > hypervisor | storage_provider_name | cluster_id | cluster_uuid > | cluster_name | cluster_type | data_center_id | data_center_uuid > | data_center_name | data_center_type | pod_id | pod_uuid > | pod_name | tag | disk_used_capacity | > disk_reserved_capacity | job_id | job_uuid | job_status | job_account_id | > +----+--------------------------------------+---------+--------+--------------------------------------------------------+-------------------+----------------+---------------------+---------+----------------+---------------+---------+------------+-----------------------+------------+--------------------------------------+----------------+--------------+----------------+--------------------------------------+------------------+------------------+--------+--------------------------------------+-----------+------+--------------------+------------------------+--------+----------+------------+----------------+ > | 16 | 15027b8a-bb16-3f01-b5af-e980a0060f25 | primary | Up | > /export/home/parth/contrail/cloudstack_248_196/primary | NetworkFilesystem | > 10.223.110.232 | 2014-01-31 20:53:34 | NULL | 11810778316800 | > NULL | CLUSTER | NULL | DefaultPrimary | 1 | > 878b925d-9373-4dc5-88f2-9a83d6d924b2 | demo-pod2-xen1 | CloudManaged | > 1 | 8d2fe185-7f85-45e0-b523-cf318c66e799 | default | Advanced > | 1 | fdd9ed6b-3618-4f9f-afe1-191eb290da22 | demo-pod2 | NULL | > 0 | 0 | NULL | NULL | NULL | > NULL | > +----+--------------------------------------+---------+--------+--------------------------------------------------------+-------------------+----------------+---------------------+---------+----------------+---------------+---------+------------+-----------------------+------------+--------------------------------------+----------------+--------------+----------------+--------------------------------------+------------------+------------------+--------+--------------------------------------+-----------+------+--------------------+------------------------+--------+----------+------------+----------------+ > 1 row in set (0.00 sec) > 2014-02-02 14:22:33,377 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] > (secstorage-1:ctx-61d52c4c) template 1 > is already in store:16, type:Primary > 2014-02-02 14:22:33,379 DEBUG [o.a.c.s.v.VolumeServiceImpl] > (secstorage-1:ctx-61d52c4c) Found template ro > uting-1 in storage pool 16 with VMTemplateStoragePool id: 11868 > 2014-02-02 14:22:37,360 INFO [o.a.c.s.v.VolumeServiceImpl] > (consoleproxy-1:ctx-c605fca1) Unable to acqui > re lock on VMTemplateStoragePool 11867 > 2014-02-02 14:22:49,487 DEBUG [c.c.s.StatsCollector] > (StatsCollector-2:ctx-b70366de) VmStatsCollector is > running... > 2014-02-02 14:24:09,967 DEBUG [c.c.s.StatsCollector] > (StatsCollector-1:ctx-f7f09a60) HostStatsCollector i > s running... > 2014-02-02 14:24:09,966 DEBUG [c.c.s.StatsCollector] > (StatsCollector-3:ctx-0ec2cf0e) StorageCollector is > running... > 2014-02-02 14:24:09,966 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-7:ctx-350f3bc0) Seq 2-760414210: > Executing request > 2014-02-02 14:23:58,075 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-333:ctx-b500c0a9) Seq 1-100250419 > 4: Executing request > 2014-02-02 14:23:58,075 DEBUG [c.c.c.ClusterManagerImpl] > (Cluster-Heartbeat-1:ctx-3763fe2a) Management se > rver heartbeat takes too long to finish. profiler: Done. Duration: 75297ms, > profilerHeartbeatUpdate: Done > . Duration: 7251ms, profilerPeerScan: Started but not stopped > 2014-02-02 14:24:34,629 DEBUG [c.c.v.VirtualMachineManagerImpl] > (secstorage-1:ctx-61d52c4c) Cleaning up r > esources for the vm VM[SecondaryStorageVm|s-2968-VM] in Starting state > 2014-02-02 14:24:34,629 DEBUG [c.c.v.VirtualMachineManagerImpl] > (consoleproxy-1:ctx-c605fca1) Cleaning up > resources for the vm VM[ConsoleProxy|v-2-VM] in Starting state > 2014-02-02 14:24:34,887 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-61d52c4c) > Seq 1-1002523328: Sending { > Cmd , MgmtId: 214076393326387, via: 1(xenserver-2), Ver: v1, Flags: 100011, > [{"com.cloud.agent.api.StopCo > mmand":{"isProxy":false,"executeInSequence":false,"vmName":"s-2968-VM","wait":0}}] > } > 2014-02-02 14:24:34,887 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-61d52c4c) > Seq 1-1002523328: Executing: > { Cmd , MgmtId: 214076393326387, via: 1(xenserver-2), Ver: v1, Flags: > 100011, [{"com.cloud.agent.api.Sto > pCommand":{"isProxy":false,"executeInSequence":false,"vmName":"s-2968-VM","wait":0}}] > } > 2014-02-02 14:24:39,397 DEBUG [c.c.a.t.Request] (consoleproxy-1:ctx-c605fca1) > Seq 2-760433604: Sending { > Cmd , MgmtId: 214076393326387, via: 2(xenserver-1), Ver: v1, Flags: 100011, > [{"com.cloud.agent.api.StopC > ommand":{"isProxy":false,"executeInSequence":false,"vmName":"v-2-VM","wait":0}}] > } > 2014-02-02 14:24:39,397 DEBUG [c.c.a.t.Request] (consoleproxy-1:ctx-c605fca1) > Seq 2-760433604: Executing: > { Cmd , MgmtId: 214076393326387, via: 2(xenserver-1), Ver: v1, Flags: > 100011, [{"com.cloud.agent.api.St > opCommand":{"isProxy":false,"executeInSequence":false,"vmName":"v-2-VM","wait":0}}] > } > 2014-02-02 14:24:39,414 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] > (RouterStatusMonitor-1:ctx-b0e > c7c97) Found 0 routers to update status. > 2014-02-02 14:24:47,366 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-153:ctx-bd2d4c7f) Seq 1-100252332 > 8: Executing request > 2014-02-02 14:24:47,366 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-275:ctx-d43e2414) Seq 2-760433604 > : Executing request > 2014-02-02 14:24:51,658 ERROR [c.c.s.StatsCollector] > (StatsCollector-3:ctx-0ec2cf0e) Error trying to retr > ieve storage stats > java.lang.OutOfMemoryError: PermGen space > 2014-02-02 14:24:59,201 ERROR [o.a.c.f.j.i.AsyncJobManagerImpl] > (AsyncJobMgr-Heartbeat-1:ctx-4ca44bf9) Unexpected exception when trying to > execute queue item, > java.lang.OutOfMemoryError: PermGen space > 2014-02-02 14:24:59,215 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-7:ctx-350f3bc0) Seq 2-760414210: Response Received: > 2014-02-02 14:24:59,215 DEBUG [c.c.a.t.Request] (DirectAgent-7:ctx-350f3bc0) > Seq 2-760414210: Processing: { Ans: , MgmtId: 214076393326387, via: 2, Ver: > v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] } > 2014-02-02 14:24:59,216 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-7:ctx-de693d5b) Seq 2-760414210: Executing request > 2014-02-02 14:25:11,353 ERROR [c.c.s.StatsCollector] > (StatsCollector-1:ctx-f7f09a60) Error trying to retrieve host stats > java.lang.OutOfMemoryError: PermGen space > 2014-02-02 14:25:39,109 DEBUG [o.a.c.n.c.m.ContrailManager] > (DBSyncTimer:null) DB Sync task is running > 2014-02-02 14:25:39,109 DEBUG [o.a.c.n.c.m.ServerDBSync] (DBSyncTimer:null) > syncing cloudstack db with v -- This message was sent by Atlassian JIRA (v6.2#6252)