Re: [Users] SPM is always contending
Hi Andy, I saw in another thread that you have resolved the issue, but I would like to further investigate the cause. Can you please send an sos report as soon as possible? Thanks, Yeela - Original Message - > From: "Andy Singleton" > To: users@ovirt.org > Sent: Friday, April 5, 2013 12:07:24 PM > Subject: [Users] SPM is always contending > > After the node acting as spm was (accidentally) put into maintenance > mode, the spm role is always contending. The original node doesnt take > the role back either. > About 10 instances were still running on the node when it was put into > maintenance. > > I have ovirt 3.1.0-4 with a targetcli iscsi store. > > Here are the contents of the engine.log. > > Andy > > 2013-04-04 15:25:23,979 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] > (QuartzScheduler_Worker-96) [5449512a] spmStart polling ended. spm status: > Free > 2013-04-04 15:25:23,993 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] > (QuartzScheduler_Worker-96) [5449512a] START, HSMClearTaskVDSCommand(vd > sId = d265245e-3cc5-11e2-bce7-001018fc3b14, > taskId=7c1cf7c6-21d4-4504-882d-3d465742b6ef), log id: d2da8ba > 2013-04-04 15:25:24,011 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] > (QuartzScheduler_Worker-96) [5449512a] FINISH, HSMClearTaskVDSCommand, > log id: d2da8ba > 2013-04-04 15:25:24,012 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] > (QuartzScheduler_Worker-96) [5449512a] FINISH, SpmStartVDSCommand, return: > org.ovirt.engine.core.common.businessentities.SpmStatusResult@30a05218, > log id: 5980d6f4 > 2013-04-04 15:25:24,014 INFO > [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] > (QuartzScheduler_Worker-96) [6d6dbe98] Running command: SetStoragePoolStat > usCommand internal: true. Entities affected : ID: > 0ba15357-9b3b-4a76-8dac-b2b66b922174 Type: StoragePool > 2013-04-04 15:25:24,843 ERROR > [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] > (QuartzScheduler_Worker-96) [6d6dbe98] > IrsBroker::Failed::GetStoragePoolInfoV > DS due to: IrsSpmStartFailedException: IRSGenericException: > IRSErrorException: SpmStart failed > 2013-04-04 15:25:24,900 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] > (QuartzScheduler_Worker-96) [6d6dbe98] Irs placed on server null failed. > Proc > eed Failover > 2013-04-04 15:25:34,977 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] > (QuartzScheduler_Worker-96) [6d6dbe98] hostFromVds::selectedVds - > moon-palace > , spmStatus Free, storage pool Primary > 2013-04-04 15:25:35,806 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] > (QuartzScheduler_Worker-96) [6d6dbe98] starting spm on vds moon-palace, > stora > ge pool Primary, prevId -1, LVER 2520 > 2013-04-04 15:25:35,851 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] > (QuartzScheduler_Worker-96) [6d6dbe98] START, SpmStartVDSCommand(vdsId = 3d > 88c8b0-84bc-11e2-96b1-001018fc3b14, storagePoolId = > 0ba15357-9b3b-4a76-8dac-b2b66b922174, prevId=-1, prevLVER=2520, > storagePoolFormatType=V2, recoveryMode=Manual, SCSIF > encing=false), log id: 30f4f64e > 2013-04-04 15:25:45,901 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] > (QuartzScheduler_Worker-96) [6d6dbe98] spmStart polling started: taskId = 6 > 8002cff-478d-4c2c-8839-b935d8a858b1 > 2013-04-04 15:26:11,903 INFO [org.ovirt.engine.core.bll.VdsLoadBalancer] > (QuartzScheduler_Worker-81) [45db3aaa] VdsLoadBalancer: Starting load > balance for cluster: BP_ > Primary, algorithm: EvenlyDistribute. > 2013-04-04 15:26:12,807 INFO [org.ovirt.engine.core.bll.VdsLoadBalancer] > (QuartzScheduler_Worker-81) [45db3aaa] VdsLoadBalancer: high util: 51, > low util: 0, duration: > 2, threashold: 80 > 2013-04-04 15:26:12,845 INFO > [org.ovirt.engine.core.bll.VdsLoadBalancingAlgorithm] > (QuartzScheduler_Worker-81) [45db3aaa] VdsLoadBalancer: number of > relevant vdss (no > migration, no pending): 3. > 2013-04-04 15:26:12,846 INFO > [org.ovirt.engine.core.bll.VdsCpuVdsLoadBalancingAlgorithm] > (QuartzScheduler_Worker-81) [45db3aaa] VdsLoadBalancer: number of over > utilize > d vdss found: 0. > 2013-04-04 15:26:12,846 INFO > [org.ovirt.engine.core.bll.VdsCpuVdsLoadBalancingAlgorithm] > (QuartzScheduler_Worker-81) [45db3aaa] VdsLoadBalancer: max cpu limit: > 40, num > ber of ready to migration vdss: 3 > 2013-04-04 15:26:19,024 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] > (QuartzScheduler_Worker-96) [6d6dbe98] spmStart polling ended: taskId = 680 > 02cff-478d-4c2c-8839-b935d8a858b1 task status = finished > 2013-
Re: [Users] SPM is always contending - resolved
The primary is iSCSI. On 04/07/2013 03:59 AM, Shu Ming wrote: Which type of storage domain was in your engine? NFS? iSCSI? or other types? Andy Singleton: Problem resolved. The clue was obviously in the engine.log (FSCK error) but I couldn't see what was needing an fsck or why this would prevent the storage of every vm from being mounted. I still dont understand the whole of the ovirt storage mechanism. More information was in the node vdsm.log - It pointed here. /dev/mapper/9ed5a2fe--ae4c--433f--a15f--2c710ed5156f-master One fsck later problem resolved. Thanks Andy ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] SPM is always contending - resolved
Which type of storage domain was in your engine? NFS? iSCSI? or other types? Andy Singleton: Problem resolved. The clue was obviously in the engine.log (FSCK error) but I couldn't see what was needing an fsck or why this would prevent the storage of every vm from being mounted. I still dont understand the whole of the ovirt storage mechanism. More information was in the node vdsm.log - It pointed here. /dev/mapper/9ed5a2fe--ae4c--433f--a15f--2c710ed5156f-master One fsck later problem resolved. Thanks Andy ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users -- --- 舒明 Shu Ming Open Virtualization Engineerning; CSTL, IBM Corp. Tel: 86-10-82451626 Tieline: 9051626 E-mail: shum...@cn.ibm.com or shum...@linux.vnet.ibm.com Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193, PRC ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] SPM is always contending - resolved
Problem resolved. The clue was obviously in the engine.log (FSCK error) but I couldn't see what was needing an fsck or why this would prevent the storage of every vm from being mounted. I still dont understand the whole of the ovirt storage mechanism. More information was in the node vdsm.log - It pointed here. /dev/mapper/9ed5a2fe--ae4c--433f--a15f--2c710ed5156f-master One fsck later problem resolved. Thanks Andy ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[Users] SPM is always contending
After the node acting as spm was (accidentally) put into maintenance mode, the spm role is always contending. The original node doesnt take the role back either. About 10 instances were still running on the node when it was put into maintenance. I have ovirt 3.1.0-4 with a targetcli iscsi store. Here are the contents of the engine.log. Andy 2013-04-04 15:25:23,979 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [5449512a] spmStart polling ended. spm status: Free 2013-04-04 15:25:23,993 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) [5449512a] START, HSMClearTaskVDSCommand(vd sId = d265245e-3cc5-11e2-bce7-001018fc3b14, taskId=7c1cf7c6-21d4-4504-882d-3d465742b6ef), log id: d2da8ba 2013-04-04 15:25:24,011 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) [5449512a] FINISH, HSMClearTaskVDSCommand, log id: d2da8ba 2013-04-04 15:25:24,012 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [5449512a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@30a05218, log id: 5980d6f4 2013-04-04 15:25:24,014 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-96) [6d6dbe98] Running command: SetStoragePoolStat usCommand internal: true. Entities affected : ID: 0ba15357-9b3b-4a76-8dac-b2b66b922174 Type: StoragePool 2013-04-04 15:25:24,843 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [6d6dbe98] IrsBroker::Failed::GetStoragePoolInfoV DS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed 2013-04-04 15:25:24,900 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [6d6dbe98] Irs placed on server null failed. Proc eed Failover 2013-04-04 15:25:34,977 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [6d6dbe98] hostFromVds::selectedVds - moon-palace , spmStatus Free, storage pool Primary 2013-04-04 15:25:35,806 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [6d6dbe98] starting spm on vds moon-palace, stora ge pool Primary, prevId -1, LVER 2520 2013-04-04 15:25:35,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [6d6dbe98] START, SpmStartVDSCommand(vdsId = 3d 88c8b0-84bc-11e2-96b1-001018fc3b14, storagePoolId = 0ba15357-9b3b-4a76-8dac-b2b66b922174, prevId=-1, prevLVER=2520, storagePoolFormatType=V2, recoveryMode=Manual, SCSIF encing=false), log id: 30f4f64e 2013-04-04 15:25:45,901 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [6d6dbe98] spmStart polling started: taskId = 6 8002cff-478d-4c2c-8839-b935d8a858b1 2013-04-04 15:26:11,903 INFO [org.ovirt.engine.core.bll.VdsLoadBalancer] (QuartzScheduler_Worker-81) [45db3aaa] VdsLoadBalancer: Starting load balance for cluster: BP_ Primary, algorithm: EvenlyDistribute. 2013-04-04 15:26:12,807 INFO [org.ovirt.engine.core.bll.VdsLoadBalancer] (QuartzScheduler_Worker-81) [45db3aaa] VdsLoadBalancer: high util: 51, low util: 0, duration: 2, threashold: 80 2013-04-04 15:26:12,845 INFO [org.ovirt.engine.core.bll.VdsLoadBalancingAlgorithm] (QuartzScheduler_Worker-81) [45db3aaa] VdsLoadBalancer: number of relevant vdss (no migration, no pending): 3. 2013-04-04 15:26:12,846 INFO [org.ovirt.engine.core.bll.VdsCpuVdsLoadBalancingAlgorithm] (QuartzScheduler_Worker-81) [45db3aaa] VdsLoadBalancer: number of over utilize d vdss found: 0. 2013-04-04 15:26:12,846 INFO [org.ovirt.engine.core.bll.VdsCpuVdsLoadBalancingAlgorithm] (QuartzScheduler_Worker-81) [45db3aaa] VdsLoadBalancer: max cpu limit: 40, num ber of ready to migration vdss: 3 2013-04-04 15:26:19,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [6d6dbe98] spmStart polling ended: taskId = 680 02cff-478d-4c2c-8839-b935d8a858b1 task status = finished 2013-04-04 15:26:19,024 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [6d6dbe98] Start SPM Task failed - result: clea nSuccess, message: VDSGenericException: VDSErrorException: Failed in vdscommand to HSMGetTaskStatusVDS, error = BlockSD master file system FSCK error This log sequence repeats indefinitely ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users