We prevent starting VMs with illegal images[1]

You can use "$ vdsm-tool dump-volume-chains"
to look for illegal images and then look in the engine log for the reason
they became illagal,

if it's something like this, it usually means you can remove them:
63696:2018-06-15 09:41:58,134+01 ERROR
[org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand]
(DefaultQuartzScheduler2) [6fa97ea4-8f61-4a48-8e08-a8bb1b9de826] Merging of
snapshot 'e609d6cc-2025-4cf0-ad34-03519131cdd1' images
'1d01c6c8-b61e-42bc-a054-f04c3f792b10'..'ef6f732e-2a7a-4a14-a10f-bcc88bdd805f'
failed. Images have been marked illegal and can no longer be previewed or
reverted to. Please retry Live Merge on the snapshot to complete the
operation.


On Mon, Jun 18, 2018 at 5:46 PM, <nico...@devels.es> wrote:

> Indeed, when the problem started I think the SPM was the host I added as
> VDSM log in the first e-mail. Currently it is the one I sent in the second
> mail.
>
> FWIW, if it helps to debug more fluently, we can provide VPN access to our
> infrastructure so you can access and see whateve you need (all hosts, DB,
> etc...).
>
> Right now the machines that keep running work, but once shut down they
> start showing the problem below...
>
> Thank you
>
>
> El 2018-06-18 15:20, Benny Zlotnik escribió:
>
>> I'm having trouble following the errors, I think the SPM changed or
>> the vdsm log from the right host might be missing.
>>
>> However, I believe what started the problems is this transaction
>> timeout:
>>
>> 2018-06-15 14:20:51,378+01 ERROR
>> [org.ovirt.engine.core.bll.tasks.CommandAsyncTask]
>> (org.ovirt.thread.pool-6-thread-29)
>> [1db468cb-85fd-4189-b356-d31781461504] [within thread]: endAction for
>> action type RemoveSnapshotSingleDisk threw an exception.:
>> org.springframework.jdbc.CannotGetJdbcConnectionException: Could not
>> get JDBC Connection; nested exception is java.sql.SQLException:
>> javax.resource.ResourceException: IJ000460: Error checking for a
>> transaction
>>  at
>> org.springframework.jdbc.datasource.DataSourceUtils.getConne
>> ction(DataSourceUtils.java:80)
>> [spring-jdbc.jar:4.2.4.RELEASE]
>>  at
>> org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:615)
>> [spring-jdbc.jar:4.2.4.RELEASE]
>>  at
>> org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:680)
>> [spring-jdbc.jar:4.2.4.RELEASE]
>>  at
>> org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:712)
>> [spring-jdbc.jar:4.2.4.RELEASE]
>>  at
>> org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:762)
>> [spring-jdbc.jar:4.2.4.RELEASE]
>>  at
>> org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$P
>> ostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDi
>> alect.java:152)
>> [dal.jar:]
>>
>> This looks like a bug
>>
>> Regardless, I am not sure restoring a backup would help since you
>> probably have orphaned images on the storage which need to be removed
>>
>> Adding Ala
>>
>> On Mon, Jun 18, 2018 at 4:19 PM, <nico...@devels.es> wrote:
>>
>> Hi Benny,
>>>
>>> Please find the SPM logs at [1].
>>>
>>> Thank you
>>>
>>>   [1]:
>>>
>>> https://wetransfer.com/downloads/62bf649462aabbc2ef21824682b
>> 0a08320180618131825/036b7782f58d337baf909a7220d8455320180618131825/5550ee
>>
>>> [1]
>>>
>>> El 2018-06-18 13:19, Benny Zlotnik escribió:
>>> Can you send the SPM logs as well?
>>>
>>> On Mon, Jun 18, 2018 at 1:13 PM, <nico...@devels.es> wrote:
>>>
>>> Hi Benny,
>>>
>>> Please find the logs at [1].
>>>
>>> Thank you.
>>>
>>>   [1]:
>>>
>>>
>>> https://wetransfer.com/downloads/12208fb4a6a5df3114bbbc10af1
>> 94c8820180618101223/647c066b7b91096570def304da86dbca20180618101223/583d3d
>>
>>> [2]
>>>
>>> [1]
>>>
>>> El 2018-06-18 09:28, Benny Zlotnik escribió:
>>>
>>> Can you provide full engine and vdsm logs?
>>>
>>> On Mon, Jun 18, 2018 at 11:20 AM, <nico...@devels.es> wrote:
>>>
>>> Hi,
>>>
>>> We're running oVirt 4.1.9 (we cannot upgrade at this time) and
>>> we're having a major problem in our infrastructure. On friday, a
>>> snapshots were automatically created on more than 200 VMs and as
>>> this was just a test task, all of them were deleted at the same
>>> time, which seems to have corrupted several VMs.
>>>
>>> When trying to delete a snapshot on some of the VMs, a "General
>>> error" is thrown with a NullPointerException in the engine log
>>> (attached).
>>>
>>> But the worst part is that when some of these machines is powered
>>> off and then powered on, the VMs are corrupt...
>>>
>>> VM myvm is down with error. Exit message: Bad volume specification
>>> {u'index': 0, u'domainID': u'110ea376-d789-40a1-b9f6-6b40c31afe01',
>>> 'reqsize': '0', u'format': u'cow', u'bootOrder': u'1', u'address':
>>> {u'function': u'0x0', u'bus': u'0x00', u'domain': u'0x0000',
>>> u'type': u'pci', u'slot': u'0x06'}, u'volumeID':
>>> u'1fd0f9aa-6505-45d2-a17e-859bd5dd4290', 'apparentsize':
>>> '23622320128', u'imageID': u'65519220-68e1-462a-99b3-f0763c78eae2',
>>> u'discard': False, u'specParams': {}, u'readonly': u'false',
>>> u'iface': u'virtio', u'optional': u'false', u'deviceId':
>>> u'65519220-68e1-462a-99b3-f0763c78eae2', 'truesize': '23622320128',
>>> u'poolID': u'75bf8f48-970f-42bc-8596-f8ab6efb2b63', u'device':
>>> u'disk', u'shared': u'false', u'propagateErrors': u'off', u'type':
>>> u'disk'}.
>>>
>>> We're really frustrated by now and don't know how to procceed... We
>>> have a DB backup (with engine-backup) from thursday which would
>>> have
>>> a "sane" DB definition without all the snapshots, as they were all
>>> created on friday. Would it be safe to restore this backup?
>>>
>>> Any help is really appreciated...
>>>
>>> Thanks.
>>> _______________________________________________
>>> Users mailing list -- users@ovirt.org
>>> To unsubscribe send an email to users-le...@ovirt.org
>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ [3]
>>> [2]
>>> [1]
>>> oVirt Code of Conduct:
>>> https://www.ovirt.org/community/about/community-guidelines/ [4] [3]
>>> [2]
>>> List Archives:
>>>
>>>
>>> https://lists.ovirt.org/archives/list/users@ovirt.org/messag
>> e/P5OOGBL3BRZIQ2I46FYELBUIIWT5QK4C/
>>
>>> [5]
>>> [4]
>>> [3]
>>>
>>> Links:
>>> ------
>>> [1] https://www.ovirt.org/site/privacy-policy/ [3] [2]
>>> [2] https://www.ovirt.org/community/about/community-guidelines/ [4]
>>> [3]
>>> [3]
>>>
>>>
>>> https://lists.ovirt.org/archives/list/users@ovirt.org/messag
>> e/P5OOGBL3BRZIQ2I46FYELBUIIWT5QK4C/
>>
>>> [5]
>>> [4]
>>>
>>> Links:
>>> ------
>>> [1]
>>>
>>> https://wetransfer.com/downloads/12208fb4a6a5df3114bbbc10af1
>> 94c8820180618101223/647c066b7b91096570def304da86dbca20180618101223/583d3d
>>
>>> [2]
>>> [2] https://www.ovirt.org/site/privacy-policy/ [3]
>>> [3] https://www.ovirt.org/community/about/community-guidelines/ [4]
>>> [4]
>>>
>>> https://lists.ovirt.org/archives/list/users@ovirt.org/messag
>> e/P5OOGBL3BRZIQ2I46FYELBUIIWT5QK4C/
>>
>>> [5]
>>>
>>
>>
>>
>> Links:
>> ------
>> [1]
>> https://wetransfer.com/downloads/62bf649462aabbc2ef21824682b
>> 0a08320180618131825/036b7782f58d337baf909a7220d8455320180618131825/5550ee
>> [2]
>> https://wetransfer.com/downloads/12208fb4a6a5df3114bbbc10af1
>> 94c8820180618101223/647c066b7b91096570def304da86dbca20180618101223/583d3d
>> [3] https://www.ovirt.org/site/privacy-policy/
>> [4] https://www.ovirt.org/community/about/community-guidelines/
>> [5]
>> https://lists.ovirt.org/archives/list/users@ovirt.org/messag
>> e/P5OOGBL3BRZIQ2I46FYELBUIIWT5QK4C/
>>
>
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/LPTPCZUCLLIMBHWHTZQW5CJ3PM5PVZFI/

Reply via email to