Re: [ovirt-users] Cannot start hosted engine after last 4.0.1 packages
As I have an oob network I change my routing to see if it wanted to upgrade packages, it seemed that I already was on 4.0.2 when this occured. The HE started in some strange way on one host, that host was even Non Operational but had at least the ovirtmgmt network, I changed something there earlier which was never an issue for the time being. I will investigate now if I can change the Maintenance states again because they seem to fail. 2016-08-18 20:59 GMT+02:00 Matt . : > It takes a while and this is the status output of systemctl after it, > see attachment. > > > > 2016-08-18 17:40 GMT+02:00 Simone Tiraboschi : >> On Thu, Aug 18, 2016 at 5:28 PM, Matt . wrote: >>> Sure, It's in the attachment as screenshot. >> >> can you please try with >> systemctl restart sanlock --force >> ? >> >>> 2016-08-18 17:13 GMT+02:00 Simone Tiraboschi : On Thu, Aug 18, 2016 at 5:05 PM, Matt . wrote: > There is not storage issue, there is plenty of space, around 900G and > this is shown in the past as the same issue. > > What is strange is that I cannot change the maintenance status of the > hosts, when I set it to whatever status it's not changed and the > status are various it seems from Global to Local. > > So I'm stuck. can you please post the output of sanlock client status ? > 2016-08-18 16:41 GMT+02:00 Nir Soffer : >> On Thu, Aug 18, 2016 at 5:33 PM, Matt . wrote: >>> OK, it seems to be an issue related to something like a sanlock which >>> I cannot solve: >>> >>> vdsm "Failed to acquire lock: No space left on device" is what happens >>> when I restart vdsmd and even removing the __DIRECT_IO_TEST__ file is >>> newly created on a vdsmd restart. >> >> __DIRECT_IO_TEST__ is empty file, used to check if vdsm can do direct I/O >> to this storage, there is no need to remove it. >> >> "No space left on device" seems to be your problem, you should make some >> space if you want to use this storage domain. >> >>> >>> As I'm not able to change the maintenance mode on the hosts I need to >>> figure out something else. >>> >>> >>> >>> >>> >>> 2016-08-18 10:40 GMT+02:00 Matt . : OK nice to know, I never saw that message before. I cannot do anything anymore with hosted engine. It doesn't change the maintenance state, it doesn't start the vm and this happens on all hosts. Nothing else changed then just the ovirt* and vdsm* package upgrades. The HE storage is also mounted and the agent.log shows it's information as normal. Showing logs is kinda an issue as I'm bound to ILO/Drac at the moment. 2016-08-18 10:18 GMT+02:00 Simone Tiraboschi : > On Thu, Aug 18, 2016 at 10:06 AM, Matt . > wrote: >> Hello, >> >> I'm having issues after the last 4.0.1 package just before 4.0.2 came >> out. My engine was running great, hosts started the vm also and the >> hosted-engine command was working fine. >> >> Now I get rpcxml 3.6 deprecated warning which I never got before. See >> the attachment for this output. > > That one it's just a warning and we are going to fix the root cause > for the next version, it's absolutely harmless. > >> I have checked is IPv6 was disabled, but it isn't as that is what is >> needed for this. >> >> Any clue ? > > If you have any issue, it's somewhere else. > What are the symptoms? > >> Thanks! >> >> Matt >> >> ___ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> >>> ___ >>> Users mailing list >>> Users@ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Cannot start hosted engine after last 4.0.1 packages
It takes a while and this is the status output of systemctl after it, see attachment. 2016-08-18 17:40 GMT+02:00 Simone Tiraboschi : > On Thu, Aug 18, 2016 at 5:28 PM, Matt . wrote: >> Sure, It's in the attachment as screenshot. > > can you please try with > systemctl restart sanlock --force > ? > >> 2016-08-18 17:13 GMT+02:00 Simone Tiraboschi : >>> On Thu, Aug 18, 2016 at 5:05 PM, Matt . wrote: There is not storage issue, there is plenty of space, around 900G and this is shown in the past as the same issue. What is strange is that I cannot change the maintenance status of the hosts, when I set it to whatever status it's not changed and the status are various it seems from Global to Local. So I'm stuck. >>> >>> can you please post the output of >>> sanlock client status >>> ? >>> 2016-08-18 16:41 GMT+02:00 Nir Soffer : > On Thu, Aug 18, 2016 at 5:33 PM, Matt . wrote: >> OK, it seems to be an issue related to something like a sanlock which >> I cannot solve: >> >> vdsm "Failed to acquire lock: No space left on device" is what happens >> when I restart vdsmd and even removing the __DIRECT_IO_TEST__ file is >> newly created on a vdsmd restart. > > __DIRECT_IO_TEST__ is empty file, used to check if vdsm can do direct I/O > to this storage, there is no need to remove it. > > "No space left on device" seems to be your problem, you should make some > space if you want to use this storage domain. > >> >> As I'm not able to change the maintenance mode on the hosts I need to >> figure out something else. >> >> >> >> >> >> 2016-08-18 10:40 GMT+02:00 Matt . : >>> OK nice to know, I never saw that message before. >>> >>> I cannot do anything anymore with hosted engine. It doesn't change the >>> maintenance state, it doesn't start the vm and this happens on all >>> hosts. Nothing else changed then just the ovirt* and vdsm* package >>> upgrades. >>> >>> The HE storage is also mounted and the agent.log shows it's >>> information as normal. >>> >>> Showing logs is kinda an issue as I'm bound to ILO/Drac at the moment. >>> >>> >>> >>> >>> 2016-08-18 10:18 GMT+02:00 Simone Tiraboschi : On Thu, Aug 18, 2016 at 10:06 AM, Matt . wrote: > Hello, > > I'm having issues after the last 4.0.1 package just before 4.0.2 came > out. My engine was running great, hosts started the vm also and the > hosted-engine command was working fine. > > Now I get rpcxml 3.6 deprecated warning which I never got before. See > the attachment for this output. That one it's just a warning and we are going to fix the root cause for the next version, it's absolutely harmless. > I have checked is IPv6 was disabled, but it isn't as that is what is > needed for this. > > Any clue ? If you have any issue, it's somewhere else. What are the symptoms? > Thanks! > > Matt > > ___ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > >> ___ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Cannot start hosted engine after last 4.0.1 packages
On Thu, Aug 18, 2016 at 5:28 PM, Matt . wrote: > Sure, It's in the attachment as screenshot. can you please try with systemctl restart sanlock --force ? > 2016-08-18 17:13 GMT+02:00 Simone Tiraboschi : >> On Thu, Aug 18, 2016 at 5:05 PM, Matt . wrote: >>> There is not storage issue, there is plenty of space, around 900G and >>> this is shown in the past as the same issue. >>> >>> What is strange is that I cannot change the maintenance status of the >>> hosts, when I set it to whatever status it's not changed and the >>> status are various it seems from Global to Local. >>> >>> So I'm stuck. >> >> can you please post the output of >> sanlock client status >> ? >> >>> 2016-08-18 16:41 GMT+02:00 Nir Soffer : On Thu, Aug 18, 2016 at 5:33 PM, Matt . wrote: > OK, it seems to be an issue related to something like a sanlock which > I cannot solve: > > vdsm "Failed to acquire lock: No space left on device" is what happens > when I restart vdsmd and even removing the __DIRECT_IO_TEST__ file is > newly created on a vdsmd restart. __DIRECT_IO_TEST__ is empty file, used to check if vdsm can do direct I/O to this storage, there is no need to remove it. "No space left on device" seems to be your problem, you should make some space if you want to use this storage domain. > > As I'm not able to change the maintenance mode on the hosts I need to > figure out something else. > > > > > > 2016-08-18 10:40 GMT+02:00 Matt . : >> OK nice to know, I never saw that message before. >> >> I cannot do anything anymore with hosted engine. It doesn't change the >> maintenance state, it doesn't start the vm and this happens on all >> hosts. Nothing else changed then just the ovirt* and vdsm* package >> upgrades. >> >> The HE storage is also mounted and the agent.log shows it's >> information as normal. >> >> Showing logs is kinda an issue as I'm bound to ILO/Drac at the moment. >> >> >> >> >> 2016-08-18 10:18 GMT+02:00 Simone Tiraboschi : >>> On Thu, Aug 18, 2016 at 10:06 AM, Matt . wrote: Hello, I'm having issues after the last 4.0.1 package just before 4.0.2 came out. My engine was running great, hosts started the vm also and the hosted-engine command was working fine. Now I get rpcxml 3.6 deprecated warning which I never got before. See the attachment for this output. >>> >>> That one it's just a warning and we are going to fix the root cause >>> for the next version, it's absolutely harmless. >>> I have checked is IPv6 was disabled, but it isn't as that is what is needed for this. Any clue ? >>> >>> If you have any issue, it's somewhere else. >>> What are the symptoms? >>> Thanks! Matt ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users > ___ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Cannot start hosted engine after last 4.0.1 packages
Sure, It's in the attachment as screenshot. 2016-08-18 17:13 GMT+02:00 Simone Tiraboschi : > On Thu, Aug 18, 2016 at 5:05 PM, Matt . wrote: >> There is not storage issue, there is plenty of space, around 900G and >> this is shown in the past as the same issue. >> >> What is strange is that I cannot change the maintenance status of the >> hosts, when I set it to whatever status it's not changed and the >> status are various it seems from Global to Local. >> >> So I'm stuck. > > can you please post the output of > sanlock client status > ? > >> 2016-08-18 16:41 GMT+02:00 Nir Soffer : >>> On Thu, Aug 18, 2016 at 5:33 PM, Matt . wrote: OK, it seems to be an issue related to something like a sanlock which I cannot solve: vdsm "Failed to acquire lock: No space left on device" is what happens when I restart vdsmd and even removing the __DIRECT_IO_TEST__ file is newly created on a vdsmd restart. >>> >>> __DIRECT_IO_TEST__ is empty file, used to check if vdsm can do direct I/O >>> to this storage, there is no need to remove it. >>> >>> "No space left on device" seems to be your problem, you should make some >>> space if you want to use this storage domain. >>> As I'm not able to change the maintenance mode on the hosts I need to figure out something else. 2016-08-18 10:40 GMT+02:00 Matt . : > OK nice to know, I never saw that message before. > > I cannot do anything anymore with hosted engine. It doesn't change the > maintenance state, it doesn't start the vm and this happens on all > hosts. Nothing else changed then just the ovirt* and vdsm* package > upgrades. > > The HE storage is also mounted and the agent.log shows it's > information as normal. > > Showing logs is kinda an issue as I'm bound to ILO/Drac at the moment. > > > > > 2016-08-18 10:18 GMT+02:00 Simone Tiraboschi : >> On Thu, Aug 18, 2016 at 10:06 AM, Matt . wrote: >>> Hello, >>> >>> I'm having issues after the last 4.0.1 package just before 4.0.2 came >>> out. My engine was running great, hosts started the vm also and the >>> hosted-engine command was working fine. >>> >>> Now I get rpcxml 3.6 deprecated warning which I never got before. See >>> the attachment for this output. >> >> That one it's just a warning and we are going to fix the root cause >> for the next version, it's absolutely harmless. >> >>> I have checked is IPv6 was disabled, but it isn't as that is what is >>> needed for this. >>> >>> Any clue ? >> >> If you have any issue, it's somewhere else. >> What are the symptoms? >> >>> Thanks! >>> >>> Matt >>> >>> ___ >>> Users mailing list >>> Users@ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >>> ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Cannot start hosted engine after last 4.0.1 packages
On Thu, Aug 18, 2016 at 5:05 PM, Matt . wrote: > There is not storage issue, there is plenty of space, around 900G and > this is shown in the past as the same issue. > > What is strange is that I cannot change the maintenance status of the > hosts, when I set it to whatever status it's not changed and the > status are various it seems from Global to Local. > > So I'm stuck. can you please post the output of sanlock client status ? > 2016-08-18 16:41 GMT+02:00 Nir Soffer : >> On Thu, Aug 18, 2016 at 5:33 PM, Matt . wrote: >>> OK, it seems to be an issue related to something like a sanlock which >>> I cannot solve: >>> >>> vdsm "Failed to acquire lock: No space left on device" is what happens >>> when I restart vdsmd and even removing the __DIRECT_IO_TEST__ file is >>> newly created on a vdsmd restart. >> >> __DIRECT_IO_TEST__ is empty file, used to check if vdsm can do direct I/O >> to this storage, there is no need to remove it. >> >> "No space left on device" seems to be your problem, you should make some >> space if you want to use this storage domain. >> >>> >>> As I'm not able to change the maintenance mode on the hosts I need to >>> figure out something else. >>> >>> >>> >>> >>> >>> 2016-08-18 10:40 GMT+02:00 Matt . : OK nice to know, I never saw that message before. I cannot do anything anymore with hosted engine. It doesn't change the maintenance state, it doesn't start the vm and this happens on all hosts. Nothing else changed then just the ovirt* and vdsm* package upgrades. The HE storage is also mounted and the agent.log shows it's information as normal. Showing logs is kinda an issue as I'm bound to ILO/Drac at the moment. 2016-08-18 10:18 GMT+02:00 Simone Tiraboschi : > On Thu, Aug 18, 2016 at 10:06 AM, Matt . wrote: >> Hello, >> >> I'm having issues after the last 4.0.1 package just before 4.0.2 came >> out. My engine was running great, hosts started the vm also and the >> hosted-engine command was working fine. >> >> Now I get rpcxml 3.6 deprecated warning which I never got before. See >> the attachment for this output. > > That one it's just a warning and we are going to fix the root cause > for the next version, it's absolutely harmless. > >> I have checked is IPv6 was disabled, but it isn't as that is what is >> needed for this. >> >> Any clue ? > > If you have any issue, it's somewhere else. > What are the symptoms? > >> Thanks! >> >> Matt >> >> ___ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> >>> ___ >>> Users mailing list >>> Users@ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Cannot start hosted engine after last 4.0.1 packages
There is not storage issue, there is plenty of space, around 900G and this is shown in the past as the same issue. What is strange is that I cannot change the maintenance status of the hosts, when I set it to whatever status it's not changed and the status are various it seems from Global to Local. So I'm stuck. 2016-08-18 16:41 GMT+02:00 Nir Soffer : > On Thu, Aug 18, 2016 at 5:33 PM, Matt . wrote: >> OK, it seems to be an issue related to something like a sanlock which >> I cannot solve: >> >> vdsm "Failed to acquire lock: No space left on device" is what happens >> when I restart vdsmd and even removing the __DIRECT_IO_TEST__ file is >> newly created on a vdsmd restart. > > __DIRECT_IO_TEST__ is empty file, used to check if vdsm can do direct I/O > to this storage, there is no need to remove it. > > "No space left on device" seems to be your problem, you should make some > space if you want to use this storage domain. > >> >> As I'm not able to change the maintenance mode on the hosts I need to >> figure out something else. >> >> >> >> >> >> 2016-08-18 10:40 GMT+02:00 Matt . : >>> OK nice to know, I never saw that message before. >>> >>> I cannot do anything anymore with hosted engine. It doesn't change the >>> maintenance state, it doesn't start the vm and this happens on all >>> hosts. Nothing else changed then just the ovirt* and vdsm* package >>> upgrades. >>> >>> The HE storage is also mounted and the agent.log shows it's >>> information as normal. >>> >>> Showing logs is kinda an issue as I'm bound to ILO/Drac at the moment. >>> >>> >>> >>> >>> 2016-08-18 10:18 GMT+02:00 Simone Tiraboschi : On Thu, Aug 18, 2016 at 10:06 AM, Matt . wrote: > Hello, > > I'm having issues after the last 4.0.1 package just before 4.0.2 came > out. My engine was running great, hosts started the vm also and the > hosted-engine command was working fine. > > Now I get rpcxml 3.6 deprecated warning which I never got before. See > the attachment for this output. That one it's just a warning and we are going to fix the root cause for the next version, it's absolutely harmless. > I have checked is IPv6 was disabled, but it isn't as that is what is > needed for this. > > Any clue ? If you have any issue, it's somewhere else. What are the symptoms? > Thanks! > > Matt > > ___ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > >> ___ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Cannot start hosted engine after last 4.0.1 packages
On Thu, Aug 18, 2016 at 5:33 PM, Matt . wrote: > OK, it seems to be an issue related to something like a sanlock which > I cannot solve: > > vdsm "Failed to acquire lock: No space left on device" is what happens > when I restart vdsmd and even removing the __DIRECT_IO_TEST__ file is > newly created on a vdsmd restart. __DIRECT_IO_TEST__ is empty file, used to check if vdsm can do direct I/O to this storage, there is no need to remove it. "No space left on device" seems to be your problem, you should make some space if you want to use this storage domain. > > As I'm not able to change the maintenance mode on the hosts I need to > figure out something else. > > > > > > 2016-08-18 10:40 GMT+02:00 Matt . : >> OK nice to know, I never saw that message before. >> >> I cannot do anything anymore with hosted engine. It doesn't change the >> maintenance state, it doesn't start the vm and this happens on all >> hosts. Nothing else changed then just the ovirt* and vdsm* package >> upgrades. >> >> The HE storage is also mounted and the agent.log shows it's >> information as normal. >> >> Showing logs is kinda an issue as I'm bound to ILO/Drac at the moment. >> >> >> >> >> 2016-08-18 10:18 GMT+02:00 Simone Tiraboschi : >>> On Thu, Aug 18, 2016 at 10:06 AM, Matt . wrote: Hello, I'm having issues after the last 4.0.1 package just before 4.0.2 came out. My engine was running great, hosts started the vm also and the hosted-engine command was working fine. Now I get rpcxml 3.6 deprecated warning which I never got before. See the attachment for this output. >>> >>> That one it's just a warning and we are going to fix the root cause >>> for the next version, it's absolutely harmless. >>> I have checked is IPv6 was disabled, but it isn't as that is what is needed for this. Any clue ? >>> >>> If you have any issue, it's somewhere else. >>> What are the symptoms? >>> Thanks! Matt ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users > ___ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Cannot start hosted engine after last 4.0.1 packages
OK, it seems to be an issue related to something like a sanlock which I cannot solve: vdsm "Failed to acquire lock: No space left on device" is what happens when I restart vdsmd and even removing the __DIRECT_IO_TEST__ file is newly created on a vdsmd restart. As I'm not able to change the maintenance mode on the hosts I need to figure out something else. 2016-08-18 10:40 GMT+02:00 Matt . : > OK nice to know, I never saw that message before. > > I cannot do anything anymore with hosted engine. It doesn't change the > maintenance state, it doesn't start the vm and this happens on all > hosts. Nothing else changed then just the ovirt* and vdsm* package > upgrades. > > The HE storage is also mounted and the agent.log shows it's > information as normal. > > Showing logs is kinda an issue as I'm bound to ILO/Drac at the moment. > > > > > 2016-08-18 10:18 GMT+02:00 Simone Tiraboschi : >> On Thu, Aug 18, 2016 at 10:06 AM, Matt . wrote: >>> Hello, >>> >>> I'm having issues after the last 4.0.1 package just before 4.0.2 came >>> out. My engine was running great, hosts started the vm also and the >>> hosted-engine command was working fine. >>> >>> Now I get rpcxml 3.6 deprecated warning which I never got before. See >>> the attachment for this output. >> >> That one it's just a warning and we are going to fix the root cause >> for the next version, it's absolutely harmless. >> >>> I have checked is IPv6 was disabled, but it isn't as that is what is >>> needed for this. >>> >>> Any clue ? >> >> If you have any issue, it's somewhere else. >> What are the symptoms? >> >>> Thanks! >>> >>> Matt >>> >>> ___ >>> Users mailing list >>> Users@ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >>> ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Cannot start hosted engine after last 4.0.1 packages
OK nice to know, I never saw that message before. I cannot do anything anymore with hosted engine. It doesn't change the maintenance state, it doesn't start the vm and this happens on all hosts. Nothing else changed then just the ovirt* and vdsm* package upgrades. The HE storage is also mounted and the agent.log shows it's information as normal. Showing logs is kinda an issue as I'm bound to ILO/Drac at the moment. 2016-08-18 10:18 GMT+02:00 Simone Tiraboschi : > On Thu, Aug 18, 2016 at 10:06 AM, Matt . wrote: >> Hello, >> >> I'm having issues after the last 4.0.1 package just before 4.0.2 came >> out. My engine was running great, hosts started the vm also and the >> hosted-engine command was working fine. >> >> Now I get rpcxml 3.6 deprecated warning which I never got before. See >> the attachment for this output. > > That one it's just a warning and we are going to fix the root cause > for the next version, it's absolutely harmless. > >> I have checked is IPv6 was disabled, but it isn't as that is what is >> needed for this. >> >> Any clue ? > > If you have any issue, it's somewhere else. > What are the symptoms? > >> Thanks! >> >> Matt >> >> ___ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Cannot start hosted engine after last 4.0.1 packages
On Thu, Aug 18, 2016 at 10:06 AM, Matt . wrote: > Hello, > > I'm having issues after the last 4.0.1 package just before 4.0.2 came > out. My engine was running great, hosts started the vm also and the > hosted-engine command was working fine. > > Now I get rpcxml 3.6 deprecated warning which I never got before. See > the attachment for this output. That one it's just a warning and we are going to fix the root cause for the next version, it's absolutely harmless. > I have checked is IPv6 was disabled, but it isn't as that is what is > needed for this. > > Any clue ? If you have any issue, it's somewhere else. What are the symptoms? > Thanks! > > Matt > > ___ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
