Re: [ovirt-users] node-ng update failed from ovirt-node-ng-4.1.0-0 to ovirt-node-ng-image-4.1.0-1, and NM + iscsi boo issue

2017-02-07 Thread Sergey Kulikov

There is no new lv, I can remove rpm and reinstall manually to reproduce with 
any debug options, if you want.
I left host in maintenance mode. 
should I also remove ovirt-node-ng-image ? I found something strange:
# rpm -q ovirt-node-ng-image  
ovirt-node-ng-image-4.1.0-1.el7.centos.noarch

# rpm -ql ovirt-node-ng-image
/usr/share/ovirt-node-ng/image
/usr/share/ovirt-node-ng/image/ovirt-node-ng-4.1.0-0.20170201.0.el7.squashfs.img
/usr/share/ovirt-node-ng/image/product.img

version is 4.1.0-1, but squashfs file is 4.1.0-0


About iscsi... I've disabled NM in 4.0.6, everything was ok, it was re-enabled 
only after update, I don't
think iscsi root affects NM status.
The problem is only with reboot, 1 time out of ~4 reboots it's falling to RO.
Also I have centos hosts and NM stays disabled there, maybe I should try 
"unmanaged-devices="
in NM config, but I think it's not ideal, NM is useless for this setup)

-- 



 Tuesday, February 7, 2017, 07:14:55:

> Hey Sergey -


> If you check "lvs" and ensure that there's not actually a new LV
> from the update, you can cleanly 'rpm -e
> ovirt-node-ng-image-update', and be ok without redeploying.


> Unfortunately, it's hard to tell from the logs (and '--justdb'
> hanging) what's happening here, but I'll try to reproduce.


> NetworkManager disablement should "stick" across upgrades, but it's
> possible that iscsi roots are doing something here. I'll check for a dracut 
> flag, also...

> On Mon, Feb 6, 2017 at 1:14 PM, Sandro Bonazzola  wrote:

> Adding Douglas and Ryan


> Il 06/Feb/2017 13:32, "Sergey Kulikov"  ha scritto:


>  1) I've updated from 4.0.6 to 4.1.0 (on Feb 01 node-ng was at version 
> 4.1.0-0)
>  After some time engine alerted, that this node have updates to 
> ovirt-node-ng-image-4.1.0-1,
>  but update from engine timed out, there were hanging processes in ps on this 
> node:
>  
>  root     36309  0.0  0.0 113120  1564 ?        Ss   19:04   0:00
> bash -c umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t
> ovirt-XX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null
> 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar
> --warning=no-timestamp -C "${MYTMP}" -x && 
> "${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine 
> DIALOG/customization=bool:True
>  root     36339  0.2  0.0 496700 94208 ?        S    19:04   0:21
> /bin/python /tmp/ovirt-GCmVusccfe/pythonlib/otopi/__main__.py
> "BASE/pluginPath=str:/tmp/ovirt-GCmVusccfe/otopi-plugins" 
> APPEND:BASE/pluginGroups=str:ovirt-host-common:ovirt-host-mgmt
> DIALOG/dialect=str:machine DIALOG/customization=bool:True
>  root     37498  0.0  0.0 113124  1452 ?        S    19:09   0:00 /bin/sh 
> /var/tmp/rpm-tmp.4UqJ4e 1
>  root     37560  0.0  0.0      0     0 ?        S<   21:42   0:00 
> [kworker/21:2H]
>  root     37626  0.0  0.0 174516  5996 ?        S    19:09   0:00
> rpm -Uvh --quiet --justdb
> /usr/share/imgbased/ovirt-node-ng-image-update-4.1.0-1.el7.centos.noarch.rpm
>  
>  they were hanging forever, I ended up with rebooting the node, no
> errors in log, it was just hanging at:
>  
>  2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
> dialog.__logString:204 DIALOG:SEND       ***CONFIRM GPG_KEY Confirm
> use of GPG Key userid=oVirt  hexkeyid=FE590CB7
>  2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
> dialog.__logString:204 DIALOG:SEND       ###
>  2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
> dialog.__logString:204 DIALOG:SEND       ### Please confirm
> 'GPG_KEY' Confirm use of GPG Key userid=oVirt  
> hexkeyid=FE590CB7
>  2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
> dialog.__logString:204 DIALOG:SEND       ### Response is CONFIRM 
> GPG_KEY=yes|no or ABORT GPG_KEY
>  2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
> dialog.__logString:204 DIALOG:RECEIVE    CONFIRM GPG_KEY=yes
>  2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager
> yumpackager.info:80 Yum Status: Running Test Transaction
>  Running Transaction Check
>  2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager
> yumpackager.info:80 Yum Status: Running Transaction
>  2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager
> yumpackager.info:80 Yum install: 1/2:
> ovirt-node-ng-image-4.1.0-1.el7.centos.noarch
>  2017-02-03 19:09:20 DEBUG
> otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum
> Done: ovirt-node-ng-image-4.1.0-1.el7.centos.noarch
>  2017-02-03 19:09:20 INFO otopi.plugins.otopi.packagers.yumpackager
> yumpackager.info:80 Yum install: 2/2:
> ovirt-node-ng-image-update-4.1.0-1.el7.centos.noarch
>  
>  now my node have this layout:
>  # imgbase layout
>  ovirt-node-ng-4.1.0-0.20170201.0
>   +- ovirt-node-ng-4.1.0-0.20170201.0+1
>  (so update failed)
>  but 4.1.0-1 rpms are marked as "installed" and yum can't find any
> updates, can I rollback to base layout without installed  4.1.0-1 rms ?
>  imgbase 

Re: [ovirt-users] node-ng update failed from ovirt-node-ng-4.1.0-0 to ovirt-node-ng-image-4.1.0-1, and NM + iscsi boo issue

2017-02-06 Thread Ryan Barry
Hey Sergey -

If you check "lvs" and ensure that there's not actually a new LV from the
update, you can cleanly 'rpm -e ovirt-node-ng-image-update', and be ok
without redeploying.

Unfortunately, it's hard to tell from the logs (and '--justdb' hanging)
what's happening here, but I'll try to reproduce.

NetworkManager disablement should "stick" across upgrades, but it's
possible that iscsi roots are doing something here. I'll check for a dracut
flag, also...

On Mon, Feb 6, 2017 at 1:14 PM, Sandro Bonazzola 
wrote:

> Adding Douglas and Ryan
>
> Il 06/Feb/2017 13:32, "Sergey Kulikov"  ha scritto:
>
>>
>> 1) I've updated from 4.0.6 to 4.1.0 (on Feb 01 node-ng was at version
>> 4.1.0-0)
>> After some time engine alerted, that this node have updates to
>> ovirt-node-ng-image-4.1.0-1,
>> but update from engine timed out, there were hanging processes in ps on
>> this node:
>>
>> root 36309  0.0  0.0 113120  1564 ?Ss   19:04   0:00 bash -c
>> umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t
>> ovirt-XX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm
>> -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C
>> "${MYTMP}" -x &&  "${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine
>> DIALOG/customization=bool:True
>> root 36339  0.2  0.0 496700 94208 ?S19:04   0:21
>> /bin/python /tmp/ovirt-GCmVusccfe/pythonlib/otopi/__main__.py
>> "BASE/pluginPath=str:/tmp/ovirt-GCmVusccfe/otopi-plugins"
>> APPEND:BASE/pluginGroups=str:ovirt-host-common:ovirt-host-mgmt
>> DIALOG/dialect=str:machine DIALOG/customization=bool:True
>> root 37498  0.0  0.0 113124  1452 ?S19:09   0:00 /bin/sh
>> /var/tmp/rpm-tmp.4UqJ4e 1
>> root 37560  0.0  0.0  0 0 ?S<   21:42   0:00
>> [kworker/21:2H]
>> root 37626  0.0  0.0 174516  5996 ?S19:09   0:00 rpm -Uvh
>> --quiet --justdb /usr/share/imgbased/ovirt-node
>> -ng-image-update-4.1.0-1.el7.centos.noarch.rpm
>>
>> they were hanging forever, I ended up with rebooting the node, no errors
>> in log, it was just hanging at:
>>
>> 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
>> dialog.__logString:204 DIALOG:SEND   ***CONFIRM GPG_KEY Confirm use of
>> GPG Key userid=oVirt  hexkeyid=FE590CB7
>> 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
>> dialog.__logString:204 DIALOG:SEND   ###
>> 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
>> dialog.__logString:204 DIALOG:SEND   ### Please confirm 'GPG_KEY'
>> Confirm use of GPG Key userid=oVirt  hexkeyid=FE590CB7
>> 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
>> dialog.__logString:204 DIALOG:SEND   ### Response is CONFIRM
>> GPG_KEY=yes|no or ABORT GPG_KEY
>> 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
>> dialog.__logString:204 DIALOG:RECEIVECONFIRM GPG_KEY=yes
>> 2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager
>> yumpackager.info:80 Yum Status: Running Test Transaction
>> Running Transaction Check
>> 2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager
>> yumpackager.info:80 Yum Status: Running Transaction
>> 2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager
>> yumpackager.info:80 Yum install: 1/2: ovirt-node-ng-image-4.1.0-1.el
>> 7.centos.noarch
>> 2017-02-03 19:09:20 DEBUG otopi.plugins.otopi.packagers.yumpackager
>> yumpackager.verbose:76 Yum Done: ovirt-node-ng-image-4.1.0-1.el
>> 7.centos.noarch
>> 2017-02-03 19:09:20 INFO otopi.plugins.otopi.packagers.yumpackager
>> yumpackager.info:80 Yum install: 2/2: ovirt-node-ng-image-update-4.1
>> .0-1.el7.centos.noarch
>>
>> now my node have this layout:
>> # imgbase layout
>> ovirt-node-ng-4.1.0-0.20170201.0
>>  +- ovirt-node-ng-4.1.0-0.20170201.0+1
>> (so update failed)
>> but 4.1.0-1 rpms are marked as "installed" and yum can't find any
>> updates, can I rollback to base layout without installed  4.1.0-1 rms ?
>> imgbase rollback needs at least 2 layers over base.
>>
>> Or maybe the only way is to reinstall this node?
>>
>> 2) And another question, how can I disable NetworkManger permanently, or
>> exclude some interfaces permanently?
>> I've tried to disable NetworkManger by systemctl, but after update from
>> 4.0 to 4.1 it was re-enabled(so it's not persistent between updates).
>> I've an issue with iscsi root and enabled NetworkManger, because NM tries
>> to bring down\up my iscsi interfaces on boot, and sometimes FS remounting RO
>> because of IO errors, I can't put NM_CONTROLLED=no in ifcfg, because
>> ifcfg is generated by dracut at every boot.
>>
>>
>> -
>>
>> ___
>> Users mailing list
>> Users@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [ovirt-users] node-ng update failed from ovirt-node-ng-4.1.0-0 to ovirt-node-ng-image-4.1.0-1, and NM + iscsi boo issue

2017-02-06 Thread Sandro Bonazzola
Adding Douglas and Ryan

Il 06/Feb/2017 13:32, "Sergey Kulikov"  ha scritto:

>
> 1) I've updated from 4.0.6 to 4.1.0 (on Feb 01 node-ng was at version
> 4.1.0-0)
> After some time engine alerted, that this node have updates to
> ovirt-node-ng-image-4.1.0-1,
> but update from engine timed out, there were hanging processes in ps on
> this node:
>
> root 36309  0.0  0.0 113120  1564 ?Ss   19:04   0:00 bash -c
> umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t
> ovirt-XX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm
> -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C
> "${MYTMP}" -x &&  "${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine
> DIALOG/customization=bool:True
> root 36339  0.2  0.0 496700 94208 ?S19:04   0:21
> /bin/python /tmp/ovirt-GCmVusccfe/pythonlib/otopi/__main__.py
> "BASE/pluginPath=str:/tmp/ovirt-GCmVusccfe/otopi-plugins"
> APPEND:BASE/pluginGroups=str:ovirt-host-common:ovirt-host-mgmt
> DIALOG/dialect=str:machine DIALOG/customization=bool:True
> root 37498  0.0  0.0 113124  1452 ?S19:09   0:00 /bin/sh
> /var/tmp/rpm-tmp.4UqJ4e 1
> root 37560  0.0  0.0  0 0 ?S<   21:42   0:00
> [kworker/21:2H]
> root 37626  0.0  0.0 174516  5996 ?S19:09   0:00 rpm -Uvh
> --quiet --justdb /usr/share/imgbased/ovirt-node-ng-image-update-4.1.0-1.
> el7.centos.noarch.rpm
>
> they were hanging forever, I ended up with rebooting the node, no errors
> in log, it was just hanging at:
>
> 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
> dialog.__logString:204 DIALOG:SEND   ***CONFIRM GPG_KEY Confirm use of
> GPG Key userid=oVirt  hexkeyid=FE590CB7
> 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
> dialog.__logString:204 DIALOG:SEND   ###
> 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
> dialog.__logString:204 DIALOG:SEND   ### Please confirm 'GPG_KEY'
> Confirm use of GPG Key userid=oVirt  hexkeyid=FE590CB7
> 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
> dialog.__logString:204 DIALOG:SEND   ### Response is CONFIRM
> GPG_KEY=yes|no or ABORT GPG_KEY
> 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
> dialog.__logString:204 DIALOG:RECEIVECONFIRM GPG_KEY=yes
> 2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager
> yumpackager.info:80 Yum Status: Running Test Transaction
> Running Transaction Check
> 2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager
> yumpackager.info:80 Yum Status: Running Transaction
> 2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager
> yumpackager.info:80 Yum install: 1/2: ovirt-node-ng-image-4.1.0-1.
> el7.centos.noarch
> 2017-02-03 19:09:20 DEBUG otopi.plugins.otopi.packagers.yumpackager
> yumpackager.verbose:76 Yum Done: ovirt-node-ng-image-4.1.0-1.
> el7.centos.noarch
> 2017-02-03 19:09:20 INFO otopi.plugins.otopi.packagers.yumpackager
> yumpackager.info:80 Yum install: 2/2: ovirt-node-ng-image-update-4.
> 1.0-1.el7.centos.noarch
>
> now my node have this layout:
> # imgbase layout
> ovirt-node-ng-4.1.0-0.20170201.0
>  +- ovirt-node-ng-4.1.0-0.20170201.0+1
> (so update failed)
> but 4.1.0-1 rpms are marked as "installed" and yum can't find any updates,
> can I rollback to base layout without installed  4.1.0-1 rms ?
> imgbase rollback needs at least 2 layers over base.
>
> Or maybe the only way is to reinstall this node?
>
> 2) And another question, how can I disable NetworkManger permanently, or
> exclude some interfaces permanently?
> I've tried to disable NetworkManger by systemctl, but after update from
> 4.0 to 4.1 it was re-enabled(so it's not persistent between updates).
> I've an issue with iscsi root and enabled NetworkManger, because NM tries
> to bring down\up my iscsi interfaces on boot, and sometimes FS remounting RO
> because of IO errors, I can't put NM_CONTROLLED=no in ifcfg, because ifcfg
> is generated by dracut at every boot.
>
>
> -
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


[ovirt-users] node-ng update failed from ovirt-node-ng-4.1.0-0 to ovirt-node-ng-image-4.1.0-1, and NM + iscsi boo issue

2017-02-06 Thread Sergey Kulikov

1) I've updated from 4.0.6 to 4.1.0 (on Feb 01 node-ng was at version 4.1.0-0)
After some time engine alerted, that this node have updates to 
ovirt-node-ng-image-4.1.0-1,
but update from engine timed out, there were hanging processes in ps on this 
node:

root 36309  0.0  0.0 113120  1564 ?Ss   19:04   0:00 bash -c umask 
0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XX)"; trap 
"chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 
2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x &&  
"${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine 
DIALOG/customization=bool:True
root 36339  0.2  0.0 496700 94208 ?S19:04   0:21 /bin/python 
/tmp/ovirt-GCmVusccfe/pythonlib/otopi/__main__.py 
"BASE/pluginPath=str:/tmp/ovirt-GCmVusccfe/otopi-plugins"  
APPEND:BASE/pluginGroups=str:ovirt-host-common:ovirt-host-mgmt 
DIALOG/dialect=str:machine DIALOG/customization=bool:True
root 37498  0.0  0.0 113124  1452 ?S19:09   0:00 /bin/sh 
/var/tmp/rpm-tmp.4UqJ4e 1
root 37560  0.0  0.0  0 0 ?S<   21:42   0:00 [kworker/21:2H]
root 37626  0.0  0.0 174516  5996 ?S19:09   0:00 rpm -Uvh 
--quiet --justdb 
/usr/share/imgbased/ovirt-node-ng-image-update-4.1.0-1.el7.centos.noarch.rpm

they were hanging forever, I ended up with rebooting the node, no errors in 
log, it was just hanging at:

2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine 
dialog.__logString:204 DIALOG:SEND   ***CONFIRM GPG_KEY Confirm use of GPG 
Key userid=oVirt  hexkeyid=FE590CB7
2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine 
dialog.__logString:204 DIALOG:SEND   ###
2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine 
dialog.__logString:204 DIALOG:SEND   ### Please confirm 'GPG_KEY' Confirm 
use of GPG Key userid=oVirt  hexkeyid=FE590CB7
2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine 
dialog.__logString:204 DIALOG:SEND   ### Response is CONFIRM GPG_KEY=yes|no 
or ABORT GPG_KEY
2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine 
dialog.__logString:204 DIALOG:RECEIVECONFIRM GPG_KEY=yes
2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager 
yumpackager.info:80 Yum Status: Running Test Transaction
Running Transaction Check
2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager 
yumpackager.info:80 Yum Status: Running Transaction
2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager 
yumpackager.info:80 Yum install: 1/2: 
ovirt-node-ng-image-4.1.0-1.el7.centos.noarch
2017-02-03 19:09:20 DEBUG otopi.plugins.otopi.packagers.yumpackager 
yumpackager.verbose:76 Yum Done: ovirt-node-ng-image-4.1.0-1.el7.centos.noarch
2017-02-03 19:09:20 INFO otopi.plugins.otopi.packagers.yumpackager 
yumpackager.info:80 Yum install: 2/2: 
ovirt-node-ng-image-update-4.1.0-1.el7.centos.noarch

now my node have this layout:
# imgbase layout
ovirt-node-ng-4.1.0-0.20170201.0
 +- ovirt-node-ng-4.1.0-0.20170201.0+1
(so update failed)
but 4.1.0-1 rpms are marked as "installed" and yum can't find any updates, can 
I rollback to base layout without installed  4.1.0-1 rms ?
imgbase rollback needs at least 2 layers over base.

Or maybe the only way is to reinstall this node?

2) And another question, how can I disable NetworkManger permanently, or 
exclude some interfaces permanently?
I've tried to disable NetworkManger by systemctl, but after update from 4.0 to 
4.1 it was re-enabled(so it's not persistent between updates).
I've an issue with iscsi root and enabled NetworkManger, because NM tries to 
bring down\up my iscsi interfaces on boot, and sometimes FS remounting RO
because of IO errors, I can't put NM_CONTROLLED=no in ifcfg, because ifcfg is 
generated by dracut at every boot.


-

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users