Re: [ovirt-users] node-ng update failed from ovirt-node-ng-4.1.0-0 to ovirt-node-ng-image-4.1.0-1, and NM + iscsi boo issue

2017-02-07 Thread Sergey Kulikov 

There is no new lv, I can remove rpm and reinstall manually to reproduce with 
any debug options, if you want.
I left host in maintenance mode. 
should I also remove ovirt-node-ng-image ? I found something strange:
# rpm -q ovirt-node-ng-image  
ovirt-node-ng-image-4.1.0-1.el7.centos.noarch

# rpm -ql ovirt-node-ng-image
/usr/share/ovirt-node-ng/image
/usr/share/ovirt-node-ng/image/ovirt-node-ng-4.1.0-0.20170201.0.el7.squashfs.img
/usr/share/ovirt-node-ng/image/product.img

version is 4.1.0-1, but squashfs file is 4.1.0-0


About iscsi... I've disabled NM in 4.0.6, everything was ok, it was re-enabled 
only after update, I don't
think iscsi root affects NM status.
The problem is only with reboot, 1 time out of ~4 reboots it's falling to RO.
Also I have centos hosts and NM stays disabled there, maybe I should try 
"unmanaged-devices="
in NM config, but I think it's not ideal, NM is useless for this setup)

-- 



 Tuesday, February 7, 2017, 07:14:55:

> Hey Sergey -


> If you check "lvs" and ensure that there's not actually a new LV
> from the update, you can cleanly 'rpm -e
> ovirt-node-ng-image-update', and be ok without redeploying.


> Unfortunately, it's hard to tell from the logs (and '--justdb'
> hanging) what's happening here, but I'll try to reproduce.


> NetworkManager disablement should "stick" across upgrades, but it's
> possible that iscsi roots are doing something here. I'll check for a dracut 
> flag, also...

> On Mon, Feb 6, 2017 at 1:14 PM, Sandro Bonazzola  wrote:

> Adding Douglas and Ryan


> Il 06/Feb/2017 13:32, "Sergey Kulikov"  ha scritto:


>  1) I've updated from 4.0.6 to 4.1.0 (on Feb 01 node-ng was at version 
> 4.1.0-0)
>  After some time engine alerted, that this node have updates to 
> ovirt-node-ng-image-4.1.0-1,
>  but update from engine timed out, there were hanging processes in ps on this 
> node:
>  
>  root     36309  0.0  0.0 113120  1564 ?        Ss   19:04   0:00
> bash -c umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t
> ovirt-XX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null
> 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar
> --warning=no-timestamp -C "${MYTMP}" -x && 
> "${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine 
> DIALOG/customization=bool:True
>  root     36339  0.2  0.0 496700 94208 ?        S    19:04   0:21
> /bin/python /tmp/ovirt-GCmVusccfe/pythonlib/otopi/__main__.py
> "BASE/pluginPath=str:/tmp/ovirt-GCmVusccfe/otopi-plugins" 
> APPEND:BASE/pluginGroups=str:ovirt-host-common:ovirt-host-mgmt
> DIALOG/dialect=str:machine DIALOG/customization=bool:True
>  root     37498  0.0  0.0 113124  1452 ?        S    19:09   0:00 /bin/sh 
> /var/tmp/rpm-tmp.4UqJ4e 1
>  root     37560  0.0  0.0      0     0 ?        S<   21:42   0:00 
> [kworker/21:2H]
>  root     37626  0.0  0.0 174516  5996 ?        S    19:09   0:00
> rpm -Uvh --quiet --justdb
> /usr/share/imgbased/ovirt-node-ng-image-update-4.1.0-1.el7.centos.noarch.rpm
>  
>  they were hanging forever, I ended up with rebooting the node, no
> errors in log, it was just hanging at:
>  
>  2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
> dialog.__logString:204 DIALOG:SEND       ***CONFIRM GPG_KEY Confirm
> use of GPG Key userid=oVirt  hexkeyid=FE590CB7
>  2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
> dialog.__logString:204 DIALOG:SEND       ###
>  2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
> dialog.__logString:204 DIALOG:SEND       ### Please confirm
> 'GPG_KEY' Confirm use of GPG Key userid=oVirt  
> hexkeyid=FE590CB7
>  2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
> dialog.__logString:204 DIALOG:SEND       ### Response is CONFIRM 
> GPG_KEY=yes|no or ABORT GPG_KEY
>  2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
> dialog.__logString:204 DIALOG:RECEIVE    CONFIRM GPG_KEY=yes
>  2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager
> yumpackager.info:80 Yum Status: Running Test Transaction
>  Running Transaction Check
>  2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager
> yumpackager.info:80 Yum Status: Running Transaction
>  2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager
> yumpackager.info:80 Yum install: 1/2:
> ovirt-node-ng-image-4.1.0-1.el7.centos.noarch
>  2017-02-03 19:09:20 DEBUG
> otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum
> Done: ovirt-node-ng-image-4.1.0-1.el7.centos.noarch
>  2017-02-03 19:09:20 INFO otopi.plugins.otopi.packagers.yumpackager
> yumpackager.info:80 Yum install: 2/2:
> ovirt-node-ng-image-update-4.1.0-1.el7.centos.noarch
>  
>  now my node have this layout:
>  # imgbase layout
>  ovirt-node-ng-4.1.0-0.20170201.0
>   +- ovirt-node-ng-4.1.0-0.20170201.0+1
>  (so update failed)
>  but 4.1.0-1 rpms are marked as "installed" and yum can't find any
> updates, can I rollback to base layout without installed  4.1.0-1 rms ?
>  imgbase

Re: [ovirt-users] node-ng update failed from ovirt-node-ng-4.1.0-0 to ovirt-node-ng-image-4.1.0-1, and NM + iscsi boo issue

2017-02-06 Thread Ryan Barry 
Hey Sergey -

If you check "lvs" and ensure that there's not actually a new LV from the
update, you can cleanly 'rpm -e ovirt-node-ng-image-update', and be ok
without redeploying.

Unfortunately, it's hard to tell from the logs (and '--justdb' hanging)
what's happening here, but I'll try to reproduce.

NetworkManager disablement should "stick" across upgrades, but it's
possible that iscsi roots are doing something here. I'll check for a dracut
flag, also...

On Mon, Feb 6, 2017 at 1:14 PM, Sandro Bonazzola 
wrote:

> Adding Douglas and Ryan
>
> Il 06/Feb/2017 13:32, "Sergey Kulikov"  ha scritto:
>
>>
>> 1) I've updated from 4.0.6 to 4.1.0 (on Feb 01 node-ng was at version
>> 4.1.0-0)
>> After some time engine alerted, that this node have updates to
>> ovirt-node-ng-image-4.1.0-1,
>> but update from engine timed out, there were hanging processes in ps on
>> this node:
>>
>> root 36309  0.0  0.0 113120  1564 ?Ss   19:04   0:00 bash -c
>> umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t
>> ovirt-XX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm
>> -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C
>> "${MYTMP}" -x &&  "${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine
>> DIALOG/customization=bool:True
>> root 36339  0.2  0.0 496700 94208 ?S19:04   0:21
>> /bin/python /tmp/ovirt-GCmVusccfe/pythonlib/otopi/__main__.py
>> "BASE/pluginPath=str:/tmp/ovirt-GCmVusccfe/otopi-plugins"
>> APPEND:BASE/pluginGroups=str:ovirt-host-common:ovirt-host-mgmt
>> DIALOG/dialect=str:machine DIALOG/customization=bool:True
>> root 37498  0.0  0.0 113124  1452 ?S19:09   0:00 /bin/sh
>> /var/tmp/rpm-tmp.4UqJ4e 1
>> root 37560  0.0  0.0  0 0 ?S<   21:42   0:00
>> [kworker/21:2H]
>> root 37626  0.0  0.0 174516  5996 ?S19:09   0:00 rpm -Uvh
>> --quiet --justdb /usr/share/imgbased/ovirt-node
>> -ng-image-update-4.1.0-1.el7.centos.noarch.rpm
>>
>> they were hanging forever, I ended up with rebooting the node, no errors
>> in log, it was just hanging at:
>>
>> 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
>> dialog.__logString:204 DIALOG:SEND   ***CONFIRM GPG_KEY Confirm use of
>> GPG Key userid=oVirt  hexkeyid=FE590CB7
>> 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
>> dialog.__logString:204 DIALOG:SEND   ###
>> 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
>> dialog.__logString:204 DIALOG:SEND   ### Please confirm 'GPG_KEY'
>> Confirm use of GPG Key userid=oVirt  hexkeyid=FE590CB7
>> 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
>> dialog.__logString:204 DIALOG:SEND   ### Response is CONFIRM
>> GPG_KEY=yes|no or ABORT GPG_KEY
>> 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
>> dialog.__logString:204 DIALOG:RECEIVECONFIRM GPG_KEY=yes
>> 2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager
>> yumpackager.info:80 Yum Status: Running Test Transaction
>> Running Transaction Check
>> 2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager
>> yumpackager.info:80 Yum Status: Running Transaction
>> 2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager
>> yumpackager.info:80 Yum install: 1/2: ovirt-node-ng-image-4.1.0-1.el
>> 7.centos.noarch
>> 2017-02-03 19:09:20 DEBUG otopi.plugins.otopi.packagers.yumpackager
>> yumpackager.verbose:76 Yum Done: ovirt-node-ng-image-4.1.0-1.el
>> 7.centos.noarch
>> 2017-02-03 19:09:20 INFO otopi.plugins.otopi.packagers.yumpackager
>> yumpackager.info:80 Yum install: 2/2: ovirt-node-ng-image-update-4.1
>> .0-1.el7.centos.noarch
>>
>> now my node have this layout:
>> # imgbase layout
>> ovirt-node-ng-4.1.0-0.20170201.0
>>  +- ovirt-node-ng-4.1.0-0.20170201.0+1
>> (so update failed)
>> but 4.1.0-1 rpms are marked as "installed" and yum can't find any
>> updates, can I rollback to base layout without installed  4.1.0-1 rms ?
>> imgbase rollback needs at least 2 layers over base.
>>
>> Or maybe the only way is to reinstall this node?
>>
>> 2) And another question, how can I disable NetworkManger permanently, or
>> exclude some interfaces permanently?
>> I've tried to disable NetworkManger by systemctl, but after update from
>> 4.0 to 4.1 it was re-enabled(so it's not persistent between updates).
>> I've an issue with iscsi root and enabled NetworkManger, because NM tries
>> to bring down\up my iscsi interfaces on boot, and sometimes FS remounting RO
>> because of IO errors, I can't put NM_CONTROLLED=no in ifcfg, because
>> ifcfg is generated by dracut at every boot.
>>
>>
>> -
>>
>> ___
>> Users mailing list
>> Users@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] node-ng update failed from ovirt-node-ng-4.1.0-0 to ovirt-node-ng-image-4.1.0-1, and NM + iscsi boo issue

2017-02-06 Thread Sandro Bonazzola 
Adding Douglas and Ryan

Il 06/Feb/2017 13:32, "Sergey Kulikov"  ha scritto:

>
> 1) I've updated from 4.0.6 to 4.1.0 (on Feb 01 node-ng was at version
> 4.1.0-0)
> After some time engine alerted, that this node have updates to
> ovirt-node-ng-image-4.1.0-1,
> but update from engine timed out, there were hanging processes in ps on
> this node:
>
> root 36309  0.0  0.0 113120  1564 ?Ss   19:04   0:00 bash -c
> umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t
> ovirt-XX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm
> -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C
> "${MYTMP}" -x &&  "${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine
> DIALOG/customization=bool:True
> root 36339  0.2  0.0 496700 94208 ?S19:04   0:21
> /bin/python /tmp/ovirt-GCmVusccfe/pythonlib/otopi/__main__.py
> "BASE/pluginPath=str:/tmp/ovirt-GCmVusccfe/otopi-plugins"
> APPEND:BASE/pluginGroups=str:ovirt-host-common:ovirt-host-mgmt
> DIALOG/dialect=str:machine DIALOG/customization=bool:True
> root 37498  0.0  0.0 113124  1452 ?S19:09   0:00 /bin/sh
> /var/tmp/rpm-tmp.4UqJ4e 1
> root 37560  0.0  0.0  0 0 ?S<   21:42   0:00
> [kworker/21:2H]
> root 37626  0.0  0.0 174516  5996 ?S19:09   0:00 rpm -Uvh
> --quiet --justdb /usr/share/imgbased/ovirt-node-ng-image-update-4.1.0-1.
> el7.centos.noarch.rpm
>
> they were hanging forever, I ended up with rebooting the node, no errors
> in log, it was just hanging at:
>
> 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
> dialog.__logString:204 DIALOG:SEND   ***CONFIRM GPG_KEY Confirm use of
> GPG Key userid=oVirt  hexkeyid=FE590CB7
> 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
> dialog.__logString:204 DIALOG:SEND   ###
> 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
> dialog.__logString:204 DIALOG:SEND   ### Please confirm 'GPG_KEY'
> Confirm use of GPG Key userid=oVirt  hexkeyid=FE590CB7
> 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
> dialog.__logString:204 DIALOG:SEND   ### Response is CONFIRM
> GPG_KEY=yes|no or ABORT GPG_KEY
> 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
> dialog.__logString:204 DIALOG:RECEIVECONFIRM GPG_KEY=yes
> 2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager
> yumpackager.info:80 Yum Status: Running Test Transaction
> Running Transaction Check
> 2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager
> yumpackager.info:80 Yum Status: Running Transaction
> 2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager
> yumpackager.info:80 Yum install: 1/2: ovirt-node-ng-image-4.1.0-1.
> el7.centos.noarch
> 2017-02-03 19:09:20 DEBUG otopi.plugins.otopi.packagers.yumpackager
> yumpackager.verbose:76 Yum Done: ovirt-node-ng-image-4.1.0-1.
> el7.centos.noarch
> 2017-02-03 19:09:20 INFO otopi.plugins.otopi.packagers.yumpackager
> yumpackager.info:80 Yum install: 2/2: ovirt-node-ng-image-update-4.
> 1.0-1.el7.centos.noarch
>
> now my node have this layout:
> # imgbase layout
> ovirt-node-ng-4.1.0-0.20170201.0
>  +- ovirt-node-ng-4.1.0-0.20170201.0+1
> (so update failed)
> but 4.1.0-1 rpms are marked as "installed" and yum can't find any updates,
> can I rollback to base layout without installed  4.1.0-1 rms ?
> imgbase rollback needs at least 2 layers over base.
>
> Or maybe the only way is to reinstall this node?
>
> 2) And another question, how can I disable NetworkManger permanently, or
> exclude some interfaces permanently?
> I've tried to disable NetworkManger by systemctl, but after update from
> 4.0 to 4.1 it was re-enabled(so it's not persistent between updates).
> I've an issue with iscsi root and enabled NetworkManger, because NM tries
> to bring down\up my iscsi interfaces on boot, and sometimes FS remounting RO
> because of IO errors, I can't put NM_CONTROLLED=no in ifcfg, because ifcfg
> is generated by dracut at every boot.
>
>
> -
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users