Re: [ovirt-users] supervdsmd IOError to /dev/stdout
Did it change the file permissions when you copied svdsm.logger.conf.rpmnew? No, the file permission were the same root:root 0644 Yes reproducible with 3.6 config file - it is definitely due to the 3.6 svdsm.logger.conf: when I copied the rpmnew file from 4.1, it works again. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] supervdsmd IOError to /dev/stdout
Did it change the file permissions when you copied svdsm.logger.conf.rpmnew? can you reproduce it? On Tue, Aug 1, 2017 at 3:06 PM Richard Chan wrote: > Hi, > > Could this be the cause of the access issue (old system upgraded from 3.4 > all the way to 4.1)? > ## change in python class name?? > > [handler_logfile] > -class=logUtils.UserGroupEnforcingHandler > +class=vdsm.logUtils.UserGroupEnforcingHandler > > When I copied svdsm.logger.conf.rpmnew over svdsm.logger.conf, it could > start up. > > Thanks! > > > > On Tue, Aug 1, 2017 at 5:02 PM, Yaniv Bronheim > wrote: > >> Hi, >> >> Seems like I already bumped in such issue awhile ago - >> https://bugzilla.redhat.com/show_bug.cgi?id=1216880 >> but now I see what's wrong - it happens after failing to read >> /etc/vdsm/svdsm.logger.conf for some reason >> then we have a bug in our fallback which tries to run >> logging.basicConfig(filename='/dev/stdout', filemode='w+', >> level=logging.DEBUG) >> >> and this fails while running as systemd daemon >> >> sorry about that. check why it can't load /etc/vdsm/svdsm.logger.conf >> and update us >> after fixing the access issue check if you can run >> logging.config.fileConfig("/etc/vdsm/svdsm.logger.conf" , >> disable_existing_loggers=False) >> Then supervdsmd should start properly >> >> I'll reopen the bugzilla and fix the wrong fallback to syslog print >> >> Thanks! >> >> >> On Sun, Jul 30, 2017 at 3:23 PM Richard Chan < >> rich...@treeboxsolutions.com> wrote: >> >>> Hi Yaniv, >>> >>> On this one node, it happened from 3.6 -> 4.0. It persisted after >>> 4.0->4.1. >>> Current: vdsm-4.19.24-1.el7.centos.x86_64 >>> >>> >>> In systemd override I now have to have: >>> StandardOutput=null >>> >>> >>> >>> >>> Jul 29 01:19:21 systemd[1]: Starting Auxiliary vdsm service for >>> running helper functions as root... >>> Jul 29 01:19:21 python2[39124]: detected unhandled Python >>> exception in '/usr/share/vdsm/supervdsmServer' >>> Jul 29 01:19:21 daemonAdapter[39124]: Traceback (most recent >>> call last): >>> Jul 29 01:19:21 daemonAdapter[39124]: File >>> "/usr/share/vdsm/supervdsmServer", line 45, in >>> Jul 29 01:19:21 daemonAdapter[39124]: level=logging.DEBUG) >>> Jul 29 01:19:21 daemonAdapter[39124]: File >>> "/usr/lib64/python2.7/logging/__init__.py", line 1529, in basicConfig >>> Jul 29 01:19:21 daemonAdapter[39124]: hdlr = >>> FileHandler(filename, mode) >>> Jul 29 01:19:21 daemonAdapter[39124]: File >>> "/usr/lib64/python2.7/logging/__init__.py", line 902, in __init__ >>> Jul 29 01:19:21 daemonAdapter[39124]: >>> StreamHandler.__init__(self, self._open()) >>> Jul 29 01:19:21 daemonAdapter[39124]: File >>> "/usr/lib64/python2.7/logging/__init__.py", line 925, in _open >>> Jul 29 01:19:21 daemonAdapter[39124]: stream = >>> open(self.baseFilename, self.mode) >>> Jul 29 01:19:21 daemonAdapter[39124]: IOError: [Errno 6] No >>> such device or address: '/dev/stdout' >>> Jul 29 01:19:21 systemd[1]: supervdsmd.service: main process >>> exited, code=exited, status=1/FAILURE >>> >>> >>> >>> On Sun, Jul 30, 2017 at 3:47 PM, Yaniv Kaul wrote: >>> On Fri, Jul 28, 2017 at 8:37 PM, Richard Chan < rich...@treeboxsolutions.com> wrote: > After an upgrade to 4.0 I have a single host that cannot start > supervdsmd because of IOError on /dev/stdout. All other hosts upgraded > correctly. > Upgrade from which version? 3.6? Did you stay on 4.0 or continued to 4.1? > > In the systemd unit I have to hack StandardOutput=null. > > Any thing I have overlooked? The hosts are all identical and it is > just this one > that has this weird behaviour. > Any logs that you can share? Y. > > -- > Richard Chan > > > ___ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > >>> >>> >>> -- >>> Richard Chan >>> Chief Architect >>> >>> TreeBox Solutions Pte Ltd >>> 1 Commonwealth Lane #03-01 >>> Singapore 149544 >>> Tel: 6570 3725 >>> http://www.treeboxsolutions.com >>> >>> Co.Reg.No. 201100585R >>> ___ >>> Users mailing list >>> Users@ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >>> >> -- >> Yaniv Bronhaim. >> > > > > -- > Richard Chan > Chief Architect > > TreeBox Solutions Pte Ltd > 1 Commonwealth Lane #03-01 > Singapore 149544 > Tel: 6570 3725 > http://www.treeboxsolutions.com > > Co.Reg.No. 201100585R > -- Yaniv Bronhaim. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] supervdsmd IOError to /dev/stdout
On Tue, Aug 1, 2017 at 3:07 PM Richard Chan wrote: > Hi, > > Could this be the cause of the access issue (old system upgraded from 3.4 > all the way to 4.1)? > ## change in python class name?? > > [handler_logfile] > -class=logUtils.UserGroupEnforcingHandler > +class=vdsm.logUtils.UserGroupEnforcingHandler > > When I copied svdsm.logger.conf.rpmnew over svdsm.logger.conf, it could > start up. > If this is the case, I don't think we should change anything. Logging to syslog by default is problematic, we don't want to spam systlog with unimportant debug logs. Nir > > Thanks! > > > > On Tue, Aug 1, 2017 at 5:02 PM, Yaniv Bronheim > wrote: > >> Hi, >> >> Seems like I already bumped in such issue awhile ago - >> https://bugzilla.redhat.com/show_bug.cgi?id=1216880 >> but now I see what's wrong - it happens after failing to read >> /etc/vdsm/svdsm.logger.conf for some reason >> then we have a bug in our fallback which tries to run >> logging.basicConfig(filename='/dev/stdout', filemode='w+', >> level=logging.DEBUG) >> >> and this fails while running as systemd daemon >> >> sorry about that. check why it can't load /etc/vdsm/svdsm.logger.conf >> and update us >> after fixing the access issue check if you can run >> logging.config.fileConfig("/etc/vdsm/svdsm.logger.conf" , >> disable_existing_loggers=False) >> Then supervdsmd should start properly >> >> I'll reopen the bugzilla and fix the wrong fallback to syslog print >> >> Thanks! >> >> >> On Sun, Jul 30, 2017 at 3:23 PM Richard Chan < >> rich...@treeboxsolutions.com> wrote: >> >>> Hi Yaniv, >>> >>> On this one node, it happened from 3.6 -> 4.0. It persisted after >>> 4.0->4.1. >>> Current: vdsm-4.19.24-1.el7.centos.x86_64 >>> >>> >>> In systemd override I now have to have: >>> StandardOutput=null >>> >>> >>> >>> >>> Jul 29 01:19:21 systemd[1]: Starting Auxiliary vdsm service for >>> running helper functions as root... >>> Jul 29 01:19:21 python2[39124]: detected unhandled Python >>> exception in '/usr/share/vdsm/supervdsmServer' >>> Jul 29 01:19:21 daemonAdapter[39124]: Traceback (most recent >>> call last): >>> Jul 29 01:19:21 daemonAdapter[39124]: File >>> "/usr/share/vdsm/supervdsmServer", line 45, in >>> Jul 29 01:19:21 daemonAdapter[39124]: level=logging.DEBUG) >>> Jul 29 01:19:21 daemonAdapter[39124]: File >>> "/usr/lib64/python2.7/logging/__init__.py", line 1529, in basicConfig >>> Jul 29 01:19:21 daemonAdapter[39124]: hdlr = >>> FileHandler(filename, mode) >>> Jul 29 01:19:21 daemonAdapter[39124]: File >>> "/usr/lib64/python2.7/logging/__init__.py", line 902, in __init__ >>> Jul 29 01:19:21 daemonAdapter[39124]: >>> StreamHandler.__init__(self, self._open()) >>> Jul 29 01:19:21 daemonAdapter[39124]: File >>> "/usr/lib64/python2.7/logging/__init__.py", line 925, in _open >>> Jul 29 01:19:21 daemonAdapter[39124]: stream = >>> open(self.baseFilename, self.mode) >>> Jul 29 01:19:21 daemonAdapter[39124]: IOError: [Errno 6] No >>> such device or address: '/dev/stdout' >>> Jul 29 01:19:21 systemd[1]: supervdsmd.service: main process >>> exited, code=exited, status=1/FAILURE >>> >>> >>> >>> On Sun, Jul 30, 2017 at 3:47 PM, Yaniv Kaul wrote: >>> On Fri, Jul 28, 2017 at 8:37 PM, Richard Chan < rich...@treeboxsolutions.com> wrote: > After an upgrade to 4.0 I have a single host that cannot start > supervdsmd because of IOError on /dev/stdout. All other hosts upgraded > correctly. > Upgrade from which version? 3.6? Did you stay on 4.0 or continued to 4.1? > > In the systemd unit I have to hack StandardOutput=null. > > Any thing I have overlooked? The hosts are all identical and it is > just this one > that has this weird behaviour. > Any logs that you can share? Y. > > -- > Richard Chan > > > ___ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > >>> >>> >>> -- >>> Richard Chan >>> Chief Architect >>> >>> TreeBox Solutions Pte Ltd >>> 1 Commonwealth Lane #03-01 >>> Singapore 149544 >>> Tel: 6570 3725 >>> http://www.treeboxsolutions.com >>> >>> Co.Reg.No. 201100585R >>> ___ >>> Users mailing list >>> Users@ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >>> >> -- >> Yaniv Bronhaim. >> > > > > -- > Richard Chan > Chief Architect > > TreeBox Solutions Pte Ltd > 1 Commonwealth Lane #03-01 > Singapore 149544 > Tel: 6570 3725 > http://www.treeboxsolutions.com > > Co.Reg.No. 201100585R > ___ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > ___ Users mailing list Users@ovirt.org http://lis
Re: [ovirt-users] supervdsmd IOError to /dev/stdout
Hi, Could this be the cause of the access issue (old system upgraded from 3.4 all the way to 4.1)? ## change in python class name?? [handler_logfile] -class=logUtils.UserGroupEnforcingHandler +class=vdsm.logUtils.UserGroupEnforcingHandler When I copied svdsm.logger.conf.rpmnew over svdsm.logger.conf, it could start up. Thanks! On Tue, Aug 1, 2017 at 5:02 PM, Yaniv Bronheim wrote: > Hi, > > Seems like I already bumped in such issue awhile ago - > https://bugzilla.redhat.com/show_bug.cgi?id=1216880 > but now I see what's wrong - it happens after failing to read > /etc/vdsm/svdsm.logger.conf for some reason > then we have a bug in our fallback which tries to run > logging.basicConfig(filename='/dev/stdout', filemode='w+', > level=logging.DEBUG) > > and this fails while running as systemd daemon > > sorry about that. check why it can't load /etc/vdsm/svdsm.logger.conf > and update us > after fixing the access issue check if you can run > logging.config.fileConfig("/etc/vdsm/svdsm.logger.conf" , > disable_existing_loggers=False) > Then supervdsmd should start properly > > I'll reopen the bugzilla and fix the wrong fallback to syslog print > > Thanks! > > > On Sun, Jul 30, 2017 at 3:23 PM Richard Chan > wrote: > >> Hi Yaniv, >> >> On this one node, it happened from 3.6 -> 4.0. It persisted after >> 4.0->4.1. >> Current: vdsm-4.19.24-1.el7.centos.x86_64 >> >> >> In systemd override I now have to have: >> StandardOutput=null >> >> >> >> >> Jul 29 01:19:21 systemd[1]: Starting Auxiliary vdsm service for >> running helper functions as root... >> Jul 29 01:19:21 python2[39124]: detected unhandled Python >> exception in '/usr/share/vdsm/supervdsmServer' >> Jul 29 01:19:21 daemonAdapter[39124]: Traceback (most recent >> call last): >> Jul 29 01:19:21 daemonAdapter[39124]: File >> "/usr/share/vdsm/supervdsmServer", >> line 45, in >> Jul 29 01:19:21 daemonAdapter[39124]: level=logging.DEBUG) >> Jul 29 01:19:21 daemonAdapter[39124]: File >> "/usr/lib64/python2.7/logging/__init__.py", line 1529, in basicConfig >> Jul 29 01:19:21 daemonAdapter[39124]: hdlr = >> FileHandler(filename, mode) >> Jul 29 01:19:21 daemonAdapter[39124]: File >> "/usr/lib64/python2.7/logging/__init__.py", line 902, in __init__ >> Jul 29 01:19:21 daemonAdapter[39124]: >> StreamHandler.__init__(self, self._open()) >> Jul 29 01:19:21 daemonAdapter[39124]: File >> "/usr/lib64/python2.7/logging/__init__.py", line 925, in _open >> Jul 29 01:19:21 daemonAdapter[39124]: stream = >> open(self.baseFilename, self.mode) >> Jul 29 01:19:21 daemonAdapter[39124]: IOError: [Errno 6] No such >> device or address: '/dev/stdout' >> Jul 29 01:19:21 systemd[1]: supervdsmd.service: main process >> exited, code=exited, status=1/FAILURE >> >> >> >> On Sun, Jul 30, 2017 at 3:47 PM, Yaniv Kaul wrote: >> >>> >>> >>> On Fri, Jul 28, 2017 at 8:37 PM, Richard Chan < >>> rich...@treeboxsolutions.com> wrote: >>> After an upgrade to 4.0 I have a single host that cannot start supervdsmd because of IOError on /dev/stdout. All other hosts upgraded correctly. >>> >>> Upgrade from which version? 3.6? Did you stay on 4.0 or continued to 4.1? >>> >>> In the systemd unit I have to hack StandardOutput=null. Any thing I have overlooked? The hosts are all identical and it is just this one that has this weird behaviour. >>> >>> Any logs that you can share? >>> Y. >>> >>> -- Richard Chan ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users >>> >> >> >> -- >> Richard Chan >> Chief Architect >> >> TreeBox Solutions Pte Ltd >> 1 Commonwealth Lane #03-01 >> Singapore 149544 >> Tel: 6570 3725 >> http://www.treeboxsolutions.com >> >> Co.Reg.No. 201100585R >> ___ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> > -- > Yaniv Bronhaim. > -- Richard Chan Chief Architect TreeBox Solutions Pte Ltd 1 Commonwealth Lane #03-01 Singapore 149544 Tel: 6570 3725 http://www.treeboxsolutions.com Co.Reg.No. 201100585R ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] supervdsmd IOError to /dev/stdout
Hi, Seems like I already bumped in such issue awhile ago - https://bugzilla.redhat.com/show_bug.cgi?id=1216880 but now I see what's wrong - it happens after failing to read /etc/vdsm/svdsm.logger.conf for some reason then we have a bug in our fallback which tries to run logging.basicConfig(filename='/dev/stdout', filemode='w+', level=logging.DEBUG) and this fails while running as systemd daemon sorry about that. check why it can't load /etc/vdsm/svdsm.logger.conf and update us after fixing the access issue check if you can run logging.config.fileConfig("/etc/vdsm/svdsm.logger.conf" , disable_existing_loggers=False) Then supervdsmd should start properly I'll reopen the bugzilla and fix the wrong fallback to syslog print Thanks! On Sun, Jul 30, 2017 at 3:23 PM Richard Chan wrote: > Hi Yaniv, > > On this one node, it happened from 3.6 -> 4.0. It persisted after 4.0->4.1. > Current: vdsm-4.19.24-1.el7.centos.x86_64 > > > In systemd override I now have to have: > StandardOutput=null > > > > > Jul 29 01:19:21 systemd[1]: Starting Auxiliary vdsm service for > running helper functions as root... > Jul 29 01:19:21 python2[39124]: detected unhandled Python > exception in '/usr/share/vdsm/supervdsmServer' > Jul 29 01:19:21 daemonAdapter[39124]: Traceback (most recent call > last): > Jul 29 01:19:21 daemonAdapter[39124]: File > "/usr/share/vdsm/supervdsmServer", line 45, in > Jul 29 01:19:21 daemonAdapter[39124]: level=logging.DEBUG) > Jul 29 01:19:21 daemonAdapter[39124]: File > "/usr/lib64/python2.7/logging/__init__.py", line 1529, in basicConfig > Jul 29 01:19:21 daemonAdapter[39124]: hdlr = > FileHandler(filename, mode) > Jul 29 01:19:21 daemonAdapter[39124]: File > "/usr/lib64/python2.7/logging/__init__.py", line 902, in __init__ > Jul 29 01:19:21 daemonAdapter[39124]: > StreamHandler.__init__(self, self._open()) > Jul 29 01:19:21 daemonAdapter[39124]: File > "/usr/lib64/python2.7/logging/__init__.py", line 925, in _open > Jul 29 01:19:21 daemonAdapter[39124]: stream = > open(self.baseFilename, self.mode) > Jul 29 01:19:21 daemonAdapter[39124]: IOError: [Errno 6] No such > device or address: '/dev/stdout' > Jul 29 01:19:21 systemd[1]: supervdsmd.service: main process > exited, code=exited, status=1/FAILURE > > > > On Sun, Jul 30, 2017 at 3:47 PM, Yaniv Kaul wrote: > >> >> >> On Fri, Jul 28, 2017 at 8:37 PM, Richard Chan < >> rich...@treeboxsolutions.com> wrote: >> >>> After an upgrade to 4.0 I have a single host that cannot start >>> supervdsmd because of IOError on /dev/stdout. All other hosts upgraded >>> correctly. >>> >> >> Upgrade from which version? 3.6? Did you stay on 4.0 or continued to 4.1? >> >> >>> >>> In the systemd unit I have to hack StandardOutput=null. >>> >>> Any thing I have overlooked? The hosts are all identical and it is just >>> this one >>> that has this weird behaviour. >>> >> >> Any logs that you can share? >> Y. >> >> >>> >>> -- >>> Richard Chan >>> >>> >>> ___ >>> Users mailing list >>> Users@ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >>> >>> >> > > > -- > Richard Chan > Chief Architect > > TreeBox Solutions Pte Ltd > 1 Commonwealth Lane #03-01 > Singapore 149544 > Tel: 6570 3725 > http://www.treeboxsolutions.com > > Co.Reg.No. 201100585R > ___ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > -- Yaniv Bronhaim. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] supervdsmd IOError to /dev/stdout
Hi Yaniv, On this one node, it happened from 3.6 -> 4.0. It persisted after 4.0->4.1. Current: vdsm-4.19.24-1.el7.centos.x86_64 In systemd override I now have to have: StandardOutput=null Jul 29 01:19:21 systemd[1]: Starting Auxiliary vdsm service for running helper functions as root... Jul 29 01:19:21 python2[39124]: detected unhandled Python exception in '/usr/share/vdsm/supervdsmServer' Jul 29 01:19:21 daemonAdapter[39124]: Traceback (most recent call last): Jul 29 01:19:21 daemonAdapter[39124]: File "/usr/share/vdsm/supervdsmServer", line 45, in Jul 29 01:19:21 daemonAdapter[39124]: level=logging.DEBUG) Jul 29 01:19:21 daemonAdapter[39124]: File "/usr/lib64/python2.7/logging/__init__.py", line 1529, in basicConfig Jul 29 01:19:21 daemonAdapter[39124]: hdlr = FileHandler(filename, mode) Jul 29 01:19:21 daemonAdapter[39124]: File "/usr/lib64/python2.7/logging/__init__.py", line 902, in __init__ Jul 29 01:19:21 daemonAdapter[39124]: StreamHandler.__init__(self, self._open()) Jul 29 01:19:21 daemonAdapter[39124]: File "/usr/lib64/python2.7/logging/__init__.py", line 925, in _open Jul 29 01:19:21 daemonAdapter[39124]: stream = open(self.baseFilename, self.mode) Jul 29 01:19:21 daemonAdapter[39124]: IOError: [Errno 6] No such device or address: '/dev/stdout' Jul 29 01:19:21 systemd[1]: supervdsmd.service: main process exited, code=exited, status=1/FAILURE On Sun, Jul 30, 2017 at 3:47 PM, Yaniv Kaul wrote: > > > On Fri, Jul 28, 2017 at 8:37 PM, Richard Chan < > rich...@treeboxsolutions.com> wrote: > >> After an upgrade to 4.0 I have a single host that cannot start supervdsmd >> because of IOError on /dev/stdout. All other hosts upgraded correctly. >> > > Upgrade from which version? 3.6? Did you stay on 4.0 or continued to 4.1? > > >> >> In the systemd unit I have to hack StandardOutput=null. >> >> Any thing I have overlooked? The hosts are all identical and it is just >> this one >> that has this weird behaviour. >> > > Any logs that you can share? > Y. > > >> >> -- >> Richard Chan >> >> >> ___ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> >> > -- Richard Chan Chief Architect TreeBox Solutions Pte Ltd 1 Commonwealth Lane #03-01 Singapore 149544 Tel: 6570 3725 http://www.treeboxsolutions.com Co.Reg.No. 201100585R ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] supervdsmd IOError to /dev/stdout
On Fri, Jul 28, 2017 at 8:37 PM, Richard Chan wrote: > After an upgrade to 4.0 I have a single host that cannot start supervdsmd > because of IOError on /dev/stdout. All other hosts upgraded correctly. > Upgrade from which version? 3.6? Did you stay on 4.0 or continued to 4.1? > > In the systemd unit I have to hack StandardOutput=null. > > Any thing I have overlooked? The hosts are all identical and it is just > this one > that has this weird behaviour. > Any logs that you can share? Y. > > -- > Richard Chan > > > ___ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] supervdsmd IOError to /dev/stdout
After an upgrade to 4.0 I have a single host that cannot start supervdsmd because of IOError on /dev/stdout. All other hosts upgraded correctly. In the systemd unit I have to hack StandardOutput=null. Any thing I have overlooked? The hosts are all identical and it is just this one that has this weird behaviour. -- Richard Chan ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users