Re: Regression in Frameworks - DBus Hangs
On Sat, Nov 17, 2018 at 9:31 PM Albert Astals Cid wrote: > > El diumenge, 11 de novembre de 2018, a les 11:29:51 CET, Albert Astals Cid va > escriure: > > El diumenge, 11 de novembre de 2018, a les 6:51:39 CET, Ben Cooksley va > > escriure: > > > On Thu, Nov 8, 2018 at 8:15 PM Ben Cooksley wrote: > > > > > > > > On Sat, Nov 3, 2018 at 4:41 PM Ben Cooksley wrote: > > > > > > > > > > Hi all, > > > > > > > > > > Following commits made to Frameworks somewhere in the October 21 to > > > > > October 28 time frame, the unit tests of Konsole on the CI system have > > > > > started to hang following completion. > > > > > > > > > > This hang is occurring due to a behaviour of CTest, where it will wait > > > > > (indefinitely) following the conclusion of a test for processes > > > > > spawned by that test to exit. > > > > > > > > > > Previously, the "konsole --separate" process launched by Konsole's > > > > > dbus unit test would exit normally, however following the changes in > > > > > Frameworks it now hangs and does not exit as expected. > > > > > > > > > > This is causing quite a bit of trouble as these builds require manual > > > > > Sysadmin intervention in order to have the jobs complete successfully. > > > > > > > > > > Could someone take a look into this regression please? > > > > > > > > > > Regretfully it isn't possible to run GDB within the CI system > > > > > environment (Docker drops CAP_SYS_PTRACE) so i'm unable to get a > > > > > backtrace out of Konsole to find where it is stuck. > > > > > > > > Ping? > > > > > > > > Anyone got any idea why this behaviour has regressed? > > > > > > The complete lack of response to this issue (and any mail to this > > > mailing list i've noticed, I highly suspect everyone just filters it > > > into a corner in their mail client and doesn't read it at all) is > > > really disappointing. > > > > > > Given that Kurt has disabled the test within Konsole this doesn't have > > > any urgency anymore. > > > > > > None the less though the fact a regression with > > > hang-on-application-close implications was detected by the CI system > > > and then completely ignored by those who broke it brings up other > > > concerns, and essentially means the CI system is given no regard by > > > the Frameworks developers. > > > > > > As a consequence i'll be terminating CI and Phabricator notification > > > mails to the Frameworks mailing list, effective immediately. > > > > What no. Who are you to decide that. Restore those emails. > > We're still missing the CI emails. This has now been corrected. > > Cheers, > Albert > Cheers, Ben > > > > Albert > > > > > > > > > > > > > > > > > > > Thanks, > > > > > Ben > > > > > > > > Regards, > > > > Ben > > > > > > Regards, > > > Ben Cooksley > > > KDE Sysadmin > > > > > > > > > > > > > > > > >
Re: Regression in Frameworks - DBus Hangs
El diumenge, 11 de novembre de 2018, a les 11:29:51 CET, Albert Astals Cid va escriure: > El diumenge, 11 de novembre de 2018, a les 6:51:39 CET, Ben Cooksley va > escriure: > > On Thu, Nov 8, 2018 at 8:15 PM Ben Cooksley wrote: > > > > > > On Sat, Nov 3, 2018 at 4:41 PM Ben Cooksley wrote: > > > > > > > > Hi all, > > > > > > > > Following commits made to Frameworks somewhere in the October 21 to > > > > October 28 time frame, the unit tests of Konsole on the CI system have > > > > started to hang following completion. > > > > > > > > This hang is occurring due to a behaviour of CTest, where it will wait > > > > (indefinitely) following the conclusion of a test for processes > > > > spawned by that test to exit. > > > > > > > > Previously, the "konsole --separate" process launched by Konsole's > > > > dbus unit test would exit normally, however following the changes in > > > > Frameworks it now hangs and does not exit as expected. > > > > > > > > This is causing quite a bit of trouble as these builds require manual > > > > Sysadmin intervention in order to have the jobs complete successfully. > > > > > > > > Could someone take a look into this regression please? > > > > > > > > Regretfully it isn't possible to run GDB within the CI system > > > > environment (Docker drops CAP_SYS_PTRACE) so i'm unable to get a > > > > backtrace out of Konsole to find where it is stuck. > > > > > > Ping? > > > > > > Anyone got any idea why this behaviour has regressed? > > > > The complete lack of response to this issue (and any mail to this > > mailing list i've noticed, I highly suspect everyone just filters it > > into a corner in their mail client and doesn't read it at all) is > > really disappointing. > > > > Given that Kurt has disabled the test within Konsole this doesn't have > > any urgency anymore. > > > > None the less though the fact a regression with > > hang-on-application-close implications was detected by the CI system > > and then completely ignored by those who broke it brings up other > > concerns, and essentially means the CI system is given no regard by > > the Frameworks developers. > > > > As a consequence i'll be terminating CI and Phabricator notification > > mails to the Frameworks mailing list, effective immediately. > > What no. Who are you to decide that. Restore those emails. We're still missing the CI emails. Cheers, Albert > > Albert > > > > > > > > > > > > > > Thanks, > > > > Ben > > > > > > Regards, > > > Ben > > > > Regards, > > Ben Cooksley > > KDE Sysadmin > > > > > > >
Re: Regression in Frameworks - DBus Hangs
On Sat, Nov 17, 2018 at 4:30 AM David Edmundson wrote: > > > Don't thank me yet, I locked up one of the build jobs, sorry. > https://build.kde.org/job/Applications/job/konsole/job/kf5-qt5%20SUSEQt5.9/380/console Diagnostic output available from D-Bus: jenkins@6934489e2599:~> qdbus org.kde.konsole-2448 /MainApplication org.qtproject.Qt.QCoreApplication.quitLockEnabled true jenkins@6934489e2599:~> qdbus org.kde.konsole-2448 /MainApplication org.qtproject.Qt.QGuiApplication.quitOnLastWindowClosed true (Because Jenkins started the container, running gdb to get a backtrace isn't possible) Based on this i'd say your theory we have a QEventLoopLocker in play is probably the right one here. It's a bit of a workaround, but could the test change to just asking Konsole to exit rather than close it's window? > > >Pipeline kf5-qt5 SUSEQt5.9 > > Output isn't too helpful, though I am surprised to see > kf5.kinit.klauncher in the logs when I'm supposedly invoking the process > directly. That happens whenever it sees a new name on the D-Bus server, so is unrelated (you'll see a few towards the start of the DBusTest output - that was me doing the above and asking it to quit nicely) > > David Cheers, Ben
Re: Regression in Frameworks - DBus Hangs
Don't thank me yet, I locked up one of the build jobs, sorry. https://build.kde.org/job/Applications/job/konsole/job/kf5-qt5%20SUSEQt5.9/380/console >Pipeline kf5-qt5 SUSEQt5.9 Output isn't too helpful, though I am surprised to see kf5.kinit.klauncher in the logs when I'm supposedly invoking the process directly. David
Re: Regression in Frameworks - DBus Hangs
On Fri, Nov 16, 2018 at 11:04 PM David Edmundson wrote: > > A running KJob would not show in a backtrace. > > Update at https://phabricator.kde.org/D16919 Ah I see. Thanks for taking a look into that. Cheers, Ben
Re: Regression in Frameworks - DBus Hangs
A running KJob would not show in a backtrace. Update at https://phabricator.kde.org/D16919
Re: Regression in Frameworks - DBus Hangs
On Wed, Nov 14, 2018 at 8:17 AM Ben Cooksley wrote: > > On Tue, Nov 13, 2018 at 2:25 PM David Edmundson > wrote: > > > > I don't think application CI jobs run with workspace available. I also > > can't think of anything that's been merged in workspace. > > That's correct. CI Jobs are only provided with the things they depend on. > > > > > --- > > quitOnLastWindowClosed is also blocked if a QEventLoopLocker is placed on > > the main application. > > > > All KJobs do this and a Kjob existing and hanging seems plausible, but > > that's a lot harder to grep for. > > Would it show in a backtrace perhaps? > > > > > All stdout/stderr of our newly spawned konsole should be forwarded to the > > test. > > @Ben is that available? > > Ideally with QT_LOGGING_RULES=*.debug=true > > The CI system sets QT_LOGGING_RULES=*.debug=true globally prior to > executing tests [1], so that should definitely be the case, so i'd say > the test is silencing the messages :( > > Kurt, any ideas here on what would be the best way to get at the > output from the Konsole process launched by the test? Ping? > > > > > > > > > > > > > > > [1]: > https://cgit.kde.org/sysadmin/ci-tooling.git/tree/helpers/run-tests.py#n94 > > Thanks, > Ben Cheers, Ben
Re: Regression in Frameworks - DBus Hangs
On Tue, Nov 13, 2018 at 2:25 PM David Edmundson wrote: > > I don't think application CI jobs run with workspace available. I also can't > think of anything that's been merged in workspace. That's correct. CI Jobs are only provided with the things they depend on. > > --- > quitOnLastWindowClosed is also blocked if a QEventLoopLocker is placed on the > main application. > > All KJobs do this and a Kjob existing and hanging seems plausible, but that's > a lot harder to grep for. Would it show in a backtrace perhaps? > > All stdout/stderr of our newly spawned konsole should be forwarded to the > test. > @Ben is that available? > Ideally with QT_LOGGING_RULES=*.debug=true The CI system sets QT_LOGGING_RULES=*.debug=true globally prior to executing tests [1], so that should definitely be the case, so i'd say the test is silencing the messages :( Kurt, any ideas here on what would be the best way to get at the output from the Konsole process launched by the test? > > > > > > [1]: https://cgit.kde.org/sysadmin/ci-tooling.git/tree/helpers/run-tests.py#n94 Thanks, Ben
Re: Regression in Frameworks - DBus Hangs
I don't think application CI jobs run with workspace available. I also can't think of anything that's been merged in workspace. --- quitOnLastWindowClosed is also blocked if a QEventLoopLocker is placed on the main application. All KJobs do this and a Kjob existing and hanging seems plausible, but that's a lot harder to grep for. All stdout/stderr of our newly spawned konsole should be forwarded to the test. @Ben is that available? Ideally with QT_LOGGING_RULES=*.debug=true
Re: Regression in Frameworks - DBus Hangs
On Mon, Nov 12, 2018 at 10:52:17PM +1300, Ben Cooksley wrote:
> In this instance I hoped someone would know of a behaviour change they
> introduced around D-Bus or application shutdown stuff, particularly
> given the time window i'd provided.
>
> >
> > I imagine the majority of people on this list is in the same situation. I
> > don't know what can be realistically expected.
> > It's quite a jump to go from that to saying people don't check emails.
> >
> > David
>
> Based on this i've run some additional diagnostics on a CI worker this
> evening, and based on my reading of the test code it would appear that
> the culprit is Konsole no longer exiting when it's last window is
> closed. Konsole's code didn't change at all between the test working
> and it breaking, which indicates this is a Frameworks regression.
>
> The test does this:
>
> QDBusInterface iface(_interfaceName,
> QStringLiteral("/konsole/MainWindow_1"),
> QStringLiteral("org.qtproject.Qt.QWidget"));
> QVERIFY2(iface.isValid(), "Unable to get a dbus interface to Konsole!");
>
> QDBusReply instanceReply = iface.call(QStringLiteral("close"));
>
> Which indeed results in the Window closing, at least as far as D-Bus
> is concerned:
>
> jenkins@0f9cdb19eeb8:/home/jenkins/konsole> qdbus org.kde.konsole-3065
> /
> /KBookmarkManager
> /KBookmarkManager/konsole
> /MainApplication
> /org
> /org/kde
> /org/kde/konsole
>
> I also confirmed this by taking a screenshot of the Xvfb instance
> being used by this session, which showed nothing but black -
> confirming the window was indeed closed.
>
> Instructing Konsole to close by running "qdbus org.kde.konsole-3065
> /MainApplication quit" caused the test to finish (as expected).
>
> Thoughts anyone?
There is a Qt property on QApplication which is devoted to precisely
this feature. The property, "quitOnLastWindowClosed" will cause the
application to automatically quit if the last window has been closed.
It dates back to Qt 4 and a cursory search of qtbase, KF5 and
kde/workspace for any commits that may have changed the default for this
property [1] don't show any recent changes (disabling it when it was
previously enabled by default or vice versa).
Manually enabling the property in konsole might be enough to make the
test pass. That wouldn't explain what the "regression" was but it seems
strictly more correct either way.
If it's a recent change then it may be related to distro
changes to Qt, something to do with Plasma integration (which is why I
searched in kde/workspace), or some other aspect of the way KF5-based
applications configure themselves by default based on the settings of
the desktop they are being run within.
The only reference of any sort that I see in either KF5 or kde/workspace
for "quitOnLastWindowClosed" is in kxmlgui (for KMainWindow). There
aren't many recent commits so anyone who can reproduce the hang should
have a quick job verifying if reverting some of those would help or not.
[1] Using git -P log --since="8 weeks ago" -G quitOnLastWindow --oneline
for each git repository
Regards,
- Michael Pyne
Re: Regression in Frameworks - DBus Hangs
On Mon, Nov 12, 2018 at 7:27 AM David Edmundson
wrote:
>
> I ran the test locally which passed, then left it for someone else as:
> - I can't reproduce locally.
> - There are absolutely no logs on build.k.o to give a clue.
> - I have no access to build.k.o to do anything (AFAIK).
> - Even the phab ticket cited in the commit that disables the test is not
> accessible to the public.
I see. Had I received a response saying "I can't reproduce this" then
we could have gone further in this.
>From my perspective though it looks like nobody has even looked into
this at all as i've no way of knowing if people have read it or tried
to do something about it.
Knowing that others can't reproduce it locally is important and makes
quite the difference.
In this instance I hoped someone would know of a behaviour change they
introduced around D-Bus or application shutdown stuff, particularly
given the time window i'd provided.
>
> I imagine the majority of people on this list is in the same situation. I
> don't know what can be realistically expected.
> It's quite a jump to go from that to saying people don't check emails.
>
> David
Based on this i've run some additional diagnostics on a CI worker this
evening, and based on my reading of the test code it would appear that
the culprit is Konsole no longer exiting when it's last window is
closed. Konsole's code didn't change at all between the test working
and it breaking, which indicates this is a Frameworks regression.
The test does this:
QDBusInterface iface(_interfaceName,
QStringLiteral("/konsole/MainWindow_1"),
QStringLiteral("org.qtproject.Qt.QWidget"));
QVERIFY2(iface.isValid(), "Unable to get a dbus interface to Konsole!");
QDBusReply instanceReply = iface.call(QStringLiteral("close"));
Which indeed results in the Window closing, at least as far as D-Bus
is concerned:
jenkins@0f9cdb19eeb8:/home/jenkins/konsole> qdbus org.kde.konsole-3065
/
/KBookmarkManager
/KBookmarkManager/konsole
/MainApplication
/org
/org/kde
/org/kde/konsole
I also confirmed this by taking a screenshot of the Xvfb instance
being used by this session, which showed nothing but black -
confirming the window was indeed closed.
Instructing Konsole to close by running "qdbus org.kde.konsole-3065
/MainApplication quit" caused the test to finish (as expected).
Thoughts anyone?
Regards,
Ben
Re: Regression in Frameworks - DBus Hangs
I ran the test locally which passed, then left it for someone else as: - I can't reproduce locally. - There are absolutely no logs on build.k.o to give a clue. - I have no access to build.k.o to do anything (AFAIK). - Even the phab ticket cited in the commit that disables the test is not accessible to the public. I imagine the majority of people on this list is in the same situation. I don't know what can be realistically expected. It's quite a jump to go from that to saying people don't check emails. David
Re: Regression in Frameworks - DBus Hangs
Hi, > Ping? >> > >> > Anyone got any idea why this behaviour has regressed? >> >> The complete lack of response to this issue (and any mail to this >> mailing list i've noticed, I highly suspect everyone just filters it >> into a corner in their mail client and doesn't read it at all) is >> really disappointing. >> >> Given that Kurt has disabled the test within Konsole this doesn't have >> any urgency anymore. >> >> None the less though the fact a regression with >> hang-on-application-close implications was detected by the CI system >> and then completely ignored by those who broke it brings up other >> concerns, and essentially means the CI system is given no regard by >> the Frameworks developers. >> >> As a consequence i'll be terminating CI and Phabricator notification >> mails to the Frameworks mailing list, effective immediately. I think that is a bit harsh. I assume nobody replied because nobody has an idea which commit it was (at least I have none). The mails are really useful, I am not sure that cutting them off will help. Greetings Christoph P.S. I appreciate all the work you put in the CI (and all other stuff you admin)! -- - Dr.-Ing. Christoph Cullmann - AbsInt Angewandte Informatik GmbH Email: cullm...@absint.com Science Park 1 Tel: +49-681-38360-22 66123 Saarbrücken Fax: +49-681-38360-20 GERMANYWWW: http://www.AbsInt.com Geschäftsführung: Dr.-Ing. Christian Ferdinand Eingetragen im Handelsregister des Amtsgerichts Saarbrücken, HRB 11234
Re: Regression in Frameworks - DBus Hangs
El diumenge, 11 de novembre de 2018, a les 6:51:39 CET, Ben Cooksley va escriure: > On Thu, Nov 8, 2018 at 8:15 PM Ben Cooksley wrote: > > > > On Sat, Nov 3, 2018 at 4:41 PM Ben Cooksley wrote: > > > > > > Hi all, > > > > > > Following commits made to Frameworks somewhere in the October 21 to > > > October 28 time frame, the unit tests of Konsole on the CI system have > > > started to hang following completion. > > > > > > This hang is occurring due to a behaviour of CTest, where it will wait > > > (indefinitely) following the conclusion of a test for processes > > > spawned by that test to exit. > > > > > > Previously, the "konsole --separate" process launched by Konsole's > > > dbus unit test would exit normally, however following the changes in > > > Frameworks it now hangs and does not exit as expected. > > > > > > This is causing quite a bit of trouble as these builds require manual > > > Sysadmin intervention in order to have the jobs complete successfully. > > > > > > Could someone take a look into this regression please? > > > > > > Regretfully it isn't possible to run GDB within the CI system > > > environment (Docker drops CAP_SYS_PTRACE) so i'm unable to get a > > > backtrace out of Konsole to find where it is stuck. > > > > Ping? > > > > Anyone got any idea why this behaviour has regressed? > > The complete lack of response to this issue (and any mail to this > mailing list i've noticed, I highly suspect everyone just filters it > into a corner in their mail client and doesn't read it at all) is > really disappointing. > > Given that Kurt has disabled the test within Konsole this doesn't have > any urgency anymore. > > None the less though the fact a regression with > hang-on-application-close implications was detected by the CI system > and then completely ignored by those who broke it brings up other > concerns, and essentially means the CI system is given no regard by > the Frameworks developers. > > As a consequence i'll be terminating CI and Phabricator notification > mails to the Frameworks mailing list, effective immediately. What no. Who are you to decide that. Restore those emails. Albert > > > > > > > > > Thanks, > > > Ben > > > > Regards, > > Ben > > Regards, > Ben Cooksley > KDE Sysadmin >
Re: Regression in Frameworks - DBus Hangs
On Thu, Nov 8, 2018 at 8:15 PM Ben Cooksley wrote: > > On Sat, Nov 3, 2018 at 4:41 PM Ben Cooksley wrote: > > > > Hi all, > > > > Following commits made to Frameworks somewhere in the October 21 to > > October 28 time frame, the unit tests of Konsole on the CI system have > > started to hang following completion. > > > > This hang is occurring due to a behaviour of CTest, where it will wait > > (indefinitely) following the conclusion of a test for processes > > spawned by that test to exit. > > > > Previously, the "konsole --separate" process launched by Konsole's > > dbus unit test would exit normally, however following the changes in > > Frameworks it now hangs and does not exit as expected. > > > > This is causing quite a bit of trouble as these builds require manual > > Sysadmin intervention in order to have the jobs complete successfully. > > > > Could someone take a look into this regression please? > > > > Regretfully it isn't possible to run GDB within the CI system > > environment (Docker drops CAP_SYS_PTRACE) so i'm unable to get a > > backtrace out of Konsole to find where it is stuck. > > Ping? > > Anyone got any idea why this behaviour has regressed? The complete lack of response to this issue (and any mail to this mailing list i've noticed, I highly suspect everyone just filters it into a corner in their mail client and doesn't read it at all) is really disappointing. Given that Kurt has disabled the test within Konsole this doesn't have any urgency anymore. None the less though the fact a regression with hang-on-application-close implications was detected by the CI system and then completely ignored by those who broke it brings up other concerns, and essentially means the CI system is given no regard by the Frameworks developers. As a consequence i'll be terminating CI and Phabricator notification mails to the Frameworks mailing list, effective immediately. > > > > > Thanks, > > Ben > > Regards, > Ben Regards, Ben Cooksley KDE Sysadmin
Re: Regression in Frameworks - DBus Hangs
On Sat, Nov 3, 2018 at 4:41 PM Ben Cooksley wrote: > > Hi all, > > Following commits made to Frameworks somewhere in the October 21 to > October 28 time frame, the unit tests of Konsole on the CI system have > started to hang following completion. > > This hang is occurring due to a behaviour of CTest, where it will wait > (indefinitely) following the conclusion of a test for processes > spawned by that test to exit. > > Previously, the "konsole --separate" process launched by Konsole's > dbus unit test would exit normally, however following the changes in > Frameworks it now hangs and does not exit as expected. > > This is causing quite a bit of trouble as these builds require manual > Sysadmin intervention in order to have the jobs complete successfully. > > Could someone take a look into this regression please? > > Regretfully it isn't possible to run GDB within the CI system > environment (Docker drops CAP_SYS_PTRACE) so i'm unable to get a > backtrace out of Konsole to find where it is stuck. Ping? Anyone got any idea why this behaviour has regressed? > > Thanks, > Ben Regards, Ben
Regression in Frameworks - DBus Hangs
Hi all, Following commits made to Frameworks somewhere in the October 21 to October 28 time frame, the unit tests of Konsole on the CI system have started to hang following completion. This hang is occurring due to a behaviour of CTest, where it will wait (indefinitely) following the conclusion of a test for processes spawned by that test to exit. Previously, the "konsole --separate" process launched by Konsole's dbus unit test would exit normally, however following the changes in Frameworks it now hangs and does not exit as expected. This is causing quite a bit of trouble as these builds require manual Sysadmin intervention in order to have the jobs complete successfully. Could someone take a look into this regression please? Regretfully it isn't possible to run GDB within the CI system environment (Docker drops CAP_SYS_PTRACE) so i'm unable to get a backtrace out of Konsole to find where it is stuck. Thanks, Ben
