Re: [Libreoffice-qa] minutes of ESC call ...
Hi, On Thu, May 28, 2015 at 06:01:33PM +, Noel Grandin wrote: Wasn't someone going to look at speeding that up? Make check is not currently making very good use of multiple cores and it has a long tail where only one or two cores are busy. I cant fully confirm that -- at least not combined with Michaels 15 minutes number. Good ol' big Bertha with two Opteron 6272s has 32 cores, but mediocre single-threaded performance compared with some of todays offerings. Still[1]: -j real 5m 9.091s user25m12.250s sys 2m53.420s -j32 real 5m30.133s user29m46.310s sys 3m10.660s -j8 real 6m35.434s user19m36.700s sys 2m 6.390s -j1 real34m11.814s user17m51.990s sys 1m52.470s (all above along with moderate desktop use alongside: Firefox, mplayer audio streaming etc. -- so not too scientific.) And with this I see a lot of processes running parallel till the end. This suggests to me that the stuff is quite parallelized -- however none of the testing threads seem to be CPU-bound rather the Java-stuff seems to be IO bound (for IPC?) with the slowest test unit taking ~5 minutes. Thus even 32 jobs or more dont seem to keep more than one core busy. At least for people with a meager duo-core laptop or somesuch, this suggests you should try: time make check PARALLELISM= # aka forkbomb me please And if that doesnt explode your RAM, it should help you quite a bit and you should end up with 5 minutes for 'make check' -- quite independant of your hardware, as much of that seems IO-bound idling. If we want to cut down on that 5 minutes, we would need to touch the JunitTest internals. Best, Bjoern [1] As a sidenote: I noted that linking with debug symbols takes at least 4 times as long as without debug symbols. Given that linking happens at the end of the build with a narrow dependency tree left (as libraries are daisy-chained dependency-wise for linking), you are really penalized for building with symbols when you dont need those, as it can easily dry out paralellism. So dont do that: Rather than building with symbols everywhere and then skipping on tests, better build without symbols unless you are debugging, but _run the tests_. [2] On my ancient notebook with a i7-2720QM (a modern 4th gen i7-4770HQ is at least 50% faster at the same speed) I find the following with disabling Junittest_toolkit_unoapi as it was failing on that checkout. time make check PARALLELISM=: reall5m15.351s user15m52.808s sys 1m10.073s time make check (thus -j8): real 6m2.806s user14m32.157s sys 1m1.861s time make check PARALLELISM=1: real41m 7.984s user27m33.174s sys 1m13.949s ___ List Name: Libreoffice-qa mailing list Mail address: Libreoffice-qa@lists.freedesktop.org Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://lists.freedesktop.org/archives/libreoffice-qa/
Re: [Libreoffice-qa] minutes of ESC call ...
Hi, On Thu, May 28, 2015 at 10:40:40PM +0200, Bjoern Michaelsen wrote: And with this I see a lot of processes running parallel till the end. This suggests to me that the stuff is quite parallelized -- however none of the testing threads seem to be CPU-bound rather the Java-stuff seems to be IO bound (for IPC?) with the slowest test unit taking ~5 minutes. Thus even 32 jobs or more dont seem to keep more than one core busy. So, I has a look at what is so slow in subsequentcheck: sc_unoapi4m21.137s sw_unoapi3m34.494s toolkit_unoapi 3m18.056s forms_unoapi 2m51.588s sw_complex 1m 6.937s framework_unoapi 1m 2.938s svx_unoapi 0m40.205s sd_unoapi0m38.568s chart2_unoapi0m37.695s xmloff_unoapi0m29.696s ucb_unoapi 0m25.288s starmath_unoapi 0m24.856s pragmatically splitting up the first four: https://gerrit.libreoffice.org/#/c/15959/ brings 'make check' down to 2m6s on big Bertha. For comparison: find instdir/ -name '*lo.so'|xargs rm; time make that is, just relinking (most) LibreOffice libs takes 2m26s on that machine without symbols (assume more than 10 minutes with symbols). Given this relation, there is no excuse to not run 'make check' anymore at least before pushing -- if there ever was one. Best, Bjoern ___ List Name: Libreoffice-qa mailing list Mail address: Libreoffice-qa@lists.freedesktop.org Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://lists.freedesktop.org/archives/libreoffice-qa/
Re: [Libreoffice-qa] organizing our crasher bugs ?
On 05/27/2015 07:25 AM, Caolán McNamara wrote: There are ~190 bugs with the word crash in the summary in the states new/assigned/reopened/unconfirmed. As with the coverity, import-testing, export-testing stuff I think it would be helpful to start chewing into crashers in some systematic way where progress can be measured. 190 isn't a vast number in the over all scheme of things. What would work ideally for me is to someone get the subset of all crashes that are ~100% reproducible under Linux and ordered by the number of steps required to reproduce. Any ideas on how to generate that subset ? Bjoern already largely addressed this but bugzilla is hugely limited for organizing. What I think would be minimum (and can be done) is: Confirm each crash on 5.0; _Ensure_ that each has easy steps; _Ensure_ that each has been checked for regressions (and tagged accordingly in whiteboard/version field); _Ensure_ that each has a debug log; Prioritize correctly (Major/Critical High/Highest) For the most straight forward ones that have easy steps we can just mark them as critical + highest - signaling that they are easy to reproduce and everything is there for devs to tackle them. Best, Joel FYIW: I've been pushing this for about a year now ;) ___ List Name: Libreoffice-qa mailing list Mail address: Libreoffice-qa@lists.freedesktop.org Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://lists.freedesktop.org/archives/libreoffice-qa/
Re: [Libreoffice-qa] minutes of ESC call ...
On Thu, May 28, 2015 at 7:37 PM, Bjoern Michaelsen bjoern.michael...@canonical.com wrote: there is no excuse to not run 'make check' anymore at least before pushing -- if there ever was one. you mean beside the fact that make check fails on Windows _and_ Mac ? ___ List Name: Libreoffice-qa mailing list Mail address: Libreoffice-qa@lists.freedesktop.org Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://lists.freedesktop.org/archives/libreoffice-qa/
Re: [Libreoffice-qa] minutes of ESC call ...
On Thu, May 28, 2015 at 11:16 AM, Michael Meeks michael.me...@collabora.com wrote: + Windows 64 status (David O) + all fine, poking some Base test with Stephan + an OLE test, failing on some machines not others; environment dependent. Thorsten's machine doesn't crash. That is the exception not the rule. I could not get a build to finish, and cloph also reproduce the ole bug. Until Win64 can reliably build, I can't pull that platform under jenkins/gerrit-build (Not that I could right this minutes anyway, still working on bring hardware online to cope with growing demand) Norbert ___ List Name: Libreoffice-qa mailing list Mail address: Libreoffice-qa@lists.freedesktop.org Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://lists.freedesktop.org/archives/libreoffice-qa/
[Libreoffice-qa] minutes of ESC call ...
* Present: + Andras, Caolan, Kendy, Sophie, Michael M, David T, Bjoern, David O, Robinson, Miklos, Muthu, Thorsten, Lionel * Completed Action Items: + add ooenv bits for cppunittests (Michael) + take pretty Cambridge Hackfest pictures (Thorsten) [ pending permission from those included ] + firebird_sdbc disable by default in the Win64 config (Thorsten) + disable by default in thunderbird in favour of mork on Win64 (Thorsten) + please forward GSOC phone numbers to Fridrich/Thorsten/Cedric (Thorsten, Kendy) + provide a new Easy Hacks query for the minutes (Robinson) [ not a case sensitive query = not needed ] + make --enable-pch a dummy option for windows (Norbert) + same for ccache on windows (Norbert) * Pending Action Items: + find some developer/sysadmin'y type people to train in CI maintenance (Cloph) [ no time scheduled; after the call ] + move problem writer tests from check - subsequentcheck for sw.all (Michael S) + add ooenv MALLOC_CHECK etc. to script wrapper for tinderboxes (Norbert) * Release Engineering update (Robinson) + 5.0.0 - Beta 0 + not much heard. + Late Feature Status (Michael) + LibreOfficeKit / Online tweaks (Kendy) + unstable API and/or reviewed. + OpenGL / double-buffer RenderContext (Kendy/Quikee) + re-review in a week - see if we can fix the rendering. + gtk3+ theming, embedded video, cut/paste (Caolan) + got popups working at hackfest + that also fixes a bunch of platform's undo / color selector popup placement problems. + no concerns. + change-tracking UI improvements @ hackfest (Kendy) [ punted in favour of rendercontext work @ hackfest ] + toolbar search / wrap-around (Kendy) [ a somewhat feature'y change to push to master; in writer/impress - when searching instead of the wrap-around / dialog. Say so in the search toolbar; reviews solicited appreciated ]. + text wrapping for chart labels (Marco) [ punted to 5.1 ] + orcus with gnumeric import (Markus) + no concerns. + Windows 64 status (David O) + all fine, poking some Base test with Stephan + an OLE test, failing on some machines not others; environment dependent. Thorsten's machine doesn't crash. + firebird now disabled; didn't fix (Stephan) + mork driver on Win64 - no-one ever reported a bug (David O) AI: + encourage QA guys to test thunderbird / Win64 (Robinson) + 4.4.4 RC1 status + will push that out later today + Android iOS Remote * Killed gcc 4.6 on master (post 5.0) (Stephan) + at least one commit in master now that is incompatible with 4.6 + but avoiding large scale changes for some months. + internal tinderbox switched to libreoffice-5-0 (Thorsten) * Crashtest update (Caolán) + import is reporting no crashers at all. + prolly timing out on 1x doc. + exports - terrible currently + ~all from recent DrawingML export fix (Andras) + beautiful screenshots; hopefully just some corner case (Michael) + ignoring those plus ignoring ~20x asserts from last discussion (Caolan) + then the legacy issues are dropping nicely. + also intersted in building a list of easy-to-fix regression crashers (Caolan) * GSoC (Thorsten) + make sure students on-track; coding started on monday ... + mid-term is not so far out - June 26th, deadline July 3rd. + seen some good commits already there. * Cambridge Hackfest retrospective (~Many) + a good number of guys; only 2x students - 1x already known + Photo approval: Miklos, Caolan, Michael, Kendy, Andras + Bjoern punted too (though no pictures exist) * Hackfests (Bjoern) + Hamburg, Germany: Oct 23rd - 25th + https://wiki.documentfoundation.org/Hackfest/Hamburg2015 + draft page; to be filled out + great to see everyone there. + next venues / suggestions + Belgium another candidate ... * Consolidating debug levels via EasyHacks (Bjoern) + first thing to do - to kill all OSL_DEBUG_ levels above 2, so have only 0 and 1. + questions answered (Thorsten) + getting rid of level 2 is great; as much - SAL_INFOS as possible (Stephan) * UX Update (Kendy) + closed submissions for 5.0 branding proposals + sent the list to the marketing design representatives + how should we proceed ? / should we amend it with a designer ? + hope to have new graphics soon (by RC2) * l10n Update (Sophie) + Olivier H found an English string in 4.4 + macros-security warning dialog +
