Re: [Interest] Tricks to improve moc performance?
On Mon, Dec 23, 2019 at 2:26 PM Adam Light wrote: > > One idea I had was to combine all files in HEADERS into a single giant .h > file, and then call moc.exe once on that file. The single call to moc.exe > would be executed in a single thread, but it would avoid the need to > rediscover all of the header files from each of the 600 or so moc processes > that would otherwise run. > > I've come up with a feature that replaces the standard moc feature that does exactly this. It's a bit fragile--it works for our project, but I tried using it to build Qt Creator and there were errors. Someone better with qmake might have more success making it more broadly useful. Ultimately, I think the best approach would be to allow moc to accept a list of files to be mocced, but as is I don't think qmake can (easily?) be used to supply that list. Using this new feature, it takes only 2-3 seconds to moc all of the 550 or so mocable header files in our project, versus over 60 seconds without the feature. So builds are much faster, and the feature almost eliminates the performance killing effects of the bindflt minifilter driver I mentioned earlier in the thread. I've created a feature request at https://bugreports.qt.io/browse/QTBUG-81348 that contains two versions of the new feature as well as a detailed description of the problem and what I have learned while investigating it. Adam ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] Tricks to improve moc performance?
On Thu, Dec 5, 2019 at 5:09 PM Adam Light wrote:
>
> We have almost 600 classes that use Q_OBJECT, so there are a lot of calls
> to moc. We probably don't strictly need the Q_OBJECT macro in some of those
> classes, but I would prefer not to start removing Q_OBJECT unless that's
> the last option.
>
> We also have around 200 paths in our INCLUDEPATH variable. Moving header
> files into fewer directories would allow that to decrease, which should
> improve moc performance.
>
> Does anyone else have any ideas of how we could change our build to
> improve moc performance when Windows decides to be "slow"? Like, for
> example, is there any way to have the moc calls run with only a few moc
> processes running at once but have the rest of the build done with all
> threads running. I'm pretty sure that the OS slowdown has something to do
> with thread contention of some sort.
>
>
To follow up with my original post, I've made moderate progress by pruning
the number of paths in INCLUDEPATH, removing #include statements from our
header files that aren't actually necessary, and tweaking qmake CONFIG
settings. That has dropped the build time down from close to 5 minutes to
just under 3 minutes.
One idea I had was to combine all files in HEADERS into a single giant .h
file, and then call moc.exe once on that file. The single call to moc.exe
would be executed in a single thread, but it would avoid the need to
rediscover all of the header files from each of the 600 or so moc processes
that would otherwise run.
I was able to get qmake to generate the concatenated .h file for me with
the following commands:
COMBINED_HEADERS = $$shell_path($$absolute_path($${MOC_DIR}/allheaders.h,
$$OUT_PWD))
for (oneHeader, HEADERS): {
CMD = "type $$shell_quote($$shell_path($$absolute_path($${oneHeader},
$$_PRO_FILE_PWD_))) >> $${COMBINED_HEADERS}"
message($${CMD})
system("$${CMD}")
}
I then manually executed the moc.exe command on the command line, using the
same flags used for all of the other regular moc calls, and it took moc
about 2 seconds, versus over 60 seconds when moc is called separately for
each mocable file. Concatenating all of the header files together did take
some time (maybe 20 seconds), but it's still a huge decrease in build time.
I haven't tried to link in this single moc output file, so there may still
be troubles ahead.
Does anyone know of an existing example of doing the concatenating of
headers properly so that the mega header file is updated when necessary? My
quick and dirty approach given above seems to work so far but I'd be
surprised if it's the best way.
Thanks
Adam
___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest
Re: [Interest] Tricks to improve moc performance?
On Monday, 9 December 2019 14:51:24 PST Hamish Moffatt wrote: > If my memory serves me correctly, we found MOC drastically slower when > we upgraded from Qt 4.8 to 5.x. We had quite a few directories in > INCLUDEPATH, though nothing like the number you mentioned. Qt 5's moc, unlike Qt 4's, does not ignore #includes. It actually does parse them. But it ignores any #include it cannot find, for compatibility. Qt 6's will require them to be found. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel System Software Products ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] Tricks to improve moc performance?
On Tue, Dec 10, 2019 at 5:09 AM Michael Jackson wrote: > What did you use to collect the sampling data? Our project also > experiences long build times (30 minutes on a 12 Core Xeon machine) using > Qt5. I would like to be able to zero in on some of the issues instead of > guessing at them. > > > I use UIforETW (https://github.com/google/UIforETW/releases) to collect the traces. This is a powerful but relatively simple user interface for Microsoft's ETW (event tracing for Windows). Bruce Dawson, the author, works for Google on the Chrome Browser and therefore UIforETW has a lot of Chrome specific options, but you can ignore all of those and use it for tracing any application. Collecting the traces is pretty simple, but analyzing them in WPA (Windows Performance Analyzer) is not because there is just an immense amount of data and so many options for analyzing it. I recommend that you start by reading this blog post: https://randomascii.wordpress.com/2015/09/01/xperf-basics-recording-a-trace-the-ultimate-easy-way/. He also has several videos that cover some important analysis topics at https://randomascii.wordpress.com/2014/08/19/etw-training-videos-available-now/, but they're behind a paywall. When I first started using the tool there was a free trial subscription of some sort that I signed up for so that I could view the videos. After that, reading his ETW related blog posts is very informative. He runs into some very strange performance problems, but the analysis techniques he discusses have been useful to me in tracking down less obscure problems in our application. The post at https://randomascii.wordpress.com/2019/10/20/63-cores-blocked-by-seven-instructions/ is even compiling related. I was excited when I saw that post since it's a similar problem to what I'm running into with moc but the sources of the problems are different. For Windows minifilter driver problems, which is what I think I'm dealing with here, I found a blog post at https://blogs.msdn.microsoft.com/ntdebugging/2012/05/31/hotfix-to-enable-mini-filter-performance-diagnostics-with-xperf-for-windows-server-2008r2/ that discusses how to collect trace information on minifilter delays. Instead of using the command line tool described there, I used UIforETW's Settings dialog and added the flags given in the blog post to the Extra Kernel Flags and Extra Kernel Stackwalk settings. If you want to profile anything Qt related, you'll need the PDB symbols for the libraries and the tools. We use our own build of Qt 5.12, so we have all of the symbols available, EXCEPT that I can't figure out how to generate PDB files for qmake.exe itself, so I've been unable to analyze why qmake takes longer to run than it seems to me it should. If anyone knows how to force generation of PDB files for qmake, that would be really helpful. Adam ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] Tricks to improve moc performance?
What did you use to collect the sampling data? Our project also experiences long build times (30 minutes on a 12 Core Xeon machine) using Qt5. I would like to be able to zero in on some of the issues instead of guessing at them. -- Mike Jackson From: Interest on behalf of Adam Light Date: Monday, December 9, 2019 at 5:15 PM To: Qt Interest Subject: Re: [Interest] Tricks to improve moc performance? On Thu, Dec 5, 2019 at 5:09 PM Adam Light wrote: Does anyone else have any ideas of how we could change our build to improve moc performance when Windows decides to be "slow"? Like, for example, is there any way to have the moc calls run with only a few moc processes running at once but have the rest of the build done with all threads running. I'm pretty sure that the OS slowdown has something to do with thread contention of some sort. The problem with moc.exe is even worse than I first realized. Even when Windows is in the "fast" state (that is, a full rebuild takes 4-5 minutes instead of 16-18 minutes), the vast majority of the build time is spent in moc.exe. I just collected sampling data for an entire build, excluding the qmake step. So this is the entirety of jom.exe and everything it calls during the build. Windows 10, debug build of our application, VS2017. I'm not using any /j flag with jom, so it will use all available threads (16 core/32 threads). The build took 4:54. During that time, there were a total of 551 moc.exe processes that ran, and those processes took a total of 3,896 seconds of CPU time. There were also 1270 cl.exe processes which took a total of 965 seconds of CPU time. Everything else (link.exe, ui.exe, jom.exe, was minimal) compared to these two. So moc.exe was running almost 4 times as long as cl.exe. That seems crazy to me. When I previously analyzed the sampling data during a moc.exe run, it was clear that the vast majority of the time was spent satisfying the includes, and most of that was checking whether a concatenation of an include path and a file name are a valid file (eg. include/path1/stdio, include/path2/stdio, etc.). It seems like the compiler (cl.exe) would need to do essentially the same thing when compiling each individual file, but it's not nearly as slow as moc. Adam ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] Tricks to improve moc performance?
On 10/12/19 9:14 am, Adam Light wrote: When I previously analyzed the sampling data during a moc.exe run, it was clear that the vast majority of the time was spent satisfying the includes, and most of that was checking whether a concatenation of an include path and a file name are a valid file (eg. include/path1/stdio, include/path2/stdio, etc.). It seems like the compiler (cl.exe) would need to do essentially the same thing when compiling each individual file, but it's not nearly as slow as moc. If my memory serves me correctly, we found MOC drastically slower when we upgraded from Qt 4.8 to 5.x. We had quite a few directories in INCLUDEPATH, though nothing like the number you mentioned. To get acceptable performance we trimmed INCLUDEPATH to a higher level directory and started referencing instead of just . At development time we build in Visual Studio using the vcxproj generator in qmake, which won't run multiple MOC steps in paralle. If you think your MOC time is bad, imagine running them serially... We had to rewrite our build scripts to use JOM for release builds due to this. Hamish ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] Tricks to improve moc performance?
On Thu, Dec 5, 2019 at 5:09 PM Adam Light wrote: > > Does anyone else have any ideas of how we could change our build to > improve moc performance when Windows decides to be "slow"? Like, for > example, is there any way to have the moc calls run with only a few moc > processes running at once but have the rest of the build done with all > threads running. I'm pretty sure that the OS slowdown has something to do > with thread contention of some sort. > > The problem with moc.exe is even worse than I first realized. Even when Windows is in the "fast" state (that is, a full rebuild takes 4-5 minutes instead of 16-18 minutes), the vast majority of the build time is spent in moc.exe. I just collected sampling data for an entire build, excluding the qmake step. So this is the entirety of jom.exe and everything it calls during the build. Windows 10, debug build of our application, VS2017. I'm not using any /j flag with jom, so it will use all available threads (16 core/32 threads). The build took 4:54. During that time, there were a total of 551 moc.exe processes that ran, and those processes took a total of 3,896 seconds of CPU time. There were also 1270 cl.exe processes which took a total of 965 seconds of CPU time. Everything else (link.exe, ui.exe, jom.exe, was minimal) compared to these two. So moc.exe was running almost 4 times as long as cl.exe. That seems crazy to me. When I previously analyzed the sampling data during a moc.exe run, it was clear that the vast majority of the time was spent satisfying the includes, and most of that was checking whether a concatenation of an include path and a file name are a valid file (eg. include/path1/stdio, include/path2/stdio, etc.). It seems like the compiler (cl.exe) would need to do essentially the same thing when compiling each individual file, but it's not nearly as slow as moc. Adam ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] Tricks to improve moc performance?
On Fri, Dec 6, 2019 at 6:08 PM Thiago Macieira wrote: > On Friday, 6 December 2019 15:10:41 PST Adam Light wrote: > > > Yes: > https://github.com/qt/qtbase/blob/dev/util/includemocs/includemocs.pl > > > > Thanks. That saved a lot of time. > > > > For what it's worth, after changing all of our code to directly #include > > the moc output, the total build time dropped by around 10%. That's > helpful. > > Nice. Did you notice a size improvement in your optimised binaries too? > > On Macintosh, with Xcode 10.2, the application's executable went from 202627092 bytes to 202587956 bytes. This is a release build (-O2). So the decrease in size is small, but measurable. I haven't tested the executable sizes on Windows yet. Strangely, on Windows, this change has almost zero effect on build time. The time to run qmake has increased slightly (about 3:10 to 3:20) but the remaining build time (everything within jom.exe) only dropped by about 8 seconds (total of about 7 minutes). On Macintosh, the qmake time also increases slightly but the rest of the build is about 45 seconds faster (6:15 to 5:29). The Windows builds are release builds with VS2017. Adam ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] Tricks to improve moc performance?
On 12/6/19 7:02 PM, Richard Weickelt wrote: By the way: does anyone know about an implementation that avoids moc runs in a similar way how ccache avoids compile runs ? You mean concatenating header and source files containing a Q_OBJECT macro and feeding moc the result? No - the ideas would be to calculate a hashsum from the files, that have an effect on the moc file and store it together with the moc file in a cache. Then you have a replacement for moc, that compares the hashsum of the cache with the current hashsum and copies the cached moc file if they match - otherwise it is calling moc and copies the result into its cache. Not 100% sure but IIRC this is how ccache works. Uwe ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] Tricks to improve moc performance?
On Friday, 6 December 2019 15:10:41 PST Adam Light wrote: > > Yes: https://github.com/qt/qtbase/blob/dev/util/includemocs/includemocs.pl > > Thanks. That saved a lot of time. > > For what it's worth, after changing all of our code to directly #include > the moc output, the total build time dropped by around 10%. That's helpful. Nice. Did you notice a size improvement in your optimised binaries too? -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel System Software Products ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] Tricks to improve moc performance?
On Friday, 6 December 2019 10:06:24 PST Richard Weickelt wrote: > > This solution gets you a single build for all the the mocs, which is good, > > but won't generate the best code that Peppe was talking about. You want > > the moc for a given class to be in the class's own .cpp. > > Unless you build with link-time optimization enabled, I suppose. Because > then translation units are no longer an optimization boundary. I don't know if it triggers this warning in LTO mode. I usually consider them complementary options, so best to use both. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel System Software Products ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] Tricks to improve moc performance?
On Friday, 6 December 2019 10:21:24 PST Michael Jackson wrote: > Dear Thiago, > Could you expand a bit on your comment? We use CMake for our build > system and the AUOT_MOC feature. I just would like to know what exactly I > am missing by doing this. Run the includemocs script that I linked to: make every .cpp of a class that has a Q_OBJECT in aheder #include the moc_*.cpp file that moc generates. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel System Software Products ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] Tricks to improve moc performance?
On Fri, Dec 6, 2019 at 9:57 AM Thiago Macieira wrote: > On Friday, 6 December 2019 06:30:36 PST Adam Light wrote: > > Yes, that's definitely on my TODO list, though I don't think it's going > to > > do much to address the current situation in which moc itself is the main > > bottleneck. But any improvement will help. > > > > Does anyone know of a script or other automated way to add these > includes? > > Yes: https://github.com/qt/qtbase/blob/dev/util/includemocs/includemocs.pl > > Thanks. That saved a lot of time. For what it's worth, after changing all of our code to directly #include the moc output, the total build time dropped by around 10%. That's helpful. Adam ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] Tricks to improve moc performance?
On 12/6/19, 12:52 PM, "Interest on behalf of Thiago Macieira" wrote: On Friday, 6 December 2019 01:16:31 PST Kevin Funk via Interest wrote: > On that note, CMake goes one step further and removes the need to do this > manually. Using CMake's AUTOMOC feature, CMake will automatically create ONE > mocs_compilations.cpp file per target which in turn includes all generated > "moc_XYZ.cpp" files. > > Quoting: > > "Header files that are not included by an #include "moc_.cpp" > statement are nonetheless scanned for Q_OBJECT or Q_GADGET macros. The > resulting moc_.cpp files are generated in custom directories and > automatically included in a generated > /mocs_compilation.cpp file, which is compiled as part of > the target." That's good, but not ideal. This solution gets you a single build for all the the mocs, which is good, but won't generate the best code that Peppe was talking about. You want the moc for a given class to be in the class's own .cpp. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel System Software Products Dear Thiago, Could you expand a bit on your comment? We use CMake for our build system and the AUOT_MOC feature. I just would like to know what exactly I am missing by doing this. -- Mike Jackson ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] Tricks to improve moc performance?
> This solution gets you a single build for all the the mocs, which is good, > but > won't generate the best code that Peppe was talking about. You want the moc > for a given class to be in the class's own .cpp. Unless you build with link-time optimization enabled, I suppose. Because then translation units are no longer an optimization boundary. ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] Tricks to improve moc performance?
> By the way: does anyone know about an implementation that avoids moc runs in > a similar way how ccache avoids compile runs ? You mean concatenating header and source files containing a Q_OBJECT macro and feeding moc the result? I cannot imagine that this would be a safe transformation in general. Moc accepts defines and include paths on command line so I would suppose that it tries very hard to parse the C++ code, to expand macros and to resolve includes. It might work for limited use-cases though. I'd say concatenating the output of moc and compiling that in one go is the only safe shortcut in this process. Richard ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] Tricks to improve moc performance?
On Friday, 6 December 2019 06:30:36 PST Adam Light wrote: > Yes, that's definitely on my TODO list, though I don't think it's going to > do much to address the current situation in which moc itself is the main > bottleneck. But any improvement will help. > > Does anyone know of a script or other automated way to add these includes? Yes: https://github.com/qt/qtbase/blob/dev/util/includemocs/includemocs.pl -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel System Software Products ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] Tricks to improve moc performance?
On Friday, 6 December 2019 01:52:52 PST Giuseppe D'Angelo via Interest wrote: > Sorry, not the code generated by moc -- the code generated by the > > compiler/linker. See e.g. here: > > https://codereview.qt-project.org/c/qt/qtbase/+/152423 it also enables the "private member never used' warning in Clang: if it can see the definition of all functions in a class, it can tell whether none of them accessed a given member. If your moc's output in a separate translation unit, then neither had all member functions. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel System Software Products ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] Tricks to improve moc performance?
On Friday, 6 December 2019 01:16:31 PST Kevin Funk via Interest wrote: > On that note, CMake goes one step further and removes the need to do this > manually. Using CMake's AUTOMOC feature, CMake will automatically create ONE > mocs_compilations.cpp file per target which in turn includes all generated > "moc_XYZ.cpp" files. > > Quoting: > > "Header files that are not included by an #include "moc_.cpp" > statement are nonetheless scanned for Q_OBJECT or Q_GADGET macros. The > resulting moc_.cpp files are generated in custom directories and > automatically included in a generated > /mocs_compilation.cpp file, which is compiled as part of > the target." That's good, but not ideal. This solution gets you a single build for all the the mocs, which is good, but won't generate the best code that Peppe was talking about. You want the moc for a given class to be in the class's own .cpp. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel System Software Products ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] Tricks to improve moc performance?
On Friday, 6 December 2019 00:42:22 PST Uwe Rathmann wrote: > A side note: the code of Qwt project runs with all Qt versions >= 4.4 > and therefore allows to compare the impact of the Qt headers on the > compile time. It is some time ago, when I did this, but the differences > were significant. IIRC there was a major slowdown around ~4.7 that gets > worse during the Qt5 cycle. The biggest slowdown of Qt 5.0 was the removal of QT_NO_STL. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel System Software Products ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] Tricks to improve moc performance?
On Fri, Dec 6, 2019 at 12:42 AM Uwe Rathmann wrote: > On 12/6/19 2:09 AM, Adam Light wrote: > > > Does anyone else have any ideas of how we could change our build to > > improve moc performance when Windows decides to be "slow"? > > Something you can try is to include the moc file at the end of your cpp > file. This can be done like this: > > #include "moc_XYZ.cpp" > > Yes, that's definitely on my TODO list, though I don't think it's going to do much to address the current situation in which moc itself is the main bottleneck. But any improvement will help. Does anyone know of a script or other automated way to add these includes? Adam ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] Tricks to improve moc performance?
On 12/6/19 9:48 AM, Giuseppe D'Angelo via Interest wrote: This trick is used in several places in Qt itself (look for "includemoc" in commits). Not only it helps build times but also it produces slightly better code overall. By the way: does anyone know about an implementation that avoids moc runs in a similar way how ccache avoids compile runs ? Uwe ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] Tricks to improve moc performance?
Hi, Il 06/12/19 10:48, Dmitriy Purgin ha scritto: Hi Giuseppe, > This trick is used in several places in Qt itself (look for "includemoc" > in commits). Not only it helps build times but also it produces slightly > better code overall. Could you please elaborate what exactly do you mean under 'better code'? How does including the xxx_moc.cpp affect the code generation by moc? If moc decides to generate a better code when included explicitly in a cpp file, why can't it generate the same better code otherwise? Is there a technical reason for it? Sorry, not the code generated by moc -- the code generated by the compiler/linker. See e.g. here: https://codereview.qt-project.org/c/qt/qtbase/+/152423 HTH, -- Giuseppe D'Angelo | giuseppe.dang...@kdab.com | Senior Software Engineer KDAB (France) S.A.S., a KDAB Group company Tel. France +33 (0)4 90 84 08 53, http://www.kdab.com KDAB - The Qt, C++ and OpenGL Experts smime.p7s Description: Firma crittografica S/MIME ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] Tricks to improve moc performance?
Hi Giuseppe, > This trick is used in several places in Qt itself (look for "includemoc" > in commits). Not only it helps build times but also it produces slightly > better code overall. Could you please elaborate what exactly do you mean under 'better code'? How does including the xxx_moc.cpp affect the code generation by moc? If moc decides to generate a better code when included explicitly in a cpp file, why can't it generate the same better code otherwise? Is there a technical reason for it? Thanks! Cheers Dmitriy On Fri, Dec 6, 2019 at 9:49 AM Giuseppe D'Angelo via Interest < interest@qt-project.org> wrote: > Il 06/12/19 09:42, Uwe Rathmann ha scritto: > > Something you can try is to include the moc file at the end of your cpp > > file. This can be done like this: > > > > #include "moc_XYZ.cpp" > > > > This type of construction is at least supported by qmake. > > > > This does not reduce the number of moc runs, but it reduces the number > > of compiler runs. As a rule of thumb for not too large classes I would > > expect, that ~50% of the compile time is spent inside the Qt headers and > > by including the moc file the overall compile time goes down by 33% as > > the headers need to be processed only once. > > This trick is used in several places in Qt itself (look for "includemoc" > in commits). Not only it helps build times but also it produces slightly > better code overall. > > HTH, > -- > Giuseppe D'Angelo | giuseppe.dang...@kdab.com | Senior Software Engineer > KDAB (France) S.A.S., a KDAB Group company > Tel. France +33 (0)4 90 84 08 53, http://www.kdab.com > KDAB - The Qt, C++ and OpenGL Experts > > ___ > Interest mailing list > Interest@qt-project.org > https://lists.qt-project.org/listinfo/interest > ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] Tricks to improve moc performance?
On Friday, 6 December 2019 09:48:51 CET Giuseppe D'Angelo via Interest wrote: > Il 06/12/19 09:42, Uwe Rathmann ha scritto: > > Something you can try is to include the moc file at the end of your cpp > > file. This can be done like this: > > > > #include "moc_XYZ.cpp" > > > > This type of construction is at least supported by qmake. > > > > This does not reduce the number of moc runs, but it reduces the number > > of compiler runs. As a rule of thumb for not too large classes I would > > expect, that ~50% of the compile time is spent inside the Qt headers and > > by including the moc file the overall compile time goes down by 33% as > > the headers need to be processed only once. > > This trick is used in several places in Qt itself (look for "includemoc" > in commits). Not only it helps build times but also it produces slightly > better code overall. Heya, On that note, CMake goes one step further and removes the need to do this manually. Using CMake's AUTOMOC feature, CMake will automatically create ONE mocs_compilations.cpp file per target which in turn includes all generated "moc_XYZ.cpp" files. Quoting: "Header files that are not included by an #include "moc_.cpp" statement are nonetheless scanned for Q_OBJECT or Q_GADGET macros. The resulting moc_.cpp files are generated in custom directories and automatically included in a generated /mocs_compilation.cpp file, which is compiled as part of the target." ... from: https://cmake.org/cmake/help/v3.9/prop_tgt/AUTOMOC.html Just as a side note. So if you thought about moving to CMake anyway this might be a good reason for a start. In general switching to e.g. CMake with the Ninja generator will give you better overall build times, esp. on Windows. And you said: > We probably don't strictly need the Q_OBJECT macro in some of those > classes, but I would prefer not to start removing Q_OBJECT unless that's > the last option. I'd rethink that opinion: Adding the Q_OBJECT (and thus inheriting from QObject) everywhere adds the dependency to moc (=> slower compilation time) and has a runtime performance impact (QObject creation is not cheap!). I'd consider only requiring Q_OBJECT where needed, it leads to overall better code. Regards, Kevin > HTH, -- Kevin Funk | kevin.f...@kdab.com | Senior Software Engineer KDAB (Deutschland) GmbH, a KDAB Group company Tel: +49-30-521325470 KDAB - The Qt, C++ and OpenGL Experts smime.p7s Description: S/MIME cryptographic signature ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] Tricks to improve moc performance?
Il 06/12/19 09:42, Uwe Rathmann ha scritto: Something you can try is to include the moc file at the end of your cpp file. This can be done like this: #include "moc_XYZ.cpp" This type of construction is at least supported by qmake. This does not reduce the number of moc runs, but it reduces the number of compiler runs. As a rule of thumb for not too large classes I would expect, that ~50% of the compile time is spent inside the Qt headers and by including the moc file the overall compile time goes down by 33% as the headers need to be processed only once. This trick is used in several places in Qt itself (look for "includemoc" in commits). Not only it helps build times but also it produces slightly better code overall. HTH, -- Giuseppe D'Angelo | giuseppe.dang...@kdab.com | Senior Software Engineer KDAB (France) S.A.S., a KDAB Group company Tel. France +33 (0)4 90 84 08 53, http://www.kdab.com KDAB - The Qt, C++ and OpenGL Experts smime.p7s Description: Firma crittografica S/MIME ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] Tricks to improve moc performance?
On 12/6/19 2:09 AM, Adam Light wrote: Does anyone else have any ideas of how we could change our build to improve moc performance when Windows decides to be "slow"? Something you can try is to include the moc file at the end of your cpp file. This can be done like this: #include "moc_XYZ.cpp" This type of construction is at least supported by qmake. This does not reduce the number of moc runs, but it reduces the number of compiler runs. As a rule of thumb for not too large classes I would expect, that ~50% of the compile time is spent inside the Qt headers and by including the moc file the overall compile time goes down by 33% as the headers need to be processed only once. A side note: the code of Qwt project runs with all Qt versions >= 4.4 and therefore allows to compare the impact of the Qt headers on the compile time. It is some time ago, when I did this, but the differences were significant. IIRC there was a major slowdown around ~4.7 that gets worse during the Qt5 cycle. HTH, Uwe ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] Tricks to improve moc performance?
On 2019-12-06 02:09, Adam Light wrote: We have several machines running Windows 10 that are quite powerful (eg. 8 or 16 core (16 or 32 thread) processors, lots of RAM, NVMe SSD storage). However, since upgrading to Windows 1903, we have noticed that sometimes the compile time of our large Qt 5.12 based application increases dramatically. As an example, on one 16 core/32 thread machine, a debug build can take as little as 4 minutes. However it sometimes takes 18 minutes. This is for the exact same code, compiler, settings, etc. I can literally do a full build that takes 4 minutes, do something, delete the build directory and rebuild and now it takes 16 minutes. The "do something" may be going to lunch or making a trivial change in the code. Otherwise the state of the machine is the same (no other heavyweight processes running, etc.) We see this behavior both when using Qt Creator 4.10.2 to do the build (using jom.exe) and also when done outside of Creator with calls to qmake and then jom. All files involved in anything I mention here are on and/or written to the local SSD. I see these issues with no antivirus running, no backup, etc. ... Does anyone else have any ideas of how we could change our build to improve moc performance when Windows decides to be "slow"? Like, for example, is there any way to have the moc calls run with only a few moc processes running at once but have the rest of the build done with all threads running. I'm pretty sure that the OS slowdown has something to do with thread contention of some sort. If anyone thinks they have run into this situation or is intimately familiar with Windows minifilter drivers, I'm happy to share the details behind why I think they are involved here. Thanks for any ideas Adam Hi, just guessing, but it could be network related. For a quick test, when you experience that slowdown, try pulling out the network cable from the PC and see if the build time changes. For a more elaborate test, consider moving your Windows build PC into the virtual realm, say like this: Use Disk2vhd https://docs.microsoft.com/en-us/sysinternals/downloads/disk2vhd to migrate your C: into a .vhd file. Install Ubuntu 19.10 on the PC and VirtualBox (or VMWare Workstation player, also free). Then load up the .vhd file into Ubuntu and, in my case, that combo (a virtual Windows running inside Ubuntu) builds faster than my vanilla Windows 10 ever did. This year when 1903 arrived I saw some changes as well, the boot/startup time is better but I felt it was slower in general than 1809. So I dug up an old Windows 7 DVD and installed it into VMWare and reinstalled Qt on that Windows 7. Upgrading from Windows 10 to Windows 7 made jom happy again. So while it's impossible today to find a new PC that you can install Windows 7 on, it is still very much possible on VirtualBox or VMWare. Finally, since Microsoft will cease support of Windows 7 next month, make sure *never* to browse the internet or do your email from Windows 7, always use Ubuntu. Rgrds Henry ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
[Interest] Tricks to improve moc performance?
We have several machines running Windows 10 that are quite powerful (eg. 8 or 16 core (16 or 32 thread) processors, lots of RAM, NVMe SSD storage). However, since upgrading to Windows 1903, we have noticed that sometimes the compile time of our large Qt 5.12 based application increases dramatically. As an example, on one 16 core/32 thread machine, a debug build can take as little as 4 minutes. However it sometimes takes 18 minutes. This is for the exact same code, compiler, settings, etc. I can literally do a full build that takes 4 minutes, do something, delete the build directory and rebuild and now it takes 16 minutes. The "do something" may be going to lunch or making a trivial change in the code. Otherwise the state of the machine is the same (no other heavyweight processes running, etc.) We see this behavior both when using Qt Creator 4.10.2 to do the build (using jom.exe) and also when done outside of Creator with calls to qmake and then jom. All files involved in anything I mention here are on and/or written to the local SSD. I see these issues with no antivirus running, no backup, etc. I have collected some Windows Performance Tracing traces while building and during a "slow" build, most of the time is spent in moc, specifically searchIncludePaths. This function results in a lot of calls to QFileInfo::exists() and QFileInfo::isDir(), both of which ultimately call GetFileAttributesExW. The vast majority of the time is spent in downstream calls that GetFileAttributesExW makes, so I am not blaming moc itself for the slowdown. I don't want to get too much into the weeds, but I believe that the slowdown is a bug in one or more Windows minifilter drivers, which get invoked by the OS in the file API calls. I suspect that pinpointing the problem in such a way that I could report it to Microsoft is unlikely, and so for practical reasons I am hoping that there is a way we could change our build process to work around this slowdown. Our application is one that was ported into Qt and we are building a few very small libraries and one gigantic application that links to those libraries. This isn't ideal, I'm sure, but I don't think it would be practical to disentangle everything to have separate projects that are first built and then linked into the main application. We have almost 600 classes that use Q_OBJECT, so there are a lot of calls to moc. We probably don't strictly need the Q_OBJECT macro in some of those classes, but I would prefer not to start removing Q_OBJECT unless that's the last option. We also have around 200 paths in our INCLUDEPATH variable. Moving header files into fewer directories would allow that to decrease, which should improve moc performance. Does anyone else have any ideas of how we could change our build to improve moc performance when Windows decides to be "slow"? Like, for example, is there any way to have the moc calls run with only a few moc processes running at once but have the rest of the build done with all threads running. I'm pretty sure that the OS slowdown has something to do with thread contention of some sort. If anyone thinks they have run into this situation or is intimately familiar with Windows minifilter drivers, I'm happy to share the details behind why I think they are involved here. Thanks for any ideas Adam ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
