On Mon, Oct 29, 2018, 6:30 PM Mats Wichmann <m...@wichmann.us> wrote:
> > Hi, folks. I'm still around even if not very active, wanted to run > something by you all. > > While working on the scons project, something else I do on the side, I > notice people keep asking about the caching capabilities of SCons. Some > are able to use it to considerable benefit. > > Tl:dr version of caching: if there exists in the cache a file which > could be used instead of building a target, use the cached version. > Sorry, I think this is very misleading. Scons "caching" really means something like "shared remote caching", with multiple devs sharing the cache. https://scons.org/doc/1.3.0/HTML/scons-user/c4154.html Or maybe that's not what you mean, dunno. Seems obvious that every build system involves caching in some way. > > The longer version of caching: for every target scons calculates a > cryptographic hash which is based on a variety of contributors which > include hashes of the things it depends on, as well as some of the build > environment (the specific compiler has a hash, arguments, etc). This > goes into the .sconsign.dblite file (if you care to look, the sconsign > program can dump it if you give the name as an argument, there is no > default file). If you enable caching during a build, the derived files > are put into the specified cache directory using the signature as a > filename. There is also a cache-warming option to scons which populates > the cache from a built tree. On a subsequent build, normal behavior is > if the computed file signature for a derived file (target) matches the > signature on the file at the target location, then no rebuilding has to > be done for that file. If caching is enabled, as an additional check if > a file named by that signature exists in the cache, then the file is > copied from the cache. > > This is maybe mildly interesting to an individual iotivity developer. > If you rebuild after making a change, scons should only rebuild the > affected targets, like any decent buildsystem, and the cache is no help. > If you cleaned in between builds, the cache would help. > > When the builds are done by the CI system (Jenkins), a fresh environment > is used every time, so there are no existing files from a previous build > and you have to pay the price of a complete build every time. In > checking with LF, it should be possible to provision the OpenStack > instances in a way that they mount a persistent location from outside > the instance which could contain an SCons cache. A quick local test > here suggests that using cached data speeds things up quite a bit: a > relatively parallel build with 8 threads takes 7 minutes or so on my > machine; the same build with a preloaded cache takes about 1/4 of the > time. My setup is fairly quick, the performance gains in other > scenarios could be a whole lot more. Or not. Don't really know until > tried. If an impressive ratio holds, that could mean a substantial > speedup in the turnaround for a "vote" from the CI system on whether a > patch passes builds or not. > > So what is the purpose of this email? To ask whether this is something > the team wants to pursue. > > It's not a slam dunk: if we could just turns on caching in the project > and that's it, the experiment would be painless. But there are plenty > of infrastructure questions - setting up the sharing as mentioned above, > whether to share one cache between all builders or use one for each type > (there are unlikely to be commonalities among builder types - a Linux > x64 object will always be different, and even have a different name than > a Windows object, and the ARM object for Tizen, or Android, or iOS will > also always differ), how to keep the cache fresh and not growing > endlessly over time, etc. And it's not clear if the "Jenkins vote" will > actually come any faster: a couple of the builders run unit tests, and > the actual running of the tests is not affected by the cache at all, if > the majority of time is running the tests, it doesn't help them. I don't > know how much the unit test run time gates the overall full CI > completion time. > > Thoughts? > > > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#9989): https://lists.iotivity.org/g/iotivity-dev/message/9989 Mute This Topic: https://lists.iotivity.org/mt/27787640/21656 Group Owner: iotivity-dev+ow...@lists.iotivity.org Unsubscribe: https://lists.iotivity.org/g/iotivity-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
