Re: [ccache] ccacje direct mode without fall back to running the preprocessor
Sorry. It is not the manifest file that it tries to update. But the ".d"
files. I think it is a real bug.
In read-only mode, when ccache tries to run cache look up in preprocessor
mode, it tries to put_file_in_cache for .d file unconditionally.
if (generating_dependencies && mode == FROMCACHE_CPP_MODE) {
put_file_in_cache(output_dep, cached_dep);
}
This is buggy, as it should check also check we are in
read-only/read-only-direct mode, like so:
if (generating_dependencies && mode == FROMCACHE_CPP_MODE &&
(!conf->read_only && !conf->read_only_direct)) {
Or even better, if put_file_in_cache returns if we are in
read-only/read-only-direct mode, which should fix more such hidden cases,
in future.
What do you think? Which one is better?
regards,
Venkat.
On Thu, Dec 10, 2015 at 6:00 PM, Andrew Stubbs wrote:
> On 10/12/15 11:38, vkr wrote:
>
>> * >> (*) If I use CCACHE_READONLY - then, there is a huge number of
>> files not hitting cache(direct-cache look up fails), and build fails
>> because of the cache storage being read-only, and the preprocessor
>> lookup that ran after direct-lookup failure, tries to update the
>> manifest file back in the cache. We can't go by this, as we want a
>> pure-read-only cache.
>>
>
> Huh? It still tries to write the manifest even in CCACHE_READONLY mode?
>
> I just tested it and no, it doesn't do that. If I edit a comment in a
> header file then I see "cache hit (preprocessed)" increment when I
> recompile. If I turn off CCACHE_READONLY then it rewrites the manifest, as
> expected, and the direct-mode starts working again.
>
> If I make the cache files read-only (chmod -R -w .ccache) then I see no
> error when CCACHE_READONLY is set (and CCACHE_TEMPDIR also), and the
> expected error writing the manifest otherwise.
>
> If I set the read-only in the config file (ccache -o read_only=true) then
> it works without the environment variable set.
>
> Are you sure your configuration is set right? Perhaps you should have
> "read_only=true" in your cache config, and then CCACHE_NOREADONLY=1 in your
> jenkins environment?
>
> I'm using the dev version of ccache, but this ought to work for all 3.2.x
> releases.
>
> Thinking about it, if you have multiple users sharing a writeable cache
> and one of them has a modified header that doesn't affect the preprocessed
> code, then you would end up with the manifest file flip-flopping between
> the two states. Maybe the read-only cache is a good idea, for the majority
> of users who have not modified any files?
>
> Andrew
>
>
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] ccacje direct mode without fall back to running the preprocessor
Andrew Stubbs wrote: > You really are getting into shaving fractions now though. At this point > you should seriously consider linking ccache statically; run it with > callgrind and you can see that a significant proportion of time is spent > before "main" is even called, loading libraries and such. > > Most of the rest of the time is spent doing MD4. I have some ideas how > to optimize that (by sharing them across runs), but nothing ready to post. I would be interested in your thoughts on how to speed that part up. Posted some patches on how to avoid certain headers in the manifest, but it seemed like there was no obvious way to improve the MD4 hash. (did try an "optimized" version from OpenSSL instead, but it was only marginally faster. And MD4 is still better than MD5 or any SHA hash) As long as you still get those direct hits, you will avoid it though*. * running the preprocessor and hashing the output (.i/.ii), that is. You will still have to hash the arguments and such, hopefully smaller. One thing that *is* a concern is the number of simultaneous `cc -E` running... Perhaps that needs a special CCACHE_PREFIX/lock some day ? /Anders ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] ccacje direct mode without fall back to running the preprocessor
On 10/12/15 03:13, vkr wrote: Hello Andew, The answer is "no", because it is necessary to run the preprocessor in order to fill the cache and record the file list that direct mode lookups use. What are the consequences if we don't let ccache record the file list, but just run the preprocessor and do a cache lookup using the preprocessor mode? WiIl I just have additional overhead when the next direct-look up runs? I guess this is the only drawback. If you don't record the direct-mode meta-data then you can never get a direct-mode cache-hit. You will rely on preprocessor mode only, for that input anyway. I understood why you might want to skip the preprocessor mode lookup, or not bother caching compiles that will never happen again (read-only mode), but this seems like an optimization too far? If a compile is usefully cachable then you should let ccache go ahead and do so. Yes, cache-misses are expensive, and yet the ccache overhead makes them even more so (my testing showed about 10% overhead, IIRC), but it's all for the sake of the cache-hits on subsequent runs. I suppose the if you *know* that a direct mode check will always fail (modified header, perhaps), but preprocessor mode will succeed, then it might make sense to skip the direct mode recording so that you can skip the direct mode file comparison on the next run. But, in that case, you can just simply disable direct-mode for that file (via environment variables in your makefile). If ccache had a complete set of options then you could optimize for every eventuality, I suppose. Maybe its worth doing? CCACHE_DISABLE_DIRECT_LOOKUP - but still write meta-data - optimize initial builds CCACHE_DISABLE_DIRECT_UPDATE - but still allow lookup - optimize cache-misses when direct mode would be pointless on subsequent runs CCACHE_DISABLE_PREPROCESSOR_LOOKUP - direct-mode lookup only - optimize cache-misses when we know processor mode will also fail - makes the cache read-only CCACHE_DISABLE_PREPROCESSOR_UPDATE - like CCACHE_READONLY but we can still write stats You really are getting into shaving fractions now though. At this point you should seriously consider linking ccache statically; run it with callgrind and you can see that a significant proportion of time is spent before "main" is even called, loading libraries and such. Most of the rest of the time is spent doing MD4. I have some ideas how to optimize that (by sharing them across runs), but nothing ready to post. Andrew ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
