Re: [ccache] ccacje direct mode without fall back to running the preprocessor

[Joel Rosdahl]

On 10 December 2015 at 20:32, vkr  wrote:

> Sorry. It is not the manifest file that it tries to update. But the ".d"
> files. I think it is a real bug. [...]
>

Yes, good catch. Thanks!

Fixed like suggested in a7ab503f07e31ebeaaec34fbaa30e264308a299d
.
I also added a couple of suitable asserts in put_file_in_cache.

-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] ccacje direct mode without fall back to running the preprocessor

[Andrew Stubbs]

On 10/12/15 19:32, vkr wrote:

Sorry. It is not the manifest file that it tries to update. But the ".d"
files. I think it is a real bug.

In read-only mode, when ccache tries to run cache look up in
preprocessor mode, it tries to put_file_in_cache for .d file
unconditionally.

 if (generating_dependencies && mode == FROMCACHE_CPP_MODE) {
 put_file_in_cache(output_dep, cached_dep);
 }

This is buggy, as it should check also check we are in
read-only/read-only-direct mode, like so:

if (generating_dependencies && mode == FROMCACHE_CPP_MODE &&
(!conf->read_only && !conf->read_only_direct)) {

Or even better, if put_file_in_cache returns if we are in
read-only/read-only-direct mode, which should fix more such hidden
cases, in future.


I think you're right, and I'm sure Joel would appreciate a patch if you 
could cook one up and test it.


I'm not the maintainer, so my opinion on the placement of the fix is not 
interesting. I would follow whatever precedent has been set elsewhere, 
and only "fix" it if things are getting out of hand.


Andrew


___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] ccacje direct mode without fall back to running the preprocessor

[vkr]

Sorry. It is not the manifest file that it tries to update. But the ".d"
files. I think it is a real bug.

In read-only mode, when ccache tries to run cache look up in preprocessor
mode, it tries to put_file_in_cache for .d file unconditionally.

if (generating_dependencies && mode == FROMCACHE_CPP_MODE) {
put_file_in_cache(output_dep, cached_dep);
}

This is buggy, as it should check also check we are in
read-only/read-only-direct mode, like so:

if (generating_dependencies && mode == FROMCACHE_CPP_MODE &&
(!conf->read_only && !conf->read_only_direct)) {

Or even better, if put_file_in_cache returns if we are in
read-only/read-only-direct mode, which should fix more such hidden cases,
in future.

What do you think? Which one is better?

regards,
Venkat.



On Thu, Dec 10, 2015 at 6:00 PM, Andrew Stubbs  wrote:

> On 10/12/15 11:38, vkr wrote:
>
>>   *  >> (*) If I use CCACHE_READONLY - then, there is a huge number of
>> files not hitting cache(direct-cache look up fails), and build fails
>> because of the cache storage being read-only, and the preprocessor
>> lookup that ran after direct-lookup failure, tries to update the
>> manifest file back in the cache. We can't go by this, as we want a
>> pure-read-only cache.
>>
>
> Huh? It still tries to write the manifest even in CCACHE_READONLY mode?
>
> I just tested it and no, it doesn't do that. If I edit a comment in a
> header file then I see "cache hit (preprocessed)" increment when I
> recompile. If I turn off CCACHE_READONLY then it rewrites the manifest, as
> expected, and the direct-mode starts working again.
>
> If I make the cache files read-only (chmod -R -w .ccache) then I see no
> error when CCACHE_READONLY is set (and CCACHE_TEMPDIR also), and the
> expected error writing the manifest otherwise.
>
> If I set the read-only in the config file (ccache -o read_only=true) then
> it works without the environment variable set.
>
> Are you sure your configuration is set right? Perhaps you should have
> "read_only=true" in your cache config, and then CCACHE_NOREADONLY=1 in your
> jenkins environment?
>
> I'm using the dev version of ccache, but this ought to work for all 3.2.x
> releases.
>
> Thinking about it, if you have multiple users sharing a writeable cache
> and one of them has a modified header that doesn't affect the preprocessed
> code, then you would end up with the manifest file flip-flopping between
> the two states. Maybe the read-only cache is a good idea, for the majority
> of users who have not modified any files?
>
> Andrew
>
>
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] ccacje direct mode without fall back to running the preprocessor

[Anders Björklund]

Andrew Stubbs wrote:
> You really are getting into shaving fractions now though. At this point
> you should seriously consider linking ccache statically; run it with
> callgrind and you can see that a significant proportion of time is spent
> before "main" is even called, loading libraries and such.
>
> Most of the rest of the time is spent doing MD4. I have some ideas how
> to optimize that (by sharing them across runs), but nothing ready to post.

I would be interested in your thoughts on how to speed that part up.

Posted some patches on how to avoid certain headers in the manifest,
but it seemed like there was no obvious way to improve the MD4 hash.
(did try an "optimized" version from OpenSSL instead, but it was only
marginally faster. And MD4 is still better than MD5 or any SHA hash)

As long as you still get those direct hits, you will avoid it though*.

* running the preprocessor and hashing the output (.i/.ii), that is.
You will still have to hash the arguments and such, hopefully smaller.
One thing that *is* a concern is the number of simultaneous `cc -E`
running... Perhaps that needs a special CCACHE_PREFIX/lock some day ?

/Anders
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] ccacje direct mode without fall back to running the preprocessor

[Andrew Stubbs]

On 10/12/15 03:13, vkr wrote:

Hello Andew,

The answer is "no", because it is necessary to run the preprocessor in

order to fill the cache and record the file list that direct mode lookups
use.

What are the consequences if we don't let ccache record the file list, but
just run the preprocessor and
do a cache lookup using the preprocessor mode?

WiIl I just have additional overhead when the next direct-look up runs? I
guess this is the only drawback.


If you don't record the direct-mode meta-data then you can never get a 
direct-mode cache-hit. You will rely on preprocessor mode only, for that 
input anyway.


I understood why you might want to skip the preprocessor mode lookup, or 
not bother caching compiles that will never happen again (read-only 
mode), but this seems like an optimization too far?


If a compile is usefully cachable then you should let ccache go ahead 
and do so. Yes, cache-misses are expensive, and yet the ccache overhead 
makes them even more so (my testing showed about 10% overhead, IIRC), 
but it's all for the sake of the cache-hits on subsequent runs.


I suppose the if you *know* that a direct mode check will always fail 
(modified header, perhaps), but preprocessor mode will succeed, then it 
might make sense to skip the direct mode recording so that you can skip 
the direct mode file comparison on the next run. But, in that case, you 
can just simply disable direct-mode for that file (via environment 
variables in your makefile).


If ccache had a complete set of options then you could optimize for 
every eventuality, I suppose. Maybe its worth doing?


CCACHE_DISABLE_DIRECT_LOOKUP
  - but still write meta-data
  - optimize initial builds
CCACHE_DISABLE_DIRECT_UPDATE
  - but still allow lookup
  - optimize cache-misses when direct mode would be pointless on
subsequent runs
CCACHE_DISABLE_PREPROCESSOR_LOOKUP
  - direct-mode lookup only
  - optimize cache-misses when we know processor mode will also fail
  - makes the cache read-only
CCACHE_DISABLE_PREPROCESSOR_UPDATE
  - like CCACHE_READONLY but we can still write stats

You really are getting into shaving fractions now though. At this point 
you should seriously consider linking ccache statically; run it with 
callgrind and you can see that a significant proportion of time is spent 
before "main" is even called, loading libraries and such.


Most of the rest of the time is spent doing MD4. I have some ideas how 
to optimize that (by sharing them across runs), but nothing ready to post.


Andrew

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] ccacje direct mode without fall back to running the preprocessor

[vkr]

Hello Andew,
> The answer is "no", because it is necessary to run the preprocessor in
order to fill the cache and record the file list that direct mode lookups
use.

What are the consequences if we don't let ccache record the file list, but
just run the preprocessor and
do a cache lookup using the preprocessor mode?

WiIl I just have additional overhead when the next direct-look up runs? I
guess this is the only drawback.


regards,
Venkat.

On Fri, Sep 5, 2014 at 1:51 AM, Andrew Stubbs  wrote:

> On 04/09/14 08:32, vkr wrote:> I would like to know if I can avoid
> > "If there is no match, ccache falls back to running the preprocessor."
>
> The answer is "no", because it is necessary to run the preprocessor in
> order to fill the cache and record the file list that direct mode lookups
> use.
>
> It might be possible if you wanted to run the cache in read-only mode, but
> that might not be very useful, and I don't believe there's an option for
> that now.
>
> Andrew
>
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] ccacje direct mode without fall back to running the preprocessor

[vkr]

Hello.
Sure, that name is an apt one and I'm glad saved a few seconds  :)

Regards,
venkrao

On Tue, Sep 16, 2014 at 12:54 AM, Joel Rosdahl  wrote:
> Hi,
>
> I'm not opposed to introducing such an option. What would be a good name?
> CCACHE_READONLY_DIRECT (readonly_direct in the config file on master)?
>
> -- Joel
>
>
> On 10 September 2014 09:57, Andrew Stubbs  wrote:
>>
>> On 10/09/14 08:24, vkr wrote:
>>>
>>> Why is that? I mean, when drop preprocessing only when an env variable is
>>> set,
>>> why is it not the right approach?
>>
>>
>> The development sources on the master branch has all-new option and
>> environment variable processing code.
>>
>> The part where you act on the option is probably in the right place, but
>> it ought to be read elsewhere. I'm not sure how that worked in 3.1.9.
>>
>>
>> Andrew
>> ___
>> ccache mailing list
>> ccache@lists.samba.org
>> https://lists.samba.org/mailman/listinfo/ccache
>
>
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] ccacje direct mode without fall back to running the preprocessor

[Joel Rosdahl]

Hi,

I'm not opposed to introducing such an option. What would be a good name?
CCACHE_READONLY_DIRECT (readonly_direct in the config file on master)?

-- Joel


On 10 September 2014 09:57, Andrew Stubbs  wrote:

> On 10/09/14 08:24, vkr wrote:
>
>> Why is that? I mean, when drop preprocessing only when an env variable is
>> set,
>> why is it not the right approach?
>>
>
> The development sources on the master branch has all-new option and
> environment variable processing code.
>
> The part where you act on the option is probably in the right place, but
> it ought to be read elsewhere. I'm not sure how that worked in 3.1.9.
>
>
> Andrew
> ___
> ccache mailing list
> ccache@lists.samba.org
> https://lists.samba.org/mailman/listinfo/ccache
>
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] ccacje direct mode without fall back to running the preprocessor

[Andrew Stubbs]

On 10/09/14 08:24, vkr wrote:

Why is that? I mean, when drop preprocessing only when an env variable is set,
why is it not the right approach?


The development sources on the master branch has all-new option and 
environment variable processing code.


The part where you act on the option is probably in the right place, but 
it ought to be read elsewhere. I'm not sure how that worked in 3.1.9.


Andrew
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] ccacje direct mode without fall back to running the preprocessor

[vkr]

Hi Andrew,
> It's not the right implementation for the master branch, but might be ok in
> the release branch?

Why is that? I mean, when drop preprocessing only when an env variable is set,
why is it not the right approach?

Regards,
Venkrao

On Mon, Sep 8, 2014 at 4:32 PM, Andrew Stubbs  wrote:
> On 08/09/14 11:37, vkr wrote:
>>
>> If the env variable TRS_CCACHE_NO_PREPROCESSOR_ON_DIRECT_LOOKUP_MISS is
>> set
>> then, I do not run preprocessing, but just fall back to running
>> real-compiler.
>>
>> I see a lot of time saving, and I can't think of any side-effect.
>
>
> It's not the right implementation for the master branch, but might be ok in
> the release branch?
>
> That's up to Joel.
>
> Seems like a worthwhile feature, in read-only mode.
>
> Andrew
>
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] ccacje direct mode without fall back to running the preprocessor

[Andrew Stubbs]

On 08/09/14 11:37, vkr wrote:

If the env variable TRS_CCACHE_NO_PREPROCESSOR_ON_DIRECT_LOOKUP_MISS is set
then, I do not run preprocessing, but just fall back to running real-compiler.

I see a lot of time saving, and I can't think of any side-effect.


It's not the right implementation for the master branch, but might be ok 
in the release branch?


That's up to Joel.

Seems like a worthwhile feature, in read-only mode.

Andrew

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] ccacje direct mode without fall back to running the preprocessor

[Andrew Stubbs]

On 08/09/14 11:37, vkr wrote:

If the env variable TRS_CCACHE_NO_PREPROCESSOR_ON_DIRECT_LOOKUP_MISS is set
then, I do not run preprocessing, but just fall back to running real-compiler.

I see a lot of time saving, and I can't think of any side-effect.


It's not the right implementation for the master branch, but might be ok 
in the release branch?


That's up to Joel.

Seems like a worthwhile feature, in read-only mode.

Andrew

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] ccacje direct mode without fall back to running the preprocessor

[vkr]

Hi Andrew
It seems to me that I can switch off preprocessing in a read-only
ccache mode, after a direct lookup fail.
How do you see the following patch?

If the env variable TRS_CCACHE_NO_PREPROCESSOR_ON_DIRECT_LOOKUP_MISS is set
then, I do not run preprocessing, but just fall back to running real-compiler.

I see a lot of time saving, and I can't think of any side-effect.

Your comments on this are very much appreciated. Thank you!.

--- ccache-3.1.9/ccache.c   2013-01-06 22:27:59.0 +0530
+++ ccache-3.1.9_patched/ccache.c   2014-09-08 15:55:49.0 +0530
@@ -2029,6 +2029,14 @@
}
  }

+
+ if (getenv("CCACHE_READONLY")) {
+   if (getenv("TRS_CCACHE_NO_PREPROCESSOR_ON_DIRECT_LOOKUP_MISS")) {
+   cc_log("TRS; Lets no run preprocessor in read-only mode, as it
goes in vain. Running real-compiler.");
+   failed();
+  }
+ }
+
  /*
   * Find the hash using the preprocessed output. Also updates
   * included_files.


On Fri, Sep 5, 2014 at 2:51 PM, vkr  wrote:
> Hello
> First of all, sorry about the typo in the subject/title of my post.
>
> I run ccache in read-only mode itself. But, I still see that after the
> direct cache lookup fail, it falls back to running the preprocessor.
> That's exactly what I wanted to avoid.
>
> It seems to me that having an option to avoid preprocessing after a
> direct-cache-lookup fail is still a very valid option if the user
> knows that preprocesor will result in a failure/he purposefully wants
> to avoid it.
> I hope my justification is valid.
>
>  88 [2014-09-05T14:45:17.445383 25208] Hostname: host.myhost.net
>  89 [2014-09-05T14:45:17.445393 25208] Working directory: (null)
>  90 [2014-09-05T14:45:17.445399 25208] Base directory:
> /var/fpwork/workspace_venkrao/release
>  91 [2014-09-05T14:45:17.445734 25208] Source file: ../my_source.cpp
>  92 [2014-09-05T14:45:17.445745 25208] Dependency file: ../my_source.d
>  93 [2014-09-05T14:45:17.445752 25208] Object file: ../my_source.cpp.o
>  94 [2014-09-05T14:45:17.447011 25208] Trying direct lookup
>  95 [2014-09-05T14:45:17.447167 25208] Looking for object file hash in
> /disk1/ccache_cache_store/transport_release/175750/0/5/eb2b36e5e9d0bb723de833a4529cab-11859.manifest
>  96 [2014-09-05T14:45:17.479757 25204] Looking for object file hash in
> /disk1/ccache_cache_store/transport_release/175750/6/6/e186baaf3c2e9e06619878dac77fe1-114310.manifest
>  97 [2014-09-05T14:45:17.863524 25194] Did not find object file hash in 
> manifest
>  98 [2014-09-05T14:45:17.864854 25194] Running preprocessor
>
>
>
> Regards,
> venkrao
>
> On Fri, Sep 5, 2014 at 1:51 AM, Andrew Stubbs  wrote:
>> On 04/09/14 08:32, vkr wrote:> I would like to know if I can avoid
>>> "If there is no match, ccache falls back to running the preprocessor."
>>
>> The answer is "no", because it is necessary to run the preprocessor in order
>> to fill the cache and record the file list that direct mode lookups use.
>>
>> It might be possible if you wanted to run the cache in read-only mode, but
>> that might not be very useful, and I don't believe there's an option for
>> that now.
>>
>> Andrew
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] ccacje direct mode without fall back to running the preprocessor

[vkr]

Hello
First of all, sorry about the typo in the subject/title of my post.

I run ccache in read-only mode itself. But, I still see that after the
direct cache lookup fail, it falls back to running the preprocessor.
That's exactly what I wanted to avoid.

It seems to me that having an option to avoid preprocessing after a
direct-cache-lookup fail is still a very valid option if the user
knows that preprocesor will result in a failure/he purposefully wants
to avoid it.
I hope my justification is valid.

 88 [2014-09-05T14:45:17.445383 25208] Hostname: host.myhost.net
 89 [2014-09-05T14:45:17.445393 25208] Working directory: (null)
 90 [2014-09-05T14:45:17.445399 25208] Base directory:
/var/fpwork/workspace_venkrao/release
 91 [2014-09-05T14:45:17.445734 25208] Source file: ../my_source.cpp
 92 [2014-09-05T14:45:17.445745 25208] Dependency file: ../my_source.d
 93 [2014-09-05T14:45:17.445752 25208] Object file: ../my_source.cpp.o
 94 [2014-09-05T14:45:17.447011 25208] Trying direct lookup
 95 [2014-09-05T14:45:17.447167 25208] Looking for object file hash in
/disk1/ccache_cache_store/transport_release/175750/0/5/eb2b36e5e9d0bb723de833a4529cab-11859.manifest
 96 [2014-09-05T14:45:17.479757 25204] Looking for object file hash in
/disk1/ccache_cache_store/transport_release/175750/6/6/e186baaf3c2e9e06619878dac77fe1-114310.manifest
 97 [2014-09-05T14:45:17.863524 25194] Did not find object file hash in manifest
 98 [2014-09-05T14:45:17.864854 25194] Running preprocessor



Regards,
venkrao

On Fri, Sep 5, 2014 at 1:51 AM, Andrew Stubbs  wrote:
> On 04/09/14 08:32, vkr wrote:> I would like to know if I can avoid
>> "If there is no match, ccache falls back to running the preprocessor."
>
> The answer is "no", because it is necessary to run the preprocessor in order
> to fill the cache and record the file list that direct mode lookups use.
>
> It might be possible if you wanted to run the cache in read-only mode, but
> that might not be very useful, and I don't believe there's an option for
> that now.
>
> Andrew
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] ccacje direct mode without fall back to running the preprocessor

[Andrew Stubbs]

On 04/09/14 08:32, vkr wrote:> I would like to know if I can avoid
> "If there is no match, ccache falls back to running the preprocessor."

The answer is "no", because it is necessary to run the preprocessor in 
order to fill the cache and record the file list that direct mode 
lookups use.


It might be possible if you wanted to run the cache in read-only mode, 
but that might not be very useful, and I don't believe there's an option 
for that now.


Andrew
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] ccacje direct mode without fall back to running the preprocessor

[vkr]

Hello
The documentation about "direct mode" says:

The current contents of the include files are then hashed and compared
to the information in the manifest. If there is a match, ccache knows
the result of the compilation. If there is no match, ccache falls back
to running the preprocessor. The output from the preprocessor is
parsed to find the include files that were read. The paths and hash
sums of those include files are then stored in the manifest along with
information about the produced compilation result.

I would like to know if I can avoid
"If there is no match, ccache falls back to running the preprocessor."

The reason is if I know that cache lookup with preprocessor output
will also result in a cache miss, there is no need to run
preprocessor,
instead, just fall back directly to running the real compiler, as this
will be faster.

Is there a way to do it?

Thanks,
-venkrao
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache