Re: [ccache] ccache on generated C++ files...

2019-06-24 Thread Malfettone, Kris via ccache
So what I seem to be observing is only when I do a ninja -t clean or make 
clean.  This removes all of my generated files (.h and .cpp files) and of 
course the normal build outputs.  A subsequent run is then mostly cache misses. 
  If I say manually do a rm $(find -name "*.o") then I get a 100% cache hit 
rate.  I had assumed it was due to the fact that the generated files would have 
a newer mtime and ctime.  My assumption must be wrong.

I had tried using:
sloppiness = include_file_ctime,include_file_mtime

But that had the same results.  This makes sense of course because according to 
the documentation those are only for the "file too new" problem which makes 
sense.

Is there an easy way to print diagnostics on why a cache miss occurred?

-Kris

-Original Message-
From: Joel Rosdahl [mailto:j...@rosdahl.net]
Sent: Sunday, June 23, 2019 4:18 PM
To: Malfettone, Kris 
Cc: ccache@lists.samba.org
Subject: Re: [ccache] ccache on generated C++ files...

Hi Kris,

On Fri, 21 Jun 2019 at 02:43, Malfettone, Kris wrote:
> My project uses a large number of generated C++ files (.h/.cpp files). On a
> clean build these files are cleaned up as well. As a result during the next
> build since they are generated again they will all have a newer mtime and
> ctime than the previous runs. Their content is the same though. Is there a
> way for those files to disable checking both the mtime and ctime checks?

I'm not sure I understand your question. Since the content of the generated
files is the same, they will get the same hash sum and will therefore be seen
as the same for the purposes of caching compilation results.

Or are you talking about the "sloppiness = file_stat_matches" option, where the
normal content hashing is sidestepped and matching in the direct mode can be
done by checking only size/mtime/ctime? If so, the answer is that it's not
possible to only check the size. Or rather: It would of course be possible to
implement such an option, but that does not seem like a good idea – it would be
way too dangerous since files of course can differ in many ways but still have
the same size.

But if you really want to try it yourself, you can simply change

  if (fi->fsize != st->size) {
return false;
  }

to

  if (fi->fsize == st->size) {
continue;
  }

in verify_result in src/manifest.c.

Is the problem that you think that hashing the generated files takes too much
time? If so, have you in some way been able to measure that hashing indeed is a
problem?

> I would love to be able to check based on some sort of "check file size then
> check content checksum" ignoring mtime and ctime.

Again, not sure I understand what you mean. In the normal case (no "sloppiness
= file_stat_matches"), the mtime/ctime is not checked for verification (but
they are checked to rule out "too new" files; see
include_file_mtime/include_file_ctime in the manual), so what you describe is
how it already works. In the "sloppiness = file_stat_matches" case, the
verification also falls back to check the content if mtime/ctime don't match.

-- Joel



IMPORTANT: The information contained in this email and/or its attachments is 
confidential. If you are not the intended recipient, please notify the sender 
immediately by reply and immediately delete this message and all its 
attachments. Any review, use, reproduction, disclosure or dissemination of this 
message or any attachment by an unintended recipient is strictly prohibited. 
Neither this message nor any attachment is intended as or should be construed 
as an offer, solicitation or recommendation to buy or sell any security or 
other financial instrument. Neither the sender, his or her employer nor any of 
their respective affiliates makes any warranties as to the completeness or 
accuracy of any of the information contained herein or that this message or any 
of its attachments is free of viruses.
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache


Re: [ccache] ccache on generated C++ files...

2019-06-25 Thread Malfettone, Kris via ccache
I have confirmed that a few files can be different due to our generation code.  
There are no volatile pieces such as preprocessor macros and such, just good 
non-deterministic print orders in our generation code.  I believe this should 
be the culprit but unfortunately will take a bit for me to cleanup to test 
again.  My mistake was "assuming" it was all consistent.

Thanks for all the time / info everybody.  CCache has always been a favorite 
tool of mine.

-Kris

-Original Message-
From: Joel Rosdahl [mailto:j...@rosdahl.net]
Sent: Tuesday, June 25, 2019 3:03 PM
To: Malfettone, Kris 
Cc: ccache@lists.samba.org
Subject: Re: [ccache] ccache on generated C++ files...

On Tue, 25 Jun 2019 at 00:28, Malfettone, Kris via ccache
 wrote:
> So what I seem to be observing is only when I do a ninja -t clean or make
> clean. This removes all of my generated files (.h and .cpp files) and of
> course the normal build outputs. A subsequent run is then mostly cache
> misses. If I say manually do a rm $(find -name "*.o") then I get a 100% cache
> hit rate. I had assumed it was due to the fact that the generated files would
> have a newer mtime and ctime. My assumption must be wrong.

Maybe there is a timestamp in the generated files?

> Is there an easy way to print diagnostics on why a cache miss occurred?

Yes, assuming you use ccache 3.5 or newer, you can enable the debug mode and
compare the hashed input for two different builds. See "To debug why you don’t
get an expected cache hit for an object file [...]" on
<https://ccache.dev/manual/3.7.1.html#_cache_debugging>.

-- Joel



IMPORTANT: The information contained in this email and/or its attachments is 
confidential. If you are not the intended recipient, please notify the sender 
immediately by reply and immediately delete this message and all its 
attachments. Any review, use, reproduction, disclosure or dissemination of this 
message or any attachment by an unintended recipient is strictly prohibited. 
Neither this message nor any attachment is intended as or should be construed 
as an offer, solicitation or recommendation to buy or sell any security or 
other financial instrument. Neither the sender, his or her employer nor any of 
their respective affiliates makes any warranties as to the completeness or 
accuracy of any of the information contained herein or that this message or any 
of its attachments is free of viruses.
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache


[ccache] ccache on generated C++ files...

2019-06-20 Thread Malfettone, Kris via ccache
I apologize if this is answered somewhere already however I could not find it.

My project uses a large number of generated C++ files (.h/.cpp files).  On a 
clean build these files are cleaned up as well.  As a result during the next 
build since they are generated again they will all have a newer mtime and ctime 
than the previous runs.  Their content is the same though.  Is there a way for 
those files to disable checking both the mtime and ctime checks?  I would love 
to be able to check based on some sort of "check file size then check content 
checksum" ignoring mtime and ctime.

If it helps all of those files are generated to a different directory than the 
static source files (cmake source vs binary dirs.).

-Kris



IMPORTANT: The information contained in this email and/or its attachments is 
confidential. If you are not the intended recipient, please notify the sender 
immediately by reply and immediately delete this message and all its 
attachments. Any review, use, reproduction, disclosure or dissemination of this 
message or any attachment by an unintended recipient is strictly prohibited. 
Neither this message nor any attachment is intended as or should be construed 
as an offer, solicitation or recommendation to buy or sell any security or 
other financial instrument. Neither the sender, his or her employer nor any of 
their respective affiliates makes any warranties as to the completeness or 
accuracy of any of the information contained herein or that this message or any 
of its attachments is free of viruses.
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache