Re: [ccache] ccache on generated C++ files...
I have confirmed that a few files can be different due to our generation code. There are no volatile pieces such as preprocessor macros and such, just good non-deterministic print orders in our generation code. I believe this should be the culprit but unfortunately will take a bit for me to cleanup to test again. My mistake was "assuming" it was all consistent. Thanks for all the time / info everybody. CCache has always been a favorite tool of mine. -Kris -Original Message- From: Joel Rosdahl [mailto:j...@rosdahl.net] Sent: Tuesday, June 25, 2019 3:03 PM To: Malfettone, Kris Cc: ccache@lists.samba.org Subject: Re: [ccache] ccache on generated C++ files... On Tue, 25 Jun 2019 at 00:28, Malfettone, Kris via ccache wrote: > So what I seem to be observing is only when I do a ninja -t clean or make > clean. This removes all of my generated files (.h and .cpp files) and of > course the normal build outputs. A subsequent run is then mostly cache > misses. If I say manually do a rm $(find -name "*.o") then I get a 100% cache > hit rate. I had assumed it was due to the fact that the generated files would > have a newer mtime and ctime. My assumption must be wrong. Maybe there is a timestamp in the generated files? > Is there an easy way to print diagnostics on why a cache miss occurred? Yes, assuming you use ccache 3.5 or newer, you can enable the debug mode and compare the hashed input for two different builds. See "To debug why you don’t get an expected cache hit for an object file [...]" on <https://ccache.dev/manual/3.7.1.html#_cache_debugging>. -- Joel IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses. ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] ccache on generated C++ files...
So what I seem to be observing is only when I do a ninja -t clean or make clean. This removes all of my generated files (.h and .cpp files) and of course the normal build outputs. A subsequent run is then mostly cache misses. If I say manually do a rm $(find -name "*.o") then I get a 100% cache hit rate. I had assumed it was due to the fact that the generated files would have a newer mtime and ctime. My assumption must be wrong. I had tried using: sloppiness = include_file_ctime,include_file_mtime But that had the same results. This makes sense of course because according to the documentation those are only for the "file too new" problem which makes sense. Is there an easy way to print diagnostics on why a cache miss occurred? -Kris -Original Message- From: Joel Rosdahl [mailto:j...@rosdahl.net] Sent: Sunday, June 23, 2019 4:18 PM To: Malfettone, Kris Cc: ccache@lists.samba.org Subject: Re: [ccache] ccache on generated C++ files... Hi Kris, On Fri, 21 Jun 2019 at 02:43, Malfettone, Kris wrote: > My project uses a large number of generated C++ files (.h/.cpp files). On a > clean build these files are cleaned up as well. As a result during the next > build since they are generated again they will all have a newer mtime and > ctime than the previous runs. Their content is the same though. Is there a > way for those files to disable checking both the mtime and ctime checks? I'm not sure I understand your question. Since the content of the generated files is the same, they will get the same hash sum and will therefore be seen as the same for the purposes of caching compilation results. Or are you talking about the "sloppiness = file_stat_matches" option, where the normal content hashing is sidestepped and matching in the direct mode can be done by checking only size/mtime/ctime? If so, the answer is that it's not possible to only check the size. Or rather: It would of course be possible to implement such an option, but that does not seem like a good idea – it would be way too dangerous since files of course can differ in many ways but still have the same size. But if you really want to try it yourself, you can simply change if (fi->fsize != st->size) { return false; } to if (fi->fsize == st->size) { continue; } in verify_result in src/manifest.c. Is the problem that you think that hashing the generated files takes too much time? If so, have you in some way been able to measure that hashing indeed is a problem? > I would love to be able to check based on some sort of "check file size then > check content checksum" ignoring mtime and ctime. Again, not sure I understand what you mean. In the normal case (no "sloppiness = file_stat_matches"), the mtime/ctime is not checked for verification (but they are checked to rule out "too new" files; see include_file_mtime/include_file_ctime in the manual), so what you describe is how it already works. In the "sloppiness = file_stat_matches" case, the verification also falls back to check the content if mtime/ctime don't match. -- Joel IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses. ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
[ccache] ccache on generated C++ files...
I apologize if this is answered somewhere already however I could not find it. My project uses a large number of generated C++ files (.h/.cpp files). On a clean build these files are cleaned up as well. As a result during the next build since they are generated again they will all have a newer mtime and ctime than the previous runs. Their content is the same though. Is there a way for those files to disable checking both the mtime and ctime checks? I would love to be able to check based on some sort of "check file size then check content checksum" ignoring mtime and ctime. If it helps all of those files are generated to a different directory than the static source files (cmake source vs binary dirs.). -Kris IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses. ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache