[ccache] ccacje direct mode without fall back to running the preprocessor
Hello The documentation about "direct mode" says: The current contents of the include files are then hashed and compared to the information in the manifest. If there is a match, ccache knows the result of the compilation. If there is no match, ccache falls back to running the preprocessor. The output from the preprocessor is parsed to find the include files that were read. The paths and hash sums of those include files are then stored in the manifest along with information about the produced compilation result. I would like to know if I can avoid "If there is no match, ccache falls back to running the preprocessor." The reason is if I know that cache lookup with preprocessor output will also result in a cache miss, there is no need to run preprocessor, instead, just fall back directly to running the real compiler, as this will be faster. Is there a way to do it? Thanks, -venkrao ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] ccacje direct mode without fall back to running the preprocessor
Hello First of all, sorry about the typo in the subject/title of my post. I run ccache in read-only mode itself. But, I still see that after the direct cache lookup fail, it falls back to running the preprocessor. That's exactly what I wanted to avoid. It seems to me that having an option to avoid preprocessing after a direct-cache-lookup fail is still a very valid option if the user knows that preprocesor will result in a failure/he purposefully wants to avoid it. I hope my justification is valid. 88 [2014-09-05T14:45:17.445383 25208] Hostname: host.myhost.net 89 [2014-09-05T14:45:17.445393 25208] Working directory: (null) 90 [2014-09-05T14:45:17.445399 25208] Base directory: /var/fpwork/workspace_venkrao/release 91 [2014-09-05T14:45:17.445734 25208] Source file: ../my_source.cpp 92 [2014-09-05T14:45:17.445745 25208] Dependency file: ../my_source.d 93 [2014-09-05T14:45:17.445752 25208] Object file: ../my_source.cpp.o 94 [2014-09-05T14:45:17.447011 25208] Trying direct lookup 95 [2014-09-05T14:45:17.447167 25208] Looking for object file hash in /disk1/ccache_cache_store/transport_release/175750/0/5/eb2b36e5e9d0bb723de833a4529cab-11859.manifest 96 [2014-09-05T14:45:17.479757 25204] Looking for object file hash in /disk1/ccache_cache_store/transport_release/175750/6/6/e186baaf3c2e9e06619878dac77fe1-114310.manifest 97 [2014-09-05T14:45:17.863524 25194] Did not find object file hash in manifest 98 [2014-09-05T14:45:17.864854 25194] Running preprocessor Regards, venkrao On Fri, Sep 5, 2014 at 1:51 AM, Andrew Stubbs wrote: > On 04/09/14 08:32, vkr wrote:> I would like to know if I can avoid >> "If there is no match, ccache falls back to running the preprocessor." > > The answer is "no", because it is necessary to run the preprocessor in order > to fill the cache and record the file list that direct mode lookups use. > > It might be possible if you wanted to run the cache in read-only mode, but > that might not be very useful, and I don't believe there's an option for > that now. > > Andrew ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] ccacje direct mode without fall back to running the preprocessor
Hi Andrew
It seems to me that I can switch off preprocessing in a read-only
ccache mode, after a direct lookup fail.
How do you see the following patch?
If the env variable TRS_CCACHE_NO_PREPROCESSOR_ON_DIRECT_LOOKUP_MISS is set
then, I do not run preprocessing, but just fall back to running real-compiler.
I see a lot of time saving, and I can't think of any side-effect.
Your comments on this are very much appreciated. Thank you!.
--- ccache-3.1.9/ccache.c 2013-01-06 22:27:59.0 +0530
+++ ccache-3.1.9_patched/ccache.c 2014-09-08 15:55:49.0 +0530
@@ -2029,6 +2029,14 @@
}
}
+
+ if (getenv("CCACHE_READONLY")) {
+ if (getenv("TRS_CCACHE_NO_PREPROCESSOR_ON_DIRECT_LOOKUP_MISS")) {
+ cc_log("TRS; Lets no run preprocessor in read-only mode, as it
goes in vain. Running real-compiler.");
+ failed();
+ }
+ }
+
/*
* Find the hash using the preprocessed output. Also updates
* included_files.
On Fri, Sep 5, 2014 at 2:51 PM, vkr wrote:
> Hello
> First of all, sorry about the typo in the subject/title of my post.
>
> I run ccache in read-only mode itself. But, I still see that after the
> direct cache lookup fail, it falls back to running the preprocessor.
> That's exactly what I wanted to avoid.
>
> It seems to me that having an option to avoid preprocessing after a
> direct-cache-lookup fail is still a very valid option if the user
> knows that preprocesor will result in a failure/he purposefully wants
> to avoid it.
> I hope my justification is valid.
>
> 88 [2014-09-05T14:45:17.445383 25208] Hostname: host.myhost.net
> 89 [2014-09-05T14:45:17.445393 25208] Working directory: (null)
> 90 [2014-09-05T14:45:17.445399 25208] Base directory:
> /var/fpwork/workspace_venkrao/release
> 91 [2014-09-05T14:45:17.445734 25208] Source file: ../my_source.cpp
> 92 [2014-09-05T14:45:17.445745 25208] Dependency file: ../my_source.d
> 93 [2014-09-05T14:45:17.445752 25208] Object file: ../my_source.cpp.o
> 94 [2014-09-05T14:45:17.447011 25208] Trying direct lookup
> 95 [2014-09-05T14:45:17.447167 25208] Looking for object file hash in
> /disk1/ccache_cache_store/transport_release/175750/0/5/eb2b36e5e9d0bb723de833a4529cab-11859.manifest
> 96 [2014-09-05T14:45:17.479757 25204] Looking for object file hash in
> /disk1/ccache_cache_store/transport_release/175750/6/6/e186baaf3c2e9e06619878dac77fe1-114310.manifest
> 97 [2014-09-05T14:45:17.863524 25194] Did not find object file hash in
> manifest
> 98 [2014-09-05T14:45:17.864854 25194] Running preprocessor
>
>
>
> Regards,
> venkrao
>
> On Fri, Sep 5, 2014 at 1:51 AM, Andrew Stubbs wrote:
>> On 04/09/14 08:32, vkr wrote:> I would like to know if I can avoid
>>> "If there is no match, ccache falls back to running the preprocessor."
>>
>> The answer is "no", because it is necessary to run the preprocessor in order
>> to fill the cache and record the file list that direct mode lookups use.
>>
>> It might be possible if you wanted to run the cache in read-only mode, but
>> that might not be very useful, and I don't believe there's an option for
>> that now.
>>
>> Andrew
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] ccacje direct mode without fall back to running the preprocessor
Hi Andrew, > It's not the right implementation for the master branch, but might be ok in > the release branch? Why is that? I mean, when drop preprocessing only when an env variable is set, why is it not the right approach? Regards, Venkrao On Mon, Sep 8, 2014 at 4:32 PM, Andrew Stubbs wrote: > On 08/09/14 11:37, vkr wrote: >> >> If the env variable TRS_CCACHE_NO_PREPROCESSOR_ON_DIRECT_LOOKUP_MISS is >> set >> then, I do not run preprocessing, but just fall back to running >> real-compiler. >> >> I see a lot of time saving, and I can't think of any side-effect. > > > It's not the right implementation for the master branch, but might be ok in > the release branch? > > That's up to Joel. > > Seems like a worthwhile feature, in read-only mode. > > Andrew > ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] ccacje direct mode without fall back to running the preprocessor
Hello. Sure, that name is an apt one and I'm glad saved a few seconds :) Regards, venkrao On Tue, Sep 16, 2014 at 12:54 AM, Joel Rosdahl wrote: > Hi, > > I'm not opposed to introducing such an option. What would be a good name? > CCACHE_READONLY_DIRECT (readonly_direct in the config file on master)? > > -- Joel > > > On 10 September 2014 09:57, Andrew Stubbs wrote: >> >> On 10/09/14 08:24, vkr wrote: >>> >>> Why is that? I mean, when drop preprocessing only when an env variable is >>> set, >>> why is it not the right approach? >> >> >> The development sources on the master branch has all-new option and >> environment variable processing code. >> >> The part where you act on the option is probably in the right place, but >> it ought to be read elsewhere. I'm not sure how that worked in 3.1.9. >> >> >> Andrew >> ___ >> ccache mailing list >> ccache@lists.samba.org >> https://lists.samba.org/mailman/listinfo/ccache > > ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
[ccache] -Bprefix compiler option - resulting in cache miss
Hello, I believe I found a potential bug with ccache that results in cache-miss if -Bprfix compilation option is used, where prefix specifies where to find the executables, libraries, include files, and data files of the compiler itself. Example, if my compilation command is: CCACHE_BASEDIR=$PWD ccache gcc -c hello.c -o hello.o -B$PWD/linkfarm/sysroot-x86_64_cge7/usr/lib64/4.7.0/ then, from what I understand, ccache is considering this argument -B$PWD/linkfarm/sysroot-x86_64_cge7/usr/lib64/4.7.0/ as well, while creating the basic-hash for direct cache lookup, but without rewriting it into its relative path based on CCACHE_BASEDIR setting. Thus, the hash so generated is for the absolute path of this workspace, and cannot be reused from another build sandbox. I verified that the changes done in the fork https://github.com/venkrao/ccache-1/commit/59e5244dd79b0fc7df682c8f3c05b778a3d00f91 fixes the problem. To verify that -Bprefix is also put into the hash, with its absolute path, I put a debug statement on the line https://github.com/venkrao/ccache-1/blob/59e5244dd79b0fc7df682c8f3c05b778a3d00f91/ccache.c#L1594 Please help to verify my understanding and the fix. Thanks, Venkat. ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
[ccache] @file arguments to ccache - Intel Compiler
Hello, I was wondering why in ccache we are expanding @file https://github.com/venkrao/ccache-1/blob/59e5244dd79b0fc7df682c8f3c05b778a3d00f91/ccache.c#L1951 Essentially, the arguments that are to be listed in @file are those that affect the preprocessor. Is it safe to just not expand the arguments onto the commandline via ccache function above, but, just pass on the @file argument as-is to the compiler command-line? The reason I'm asking this is, Intel compiler, apparently accepts shell comments(# comments) inside the .cfg file, as well as new line characters. And as far as my test results reveal, those fail ccache, as we do not care filtering our the shell comments inside ccache, instead, read the entire file content and put it into the command-line, and this results in "Preprocessor error" So, is it safe to just pass on the @file as-is to compiler, without letting ccache read and expand the arguments? I can think of a bad case, where the user puts in an absolute path -I path inside the @file, and that results in cache miss in direct mode.. I'm unable to decide how to go about supporting Intel compiler, especially with the @file. Intel compiler apparently even accepts special char like hello.h$ or regex based input args inside the @file which might make ccache handle them more difficult. Regards, Venkat. ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
[ccache] Implementing a Read-only HTTP CCACHE_DIR(resurrect)
Hello, I stumbled across this thread - https://lists.samba.org/archive/ccache/2012q2/000879.html which is years old, Coincidentally, I did some work along similar lines already, without realizing there was this discussion about this topic here, and I appreciate some comments/suggestions on my approach so far. Having cache on NFS is comparatively the easy option from configuration point of view, however, there can be environments where for whatever the reasons, NFS server is a few hops away, while there are other machines that are closer to the build farm, in which case, having a HTTP CCACHE_DIR does seemed like a reasonably better option as it involves less configuration havoc on every machine in the build farm. Keeping the above as use case, I've implemented HTTP CCACHE_DIR in my fork - https://github.com/venkrao/ccache This is a very crude throw-away test from a beginner C Programmer, that does the following. Care has been taken to ensure it does behave like existing ccache to the extent I know so far, and I did have successful runs of modified ccache with no core/crash or surprise failures. 1. Offers ONLY read-only http ccache_dir! 2. A Server process is to be started on the build machine(as of now, independent of ccache. But plan is to integrate this into ccache)(see server.c in the fork) 3. Server process uses a single cURL handle, and listens to all the incoming cache download requests(which can be for .d/.o/.manifest or any such file that ccache looks up otherwise in the CCACHE_DIR location) 4. Server process was necessary to share the cURL handle, to reuse a single http connection as much as possible(have I got it right that this does save some overhead on cURL that not many tcp connections are opened by cURL?) 5. ccache sends the URL, and the download location to the server, and it gets a response code based on which ccache runs the next actions. Unfortunately, in our existing environment I could not see visible improvement between our NFS based cache setup and this new approach. I cannot attribute the lack of performance to anything right now. I do think that http ccache_dir is better than NFS based cache in our case because, by benchmarking for data copy say so. Example, specific amount of data copied between my webserver(used in the new approach tests), and the build machine as compared to NFS server and build machine shows NFS is no good for us. In simple terms, is it a viable feature that my fork may be integrated into ccache after careful consideration, that ccache_dir is served via http?(for both get, and put cache) Please consider reviewing the changes I made. Your comments and time are greatly appreciated. Regards, Venkat. ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] ccacje direct mode without fall back to running the preprocessor
Hello Andew, > The answer is "no", because it is necessary to run the preprocessor in order to fill the cache and record the file list that direct mode lookups use. What are the consequences if we don't let ccache record the file list, but just run the preprocessor and do a cache lookup using the preprocessor mode? WiIl I just have additional overhead when the next direct-look up runs? I guess this is the only drawback. regards, Venkat. On Fri, Sep 5, 2014 at 1:51 AM, Andrew Stubbs wrote: > On 04/09/14 08:32, vkr wrote:> I would like to know if I can avoid > > "If there is no match, ccache falls back to running the preprocessor." > > The answer is "no", because it is necessary to run the preprocessor in > order to fill the cache and record the file list that direct mode lookups > use. > > It might be possible if you wanted to run the cache in read-only mode, but > that might not be very useful, and I don't believe there's an option for > that now. > > Andrew > ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] ccacje direct mode without fall back to running the preprocessor
Sorry. It is not the manifest file that it tries to update. But the ".d"
files. I think it is a real bug.
In read-only mode, when ccache tries to run cache look up in preprocessor
mode, it tries to put_file_in_cache for .d file unconditionally.
if (generating_dependencies && mode == FROMCACHE_CPP_MODE) {
put_file_in_cache(output_dep, cached_dep);
}
This is buggy, as it should check also check we are in
read-only/read-only-direct mode, like so:
if (generating_dependencies && mode == FROMCACHE_CPP_MODE &&
(!conf->read_only && !conf->read_only_direct)) {
Or even better, if put_file_in_cache returns if we are in
read-only/read-only-direct mode, which should fix more such hidden cases,
in future.
What do you think? Which one is better?
regards,
Venkat.
On Thu, Dec 10, 2015 at 6:00 PM, Andrew Stubbs wrote:
> On 10/12/15 11:38, vkr wrote:
>
>> * >> (*) If I use CCACHE_READONLY - then, there is a huge number of
>> files not hitting cache(direct-cache look up fails), and build fails
>> because of the cache storage being read-only, and the preprocessor
>> lookup that ran after direct-lookup failure, tries to update the
>> manifest file back in the cache. We can't go by this, as we want a
>> pure-read-only cache.
>>
>
> Huh? It still tries to write the manifest even in CCACHE_READONLY mode?
>
> I just tested it and no, it doesn't do that. If I edit a comment in a
> header file then I see "cache hit (preprocessed)" increment when I
> recompile. If I turn off CCACHE_READONLY then it rewrites the manifest, as
> expected, and the direct-mode starts working again.
>
> If I make the cache files read-only (chmod -R -w .ccache) then I see no
> error when CCACHE_READONLY is set (and CCACHE_TEMPDIR also), and the
> expected error writing the manifest otherwise.
>
> If I set the read-only in the config file (ccache -o read_only=true) then
> it works without the environment variable set.
>
> Are you sure your configuration is set right? Perhaps you should have
> "read_only=true" in your cache config, and then CCACHE_NOREADONLY=1 in your
> jenkins environment?
>
> I'm using the dev version of ccache, but this ought to work for all 3.2.x
> releases.
>
> Thinking about it, if you have multiple users sharing a writeable cache
> and one of them has a modified header that doesn't affect the preprocessed
> code, then you would end up with the manifest file flip-flopping between
> the two states. Maybe the read-only cache is a good idea, for the majority
> of users who have not modified any files?
>
> Andrew
>
>
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache
[ccache] Path rewriting in ccache make_relative_path - realpath conversion and its necessity
Hello, I've a question around the function `make_relative_path()' that rewrites the given path to its relative path, and while doing so, it converts the given path to the realpath, and then returns the relative path for the realpath based on the current working directory. I was wondering why it was necessary to use the realpath while rewriting the path to relative, instead of the given path as-is? i.e When the check for `x_realpath' on a given path passes, why not just use the given path to rewrite it to relative? like so: relpath = get_relative_path(get_current_working_dir(), canon_path); // Current relpath = get_relative_path(get_current_working_dir(), path); // proposed The reason I'm asking this is, while technically there is nothing wrong in first converting the given path to realpath, and then rewriting it to relative, in a build system where there are external tools that use the build system provided paths, but miss out that "converting to that realpath" part, then it gets difficult. Handling it at ccache itself is perhaps more faster, and convenient. Your comments are appreciated. regards, Venkat. ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Path rewriting in ccache make_relative_path - realpath conversion and its necessity
While I still look forward to hearing if the above is acceptable or not, I
noticed that
the in argument -isystem is not converted to
relative.(Unsupported, not bug :) )
if (str_startswith(argv[i], "-isystem/")) {
/* Handle the -isystem option, if prefix path comes
from inside base_dir,
*/
char *relpath = make_relative_path(x_strdup(argv[i]
+ 8));
char *option = format("-isystem%s", relpath);
args_add(stripped_args, option);
free(relpath);
free(option);
continue;
}
That inserted in ccache.c should solve the problem. Please comment on this
also.
regards,
venkat.
On Tue, Jan 12, 2016 at 3:05 PM, vkr wrote:
> Hello,
> I've a question around the function `make_relative_path()' that rewrites
> the given path to its relative path, and while doing so,
> it converts the given path to the realpath, and then returns the relative
> path for the realpath based on the current working directory.
>
> I was wondering why it was necessary to use the realpath while rewriting
> the path to relative, instead of the given path as-is?
> i.e When the check for `x_realpath' on a given path passes, why not just
> use the given path to rewrite it to relative?
> like so:
>
> relpath = get_relative_path(get_current_working_dir(),
> canon_path); // Current
> relpath = get_relative_path(get_current_working_dir(),
> path); // proposed
>
> The reason I'm asking this is, while technically there is nothing wrong in
> first converting the given path to realpath, and then
> rewriting it to relative, in a build system where there are external tools
> that use the build system provided paths,
> but miss out that "converting to that realpath" part, then it gets
> difficult.
>
> Handling it at ccache itself is perhaps more faster, and convenient.
>
> Your comments are appreciated.
>
> regards,
> Venkat.
>
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache
