Re: [ccache] Support for color diagnostics
On Thursday 26 of June 2014, Paul Smith wrote: On Thu, 2014-06-26 at 18:44 +0200, Lubos Lunak wrote: Caveats: - Compiles with and without colors are considered different from each other (so they are duplicated). This doesn't seem ideal, does it? No, it doesn't seem ideal. It doesn't seem easy to avoid either. -- Lubos Lunak ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Support for color diagnostics
On Friday 29 of November 2013, Lubos Lunak wrote: Hello, the attached patch adds ccache support for compiler color diagnostics (also reported by somebody as #10075). Ping? Any official comments on the patch? I've been using the patch for half a year by now without problems. Clang automatically uses colors for output automatically if used in terminal. Ccache's redirecting to a file disables this. GCC 4.8 has got a similar support, except that it apparently requires also $GCC_COLORS or an explicit option. The patch detects if the compiler would use colors if used without ccache and explicitly forces an option to achieve this. Note that I do not have GCC 4.8 here, so I tested with Clang's alias and the GCC_COLORS support is done based on documentation. Caveats: - GCC developers decided to roll their own name for the option when introducing it. Clang has an alias for the GCC way, but versions predating that obviously can't support it, so it's necessary to detect the compiler. As ccache doesn't do that (and I don't find it worth much effort, as it can't be 100% reliable anyway), the code merely guesses from the binary name. If the compiler used will be e.g. the 'cc' symlink, there'll be no colors. No big deal. - Since the stderr is different, obviously compiling with and without colors has different results as well. That means that such a compile is duplicated. It's hopefully not such a common case, although it's perfectly possible. I don't know if it's worth the effort to try to be smart here. A possibly simple improvement could be to search the cache with and without the option set and if stderr is empty, reuse the result regardless of the option. I'm not quite sure where exactly this should happen in the code. I expect it'd make sense to add $CCACHE_NOCOLORS to disable this support? I can also create manpage section for this color support, but I first wanted to check here with the code. -- Lubos Lunak From cacb14929748ae93eacefcfa194aa93689d217eb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Lubo=C5=A1=20Lu=C5=88=C3=A1k?= l.lu...@centrum.cz Date: Fri, 29 Nov 2013 12:14:03 +0100 Subject: [PATCH] support compiler color diagnostics if possible Clang and GCC (starting with 4.9) support color output if possible, but since ccache redirects stderr to a file, they detect the output is not a terminal and do not enable colors. Try to detect if colors would be used and force colors explicitly. Caveats: - Compiles with and without colors are considered different from each other (so they are duplicated). - GCC decided to roll its own name for the option, so it's necessary to guess which compiler is actually used. --- ccache.c | 77 1 file changed, 77 insertions(+) diff --git a/ccache.c b/ccache.c index c395fad..e122c50 100644 --- a/ccache.c +++ b/ccache.c @@ -1065,6 +1065,24 @@ hash_compiler(struct mdfour *hash, struct stat *st, const char *path, } /* + * Note that these compiler checks are unreliable, so nothing should hard-depend on them. + */ + +static bool compiler_is_clang() +{ + const char* name = strrchr( orig_args-argv[ 0 ], '/' ); + name = name ? name + 1 : orig_args-argv[ 0 ]; + return strstr( name, clang ) != NULL; +} + +static bool compiler_is_gcc() +{ + const char* name = strrchr(orig_args-argv[ 0 ], '/' ); + name = name ? name + 1 : orig_args-argv[ 0 ]; + return strstr(name, gcc) != NULL || strstr(name, g++) != NULL; +} + +/* * Update a hash sum with information common for the direct and preprocessor * modes. */ @@ -1128,6 +1146,15 @@ calculate_common_hash(struct args *args, struct mdfour *hash) } free(p); } + + /* Possibly hash GCC_COLORS (for color diagnostics). */ + if (compiler_is_gcc()) { + const char* gcc_colors = getenv(GCC_COLORS); + if (gcc_colors != NULL) { + hash_delimiter(hash,gcccolors); + hash_string(hash, gcc_colors); + } + } } /* @@ -1633,6 +1660,13 @@ is_precompiled_header(const char *path) || str_eq(get_extension(path), .pth); } +static bool color_output_possible() +{ + const char* term_env = getenv(TERM); + + return isatty(STDERR_FILENO) term_env strcasecmp(term_env, DUMB) != 0; +} + /* * Process the compiler options into options suitable for passing to the * preprocessor and the real compiler. The preprocessor options don't include @@ -1661,6 +1695,7 @@ cc_process_args(struct args *args, struct args **preprocessor_args, int argc; char **argv; bool result = true; + bool found_color_diagnostics = false; expanded_args = args_copy(args); stripped_args = args_init(0, NULL); @@ -2017,6 +2052,26 @@ cc_process_args(struct args *args, struct args **preprocessor_args, free(arg); } + if (str_eq(argv[i], -fcolor-diagnostics) + || str_eq(argv[i], -fno-color-diagnostics) + || str_eq(argv[i], -fdiagnostics-color) + || str_eq(argv[i], -fdiagnostics-color=always) + || str_eq(argv[i], -fno
Re: [ccache] ccache has issues with current Apple compilers
On Saturday 31 of May 2014, Tom Lane wrote: I did some further experiments and concluded that ccache is in fact kind of broken: when I try to compile a large project with it, I get thousands of weird warnings that do not appear when using gcc directly (many of them the same kind of unused -I argument bleat shown above, but others are warnings about C constructs that I don't normally see any warnings about). The resulting executables seem to work, but I can't use a toolchain that's this noisy. In any case, surely it's unexpected that ccache would change the compiler's warning behavior? Use CCACHE_CPP2 when using ccache with clang (see e.g. http://www.mail-archive.com/ccache@lists.samba.org/msg01045.html). -- Lubos Lunak ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Support for color diagnostics
Ping? On Friday 29 of November 2013, Lubos Lunak wrote: Hello, the attached patch adds ccache support for compiler color diagnostics (also reported by somebody as #10075). Clang automatically uses colors for output automatically if used in terminal. Ccache's redirecting to a file disables this. GCC 4.8 has got a similar support, except that it apparently requires also $GCC_COLORS or an explicit option. The patch detects if the compiler would use colors if used without ccache and explicitly forces an option to achieve this. Note that I do not have GCC 4.8 here, so I tested with Clang's alias and the GCC_COLORS support is done based on documentation. Caveats: - GCC developers decided to roll their own name for the option when introducing it. Clang has an alias for the GCC way, but versions predating that obviously can't support it, so it's necessary to detect the compiler. As ccache doesn't do that (and I don't find it worth much effort, as it can't be 100% reliable anyway), the code merely guesses from the binary name. If the compiler used will be e.g. the 'cc' symlink, there'll be no colors. No big deal. - Since the stderr is different, obviously compiling with and without colors has different results as well. That means that such a compile is duplicated. It's hopefully not such a common case, although it's perfectly possible. I don't know if it's worth the effort to try to be smart here. A possibly simple improvement could be to search the cache with and without the option set and if stderr is empty, reuse the result regardless of the option. I'm not quite sure where exactly this should happen in the code. I expect it'd make sense to add $CCACHE_NOCOLORS to disable this support? I can also create manpage section for this color support, but I first wanted to check here with the code. -- Lubos Lunak ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Support for color diagnostics
On Saturday 30 of November 2013, Loïc Yhuel wrote: Le 30/11/2013 11:07, Lubos Lunak a écrit : On Saturday 30 of November 2013, Loïc Yhuel wrote: Le 29/11/2013 14:08, Lubos Lunak a écrit : On Friday 29 of November 2013, Lubos Lunak wrote: Hello, the attached patch adds ccache support for compiler color diagnostics (also reported by somebody as #10075). ... I think you didn't understand GCC documentation correctly. Actually I think I did. I've now tried with a chroot (openSUSE build service really is a useful tool) and it pretty much matches my understanding of the documentation. From the man page : The default GCC_COLORS is ... Setting GCC_COLORS to the empty string disables colors. GCC enable colors when GCC_COLORS is not set, and your code doesn't. From the man page: The default is ‘never’ if GCC_COLORS environment variable isn't present in the environment. In fact you don't have to test GCC_COLORS at all : when it's an empty string (not unset !), colors are disabled , and adding -fdiagnostics-color doesn't change anything. The patch does not add -fdiagnostics-color when GCC_COLORS is empty. Sorry, I didn't check : the default is auto on Fedora, and not upstream... That's why we don't see the same behavior. http://pkgs.fedoraproject.org/cgit/gcc.git/tree/gcc48-color-auto.patch I see. But given that this autodetection requires a terminal as the output, I don't see any possible way of detecting this. -- Lubos Lunak ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Support for color diagnostics
On Friday 29 of November 2013, Lubos Lunak wrote: Hello, the attached patch adds ccache support for compiler color diagnostics (also reported by somebody as #10075). ... Caveats: I forgot one: - The function color_output_possible() may look simplistic, but as far as I can tell it works just fine (it's been in Incecream for a number of years). -- Lubos Lunak ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
[ccache] Support for Clang plugins (still) not working in 3.1.9
Hello, it looks like my patch 06be1c4cc33bf2def9c6e64483b397128f57dbe3 got included incorrectly into the maint branch. I don't know how exactly the maint branch works, I expect my patch didn't make it into 3.1.8 because you didn't want to include the patch from master that soon, but even though it's been included in 3.1.9, that ccache version still sees such compiles as 'multiple source files', because it requires 8e005b067d8c2423e24ee14ffdee8343f650f1e8 too. Could you please include that commit as well? Thank you. -- Lubos Lunak l.lu...@suse.cz ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Support for Clang plugins (still) not working in 3.1.9
On Tuesday 08 of January 2013, Joel Rosdahl wrote: Could you please include that commit as well? Thank you. I've applied it to maint now, so it will be included in 3.1.10. Please try if current maint works like you expect. Yes, now maint indeed works as well, thanks. -- Lubos Lunak l.lu...@suse.cz ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
[ccache] [PATCH] Hash also Clang plugins
Hello, please review and push the attached patch, which adds Clang plugins to the hash, similarly as is already done with GCC plugins. -- Lubos Lunak l.lu...@suse.cz From 11b35610cf97b1540dc883b03e7ce059735bd04c Mon Sep 17 00:00:00 2001 From: Lubos Lunak l.lu...@suse.cz Date: Thu, 26 Jul 2012 14:45:16 +0200 Subject: [PATCH] hash also Clang plugins --- ccache.c | 10 ++ 1 files changed, 10 insertions(+), 0 deletions(-) diff --git a/ccache.c b/ccache.c index 8e36bdd..e5c2481 100644 --- a/ccache.c +++ b/ccache.c @@ -1061,6 +1061,16 @@ calculate_object_hash(struct args *args, struct mdfour *hash, int direct_mode) continue; } + if (str_eq(args-argv[i], -Xclang) + i + 3 args-argc + str_eq(args-argv[i+1], -load) + str_eq(args-argv[i+2], -Xclang) + stat(args-argv[i+3], st) == 0) { + hash_delimiter(hash, plugin); + hash_compiler(hash, st, args-argv[i+3], false); + continue; + } + /* All other arguments are included in the hash. */ hash_delimiter(hash, arg); hash_string(hash, args-argv[i]); -- 1.7.7 ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] [PATCH] Support for Clang precompiled headers
On Thursday 12 of July 2012, Joel Rosdahl wrote: On 6 July 2012 19:09, Lubos Lunak l.lu...@suse.cz wrote: the attached patches modify ccache to also support Clang PCH in addition to GCC PCH. Thanks! Regarding 0002-hash-clang-s-.pch-file-explicitly.patch: Why is the third assignment of pch_file done outside the if (stat(pchpath, st) == 0) block? This means that pch_file and included_pch_file can be set even when there is no .pch file, so remember_include_file will fail to stat the file and then turn off the direct mode. The test suite also fails after the patch, but passes if the pch_file assignment is moved into the block. Right. I guess that was an oversight when I moved the Multiple precompiled headers used error to one place. Fixed version attached. The patches look good otherwise. PS: Since I've noticed in the archives the recent mail about issues with ccache and warnings about unused arguments: The proper way to use ccache with Clang is to set CCACHE_CPP2, which not only avoids these warnings, but in general Clang works suboptimally if passed preprocessed output (warning/error messages quoting sources are affected, some warnings are not supressed in headers). Do you have any idea about the performance impact of using CCACHE_CPP2 for clang? Debug build of LibreOffice, warm disk caches, empty ccache, make -j4 in xmloff module: CCACHE_CPP2=1: 494.64user 34.99system 2:15.60elapsed 390%CPU (0avgtext+0avgdata 1309824maxresident)k 0inputs+3752560outputs (507major+13592550minor)pagefaults 0swaps not set: 457.17user 28.94system 2:06.15elapsed 385%CPU (0avgtext+0avgdata 1309312maxresident)k 0inputs+3758600outputs (444major+12098612minor)pagefaults 0swaps So in this case CCACHE_CPP2 adds about 8% overhead. It might make a difference for somebody, but I myself would prefer to get better error messages from Clang (and I actually use ccache only for a buildbot, so I don't find this worth the effort). If somebody would feel like playing with this, note that Clang 3.2 will have -E -Wp,-rewrite-includes, which only processes #include directives and nothing more (I wrote the feature to solve a similar problem when using Clang with Icecream for distributed builds, and I assume this is the only thing ccache in fact wants from -E too). So using this in ccache for Clang should presumably reduce this overhead. -- Lubos Lunak l.lu...@suse.cz From d897f0388b35ca0b8e333e7e8edbd57a0c4ba3e1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Lubo=C5=A1=20Lu=C5=88=C3=A1k?= l.lu...@suse.cz Date: Fri, 6 Jul 2012 18:45:05 +0200 Subject: [PATCH 2/2] hash clang's .pch file explicitly It does not appear anywhere in the preprocessed output, so process_preprocessed_file() would miss it. --- ccache.c | 31 +++ 1 files changed, 31 insertions(+), 0 deletions(-) diff --git a/ccache.c b/ccache.c index 12b62a4..926d6b2 100644 --- a/ccache.c +++ b/ccache.c @@ -171,6 +171,11 @@ static bool profile_generate = false; */ static bool using_precompiled_header = false; +/* + * The .gch/.pch file used for compilation. + */ +static char *included_pch_file = NULL; + /* How long (in microseconds) to wait before breaking a stale lock. */ unsigned lock_staleness_limit = 200; @@ -532,6 +537,16 @@ process_preprocessed_file(struct mdfour *hash, const char *path) hash_buffer(hash, p, (end - p)); free(data); + + /* Explicitly check the .gch/.pch file, Clang does not include any mention of it + in the preprocessed output. */ + if (included_pch_file) { + char *path = x_strdup(included_pch_file); + path = make_relative_path(path); + hash_string(hash, path); + remember_include_file(path, strlen(included_pch_file), hash); + } + return true; } @@ -1680,6 +1695,7 @@ cc_process_args(struct args *orig_args, struct args **preprocessor_args, */ if (compopt_takes_path(argv[i])) { char *relpath; + char *pch_file = NULL; if (i == argc-1) { cc_log(Missing argument to %s, argv[i]); stats_update(STATS_ARGS); @@ -1696,23 +1712,37 @@ cc_process_args(struct args *orig_args, struct args **preprocessor_args, if (stat(argv[i+1], st) == 0) { cc_log(Detected use of precompiled header: %s, argv[i+1]); found_pch = true; + pch_file = x_strdup(argv[i+1]); } } else { char* gchpath = format(%s.gch, argv[i+1]); if (stat(gchpath, st) == 0) { cc_log(Detected use of precompiled header: %s, gchpath); found_pch = true; + pch_file = x_strdup(gchpath); } else { char* pchpath = format(%s.pch, argv[i+1]); if (stat(pchpath, st) == 0) { cc_log(Detected use of precompiled header: %s, pchpath); found_pch = true; + pch_file = x_strdup(pchpath); } free(pchpath); } free(gchpath); } + if (pch_file) { +if (included_pch_file) { + cc_log(Multiple precompiled headers used: %s and %s\n, + included_pch_file, pch_file); + stats_update
Re: [ccache] Making ccache and clang compatible
On Wednesday 11 of July 2012, Max Horn wrote: On 11.07.2012, at 02:34, Martin Pool wrote: On 5 July 2012 21:35, Mike Frysinger vap...@gentoo.org wrote: On Wednesday 04 July 2012 09:53:46 Max Horn wrote: err, -I/-isystem/-D/-U/etc... are preprocessor flags and are valid when running in preprocessor mode (-E). i don't know if clang is broken, or your analysis is incorrect, but certainly that behavior you describe is wrong. -mike I agree with you there, Mike, As do I, actually -- sorry, my email was indeed nonsense and confused. What I *meant* was that on the second run (the one without -E, with preprocessed data as input), clang prints out warnings. Now, I am not saying that ccache is doing anything wrong here, But you probably could. -I, -D, -U etc. are preprocessor options and not compiler options, and the preprocessor is (at least in the separate compilation steps theory) not run when the driver is fed preprocessed source. but I think the bug originally described in https://bugzilla.samba.org/show_bug.cgi?id=8460 could possibly be valid. If you are compiling from a .i or .ii file, the -D and -I options can't have any effect. It's reasonable for clang to emit a warning about it, and it would be reasonable for ccache to strip those options when compiling a preprocessed file. Aye, if that would be acceptable for the ccache authors, that would be the best solution I think. The best option is to use CCACHE_CPP2 with Clang. Giving Clang preprocessed source causes a number of small problems, such as some warnings not being suppressed or error/warning messages quoting modified source code. As a side-effect, this problem with preprocessor options will not exist with CCACHE_CPP2 either. It is a question how much not using CCACHE_CPP2 with Clang would improve performance anyway. -- Lubos Lunak l.lu...@suse.cz ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
[ccache] [PATCH] Support for Clang precompiled headers
Hello, the attached patches modify ccache to also support Clang PCH in addition to GCC PCH. The changes are: - Clang uses .pch as the extension, not .gch - the .pch file can be directly passed using -include-pch, instead of using -include and trying to find the .pch file - the use of a .pch is not reflected in the resulting .ii , so it needs to be explicitly added to the hash for checking if used files have changed - I also believe that path_len passed to remember_include_file() is incorrect, as make_relative_path() may change the string PS: Since I've noticed in the archives the recent mail about issues with ccache and warnings about unused arguments: The proper way to use ccache with Clang is to set CCACHE_CPP2, which not only avoids these warnings, but in general Clang works suboptimally if passed preprocessed output (warning/error messages quoting sources are affected, some warnings are not supressed in headers). -- Lubos Lunak l.lu...@suse.cz From 768ffacb434d116759fc73cf0d2723aba6b5af4c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Lubo=C5=A1=20Lu=C5=88=C3=A1k?= l.lu...@suse.cz Date: Fri, 6 Jul 2012 18:09:36 +0200 Subject: [PATCH 1/3] support for precompiled headers with clang Support the clang-specific -include-pch option, which references the PCH file itself, and support the .pch extension when using the gcc -include way. --- MANUAL.txt |2 ++ ccache.c | 29 + compopt.c |1 + 3 files changed, 24 insertions(+), 8 deletions(-) diff --git a/MANUAL.txt b/MANUAL.txt index 4be33ae..478d36b 100644 --- a/MANUAL.txt +++ b/MANUAL.txt @@ -629,6 +629,8 @@ things to make it work properly: -- ** use the *-include* compiler option to include the precompiled header (i.e., don't use *#include* in the source code to include the header); or +** (Clang compiler) use the *-include-pch* compiler option to include + the PCH file generated from the precompiled header; or ** add the *-fpch-preprocess* compiler option when compiling. If you don't do this, either the non-precompiled version of the header file diff --git a/ccache.c b/ccache.c index 8b50c36..12b62a4 100644 --- a/ccache.c +++ b/ccache.c @@ -167,7 +167,7 @@ static bool profile_use = false; static bool profile_generate = false; /* - * Whether we are using a precompiled header (either via -include or #include). + * Whether we are using a precompiled header (either via -include, #include or clang's -include-pch). */ static bool using_precompiled_header = false; @@ -1355,7 +1355,7 @@ find_compiler(char** argv) bool is_precompiled_header(const char *path) { - return str_eq(get_extension(path), .gch); + return str_eq(get_extension(path), .gch) || str_eq(get_extension(path), .pch); } /* @@ -1680,7 +1680,6 @@ cc_process_args(struct args *orig_args, struct args **preprocessor_args, */ if (compopt_takes_path(argv[i])) { char *relpath; - char *pchpath; if (i == argc-1) { cc_log(Missing argument to %s, argv[i]); stats_update(STATS_ARGS); @@ -1693,13 +1692,27 @@ cc_process_args(struct args *orig_args, struct args **preprocessor_args, args_add(stripped_args, relpath); /* Try to be smart about detecting precompiled headers */ - pchpath = format(%s.gch, argv[i+1]); - if (stat(pchpath, st) == 0) { -cc_log(Detected use of precompiled header: %s, pchpath); -found_pch = true; + if (str_eq(argv[i], -include-pch)) { +if (stat(argv[i+1], st) == 0) { + cc_log(Detected use of precompiled header: %s, argv[i+1]); + found_pch = true; +} + } else { +char* gchpath = format(%s.gch, argv[i+1]); +if (stat(gchpath, st) == 0) { + cc_log(Detected use of precompiled header: %s, gchpath); + found_pch = true; +} else { + char* pchpath = format(%s.pch, argv[i+1]); + if (stat(pchpath, st) == 0) { + cc_log(Detected use of precompiled header: %s, pchpath); + found_pch = true; + } + free(pchpath); +} +free(gchpath); } - free(pchpath); free(relpath); i++; continue; diff --git a/compopt.c b/compopt.c index 77b57f5..908302e 100644 --- a/compopt.c +++ b/compopt.c @@ -61,6 +61,7 @@ static const struct compopt compopts[] = { {-imacros,AFFECTS_CPP | TAKES_ARG | TAKES_PATH}, {-imultilib, AFFECTS_CPP | TAKES_ARG | TAKES_PATH}, {-include,AFFECTS_CPP | TAKES_ARG | TAKES_PATH}, + {-include-pch,AFFECTS_CPP | TAKES_ARG | TAKES_PATH}, {-install_name, TAKES_ARG}, /* Darwin linker option */ {-iprefix,AFFECTS_CPP | TAKES_ARG | TAKES_PATH}, {-iquote, AFFECTS_CPP | TAKES_ARG | TAKES_PATH}, -- 1.7.7 From bbfc0ed55a6f730023d068773fc9fece16b94c74 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Lubo=C5=A1=20Lu=C5=88=C3=A1k?= l.lu...@suse.cz Date: Fri, 6 Jul 2012 18:46:25 +0200 Subject: [PATCH 3/3] do not pass incorrect length to remember_include_file() --- ccache.c |7 --- 1 files changed, 4 insertions(+), 3 deletions(-) diff --git