Re: [ccache] ccache-like tool

2010-08-02 Thread Wilson Snyder

I recently released a GPL tool which is, in part, an alternative to
ccache and thought I'd drop a line to this list for anyone interested.
The front page is at http://audited-objects.sourceforge.net/ and a
comparison with ccache is at
http://audited-objects.sourceforge.net/html/man/ao-ccache.html. As a
matter of fact I haven't used ccache seriously in quite a few years,
since I've been developing my own tool, so I'd appreciate any
corrections to the comparison document.

Interesting!  You should take a look at Vesta
http://www.vestasys.org/, which is based on a similar
find-all-input-files concept, though likely quite different
in implementation.
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache


Re: [ccache] ccache-like tool

2010-08-02 Thread David Boyce
On Mon, Aug 2, 2010 at 1:32 PM, Wilson Snyder wsny...@wsnyder.org wrote:
 Interesting!  You should take a look at Vesta
 http://www.vestasys.org/, which is based on a similar
 find-all-input-files concept, though likely quite different
 in implementation.

Yes, I'm aware of Vesta. Never looked into how it works though. As
with ClearCase/clearmake it seems to be a build system tied to an SCM
tool, whereas AO attempts to cut that cord.

-David
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache


Re: [ccache] ccache-like tool

2010-07-30 Thread Joel Rosdahl
On 2010-07-30 05:13, David Boyce wrote:
 I recently released a GPL tool which is, in part, an alternative to
 ccache and thought I'd drop a line to this list for anyone interested.

I've read a little about AO before, but I got the impression that it
wasn't open source so I didn't investigate it further at the time.
Thanks for releasing it! It seems very cool.

 The front page is at http://audited-objects.sourceforge.net/

I've added a link to Audited Objects under the Related projects header
on http://ccache.samba.org.

 and a comparison with ccache is at
 http://audited-objects.sourceforge.net/html/man/ao-ccache.html. As a
 matter of fact I haven't used ccache seriously in quite a few years,
 since I've been developing my own tool, so I'd appreciate any
 corrections to the comparison document.

Here are some corrections and other comments:

ccache is a compiler cache. It acts as a caching pre-processor to
C/C++ compilers, using the -E compiler switch and a hash to detect
when a compilation can be satisfied from cache. This often results
in a 5 to 10 times speedup in common compilations.

This is no longer true, since ccache doesn't always run the preprocessor
anymore (see http://ccache.samba.org/manual.html#_how_ccache_works). I
suggest that you use the current top-level summary instead: ccache is a
compiler cache. It speeds up recompilation by caching previous
compilations and detecting when the same compilation is being done
again. Supported languages are C, C++, Objective-C and Objective-C++.

Note that for simplicity we sometimes refer to CCACHE as a C
compiler speedup when it really works with C or C++.

Minor thing: Objective-C and Objective-C++ are also supported now.

CCACHE is a personal tool; it keeps its cache in the user's home
directory. Thus each user must build up and maintain his/her cache
without sharing.

This is the default, but as you mention further below, it's possible to
configure ccache with a shared cache, so the text above is misleading.

Update: this is apparently no longer true if it ever was, since the
ccache website says nowadays 'You can also share the cache between
users'. I do not know how recent

At least since version 2.4, released 2004-09-13.

or robust this is.

On a multi-user machine with a local ccache directory, I'd say that it's
as robust as for single-user usage. On NFS, probably less robust.

AO has 'selfish' and 'martyr' modes [...] Update - it appears that
CCACHE has analogues here - CCACHE_READONLY is selfish and
CCACHE_RECACHE is altruistic.

Yes.

According to the blurb above, CCACHE typically provides a 5-10
times speedup. AO has been observed to speed up some builds by much
more. Presumably, at least part of the difference comes from the
fact that AO can cache and replay the linking phase (as well as any
other tools run by the build) while CCACHE optimizes only C
compilations.

That's still true. I think it would be quite easy for ccache to support
linking as well, but I've always thought that it rarely would be useful
since the cache hit rate for linking typically will be low. Do you have
any gut feeling or statistics about this -- won't you just fill up the
cache with libraries/executables that seldom will be reused?

AO imposes a certain overhead on commands which it actually runs
(as opposed to replaying). A typical penalty for the initial build
of a piece of software might be 10%, though it is sometimes more and
often dramatically less. CCACHE does too, because it must exec the
compiler twice for each logical compilation, but the CCACHE overhead
appears to be somewhat less.

ccache (on a direct mode cache miss) first runs the preprocessor and
then (on a preprocessor mode cache miss) compiles the output from the
preprocessor, so saying that it runs the compiler twice is a bit
misleading.

CCACHE is only useful when building objects you already have. Why
would someone want to rebuild something they already have? Most
commonly because they don't fully trust their build system and only
feel safe with a clean build from scratch.

Nowadays, I think that other scenarios are at least as common as the
make clean; make case. Quoting from http://ccache.samba.org: [...]
Another reason to use ccache is that the same cache is used for builds
in different directories. If you have several versions or branches of a
software stored in different directories, many of the object files in a
build directory can probably be taken from the cache even if they were
compiled for another version or branch. A third scenario is using ccache
to speed up clean builds performed by servers or build farms that
regularly check that the code is buildable.

Regarding ACTUAL TIMINGS, I think you should mention what ccache
version you used. Compared to earlier versions, ccache 3.0 and higher is
faster for the second pass and slightly slower for the first pass. 

Re: [ccache] ccache-like tool

2010-07-30 Thread David Boyce
On Fri, Jul 30, 2010 at 7:09 AM, Joel Rosdahl j...@rosdahl.net wrote:
 This is no longer true, since ccache doesn't always run the preprocessor
 anymore (see http://ccache.samba.org/manual.html#_how_ccache_works). I
 suggest that you use the current top-level summary instead: ccache is a
 compiler cache. It speeds up recompilation by caching previous
 compilations and detecting when the same compilation is being done
 again. Supported languages are C, C++, Objective-C and Objective-C++.

Fixed for next update.

    Note that for simplicity we sometimes refer to CCACHE as a C
    compiler speedup when it really works with C or C++.

 Minor thing: Objective-C and Objective-C++ are also supported now.

Fixed for next update.

    CCACHE is a personal tool; it keeps its cache in the user's home
    directory. Thus each user must build up and maintain his/her cache
    without sharing.

 This is the default, but as you mention further below, it's possible to
 configure ccache with a shared cache, so the text above is misleading.

    Update: this is apparently no longer true if it ever was, since the
    ccache website says nowadays 'You can also share the cache between
    users'. I do not know how recent

 At least since version 2.4, released 2004-09-13.

    or robust this is.

 On a multi-user machine with a local ccache directory, I'd say that it's
 as robust as for single-user usage. On NFS, probably less robust.

I'll rework this verbiage soon.

 That's still true. I think it would be quite easy for ccache to support
 linking as well, but I've always thought that it rarely would be useful
 since the cache hit rate for linking typically will be low. Do you have
 any gut feeling or statistics about this -- won't you just fill up the
 cache with libraries/executables that seldom will be reused?

Certainly the hit rate for links will typically be low. I have no
statistics, but I agree that it may not make sense for ccache to add
this feature. In the case of AO it just kind of falls out because AO
(again, configurably, but by default) stores all generated artifacts
and is agnostic as to whether they're the result of a compile, a link,
or anything else. Since acting as a cache is only part of what AO is
intended to do, I'd say at this point that storing all artifacts has
value whether they're reused or not. But that's certainly subject to
reconsideration.

    AO imposes a certain overhead on commands which it actually runs
    (as opposed to replaying). A typical penalty for the initial build
    of a piece of software might be 10%, though it is sometimes more and
    often dramatically less. CCACHE does too, because it must exec the
    compiler twice for each logical compilation, but the CCACHE overhead
    appears to be somewhat less.

 ccache (on a direct mode cache miss) first runs the preprocessor and
 then (on a preprocessor mode cache miss) compiles the output from the
 preprocessor, so saying that it runs the compiler twice is a bit
 misleading.

Agreed. I didn't realize this about ccache. Has it always been this
way? I'll clarify.

    CCACHE is only useful when building objects you already have. Why
    would someone want to rebuild something they already have? Most
    commonly because they don't fully trust their build system and only
    feel safe with a clean build from scratch.

 Nowadays, I think that other scenarios are at least as common as the
 make clean; make case. Quoting from http://ccache.samba.org: [...]
 Another reason to use ccache is that the same cache is used for builds
 in different directories. If you have several versions or branches of a
 software stored in different directories, many of the object files in a
 build directory can probably be taken from the cache even if they were
 compiled for another version or branch. A third scenario is using ccache
 to speed up clean builds performed by servers or build farms that
 regularly check that the code is buildable.

Agree.

 Regarding ACTUAL TIMINGS, I think you should mention what ccache
 version you used. Compared to earlier versions, ccache 3.0 and higher is
 faster for the second pass and slightly slower for the first pass. See
 http://ccache.samba.org/performance.html.

Boy, I don't even remember what version of ccache I used. I know I
wrote this document a year or more ago so it was probably 2.4. I'll
put an update on the TODO list.

 One case where I think that ccache actually is more robust than AO is
 when an object file should change even if no files have changed. For
 instance, the expansion of the preprocessor macros __TIME__, __DATE__
 and __FILE__ change independently of the source code files.

__TIME__ and __DATE__, yes. __FILE__? Doesn't look like an absolute path to me.

 Also, when
 compiling with GCC and the -g option, an absolute path to the source
 file is embedded in the object file, which means that if you just cache
 the object file, the wrong path is included in the object file if it's
 compiled from 

Re: [ccache] ccache-like tool

2010-07-30 Thread Joel Rosdahl
On 2010-07-30 14:14, David Boyce wrote:
 On Fri, Jul 30, 2010 at 7:09 AM, Joel Rosdahl j...@rosdahl.net wrote:
 ccache (on a direct mode cache miss) first runs the preprocessor and
 then (on a preprocessor mode cache miss) compiles the output from the
 preprocessor, so saying that it runs the compiler twice is a bit
 misleading.

 Agreed. I didn't realize this about ccache. Has it always been this
 way? I'll clarify.

It seems it was added in 1.6 (2002-04-08).

 One case where I think that ccache actually is more robust than AO is
 when an object file should change even if no files have changed. For
 instance, the expansion of the preprocessor macros __TIME__, __DATE__
 and __FILE__ change independently of the source code files.

 __TIME__ and __DATE__, yes. __FILE__? Doesn't look like an absolute
 path to me.

It is if the source file path is absolute.

-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache