Re: [ccache] Migrating to ccache 4: a "drop-in replacement"?

[Joel Rosdahl via ccache]

On Mon, 7 Dec 2020 at 22:55, Steffen Dettmer via ccache
 wrote:
> I just tested with ccache 3.7.12 and "ccache -s" that it seems to work as I
> expect

Cool, my memory was wrong about this! Or alternatively, it might be that the
final implementation of the config file handling did not become what I first
had planned since the manual has documented the behavior I described since
added in 2012:
https://github.com/ccache/ccache/commit/99396f88#diff-426db03993d0f56a62aa3f8976b22d2424d152c49b1cb4c893e71b151d0f846cR242-R246

Hopefully improved in .

> Thank you for sharing your expertise! Although not ccache specific,
> I'm glad to get your input because your input usually is just great!
> [...systemd...]

It was just a tongue-in-cheek jab at your tongue-in-cheek characterization of
XDG. I have nothing in particular against not using systemd, but I find it
convenient that it has become the norm (and as a Debian developer I didn't vote
against choosing systemd). Let's stop here.

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] Migrating to ccache 4: a "drop-in replacement"?

[Steffen Dettmer via ccache]

Hi!

Thanks so much for all your detailed and helpful answers!

On Sat, Dec 5, 2020 at 3:30 PM Joel Rosdahl  wrote:

> On Thu, 3 Dec 2020 at 11:28, Steffen Dettmer wrote:
> > a "plug-in replacement"?
>
> Yes to all instances of the question.
>

Wow, I didn't hope that it could be that great! Thanks so much!


> There is no difference in how cleanup works between old and new versions.
> In
> other words, both new and old ccache versions will clean up files
> generated by
> both new and old versions.
>

Ohh that is great news too!


> > [...] release notes tell that if ~/.ccache/ccache.conf exists, it will be
> > used (release notes are not clear for me if only as fallack or in any
> case
> > when it exists).
>
> See
> <
> https://ccache.dev/manual/4.1.html#_location_of_the_primary_configuration_file
> >.
>

ahh silly me, I had looked there for myself. Thank you, this is very clear.

> Also CCACHE_CONFIGPATH could be set to point to it if needed, but it is
> not
> > needed.
>
> There is almost never a reason to set CCACHE_CONFIGPATH. Instead, just set
> CCACHE_DIR to ~/.ccache if you want the old behavior.
>
> >   $ grep -v '^#' ~/.ccache/ccache.conf
> >   cache_dir=/...shared_local_path.../tmp/ccache
>
> This won't have any effect. cache_dir only has an effect if set in the
> secondary (system-wide read-only) configuration file (typically
> /etc/ccache.conf) or in the environment variable form ($CCACHE_DIR).


I just tested with ccache 3.7.12 and "ccache -s" that it seems to work as I
expect,
like:

  $ $path/bin/ccache -s|grep "config"
  primary config  /nfshomes/sdettmer/.ccache/ccache.conf
  secondary config  (readonly)/usr/local/etc/ccache.conf
  $ file /usr/local/etc/ccache.conf
  /usr/local/etc/ccache.conf: cannot open `/usr/local/etc/ccache.conf' (No
such file or directory)

  $ grep ^cache_dir ~/.ccache/ccache.conf
  cache_dir=/tmp/sdettmer/ccache
  $ $path/bin/ccache -s|grep "cache directory"
  cache directory /tmp/sdettmer/ccache

  $ grep ^cache_dir ~/.ccache/ccache.conf
  cache_dir=/tmp/ccache-all/ccache
  $ $path/bin/ccache -s|grep "cache directory"
  cache directory /tmp/ccache-all/ccache

What do I miss?


(off topic)

> NB: Our reference system is "Devuan 2.0 ASCII", a Debian fork without
> > SystemD.
>
> Sorry to hear that (SCNR).
>

Thank you for sharing your expertise! Although not ccache specific,
I'm glad to get your input because your input usually is just great!

Do you think this is bad or could cause problems?
Do you have someone concrete in mind?

As far as I know, our main reason was, that our embedded systems don't
have SystemD but use init.d scripts, so we made the development ref sys
also don't having SystemD, assuming it would avoid trouble (SystemD is
said to boot faster, but boot time was no issue for us).

Much information about SystemD pros/cons in the web [1] seem to be
very polarized and was not found to be sufficient.

I'm afraid just because all major distros use it, avoiding it could
cause trouble[2].

What do you think, should we better use a mainstream distro
(Debian or any clone, Mint or whatever)?

Let me close with an old joke:
Finally SystemD now is almost feature complete!
Just an integrated kernel and a decent browser are missing. :-)

Steffen

[1] Good example is
http://0pointer.de/blog/projects/why.html which shows properties many
consider disadvantages ("Interfacing via D-Bus", "Shell-free bootup",
"fsck handling"...) as "features" -- but it contains no single feature
SystemD has not ("customisation without being C programmer",
"indepentent small building blocks leaving the choice", "deterministic",
"easy to trace", "simple to understand", "applications don't need to
depend on it", "no hardcoded DNS IPs"...).

[2] such as what we had:
that ssh-agent did not work anymore, because gnome-keyring
overwrote its env but did not work, because the xinitrc forcibly
loads gnome-keyring with shell code that was hacked to work for
Wayland and in short only works with SystemD, and all that because
Wayland with SystemD does not work with ssh-agent because user processes
are not forked from the xinit'd process.
I'm not sure but I think it somehow worked with systemd (someone else
had reported I think).
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] Migrating to ccache 4: a "drop-in replacement"?

[Joel Rosdahl via ccache]

On Thu, 3 Dec 2020 at 11:28, Steffen Dettmer wrote:
> Is upgrading to ccache 4 a "drop-in replacement" even if sharing the cache
> directory with still running ccache 3 versions?
> [...]
> Does it really make new versions a "plug-in replacement"?
> [...]
> So in short: I would not need to change anything at all. Just deploy it. Is 
> this correct?

Yes to all instances of the question.

> [...] even cleanup should work: it is backward compatible. If I understand
> correctly, this means as long as the new ccache version runs from time to
> time, it will cleanup both old and new files and avoid ccache directory grow
> (much) beyond max_size.

There is no difference in how cleanup works between old and new versions. In
other words, both new and old ccache versions will clean up files generated by
both new and old versions.

> [...] release notes tell that if ~/.ccache/ccache.conf exists, it will be
> used (release notes are not clear for me if only as fallack or in any case
> when it exists).

See
.

> Also CCACHE_CONFIGPATH could be set to point to it if needed, but it is not
> needed.

There is almost never a reason to set CCACHE_CONFIGPATH. Instead, just set
CCACHE_DIR to ~/.ccache if you want the old behavior.

>   $ grep -v '^#' ~/.ccache/ccache.conf
>   cache_dir=/...shared_local_path.../tmp/ccache

This won't have any effect. cache_dir only has an effect if set in the
secondary (system-wide read-only) configuration file (typically
/etc/ccache.conf) or in the environment variable form ($CCACHE_DIR).

>   hard_link=false

(This is already the default.)

> NB: Our reference system is "Devuan 2.0 ASCII", a Debian fork without
> SystemD.

Sorry to hear that (SCNR).

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] Migrating to ccache 4: a "drop-in replacement"?

[Steffen Dettmer via ccache]

Hi,

in short:
Is upgrading to ccache 4 a "drop-in replacement" even if sharing
the cache directory with still running ccache 3 versions?


in detail:
I consider upgrading to ccache 4 with or without sharing the
cache directory with older ccache (~3.7.7). Sharing the directory
could ease migration for users of course.

The new ccache version would be automatically deployed to the
users and used by all recent software branches, but the old
ccache would still be used by older branches (toolchain tool
versions are "pinned" for better reproducibility). On some
servers, the cache directory is shared across users.

ccache version 4.0 introduces a new cache directory structure [1]:

> The cache directory structure has changed compared to previous
> versions (more details below). This means that ccache 4.0 will
> not share cache results with earlier versions. It is however safe
> to run ccache 4.0 and earlier versions against the same cache
> directory: cache bookkeeping, statistics and cleanup are backward
> compatible, with the minor exception that some statistics
> counters incremented by ccache 4.0 won’t be visible when
> running ccache -s with an older version.

About the cache directory structure, I didn't find "more details
below", but the description seems to be clear that sharing the
cache directory wouldn't be any problem, even cleanup should
work: it is backward compatible. If I understand correctly, this
means as long as the new ccache version runs from time to time,
it will cleanup both old and new files and avoid ccache directory
grow (much) beyond max_size. That sounds great! Does it really
make new versions a "plug-in replacement"? That would be so cool!

Also config file location has changed to match the expectations
on Poetterings laptop [2] (SCNR), but release notes tell that if
~/.ccache/ccache.conf exists, it will be used (release notes are
not clear for me if only as fallack or in any case when it
exists). Also CCACHE_CONFIGPATH could be set to point to it if
needed, but it is not needed.

So in short: I would not need to change anything at all. Just
deploy it.

Is this correct?

Sorry for the long mail, but using ccache in detail can be quite
complex and I try to avoid missing an important detail.

Steffen




In case it helps our configuration:

  $ grep -v '^#' ~/.ccache/ccache.conf
  max_size = 0
  cache_dir=/...shared_local_path.../tmp/ccache
  hard_link=false
  umask=002

(only our Jenkins account runs with max_size = 50G to have a
central place to set the size)

cmake wrapper script example (it depends on the build):

  #!/bin/bash
  # Xcode generator doesn't include the compiler as the first arg:
  [ "$1" = "/.../i586-pc-linux-gnu/gcc-4.4.3/usr/bin/i586-pc-linux-gnu-g++"
] && shift

  # "Convert" cmake variable to an env (but keep present values) to allow
shared caches
  : ${CCACHE_BASEDIR="/.../sdettmer/work/xyz-src"}
  : ${CCACHE_CPP2="set"}
  # : ${CCACHE_NODIRECT="set"}
  export CCACHE_BASEDIR CCACHE_CPP2 CCACHE_NODIRECT
  unset GCC_COLORS
  { "/.../bin/ccache" "/.../gcc-4.4.3/usr/bin/i586-pc-linux-gnu-g++" "$@"
2>&1 1>&3 3>&-  | sed "s|^\(In file included from \)\?\.\./|\1$(pwd)/../|"
; exit ${PIPESTATUS[0]} ; } 3>&1 1>&2

NB: Our reference system is "Devuan 2.0 ASCII", a Debian fork without
SystemD.


[1] https://ccache.dev/releasenotes.html#_ccache_4_0
[2]
https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache