Re: [ccache] Configuration file plan
On 2011-04-07 03:17, Wilson Snyder wrote: > A read of a NFS file can cost 100 msec, which adds up across thousands > of compiles. Reading /etc is fine, but in our setup HOME is slow. > [...] So I'd suggest CCACHE_CONFIG_PATH which could be a colon > separated list of directories to search, defaulting to say > "/etc:$HOME/.ccache". I could then simplify it to "/local_ssd/ccache" > for my site. Actually, I've already thought about the "slow $HOME" case, but I forgot to describe it in the mail. :-) To avoid reading $HOME/.ccache, I'm thinking it should be enough to put "cache_dir = /local/ssd/ccache" in /etc/ccache.conf. configuration reading will be done something like this: 1. Populate a configuration struct with compile-time defaults (e.g. "cache_dir = $HOME/.ccache"). 2. Update the struct with settings from /ccache.conf. 3. Update cache_dir in the struct from the CCACHE_DIR environment variable. 4. Update the struct with settings from ccache.conf in cache_dir. 5. Update the struct with settings from other CCACHE_* environment variables. Do you think this will be good enough? > [...] Also, I'd really like to have a hierarchy as I cache stuff on > network-closer systems. [...] This would also allow creating a > compile-directory local file of ".ccache" which would be awesome for > some uses, and for testing. I guess you're talking about having multiple cache directories so that you can read cache results from (or even write to) several places? That would also be interesting to mock up, but that's not what I'm currently trying to design. > I suspect what you're aiming for is an INI file format. INI files are > a non-standard standard :) so I'd suggest copying git's. See below. I think git's configuration format is a bit overkill for ccache, and its configuration parsing code also isn't generic enough to steal. > Can -p print the output in .ini format? That way people can easily > convert an existing env var setup to .ini format! My idea was that it should print the current settings and from where each setting originates, but that's maybe overkill. But it could very well print it in ".ini format", yes. >> The -F and -M command-line options will update the configuration file >> instead of the stats file (and those entries in the stats files will >> become obsolete). > > What if there's no file or more than one? Seems to me it's much safter > to punt. As said above, there will still be one and only one cache directory, so a configuration file will always be available (unless it's read-only, of course). > In general I don't like programs rewriting config files I agree. I would be nice to be able drop the -F/-M options (and my suggested -o option) but I'm not sure about how much breakage that would cause. Perhaps it's best to just deprecate them and let them emit warnings? > Also, rather than add any new env vars, I'd suggest a CCACHE_FLAGS > which can contain arguments to the program, then people can just use > CCACHE_FLAGS="-o goo=bar" rather than your adding CCACHE_GOO. Yes, perhaps. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Configuration file plan
On 2011-04-07 02:30, Martin Pool wrote: > I would think about not putting ccache.conf inside .ccache, because > that directory might often be excluded from backups or in some other > way treated as disposable. Aside from that it sounds really good. I'd prefer the configuration to be tied to the cache, so it will be hard to do it another way given the current cache directory layout. I guess the only reasonable alternative is $(dirname $cache_dir)/ccache.conf. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Configuration file plan
On 2011-04-06 23:25, Owen Mann wrote: > Question: for an NFS-mounted cache, wouldn't different users' > CCACHE_MAX values conflict? It seems to me it's better as a > one-time or per-cache setting. I can definitely see how it would have > value in a single-user setup, though. One of my reasons for wanting to introduce a configuration file is to improve the multi-user case so that it's easier to make all users use the same settings (umask, compression, hardlink, nlevels, max_size, etc.), so I haven't considered having a per-user configuration file -- /ccache.conf is per-cache. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Configuration file plan
>I would like to improve ccache's configurability. As you >know, configuration of ccache is today done by setting a >bunch of environment variables, but that's becoming >increasingly messy; there are currently 24 CCACHE_* >variables in the development version of ccache. I think >it's time to move settings to a configuration file while >keeping the ability to override configuration by setting >environment variables. This sounds great! >I actually envision two configuration files: > >- /ccache.conf (typically /etc/ccache.conf) >- /ccache.conf (typically $HOME/.ccache/ccache.conf) A read of a NFS file can cost 100 msec, which adds up across thousands of compiles. Reading /etc is fine, but in our setup HOME is slow. Also, I'd really like to have a hierarchy as I cache stuff on network-closer systems. So I'd suggest CCACHE_CONFIG_PATH which could be a colon separated list of directories to search, defaulting to say "/etc:$HOME/.ccache". I could then simplify it to "/local_ssd/ccache" for my site. This would also allow creating a compile-directory local file of ".ccache" which would be awesome for some uses, and for testing. >I expect that the overhead of reading the configuration >file(s) will be negligible. Not really - see above. >The "max files" and "max size" settings (set by >command-line options -F/--max-files and -M/--max-size) are >today stored in the "stats" files in the cache, but these >will find a much better place in /ccache.conf. Getting these out of the cache would be great. Every few months someone resets our cache without using our special create script, and we loose the -F/-M's. >I think the configuration file syntax should be a simple >text format like this: > ># A comment. >key = value I suspect what you're aiming for is an INI file format. INI files are a non-standard standard :) so I'd suggest copying git's. See below. > -p, --print-config print current configuration settings Can -p print the output in .ini format? That way people can easily convert an existing env var setup to .ini format! >The -F and -M command-line options will update the >configuration file instead of the stats file (and those >entries in the stats files will become obsolete). What if there's no file or more than one? Seems to me it's much safter to punt. In general I don't like programs rewriting config files, and I really don't want users mucking with the config. (Actually I'd probably have the config file under version control and thus read only so it's safe - please be sure the error message is nice if a -F fails due to a non-writable file.) Also, rather than add any new env vars, I'd suggest a CCACHE_FLAGS which can contain arguments to the program, then people can just use CCACHE_FLAGS="-o goo=bar" rather than your adding CCACHE_GOO. -Wilson == === from git-config The syntax is fairly flexible and permissive; whitespaces are mostly ignored. The # and ; characters begin comments to the end of line, blank lines are ignored. The file consists of sections and variables. A section begins with the name of the section in square brackets and continues until the next section begins. Section names are not case sensitive. Only alphanumeric characters, - and . are allowed in section names. Each variable must belong to some section, which means that there must be section header before first setting of a variable. ... All the other lines are recognized as setting variables, in the form name = value. If there is no equal sign on the line, the entire line is taken as name and the variable is recognized as boolean "true". The variable names are case-insensitive and only alphanumeric characters and - are allowed. There can be more than one value for a given variable; we say then that variable is multivalued. Leading and trailing whitespace in a variable value is discarded. Internal whitespace within a variable value is retained verbatim. The values following the equals sign in variable assign are all either a string, an integer, or a boolean. Boolean values may be given as yes/no, 0/1 or true/false. Case is not significant in boolean values, when converting value to the canonical form using --bool type specifier; git-config will ensure that the output is "true" or "false". String values may be entirely or partially enclosed in double quotes. You need to enclose variable value in double quotes if you want to preserve leading or trailing whitespace, or if variable value contains beginning of comment characters (if it contains # or ;). Double quote " and backslash \ characters in variable value must be escaped: use \" for " and \\ for \. The following escape sequences (beside \" and \\) are recognized: \n
Re: [ccache] Configuration file plan
I would think about not putting ccache.conf inside .ccache, because that directory might often be excluded from backups or in some other way treated as disposable. Aside from that it sounds really good. Martin ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Configuration file plan
FWIW, it sounds pretty sexy to me! :-)
Question: for an NFS-mounted cache, wouldn't different users' CCACHE_MAX
values conflict? It seems to me it's better as a one-time or per-cache setting.
I can definitely see how it would have value in a single-user setup, though.
Original message
>Date: Wed, 06 Apr 2011 22:12:55 +0200
>From: ccache-boun...@lists.samba.org (on behalf of Joel Rosdahl
>)
>Subject: [ccache] Configuration file plan
>To: ccache list
>
>Hi,
>
>I would like to improve ccache's configurability. As you know, configuration of
>ccache is today done by setting a bunch of environment variables, but that's
>becoming increasingly messy; there are currently 24 CCACHE_* variables in the
>development version of ccache. I think it's time to move settings to a
>configuration file while keeping the ability to override configuration by
>setting environment variables.
>
>I actually envision two configuration files:
>
>- /ccache.conf (typically /etc/ccache.conf)
>- /ccache.conf (typically $HOME/.ccache/ccache.conf)
>
>ccache reads both (if they exist) each time it's run. Settings then have the
>following priority:
>
>1. Environment variables.
>2. /ccache.conf
>3. /ccache.conf
>4. Compile-time defaults.
>
>This means that:
>
>- The administrator can choose default settings for the system by editing
> /ccache.conf.
>- The user can override the system settings with his/her own default settings
> by editing /ccache.conf. (If is shared by several
> users, the settings are of course also shared.)
>- The user can override the default settings temporarily by setting environment
> variables.
>- It's backwards compatible with the existing environment variables.
>
>I expect that the overhead of reading the configuration file(s) will be
>negligible.
>
>The "max files" and "max size" settings (set by command-line options
>-F/--max-files and -M/--max-size) are today stored in the "stats" files in the
>cache, but these will find a much better place in /ccache.conf.
>
>I think the configuration file syntax should be a simple text format like this:
>
># A comment.
>key = value
>
>No quotes around strings (at least not mandatory). Whitespace is trimmed at the
>beginning and end of values.
>
>Mapping between environment variables and keys in the configuration files:
>
>Environment variable Config key Type Default Note
>---
>CCACHE_BASEDIR base_dir string ""[1]
>CCACHE_CC compiler string ""[2]
>CCACHE_COMPILERCHECK compiler_check string mtime
>CCACHE_COMPRESScompression boolean false
>CCACHE_CPP2run_second_cpp boolean false
>CCACHE_DETECT_SHEBANG detect_shebang boolean false
>CCACHE_DIR cache_dirstring $HOME/.ccache
>CCACHE_DIRECT direct_mode boolean true
>CCACHE_DISABLE disable boolean false
>CCACHE_EXTENSION cpp_extensionstring ""[3]
>CCACHE_EXTRAFILES extra_files_to_hash string ""
>CCACHE_HARDLINKhard_linkboolean false
>CCACHE_HASHDIR hash_dir boolean false
>CCACHE_LOGFILE log_file string ""[4]
>NEW CCACHE_MAXFILESmax_filesinteger 0 [5]
>NEW CCACHE_MAXSIZE max_size integer 1G[6]
>CCACHE_NLEVELS cache_dir_levels integer 2
>CCACHE_PATHpath string ""[7]
>CCACHE_PREFIX prefix_command string ""[8]
>CCACHE_READONLYread_onlyboolean false
>CCACHE_RECACHE recache boolean false
>CCACHE_SLOPPINESS sloppiness string ""
>CCACHE_STATS statsboolean true
>CCACHE_TEMPDIR temporary_dirstring ""[9]
>CCACHE_UMASK umaskinteger - [10]
>CCACHE_UNIFY unifyboolean false
>
>String and integer settings are handled in a straight-forward manner, but
>boolean settings are a bit special for backward-compatibility reasons:
>
> - A boolean CCACHE_FOO set to any value (even the empty string) means true;
>unset means false.
> - For each boolean CCACHE_FOO, there is also a CCACHE_NOFOO with the inverse
>logic. CCACHE_FOO (if set) has higher priority than CCACHE_NOFOO. (Some
>settings are quite silly in their NO variant, but I think it's worth it for
>consistency.)
> - The possible values of a boolean setting in the configuration file are
>spelled "true" and "false".
>
>$VARIABLE and ${VARIABLE} references will be expanded as expected in variables
>that specify paths.
>
>Notes:
>
>[1] Th
