> this is a feature request. I think it would make sense to add CoW (Copy-on- > Write) support for ccache (instead of using dangerous hard-links)
I think so too. > for filesystems which support it (zfs, btrfs, more to come?). I've hacked up a simple patch to do this unconditionally on btrfs on Linux, silently falling back to a regular copy if the ioctl fails. The implementation is btrfs-specific; ZFS doesn't seem to support file- level CoW at the moment and there is no cross-fs kernel interface (yet). Notes/questions: - It's been tested only by running a few 'make check's on both my native btrfs system and an ext4 loopback image. Caveat so much emptor. - I've done no performance testing at all. I hope to do so eventually, but free time is scarce. Egon? Anyone? :-) - I really don't think this ought to be run-time configurable. It's a transparent optimisation that either always works, or always falls back to a full copy. On non-Linux platforms, this should compile away. On Linux systems, I'm reasonably assuming the cost of a failed attempt to be less than that of a configuration check, without the ugliness of the latter. - I'm not too fond of the resulting copy_file(), what with the FIXME and the goto (even if goto seems to be used liberally elsewhere). IMHO, copy_file tries to be too transparent, leading to convoluted code for little benefit. I'd like to split it up into separate copy_{to,from}_cache that only do what is needed, but I'm too afraid of having missed something. - So please: tell me why I'm stupid and wrong and save me some work! Regards, T G-R >From a1e00d82ab1eae1bc39122482b35954bf2304af0 Mon Sep 17 00:00:00 2001 From: Tobias Geerinckx-Rice <tobias.geerinckx.r...@gmail.com> Date: Tue, 16 Sep 2014 20:23:14 +0200 Subject: [RFC PATCH] Use fast (copy-on-write) copies on btrfs Modern file systems like btrfs and ZFS support metadata-only file copies. These are similar to hard links, but without the risks: if either the source or destination files are later modified, the contents of the other file(s) do not change. (Currently uses hard-coded #ifdef/#defines which should go away as soon as all this is properly supported in the kernel.) --- util.c | 44 +++++++++++++++++++++++++++++++++++++------- 1 file changed, 37 insertions(+), 7 deletions(-) diff --git a/util.c b/util.c index 156c0be..b5e5ae8 100644 --- a/util.c +++ b/util.c @@ -19,6 +19,7 @@ #include "ccache.h" +#include <sys/ioctl.h> #include <zlib.h> #ifdef HAVE_PWD_H @@ -224,6 +225,26 @@ mkstemp(char *template) #endif /* + * Copy the contents of src_fd to dest_fd in one efficient operation. + * Currently only supports btrfs on Linux, returns -1 on failure. + */ +int +clone_file(int src_fd, int dest_fd) +{ +#ifdef __linux__ +#undef BTRFS_IOCTL_MAGIC +#define BTRFS_IOCTL_MAGIC 0x94 +#undef BTRFS_IOC_CLONE +#define BTRFS_IOC_CLONE _IOW (BTRFS_IOCTL_MAGIC, 9, int) + return ioctl(dest_fd, BTRFS_IOC_CLONE, src_fd); +#else /* __linux__ */ + (void) src_fd; + (void) dest_fd; + return -1; +#endif /* __linux__ */ +} + +/* * Copy src to dest, decompressing src if needed. compress_level > 0 decides * whether dest will be compressed, and with which compression level. */ @@ -260,13 +281,6 @@ copy_file(const char *src, const char *dest, int compress_level) goto error; } - gz_in = gzdopen(fd_in, "rb"); - if (!gz_in) { - cc_log("gzdopen(src) error: %s", strerror(errno)); - close(fd_in); - goto error; - } - if (compress_level > 0) { /* * A gzip file occupies at least 20 bytes, so it will always @@ -289,6 +303,21 @@ copy_file(const char *src, const char *dest, int compress_level) goto error; } gzsetparams(gz_out, compress_level, Z_DEFAULT_STRATEGY); + } else { + /* FIXME: this introduces unnecessary open/close overhead */ + if (!file_is_compressed(src)) { + /* try to create a fast CoW copy first */ + if (clone_file(fd_in, fd_out) == 0) { + goto out; + } + } + } + + gz_in = gzdopen(fd_in, "rb"); + if (!gz_in) { + cc_log("gzdopen(src) error: %s", strerror(errno)); + close(fd_in); + goto error; } while ((n = gzread(gz_in, buf, sizeof(buf))) > 0) { @@ -335,6 +364,7 @@ copy_file(const char *src, const char *dest, int compress_level) return -1; } +out: gzclose(gz_in); gz_in = NULL; if (gz_out) { -- 2.1.0 _______________________________________________ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache