Re: dd conv=nocache
Pádraig Brady [EMAIL PROTECTED] wrote: I was thinking it would be useful to add an option to dd to tell it to instruct the OS not to cache the data. I like the idea, even if it works only with very recent Linux kernels. Personally I use dd to move large files like disc images etc. around, and I know I will not need the file cached. A very quick example patch which has only been tested on linux is attached. Do you think this is required? Do you think this is the right interface? There are two separate sets of data to which we might want to apply such an optimization: input and output. So how about an interface that lets the user disable caching on either or both? E.g., no-i-cache no-o-cache. I find the --less options far less readable. Actually, I prefer the even longer names: input-cache-disable output-cache-disable nocache could still be an alias for the combination. ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: dd conv=nocache
Jim Meyering wrote: Pádraig Brady [EMAIL PROTECTED] wrote: I was thinking it would be useful to add an option to dd to tell it to instruct the OS not to cache the data. I like the idea, even if it works only with very recent Linux kernels. It's not that new. It's in since 2.5.60 according to the man page, and I verified the code is in 2.6.0 Also it uses a posix interface, so I presume this is available elsewhere. Personally I use dd to move large files like disc images etc. around, and I know I will not need the file cached. A very quick example patch which has only been tested on linux is attached. Do you think this is required? Do you think this is the right interface? There are two separate sets of data to which we might want to apply such an optimization: input and output. So how about an interface that lets the user disable caching on either or both? E.g., no-i-cache no-o-cache. I find the --less options far less readable. Actually, I prefer the even longer names: input-cache-disable output-cache-disable nocache could still be an alias for the combination. I considered this, but thought you wouldn't need this granularity. But thinking more about it I think you're right. If we want to control input and output separately perhaps we should use flags. So we have: iflag=nocache and oflag=nocache The usual case where we want to do the same on both could be handled with flag=nocache One can't specify both [io]flag=FLAGS with flag=FLAGS at present, but I think that should be supported anyway. thanks, Pádraig. ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: dd conv=nocache
Pádraig Brady [EMAIL PROTECTED] wrote: Jim Meyering wrote: Pádraig Brady [EMAIL PROTECTED] wrote: I was thinking it would be useful to add an option to dd to tell it to instruct the OS not to cache the data. I like the idea, even if it works only with very recent Linux kernels. It's not that new. It's in since 2.5.60 according to the man page, and I verified the code is in 2.6.0 Also it uses a posix interface, so I presume this is available elsewhere. Note that I said works, not exists :-) Does it work consistently even for initial versions? ... I considered this, but thought you wouldn't need this granularity. But thinking more about it I think you're right. If we want to control input and output separately perhaps we should use flags. So we have: iflag=nocache and oflag=nocache Of course! Using that mechanism is the way to go. ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
dd conv=nocache
Hi,
I was thinking it would be useful to
add an option to dd to tell it to
instruct the OS not to cache the data.
Personally I use dd to move large files like disc images etc.
around, and I know I will not need the file cached.
A very quick example patch which has only
been tested on linux is attached.
Do you think this is required?
Do you think this is the right interface?
Note, posix_fadvise(POSIX_FADV_DONTNEED) on linux,
invalidates the cache for a file rather than a process.
I.E. if a file is already cached, and a
process calls posix_fadvise(POSIX_FADV_DONTNEED),
then the file is uncached for everyone.
This is not what is usually required, and
I'm working on the kernel guys to get them
to change this.
thanks,
Pádraig.
--- coreutils-6.2/src/dd.c 2006-08-16 19:53:51.0 +
+++ coreutils-dd/src/dd.c 2007-03-06 08:55:28.0 +
@@ -117,7 +117,8 @@
C_NOCREAT = 01,
C_EXCL = 02,
C_FDATASYNC = 04,
-C_FSYNC = 010
+C_FSYNC = 010,
+C_NOCACHE = 020
};
/* Status bit masks. */
@@ -254,6 +255,7 @@
{sync, C_SYNC}, /* Pad input records to ibs with NULs. */
{fdatasync, C_FDATASYNC}, /* Synchronize output data before finishing. */
{fsync, C_FSYNC}, /* Also synchronize output metadata. */
+ {nocache, C_NOCACHE}, /* Instruct the system not to cache the data. */
{, 0}
};
@@ -453,6 +455,7 @@
with block or unblock, pad with spaces rather than NULs\n\
fdatasync physically write output file data before finishing\n\
fsync likewise, but also write metadata\n\
+ nocache instruct the system not to cache the data\n\
), stdout);
fputs (_(\
\n\
@@ -725,6 +728,20 @@
}
}
+void invalidate_cache(int fd, size_t size)
+{
+ if (! (conversions_mask C_NOCACHE))
+return;
+ /* Note be careful to invalidate only what we've read
+ so that we don't dump any readahead cache. */
+ off_t offset = lseek(fd, 0, SEEK_CUR);
+ if (offset offset!=(off_t)-1) {
+/* TODO: coalesce into at least page_size requests.
+ Currently there are 4 extra syscalls per block. */
+posix_fadvise(fd, 0, offset, POSIX_FADV_DONTNEED);
+ }
+}
+
/* Read from FD into the buffer BUF of size SIZE, processing any
signals that arrive before bytes are read. Return the number of
bytes read if successful, -1 (setting errno) on failure. */
@@ -737,8 +754,10 @@
ssize_t nread;
process_signals ();
nread = read (fd, buf, size);
- if (! (nread 0 errno == EINTR))
+ if (! (nread 0 errno == EINTR)) {
+ invalidate_cache(fd, nread);
return nread;
+ }
}
}
@@ -774,6 +793,7 @@
total_written += nwritten;
}
+ invalidate_cache(fd, total_written);
return total_written;
}
___
Bug-coreutils mailing list
Bug-coreutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-coreutils
