Re: Threaded versions of cp, mv, ls for high latency / parallel filesystems?

2008-11-15 Thread James Youngman
On Fri, Nov 14, 2008 at 5:44 AM, Andrew McGill <[EMAIL PROTECTED]> wrote: > At the risk of drifting off topic - is there ever a benefit in the shell > implementing a ">"-redirection with just O_TRUNC , rather than O_TRUNC | > O_APPEND ? That is already the existing behaviour. The >> redirection o

Re: Threaded versions of cp, mv, ls for high latency / parallel filesystems?

2008-11-13 Thread Andrew McGill
On Thursday 13 November 2008 14:52:44 Ralf Wildenhues wrote: > Hello Andrew, > > Andrew McGill lunch.za.net> writes: > >     find -type f -print0 | > > xargs -0 -n 8 --max-procs=16 md5sum >& ~/md5sums > > > >     sort -k2 < md5sums > md5sums.sorted > > To avoid losing output, use append mo

Re: Threaded versions of cp, mv, ls for high latency / parallel filesystems?

2008-11-13 Thread James Youngman
On Wed, Nov 12, 2008 at 4:01 PM, Phillip Susi <[EMAIL PROTECTED]> wrote: > James Youngman wrote: >> >> This version should be race-free: >> >> find -type f -print0 | >> xargs -0 -n 8 --max-procs=16 md5sum >> ~/md5sums 2>&1 >> >> I think that writing into a pipe should be OK, since pipes are >>

Re: Threaded versions of cp, mv, ls for high latency / parallel filesystems?

2008-11-13 Thread Ralf Wildenhues
Hello Andrew, Andrew McGill lunch.za.net> writes: > >     find -type f -print0 | > xargs -0 -n 8 --max-procs=16 md5sum >& ~/md5sums > >     sort -k2 < md5sums > md5sums.sorted To avoid losing output, use append mode for writing: : > ~/md5sums find -type f -print0 |

Re: Threaded versions of cp, mv, ls for high latency / parallel filesystems?

2008-11-12 Thread Phillip Susi
James Youngman wrote: This version should be race-free: find -type f -print0 | xargs -0 -n 8 --max-procs=16 md5sum >> ~/md5sums 2>&1 I think that writing into a pipe should be OK, since pipes are non-seekable. However, with pipes in this situation you still have a problem if processes try

Re: Threaded versions of cp, mv, ls for high latency / parallel filesystems?

2008-11-12 Thread James Youngman
[ CC ++ [EMAIL PROTECTED] ] On Tue, Nov 11, 2008 at 2:58 PM, Andrew McGill <[EMAIL PROTECTED]> wrote: > What would you expect this to do --: > > find -type f -print0 | > xargs -0 -n 8 --max-procs=16 md5sum >& ~/md5sums Produce a race condition :)It generates 16 parallel processes

Re: Threaded versions of cp, mv, ls for high latency / parallel filesystems?

2008-11-11 Thread Andrew McGill
On Saturday 08 November 2008 20:05:25 Jim Meyering wrote: > Andrew McGill <[EMAIL PROTECTED]> wrote: > > Greetings coreutils folks, > > > > There are a number of interesting filesystems (glusterfs, lustre? ... > > NFS) which could benefit from userspace utilities doing certain > > operatings in par

Re: Threaded versions of cp, mv, ls for high latency / parallel filesystems?

2008-11-09 Thread James Youngman
On Sun, Nov 9, 2008 at 11:06 PM, Dr. David Alan Gilbert <[EMAIL PROTECTED]> wrote: > I keep wondering if the OS level needs a better interface; an 'openv' or > 'statv' > or I'm currently wondering if a combined call would work - something which > would stat a path, if it's a normal file, open it,

Re: Threaded versions of cp, mv, ls for high latency / parallel filesystems?

2008-11-09 Thread Dr. David Alan Gilbert
* Andrew McGill ([EMAIL PROTECTED]) wrote: > Greetings coreutils folks, > > There are a number of interesting filesystems (glusterfs, lustre? ... NFS) > which could benefit from userspace utilities doing certain operatings in > parallel. (I have a very slow glusterfs installation that makes me

Re: Threaded versions of cp, mv, ls for high latency / parallel filesystems?

2008-11-08 Thread James Youngman
On Sat, Nov 8, 2008 at 6:05 PM, Jim Meyering <[EMAIL PROTECTED]> wrote: > How about parallelizing it via xargs, e.g., > >$ echo a b c d e f g h | xargs -t -n4 --no-run-if-empty \ > --max-procs=2 -- cp --target-directory=dest >cp --target-directory=dest a b c d >cp --target-director

Re: Threaded versions of cp, mv, ls for high latency / parallel filesystems?

2008-11-08 Thread Jim Meyering
Andrew McGill <[EMAIL PROTECTED]> wrote: > Greetings coreutils folks, > > There are a number of interesting filesystems (glusterfs, lustre? ... NFS) > which could benefit from userspace utilities doing certain operatings in > parallel. (I have a very slow glusterfs installation that makes me think

Threaded versions of cp, mv, ls for high latency / parallel filesystems?

2008-11-08 Thread Andrew McGill
Greetings coreutils folks, There are a number of interesting filesystems (glusterfs, lustre? ... NFS) which could benefit from userspace utilities doing certain operatings in parallel. (I have a very slow glusterfs installation that makes me think that some things can be done better.) For exa