Re: cp -d dir patch for review (or 'xargs'?)
Brian Somers [EMAIL PROTECTED] wrote: If I'm not mistaken, the size of the environment is already taken into account by the xargs utility (subtracted from ARG_MAX). So this isn't an issue at all. Unless xargs runs a command with lots of arguments and that command increases the environment size then tries to run another command with the same arguments - bang (E2BIG). True, but that's certainly not xarg's fault (and it cannot be fixed in the scope of xargs). xargs has no way to know if the command will enlarge its environment, and by what amount. In such a case it's probably up to the script writer to chose a sensible value for xargs -s n. Regards Oliver -- Oliver Fromme, secnetix GmbH Co KG, Oettingenstr. 2, 80538 München Any opinions expressed in this message may be personal to the author and may not necessarily reflect the opinions of secnetix in any way. All that we see or seem is just a dream within a dream (E. A. Poe) To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
Garance A Drosihn [EMAIL PROTECTED] writes:
At 1:19 PM -0700 4/21/01, Dima Dorfman wrote:
Does that mean everyone is blind and missed my arrogant
cross-post of the amazingly short patch to do this, or
are we just interested in discussing it and not testing
the implementation? ;-)
Well, I'm in the middle of a massive reorganization of
all my machines at home (to fit in a new G4 Cube!), so
I'm not paying as much attention to this as I would like.
I think it's really great that Dima has volunteered to do
the work...:-)
From what I have been following, you had one patch to add
the '-I' and '-i' options, and a different patch to add
the newly proposed '-Y' option. Right?
No, not quite. It's the same patch. The second one just has the 'Y'
option renamed to 'I' because I thought they did the same thing: they
don't.
The '-I' option is of interest because it is used in some
other OS's, and is even defined in some standards, such as
the SingleUnixSpec. From that:
-I replstr
Insert mode: utility will be executed for each line from
standard input, taking the entire line as a single argument,
inserting it in arguments for each occurrence of replstr.
A maximum of five arguments in arguments can each contain
one or more instances of replstr. Any blank characters at
the beginning of each line are ignored. Constructed arguments
cannot grow larger than 255 bytes. Option -x is forced on.
I think that if we're going to add a '-I', then we should
follow that description. Note that '-I', by definition,
forces '-n 1'. It will always pick up only one file from
the input to xargs per command that xarg will generate.
It allows things like:
Adding support for 'I' the way it's described above wouldn't be a
trivial as it was to add 'Y'. The latter adds about 15 lines, while
the former may involve some restructuring of the code.
Xargs compiles the arguments to utility as an array of pointers. It
also has assumptions that argv is only touched in the begining. It
wasn't a problem for -Y since it doesn't support the replstr being
embedded in an argument (e.g., for a replstr of {}, something{}
will not work as one arugment, only {} will), and it didn't have to
touch argv more than twice (I just added a small loop before all
invocations of run()). With -I, it'd probably be necessary to put a
large chuck of what is now main() inside a loop. It's not exactly
rocket science, but not something I can whip up in an hour, either.
I'll see what I can do probably later this week.
Dima Dorfman
[EMAIL PROTECTED]
To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
No rain here, it is ARG_MAX - 2048: -s size Set the maximum number of bytes for the command line length pro- vided to utility. The sum of the length of the utility name and the arguments passed to utility (including NULL terminators) will be less than or equal to this number. The current default value for size is ARG_MAX - 2048. 2K would be a pretty big env, root default std is about 367 bytes. Ok, I'll stop arguing. I guess I have to agree that you can script around the problem if you're careful. FWIW, the above is really ARG_MAX - 4k (the documentation is wrong - I recently updated xargs.1 in -current) and this seems to be *exactly* the right threshold. Take a look at the commit message with that xargs.1 commit... -- Rod Grimes - KD7CAX @ CN85sl - (RWG25) [EMAIL PROTECTED] -- Brian [EMAIL PROTECTED]brian@[uk.]FreeBSD.org http://www.Awfulhak.org brian@[uk.]OpenBSD.org Don't _EVER_ lose your sense of humour ! To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
Rodney W. Grimes [EMAIL PROTECTED] wrote:
Before anyone starts writing scripts, consider that {} will be
replaced by xargs with (roughly) ARG_MAX - 10 characters worth of the
stuff coming off the pipe. If your combined arguments plus
environment exceeds ARG_MAX execve(2) will give you E2BIG.
No rain here, it is ARG_MAX - 2048:
-s size
Set the maximum number of bytes for the command line length pro-
vided to utility. The sum of the length of the utility name and
the arguments passed to utility (including NULL terminators) will
be less than or equal to this number. The current default value
for size is ARG_MAX - 2048.
2K would be a pretty big env, root default std is about 367 bytes.
Yes, that is probably not a portable assumption to make, but it is
far better than using non-standard options to xargs.
If I'm not mistaken, the size of the environment is already
taken into account by the xargs utility (subtracted from
ARG_MAX). So this isn't an issue at all.
Regards
Oliver
--
Oliver Fromme, secnetix GmbH Co KG, Oettingenstr. 2, 80538 München
Any opinions expressed in this message may be personal to the author
and may not necessarily reflect the opinions of secnetix in any way.
All that we see or seem is just a dream within a dream (E. A. Poe)
To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
Rodney W. Grimes [EMAIL PROTECTED] wrote:
Before anyone starts writing scripts, consider that {} will be
replaced by xargs with (roughly) ARG_MAX - 10 characters worth of the
stuff coming off the pipe. If your combined arguments plus
environment exceeds ARG_MAX execve(2) will give you E2BIG.
No rain here, it is ARG_MAX - 2048:
-s size
Set the maximum number of bytes for the command line length pro-
vided to utility. The sum of the length of the utility name and
the arguments passed to utility (including NULL terminators) will
be less than or equal to this number. The current default value
for size is ARG_MAX - 2048.
2K would be a pretty big env, root default std is about 367 bytes.
Yes, that is probably not a portable assumption to make, but it is
far better than using non-standard options to xargs.
If I'm not mistaken, the size of the environment is already
taken into account by the xargs utility (subtracted from
ARG_MAX). So this isn't an issue at all.
Unless xargs runs a command with lots of arguments and that command
increases the environment size then tries to run another command with
the same arguments - bang (E2BIG).
Regards
Oliver
--
Oliver Fromme, secnetix GmbH Co KG, Oettingenstr. 2, 80538 München
Any opinions expressed in this message may be personal to the author
and may not necessarily reflect the opinions of secnetix in any way.
All that we see or seem is just a dream within a dream (E. A. Poe)
--
Brian [EMAIL PROTECTED]brian@[uk.]FreeBSD.org
http://www.Awfulhak.org brian@[uk.]OpenBSD.org
Don't _EVER_ lose your sense of humour !
To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message
Re: Re: cp -d dir patch for review (or 'xargs'?)
On Sat, 21 Apr 2001 20:04:31 +0100, Brian Somers wrote:
Sorry for butting in. Adding new non-portable functionality to solve the problem
which could be adequitely taken care of using existing and well known
techniquies is not appropriate, I completely agree with you on that.
And I'm still waiting to see those well known techniques.
Attached small script should solve this problem and doesn't require
introducing incompatible option in the standard tool.
For example:
find /usr/src -type f | xargs larg cp targetdir
For speed purposes it could be implemented in raw C.
-Maxim
#!/bin/sh
if [ ${#} -le 2 ]; then
echo "Usage: larg command lastarg arg1 [arg2 ...]"
exit 0
^
oops :-)
fi
COMMAND=${1}
LASTARG=${2}
shift 2
exec ${COMMAND} "${@}" "${LASTARG}"
Yes, I think this will work as long as your environment isn't
polluted by something like $ENV (any increase in the environment size
will effect xargs's calculation of how many arguments will fit on the
command line).
Of course I still prefer the xargs fix - as you said above, it'd be
nicer in C :-)
--
Brian [EMAIL PROTECTED]brian@[uk.]FreeBSD.org
http://www.Awfulhak.org brian@[uk.]OpenBSD.org
Don't _EVER_ lose your sense of humour !
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
On Sat, 21 Apr 2001 20:04:31 +0100, Brian Somers wrote:
Sorry for butting in. Adding new non-portable functionality to solve the problem
which could be adequitely taken care of using existing and well known
techniquies is not appropriate, I completely agree with you on that.
And I'm still waiting to see those well known techniques.
Attached small script should solve this problem and doesn't require
introducing incompatible option in the standard tool.
For example:
find /usr/src -type f | xargs larg cp targetdir
For speed purposes it could be implemented in raw C.
-Maxim
#!/bin/sh
if [ ${#} -le 2 ]; then
echo "Usage: larg command lastarg arg1 [arg2 ...]"
exit 0
fi
COMMAND=${1}
LASTARG=${2}
shift 2
exec ${COMMAND} "${@}" "${LASTARG}"
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
On Sun, 22 Apr 2001 13:16:31 +0100, Brian Somers wrote:
On Sat, 21 Apr 2001 20:04:31 +0100, Brian Somers wrote:
Sorry for butting in. Adding new non-portable functionality to solve the
problem
which could be adequitely taken care of using existing and well known
techniquies is not appropriate, I completely agree with you on that.
And I'm still waiting to see those well known techniques.
Attached small script should solve this problem and doesn't require
introducing incompatible option in the standard tool.
For example:
find /usr/src -type f | xargs larg cp targetdir
For speed purposes it could be implemented in raw C.
-Maxim
#!/bin/sh
if [ ${#} -le 2 ]; then
echo "Usage: larg command lastarg arg1 [arg2 ...]"
exit 0
^
oops :-)
fi
COMMAND=${1}
LASTARG=${2}
shift 2
exec ${COMMAND} "${@}" "${LASTARG}"
Yes, I think this will work as long as your environment isn't
polluted by something like $ENV (any increase in the environment size
will effect xargs's calculation of how many arguments will fit on the
command line).
I don't see why it matters. The only thing that matters here is number of
args accepted by the shell. Anyway this is a 2-minute prototype... ;)
As you can see, the problem in fact could be easily solved using "well
known techniques".
Of course I still prefer the xargs fix - as you said above, it'd be
nicer in C :-)
I still don't see why it couldn't be an separate tool (perhaps more
general that my prototype).
-Maxim
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
Brian Somers([EMAIL PROTECTED])@2001.04.20 11:29:15 +:
find something | xargs cp {} target_directory
or
find something | xargs -i '[]' cp '[]' target_directory
or
find something -exec cp {} target_directory \;
from find(1):
-exec utility [argument ...];
True if the program named utility returns a zero value as its ex-
it status. Optional arguments may be passed to the utility. The
expression must be terminated by a semicolon (``;''). If the
string ``{}'' appears anywhere in the utility name or the argu-
ments it is replaced by the pathname of the current file.
Utility will be executed from the directory from which find was
executed.
/k
--
who | grep -i blonde | date; cd ~; unzip; touch; finger; mount;\
gasp; yes; uptime; umount; sleep
KR433/KR11-RIPE -- http://www.webmonster.de -- ftp://ftp.webmonster.de
[Key] [KeyID---] [Created-] [Fingerprint-]
GnuPG 0x2964BF46 2001-03-15 42F9 9FFF 50D4 2F38 DBEE DF22 3340 4F4E 2964 BF46
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
rohrbach@WM:datasink[~]68% tar cf /dev/null src/
rohrbach@WM:datasink[~]69% find src|wc -l
2552
rohrbach@WM:datasink[~]70% du -sk src
32258 src
rohrbach@WM:datasink[~]71% mkdir src2
rohrbach@WM:datasink[~]72% time find src -exec cp {} src2 \;
find src -exec cp {} src2 ; 0.31s user 7.55s system 39% cpu 19.858 total).
rohrbach@WM:datasink[~]73% rm -rf src2
rohrbach@WM:datasink[~]74% mkdir src2
rohrbach@WM:datasink[~]75% time find src | cpio -dup src2
61025 blocks
find src 0.02s user 0.03s system 0% cpu 21.739 total
cpio -dup src2 0.26s user 4.84s system 20% cpu 24.862 total
68: warm up the filecache
69: there are 2552 files
70: they are 32.2MB total
71: ready the target dir
72: find -exec approach, all files to one dir
73: clear target area
74: ready it again
75: let find traverse the dir, cpio transfer the files
the cpio approach keeps the hierarchy which might not be what you want
but it looks more efficient, becouse it does not fork off cp for ecery
file.
/k
Brian Dean([EMAIL PROTECTED])@2001.04.21 16:24:36 +:
On Sat, Apr 21, 2001 at 05:34:31PM +0200, Sheldon Hearn wrote:
So we have two problems:
1) Calling cp(1) repetitively is inefficient.
2) The argument list is too big for cp(1).
Extending cp(1) will not solve (2). Extending xargs(1) will solve both.
So why is an extension to cp(1) being proposed?
But extending cp does solve the problem. The proposal was to make
% cp -d target src1 src2 ... srcN
Be equivalent to;
% cp src1 src2 ... srcN target
This makes cp work with xargs;
% cat ReallyBigListOfFiles | xargs cp -d target
-Brian
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message
--
Captain Hook died of jock itch.
KR433/KR11-RIPE -- http://www.webmonster.de -- ftp://ftp.webmonster.de
[Key] [KeyID---] [Created-] [Fingerprint-]
GnuPG 0x2964BF46 2001-03-15 42F9 9FFF 50D4 2F38 DBEE DF22 3340 4F4E 2964 BF46
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
At 1:19 PM -0700 4/21/01, Dima Dorfman wrote:
Does that mean everyone is blind and missed my arrogant
cross-post of the amazingly short patch to do this, or
are we just interested in discussing it and not testing
the implementation? ;-)
Well, I'm in the middle of a massive reorganization of
all my machines at home (to fit in a new G4 Cube!), so
I'm not paying as much attention to this as I would like.
I think it's really great that Dima has volunteered to do
the work...:-)
From what I have been following, you had one patch to add
the '-I' and '-i' options, and a different patch to add
the newly proposed '-Y' option. Right?
The '-I' option is of interest because it is used in some
other OS's, and is even defined in some standards, such as
the SingleUnixSpec. From that:
-I replstr
Insert mode: utility will be executed for each line from
standard input, taking the entire line as a single argument,
inserting it in arguments for each occurrence of replstr.
A maximum of five arguments in arguments can each contain
one or more instances of replstr. Any blank characters at
the beginning of each line are ignored. Constructed arguments
cannot grow larger than 255 bytes. Option -x is forced on.
I think that if we're going to add a '-I', then we should
follow that description. Note that '-I', by definition,
forces '-n 1'. It will always pick up only one file from
the input to xargs per command that xarg will generate.
It allows things like:
1. The following will move all files from directory
$1 to directory $2:
ls $1 | xargs -I {} mv $1/{} $2/{}
[that example is a subset of an example from the standard]
One might argue whether there are alternate ways to get the
same effect. However, in this case we're just trying to add
an option which is ALREADY described as a standard option to
this command. I would think the only debate is whether Dima's
patch results in behavior which does indeed conform to the
written standard(s).
The newly proposed '-Y' option is similar, except that it
does NOT force '-n 1', and it should replace only ONE
occurrence of the replstr. I'd be happy with any other
letter instead of '-Y'. I only picked that letter because
it didn't seem to be used by any version of 'xargs' that I
personally work with. We have a lot more room for debate
on this one, because we're making it up.
My thoughts on '-Y' are something like:
-Y replstr
Alternate insert mode: The number of arguments included
on the 'utility' is exactly the same as if -Y was not
specified. The only difference is that those arguments
will replace replstr in the command(s) generated by
xargs, instead of appending the arguments at the end
of those commands.
In the string to execute, there can be only one copy
of the replstr. It is an error if there is more
than one copy of the replstr in the command to
execute.
Examples:
ls -1 *2000* | xargs -Y [] cp -p [] /year/y2k
will take all files which have '2000' in their
name, and copy them to the directory /year/y2k
ls $1 | xargs -Y [] mv $1/[] $2/[]
will generate an error message, because the
replstr can only be replaced once, and we
don't want to guess which copy of [] should
be replaced.
I also like the idea of allowing:
'-i' means exactly the same as '-I {}'
and'-y' means exactly the same as '-Y []'
Note that I would not allow an optional argument to be included
on '-i', even though some standards do allow that for backward
compatibility. Since I would rather not allow that, perhaps it
would be better to not even bother with '-i' and '-y'.
The -I, -i, -Y, and -y options are mutually exclusive. If more
than one is specified, then the last one specified will take
effect. Any attempt to do this "right" should probably be
careful that '-I' only forces '-n 1' if it is not overridden
by a later '-Y'.
I must admit I haven't really looked thru Dima's patch to see
if it meets all these criteria. I did look at OpenBSD's and
NetBSD's xargs, to see if they had any other options that we
might want to add (they didn't seem to), or if they would have
any trouble adding the same option(s). My intent here is to
at least describe what I was thinking of in precise-enough
terms that people can decide if the implementation is right.
And in case it isn't obvious, I'll say once again that I wouldn't
complain if someone comes up with a better letter than '-Y' to
use for this new option... :-) If '-I' stands for "insert mode",
then what should this mode be called, and what letter would be a
good match to it? Maybe '-J', for "Just-insert-once mode"? :-)
That would also get it listed right next to -I...
--
Garance Alistair Drosehn= [EMAIL PROTECTED]
Senior Systems Programmer or [EMAIL PROTECTED]
Rensselaer Polytechnic Instituteor [EMAIL PROTECTED]
To Unsubscribe: send mail
Re: Re: cp -d dir patch for review (or 'xargs'?)
On Sun, 22 Apr 2001 13:16:31 +0100, Brian Somers wrote:
On Sat, 21 Apr 2001 20:04:31 +0100, Brian Somers wrote:
Sorry for butting in. Adding new non-portable functionality to solve the
problem
which could be adequitely taken care of using existing and well known
techniquies is not appropriate, I completely agree with you on that.
And I'm still waiting to see those well known techniques.
Attached small script should solve this problem and doesn't require
introducing incompatible option in the standard tool.
For example:
find /usr/src -type f | xargs larg cp targetdir
For speed purposes it could be implemented in raw C.
-Maxim
#!/bin/sh
if [ ${#} -le 2 ]; then
echo "Usage: larg command lastarg arg1 [arg2 ...]"
exit 0
^
oops :-)
fi
COMMAND=${1}
LASTARG=${2}
shift 2
exec ${COMMAND} "${@}" "${LASTARG}"
Yes, I think this will work as long as your environment isn't
polluted by something like $ENV (any increase in the environment size
will effect xargs's calculation of how many arguments will fit on the
command line).
I don't see why it matters. The only thing that matters here is number of
args accepted by the shell. Anyway this is a 2-minute prototype... ;)
As you can see, the problem in fact could be easily solved using "well
known techniques".
Of course I still prefer the xargs fix - as you said above, it'd be
nicer in C :-)
I still don't see why it couldn't be an separate tool (perhaps more
general that my prototype).
I don't see that such a tool would be used without xargs, whereas
users of xargs often want/expect this sort of facility - or so I
believe.
-Maxim
--
Brian [EMAIL PROTECTED]brian@[uk.]FreeBSD.org
http://www.Awfulhak.org brian@[uk.]OpenBSD.org
Don't _EVER_ lose your sense of humour !
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
I don't see a problem with adding an option to cp to treat the first
argument as the target instead of the last argument. It's a simple
solution, the code change is simple, and it produces the exact desired
result. What's the problem?
It's yet another non-portable option.
I hate to appear rude, but has anybody in this discussion actually used
xargs for what it's meant to be used ?
How do you do this in a script:
cd /topdir; find . -type f | xargs -i {} cp {} /otherdir/.
My example should work for that, just replace the initial echo
with ``cd /topdir; find . -type f'', and replace your xargs
with my script, using -n 100 or -s of some sane value.
Before anyone starts writing scripts, consider that {} will be
replaced by xargs with (roughly) ARG_MAX - 10 characters worth of the
stuff coming off the pipe. If your combined arguments plus
environment exceeds ARG_MAX execve(2) will give you E2BIG.
No rain here, it is ARG_MAX - 2048:
-s size
Set the maximum number of bytes for the command line length pro-
vided to utility. The sum of the length of the utility name and
the arguments passed to utility (including NULL terminators) will
be less than or equal to this number. The current default value
for size is ARG_MAX - 2048.
2K would be a pretty big env, root default std is about 367 bytes.
Yes, that is probably not a portable assumption to make, but it is
far better than using non-standard options to xargs.
--
Rod Grimes - KD7CAX @ CN85sl - (RWG25) [EMAIL PROTECTED]
To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
On Fri, Apr 20, 2001 at 03:14:10PM -0700, Mike Smith wrote: # # Folks, # # although there was much rejoicing, I think there's no need for a # new option to cp. Just use the toolbox, it's not too hard: # # (cat bigfilelist; echo destdir) | xargs cp # # Or even # # echo destdir bigfilelist # xargs cp bigfilelist # # should do the trick. # # No, it won't. Consider a list of files a, b, c, d. You create input to # xargs 'a b c d destdir', which it then splits into 'a b c' and 'd # destdir'. The first time cp is run, it will probably fail; the second # time only 'd' ends up where you expect it. You mean if bigfilelist list exceeds the -n limit of xargs (default 5000)? Yes, you'll be surprised then. It was a bit of POLA violation for me when I found xargs would by default use 5000 arg chunks and not all in one go. I'd rather get rid of kern.argmax and the limitations of the exec familiy. Yes, I'm dreaming :-) Regards, Jens -- Jens Schweikhardt http://www.schweikhardt.net/ SIGSIG -- signature too long (core dumped) To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
On Fri, Apr 20, 2001 at 07:26:18PM -0700, Rodney W. Grimes wrote:
(cat bigfilelist; echo destdir) | xargs cp
I like this version of the patch!! It's much much cleaner than
hacking up cp or xargs, it even follows the unix principle of
using simple tools and glueing them togeather to do bigger
jobs, is unix implementation independent, and is very clear
in what it does.
It's clean, simple, and unfortunately, totally bogus.
Try:
echo 1 2 3 4 5 6 7 8 9 | xargs -n 4 echo
Now consider what would happen with the above suggested construct with
a very long file list.
bleck... try this for your sample:
$ (echo 1 2 3 4 5 6 7 8 9 | xargs -n 4) | while read x; do
echo -n $x; echo " dst"
done
1 2 3 4 dst
5 6 7 8 dst
9 dst
$
I don't see a problem with adding an option to cp to treat the first
argument as the target instead of the last argument. It's a simple
solution, the code change is simple, and it produces the exact desired
result. What's the problem?
It's yet another non-portable option.
I hate to appear rude, but has anybody in this discussion actually used
xargs for what it's meant to be used ?
How do you do this in a script:
cd /topdir; find . -type f | xargs -i {} cp {} /otherdir/.
Before anyone starts writing scripts, consider that {} will be
replaced by xargs with (roughly) ARG_MAX - 10 characters worth of the
stuff coming off the pipe. If your combined arguments plus
environment exceeds ARG_MAX execve(2) will give you E2BIG.
--
Rod Grimes - KD7CAX @ CN85sl - (RWG25) [EMAIL PROTECTED]
--
Brian [EMAIL PROTECTED]brian@[uk.]FreeBSD.org
http://www.Awfulhak.org brian@[uk.]OpenBSD.org
Don't _EVER_ lose your sense of humour !
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
On Sat, 21 Apr 2001 14:06:04 +0100, Brian Somers wrote:
How do you do this in a script:
cd /topdir; find . -type f | xargs -i {} cp {} /otherdir/.
for i in `find /path/to/source -type f`; do
cp $i /path/to/dest/
done
What's all the fuss about?
Ciao,
Sheldon.
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
Jens Schweikhardt [EMAIL PROTECTED] wrote: You mean if bigfilelist list exceeds the -n limit of xargs (default 5000)? Yes, you'll be surprised then. It was a bit of POLA violation for me when I found xargs would by default use 5000 arg chunks and not all in one go. I'd rather get rid of kern.argmax and the limitations of the exec familiy. Yes, I'm dreaming :-) Certainly, it would cause a whole lot of other problems, the smallest of which would be that people would be starting to write non-portable scripts that rely on the feature that there is no ARG_MAX limit. By the way, the -i and -I options of xargs are specified in the SUSv2 standard, and I think it would certainly be a good thing to comply with that. At least it would be a whole lot better than hacking a non-standard option into cp which would solve the problem for one particular case only, while fixing xargs would solve the whole class of problems. Putting that option into cp seems rather GNUish to me, but not very UNIXish. :-) Just my 2 Euro cents. Regards Oliver -- Oliver Fromme, secnetix GmbH Co KG, Oettingenstr. 2, 80538 Mnchen Any opinions expressed in this message may be personal to the author and may not necessarily reflect the opinions of secnetix in any way. "All that we see or seem is just a dream within a dream" (E. A. Poe) To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
Sheldon Hearn [EMAIL PROTECTED] wrote:
On Sat, 21 Apr 2001 14:06:04 +0100, Brian Somers wrote:
How do you do this in a script:
cd /topdir; find . -type f | xargs -i {} cp {} /otherdir/.
for i in `find /path/to/source -type f`; do
cp $i /path/to/dest/
done
That can overflow your shell's command line limit (at the
"for" command). True, our /bin/sh doesn't has such a
limit, AFAIK, but there _are_ shells that do). Apart from
that it is extremely inefficient, because it runs a "cp"
for every single file. These are exactly the problems that
xargs is supposed to solve.
Actually I don't see any problem with Brian's approach
(provided that the -i option exists, of course).
xargs _does_ take the size of the environment into account,
as well as the size of all arguments, and it still leaves
much room (it only uses up to ARG_MAX - 2048 by default).
Oh by the way, in this particular example it is probably a
good idea to use cpio. This will even work with our xargs
(which doesn't support -i yet):
cd /topdir; find . -type f | cpio -dup /otherdir
should do exactly that job.
Regards
Oliver
--
Oliver Fromme, secnetix GmbH Co KG, Oettingenstr. 2, 80538 Mnchen
Any opinions expressed in this message may be personal to the author
and may not necessarily reflect the opinions of secnetix in any way.
"All that we see or seem is just a dream within a dream" (E. A. Poe)
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
On 21-Apr-2001 Sheldon Hearn wrote:
On Sat, 21 Apr 2001 14:06:04 +0100, Brian Somers wrote:
How do you do this in a script:
cd /topdir; find . -type f | xargs -i {} cp {} /otherdir/.
for i in `find /path/to/source -type f`; do
cp $i /path/to/dest/
done
What's all the fuss about?
It looks like above construct will fail horribly if any of the files in /topdir
have names with spaces in them. Think MP3 collections :)
Ciao,
Sheldon.
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message
--
E-Mail: Alexander Kabaev [EMAIL PROTECTED]
Date: 21-Apr-2001
Time: 10:49:59
--
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
Your comments have nothing to do with the issue at hand. Just wrap the first argument to cp in double-quotes, i.e. cp "$i" The point is, why bastardize tools to cope with areas beyond their focus and well within the focus of other tools? Ciao, Sheldon. Sorry for butting in. Adding new non-portable functionality to solve the problem which could be adequitely taken care of using existing and well known techniquies is not appropriate, I completely agree with you on that. -- E-Mail: Alexander Kabaev [EMAIL PROTECTED] Date: 21-Apr-2001 Time: 11:08:13 -- To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
Sheldon Hearn [EMAIL PROTECTED] wrote: On Sat, 21 Apr 2001 16:51:24 +0200, Oliver Fromme wrote: That can overflow your shell's command line limit (at the "for" command). True, our /bin/sh doesn't has such a limit, AFAIK, but there _are_ shells that do). That's actually my point. What's being proposed is a non-standard extension to work around a problem on a system that already doesn't have the problem. Maybe I didn't make myself clear enough. We _do_ have a problem. Not all users use /bin/sh. Scripts needn't be written in /bin/sh, and xargs can be used interactively, too (I use it a lot). Just because _our_ xargs works fine with _our_ /bin/sh doesn't mean there is no problem. And then there's the gross efficiency problem. Try these alternatives and compare how long they take: for i in `find /usr/ports -type f`; do cat $i /dev/null done find /usr/ports -type f | xargs cat /dev/null The latter is a hell of a lot faster. (The example uses "cat" just because it works with xargs.) By the way, the first (inefficient) approach could be rewritten like this: find /usr/ports -type f | while read i; do cat $i /dev/null done This avoid the potential line limit problem, but of course it's just as inefficient. Regards Oliver -- Oliver Fromme, secnetix GmbH Co KG, Oettingenstr. 2, 80538 Mnchen Any opinions expressed in this message may be personal to the author and may not necessarily reflect the opinions of secnetix in any way. "All that we see or seem is just a dream within a dream" (E. A. Poe) To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
On Sat, 21 Apr 2001 17:27:04 +0200, Oliver Fromme wrote: Not all users use /bin/sh. Scripts needn't be written in /bin/sh, and xargs can be used interactively, too (I use it a lot). Just because _our_ xargs works fine with _our_ /bin/sh doesn't mean there is no problem. So we have two problems: 1) Calling cp(1) repetitively is inefficient. 2) The argument list is too big for cp(1). Extending cp(1) will not solve (2). Extending xargs(1) will solve both. So why is an extension to cp(1) being proposed? Ciao, Sheldon. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
From: Oliver Fromme [EMAIL PROTECTED] Subject: Re: cp -d dir patch for review (or 'xargs'?) Date: Sat, 21 Apr 2001 17:27:04 +0200 (CEST) Not all users use /bin/sh. Scripts needn't be written in /bin/sh ... Actually, just to jump in and correct this, scripts *should* be written in /bin/sh. That's a defacto Unix standard when it comes to writing shell scripts, just for uniformities sake, and even if you use tcsh or zsh as your personal shell one is always encouraged to write in straight POSIX-conformant /bin/sh for portable scripts. If one also needs to walk entirely outside the painted lines there then that's a good indication that maybe it should be written in perl. :) - Jordan To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
On Sat, 21 Apr 2001 14:06:04 +0100, Brian Somers wrote:
How do you do this in a script:
cd /topdir; find . -type f | xargs -i {} cp {} /otherdir/.
for i in `find /path/to/source -type f`; do
cp $i /path/to/dest/
done
What's all the fuss about?
Have you tried that for values of /path/to/source with lots of files ?
Something like
find blah | while read i; do cp $i /dest/.; done
is better, but it runs cp too many times.
Ciao,
Sheldon.
--
Brian [EMAIL PROTECTED]brian@[uk.]FreeBSD.org
http://www.Awfulhak.org brian@[uk.]OpenBSD.org
Don't _EVER_ lose your sense of humour !
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
Jordan Hubbard [EMAIL PROTECTED] wrote: From: Oliver Fromme [EMAIL PROTECTED] Not all users use /bin/sh. Scripts needn't be written in /bin/sh ... Actually, just to jump in and correct this, scripts *should* be written in /bin/sh. It depends. I often happen to write zsh scripts, but only if I'm sure that they don't really have to be portable, and that I am the only one who will ever use them. When I was young, I also wrote a few tcsh scripts (*ouch*), but I discontinued doing that long ago. :-) I agree with you 100% that portable scripts should use /bin/sh and nothing else. And to come back on topic: Portable scripts also should _not_ assume that there are no limits on the length of shell commands. On the other hand, portable scripts can legitimately assume that xargs supports -i and -I, which ours doesn't. Regards Oliver PS: FWIW, I also write a lot of awk scripts, which is my favourite scripting language, but this is really getting off-topic ... -- Oliver Fromme, secnetix GmbH Co KG, Oettingenstr. 2, 80538 Mnchen Any opinions expressed in this message may be personal to the author and may not necessarily reflect the opinions of secnetix in any way. "All that we see or seem is just a dream within a dream" (E. A. Poe) To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
And to come back on topic: Portable scripts also should _not_ assume that there are no limits on the length of shell commands. On the other hand, portable scripts can legitimately assume that xargs supports -i and -I, which ours doesn't. Agreed on both counts. I guess we should fix that. PS: FWIW, I also write a lot of awk scripts, which is my favourite scripting language, but this is really getting off-topic ... So do I, and you're right. :) - Jordan To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
On Sat, 21 Apr 2001, Brian Somers wrote:
On Sat, 21 Apr 2001 14:06:04 +0100, Brian Somers wrote:
How do you do this in a script:
cd /topdir; find . -type f | xargs -i {} cp {} /otherdir/.
for i in `find /path/to/source -type f`; do
cp $i /path/to/dest/
done
What's all the fuss about?
Have you tried that for values of /path/to/source with lots of files ?
Something like
find blah | while read i; do cp $i /dest/.; done
is better, but it runs cp too many times.
cp is a bad example, since it usually does physical i/o which is much
slower than execve() of a program that is probably cached, especially
when the program is small and not dynamically linked.
Bruce
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
Jordan Hubbard [EMAIL PROTECTED] writes:
And to come back on topic: Portable scripts also should
_not_ assume that there are no limits on the length of
shell commands. On the other hand, portable scripts can
legitimately assume that xargs supports -i and -I, which
ours doesn't.
Agreed on both counts. I guess we should fix that.
I don't have a copy of SuSv2 or anything else that defines -I and -i,
but from what I can gather, -i is the same as "-I {}" and -I allows
things like this:
dima@spike% ./xargs -I [] echo CMD LINE [] ARGS test
CMD LINE this is the contents of the test file ARGS
dima@spike% ./xargs -I [] echo CMD [] LINE ARGS test
CMD this is the contents of the test file LINE ARGS
dima@spike% ./xargs -I [] echo [] CMD LINE ARGS test
this is the contents of the test file CMD LINE ARGS
Does that mean everyone is blind and missed my arrogant cross-post of
the amazingly short patch to do this, or are we just interested in
discussing it and not testing the implementation? ;-)
FWIW, I'm not sure the patch is entirely correct; xargs' processing of
this stuff looks like black magic. It works, but I'm not sure if I
failed to cater to some other weird assumptions it makes. This is why
it'd help if someone would at least look at it.
Thanks,
Dima Dorfman
[EMAIL PROTECTED]
Index: xargs.c
===
RCS file: /st/src/FreeBSD/src/usr.bin/xargs/xargs.c,v
retrieving revision 1.9
diff -u -r1.9 xargs.c
--- xargs.c 1999/08/28 01:07:50 1.9
+++ xargs.c 2001/04/21 20:15:27
@@ -73,6 +73,8 @@
int cnt, indouble, insingle, nargs, nflag, nline, xflag, wasquoted;
char **av, *argp, **ep = env;
long arg_max;
+ int apargs = 0;
+ char **avv, *replstr = NULL;
/*
* POSIX.2 limits the exec line length to ARG_MAX - 2K. Running that
@@ -96,8 +98,14 @@
nline -= strlen(*ep++) + 1 + sizeof(*ep);
}
nflag = xflag = wasquoted = 0;
- while ((ch = getopt(argc, argv, "0n:s:tx")) != -1)
+ while ((ch = getopt(argc, argv, "0I:in:s:tx")) != -1)
switch(ch) {
+ case 'I':
+ replstr = optarg;
+ break;
+ case 'i':
+ replstr = "{}";
+ break;
case 'n':
nflag = 1;
if ((nargs = atoi(optarg)) = 0)
@@ -144,6 +152,13 @@
else {
cnt = 0;
do {
+ if (replstr strcmp(*argv, replstr) == 0) {
+ apargs = 1;
+ *argv++;
+ for (avv = argv; *avv; *avv++)
+ cnt += strlen(*avv) + 1;
+ break;
+ }
cnt += strlen(*bxp++ = *argv) + 1;
} while (*++argv);
}
@@ -211,6 +226,8 @@
if (xp == exp || p ebp || ch == EOF) {
if (xflag xp != exp p ebp)
errx(1, "insufficient space for arguments");
+ for (avv = argv; apargs *avv; *avv++)
+ strlen(*xp++ = *avv) + 1;
*xp = NULL;
run(av);
if (ch == EOF)
@@ -253,6 +270,8 @@
if (xflag)
errx(1, "insufficient space for arguments");
+ for (avv = argv; apargs *avv; *avv++)
+ strlen(*xp++ = *avv) + 1;
*xp = NULL;
run(av);
xp = bxp;
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
On Sat, Apr 21, 2001 at 05:34:31PM +0200, Sheldon Hearn wrote: So we have two problems: 1) Calling cp(1) repetitively is inefficient. 2) The argument list is too big for cp(1). Extending cp(1) will not solve (2). Extending xargs(1) will solve both. So why is an extension to cp(1) being proposed? But extending cp does solve the problem. The proposal was to make % cp -d target src1 src2 ... srcN Be equivalent to; % cp src1 src2 ... srcN target This makes cp work with xargs; % cat ReallyBigListOfFiles | xargs cp -d target -Brian To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
Brian Dean [EMAIL PROTECTED] wrote: But extending cp does solve the problem. Only for cp. It wouldn't solve the problem for mv, ln and a bunch of other tools. Fixing it at _one_ place in xargs would solve all of that without touching a dozen tools. [...] This makes cp work with xargs; % cat ReallyBigListOfFiles | xargs cp -d target That's actually a bad example anyway, because you would use cpio in that case, not xargs|cp. It's also a bad example for using cat, but that's a different story. :-) cpio -dup target ReallyBigListOfFiles Regards Oliver -- Oliver Fromme, secnetix GmbH Co KG, Oettingenstr. 2, 80538 Mnchen Any opinions expressed in this message may be personal to the author and may not necessarily reflect the opinions of secnetix in any way. "All that we see or seem is just a dream within a dream" (E. A. Poe) To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
So we have two problems: 1) Calling cp(1) repetitively is inefficient. 2) The argument list is too big for cp(1). Extending cp(1) will not solve (2). Extending xargs(1) will solve both. So why is an extension to cp(1) being proposed? I wasn't proposing that cp should be changed - I don't think it should. I'm just guilty of using a stale subject line :-/ Ciao, Sheldon. -- Brian [EMAIL PROTECTED]brian@[uk.]FreeBSD.org http://www.Awfulhak.org brian@[uk.]OpenBSD.org Don't _EVER_ lose your sense of humour ! To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
Dima Dorfman [EMAIL PROTECTED] wrote:
I don't have a copy of SuSv2 or anything else that defines -I and -i,
http://www.secnetix.de/~olli/susv2/xcu/xargs.html
but from what I can gather, -i is the same as "-I {}" and -I allows
things like this:
Not exactly. The difference is that the option-argument to
-i is optional and -- if present -- has to follow without
whitespace after the -i. This is a violation of the common
utility syntax guidelines, but has been adopted by SUSv2
because it was widely implemented.
So ``-i'' is the same as ``-I {}'', and ``-i[]'' (no space!)
is the same as ``-I []''.
Unfortunately, when you use -i or -I, then each line from
stdin is used as a signle argument, and the utility is
invoked once for every line, unless I misunderstand what
SUSv2 is saying. :-(
$ cat test
foo bar
baz bla
$ xargs -i echo XXX '{}' YYY test
XXX foo bar YYY
XXX baz bla YYY
$
Does that mean everyone is blind and missed my arrogant cross-post of
the amazingly short patch to do this, or are we just interested in
discussing it and not testing the implementation? ;-)
I must have missed it, and I think it's at least a good
start. :-)
The patch looks good. At leat it would solve the problem
which this thread is about, although I think it doesn't
comply with SUSv2.
Regards
Oliver
--
Oliver Fromme, secnetix GmbH Co KG, Oettingenstr. 2, 80538 Mnchen
Any opinions expressed in this message may be personal to the author
and may not necessarily reflect the opinions of secnetix in any way.
"All that we see or seem is just a dream within a dream" (E. A. Poe)
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
I looked at your patches and immediately thought ``these patches
can't be right'' as I was expecting it to deal with things such as
xargs -I [] echo args are [], duplicated are []
I'm also dubious about the patches working for large volumes on
standard input. At this point I scrapped the email I was composing
'cos I didn't have time to look into it further :-/
I think it's important to test any patches with a large number of
large path names as input - so that ARG_MAX is reached before the
5000 argument limit and we can see that we don't end up getting E2BIG
because of an accidental overflow/miscalculation.
Sorry I don't have more time to spend on it :-/
I don't have a copy of SuSv2 or anything else that defines -I and -i,
but from what I can gather, -i is the same as "-I {}" and -I allows
things like this:
dima@spike% ./xargs -I [] echo CMD LINE [] ARGS test
CMD LINE this is the contents of the test file ARGS
dima@spike% ./xargs -I [] echo CMD [] LINE ARGS test
CMD this is the contents of the test file LINE ARGS
dima@spike% ./xargs -I [] echo [] CMD LINE ARGS test
this is the contents of the test file CMD LINE ARGS
Does that mean everyone is blind and missed my arrogant cross-post of
the amazingly short patch to do this, or are we just interested in
discussing it and not testing the implementation? ;-)
FWIW, I'm not sure the patch is entirely correct; xargs' processing of
this stuff looks like black magic. It works, but I'm not sure if I
failed to cater to some other weird assumptions it makes. This is why
it'd help if someone would at least look at it.
Thanks,
Dima Dorfman
[EMAIL PROTECTED]
--
Brian [EMAIL PROTECTED]brian@[uk.]FreeBSD.org
http://www.Awfulhak.org brian@[uk.]OpenBSD.org
Don't _EVER_ lose your sense of humour !
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
Putting that option into cp seems rather GNUish to me, but not very UNIXish. :-) Yes. I think most people agree that changing cp is not good. Just my 2 Euro cents. Regards Oliver -- Oliver Fromme, secnetix GmbH Co KG, Oettingenstr. 2, 80538 Mnchen Any opinions expressed in this message may be personal to the author and may not necessarily reflect the opinions of secnetix in any way. "All that we see or seem is just a dream within a dream" (E. A. Poe) -- Brian [EMAIL PROTECTED]brian@[uk.]FreeBSD.org http://www.Awfulhak.org brian@[uk.]OpenBSD.org Don't _EVER_ lose your sense of humour ! To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
Dima Dorfman [EMAIL PROTECTED] wrote:
I don't have a copy of SuSv2 or anything else that defines -I and -i,
http://www.secnetix.de/~olli/susv2/xcu/xargs.html
but from what I can gather, -i is the same as "-I {}" and -I allows
things like this:
Not exactly. The difference is that the option-argument to
-i is optional and -- if present -- has to follow without
whitespace after the -i. This is a violation of the common
utility syntax guidelines, but has been adopted by SUSv2
because it was widely implemented.
So ``-i'' is the same as ``-I {}'', and ``-i[]'' (no space!)
is the same as ``-I []''.
I don't think we should adopt these semantics. I'm coming around to
Dima's -Y option - which must have an argument.
Unfortunately, when you use -i or -I, then each line from
stdin is used as a signle argument, and the utility is
invoked once for every line, unless I misunderstand what
SUSv2 is saying. :-(
I guess that settles it then. This is a dumb restriction and doesn't
seem to fit in very well with how xargs works. Again, Dima's idea is
IMHO superior.
But as I said in my other follow-up, I'm not convinced that the patch
deals with ARG_MAX overflows properly (I may be wrong though).
--
Brian [EMAIL PROTECTED]brian@[uk.]FreeBSD.org
http://www.Awfulhak.org brian@[uk.]OpenBSD.org
Don't _EVER_ lose your sense of humour !
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
Brian Somers [EMAIL PROTECTED] writes: I looked at your patches and immediately thought ``these patches can't be right'' as I was expecting it to deal with things such as xargs -I [] echo args are [], duplicated are [] It deals with it. It conveniently ignores the second '[]' :-). Seriosly though, what do you expect it to do in this case? It can either read some more from stdin, or use the same input it used for the first case of '[]'. I also can't think of a case when either one of these would be useful. I guess the only reason we would want this is if SUSv2 defines it, but even that may not matter since we probably won't support the silly '-i[nospace]' semantic (other than being silly, I can't think of how to implement it without writing a custom getopt() facility). I'm also dubious about the patches working for large volumes on standard input. At this point I scrapped the email I was composing 'cos I didn't have time to look into it further :-/ I think it's important to test any patches with a large number of large path names as input - so that ARG_MAX is reached before the 5000 argument limit and we can see that we don't end up getting E2BIG because of an accidental overflow/miscalculation. Any advice on testing this (you did write rev. 1.9 of xargs.1, after all)? I created a file with 4500 words like this: /this/is/a/very/long/path/name/because/I/am/testing/some/posix/limit/10 which ended up being 330 kB. It ran the `utility' multiple times like I expected it to. That said, I don't know what kind of failure mode to expect. I.e., if the patch is wrong, should it have failed with something like, "xargs: exec: argument list too long", or would it just produce incorrect output (which I didn't really check for)? Thanks, Dima Dorfman [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
If you just want an xargs that supports --replstr/-i simply install: ftp://ftp.gnu.org/gnu/findutils or even more easily: /usr/ports/misc/findutils two comments: I don't want to enter a protracted discussion over the benefits/drawbacks of the current xargs vs an updated xargs, nor try to do a write-from-scratch. The cp -d option has runtime execution of O(1). Xargs addes O(n) due to it's manipulation of the arguement vector in -i mode. The process I'm dealing with already takes many hours to run. I want to reduce time, not increase it. Comments welcome. -john - Garance A Drosihn's Original Message - At 10:08 PM -0700 4/19/01, Dima Dorfman wrote: Garance A Drosihn [EMAIL PROTECTED] writes: Or maybe something to indicate where the list of arguments should go in a command. Hrm. Let's say '-Y replstr' or '-y[replstr]' (no blank after -y). If no [replstr] is given on -y, it defaults to the two characters '[]'. Then one might do: cat big_file_list | xargs -y cp [] target_directory This is a great idea! I'm willing to implement it if nobody else wants to. Woo-hoo! Someone to do the work! Yes! you're trying to address. On the other hand, the man page for 'xargs' on FreeBSD says: The xargs utility is expected to be IEEE Std 1003.2 (``POSIX.2'') compliant. so I don't know how we go about adding options to it. On the other hand, that same issue is faced by adding options to 'cp', as there is a similar claim made in cp's man page. I don't think it's a problem. We're adding new options here, not changing--sometimes known as breaking--what already exists. I'm pretty sure that the standards don't say anything to the effect of, "You must support this and nothing else." That'd be rather silly. Actually, it's not as silly as it sounds. If you're writing scripts, and you use those extra parameters, then you'll get into trouble when running the script on some other POSIX-based OS which does not have these new options. I really do like the idea of both the -I/-i options from solaris, and the -Y/-y options that I just dreamed up, but I'm not sure what the right procedure is to introduce them (and eventually have them standard everywhere... :-). Maybe we could initially have a 'yargs' command, which is just like 'xargs' except that it adds those four options. Maybe I'm just overly pedantic. Hmm. Checking my copy of "Single Unix Specification, v2", the -I/-i parameters are defined in THAT standard, but it doesn't have anything matching my -Y/-y suggestion. Hmm, I wonder if I should be copying this "meta-question" to the mailing list for standardizing things... -- Garance Alistair Drosehn= [EMAIL PROTECTED] Senior Systems Programmer or [EMAIL PROTECTED] Rensselaer Polytechnic Instituteor [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
Garance A Drosihn [EMAIL PROTECTED] writes:
Or maybe something to indicate where the list of arguments
should go in a command. Hrm. Let's say '-Y replstr' or
'-y[replstr]' (no blank after -y). If no [replstr] is
given on -y, it defaults to the two characters '[]'.
Then one might do:
cat big_file_list | xargs -y cp [] target_directory
This is a great idea! I'm willing to implement it if nobody else
wants to.
If you add this (which I think is a good idea), please make it option
free with {} as the default arglist and -i to override that string in
line with sysv's xargs:
find something | xargs cp {} target_directory
or
find something | xargs -i '[]' cp '[]' target_directory
Although it's possible to break something that uses a literal {} as
an argument, I think this is better than introducing semantics
that'll confuse people.
Dima Dorfman
[EMAIL PROTECTED]
Cheers.
--
Brian [EMAIL PROTECTED]brian@[uk.]FreeBSD.org
http://www.Awfulhak.org brian@[uk.]OpenBSD.org
Don't _EVER_ lose your sense of humour !
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
Dima Dorfman wrote: Garance A Drosihn [EMAIL PROTECTED] writes: Or maybe something to indicate where the list of arguments should go in a command. Hrm. Let's say '-Y replstr' or '-y[replstr]' (no blank after -y). If no [replstr] is given on -y, it defaults to the two characters '[]'. Then one might do: cat big_file_list | xargs -y cp [] target_directory This is a great idea! I'm willing to implement it if nobody else wants to. you're trying to address. On the other hand, the man page for 'xargs' on FreeBSD says: The xargs utility is expected to be IEEE Std 1003.2 (``POSIX.2'') compliant. so I don't know how we go about adding options to it. On the other hand, that same issue is faced by adding options to 'cp', as there is a similar claim made in cp's man page. I don't think it's a problem. We're adding new options here, not changing--sometimes known as breaking--what already exists. I'm pretty sure that the standards don't say anything to the effect of, "You must support this and nothing else." That'd be rather silly. I don't think that introducing a new option in the tool that expected to be compatible among several systems is a good thing. Once new option is introduced and documented, people would start using it, in many cases even without a notion that this option is FreeBSD specific, which will obviously lead to users' confusion and scripts incompatabilities. The right way to go, IMO, is to introduce a simple wrapper for xargs (say yargs), that it will be clearly documented as a FreeBSD scecific thing. -Maxim To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
On Fri, 20 Apr 2001 00:57:29 -0400, Garance A Drosihn [EMAIL PROTECTED] said: '-y[replstr]' (no blank after -y). Prohibited by POSIX. The `xargs' utility ``shall'' follow the Utility Syntax Guidelines. so I don't know how we go about adding options to it. POSIX is clear on this issue: the implementation may add any options it wishes, provided that those options are documented. Of course, if a POSIX working group happens to choose ``your'' option letter next time they add an option to that utility, you lose. -GAWollman To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
Folks, although there was much rejoicing, I think there's no need for a new option to cp. Just use the toolbox, it's not too hard: (cat bigfilelist; echo destdir) | xargs cp Or even echo destdir bigfilelist xargs cp bigfilelist should do the trick. Regards, Jens -- Jens Schweikhardt http://www.schweikhardt.net/ SIGSIG -- signature too long (core dumped) To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
Jens Schweikhardt wrote: although there was much rejoicing, I think there's no need for a new option to cp. Just use the toolbox, it's not too hard: (cat bigfilelist; echo destdir) | xargs cp Or even echo destdir bigfilelist xargs cp bigfilelist should do the trick. Err, neither of those will work if there are too many filenames for a single invokation of "cp" since none but the last will get the "destdir" argument. -- Ben Smithurst / [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
Folks, although there was much rejoicing, I think there's no need for a new option to cp. Just use the toolbox, it's not too hard: (cat bigfilelist; echo destdir) | xargs cp Or even echo destdir bigfilelist xargs cp bigfilelist should do the trick. No, it won't. Consider a list of files a, b, c, d. You create input to xargs 'a b c d destdir', which it then splits into 'a b c' and 'd destdir'. The first time cp is run, it will probably fail; the second time only 'd' ends up where you expect it. The best solution to this is actually to fix xargs to accept a "command format string", eg. echo list | xargs "cp %s destdir" Probably a better way to do it would be to have the insertion marker an option to xargs: echo list | xargs -s LIST_HERE cp LIST_HERE destdir -- ... every activity meets with opposition, everyone who acts has his rivals and unfortunately opponents also. But not because people want to be opponents, rather because the tasks and relationships force people to take different points of view. [Dr. Fritz Todt] V I C T O R Y N O T V E N G E A N C E To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
[ attempting to consolidate two identical threads into one ]
Brian Somers [EMAIL PROTECTED] writes:
I agree - the script idea doesn't seem right.
If {} isn't allowed to implicitly mean ``all the arguments that'll
fit'', then I'd vote for using -i (a version that does full grouping)
although I would not vote for the semantics whereby -i must either be
followed directly with the replacement string (with no intervening
spaces) or else have an implicit {} replacement string, ie we should
have these meaning the same thing:
xargs -i '' command blah '' blah
xargs -i'' command blah '' blah
I honestly don't understand the differnece between this and the -y
option gad described. Nevertheless, it seems that pretty much
everyone agrees that something like this is a good idea.
Attached is a patch that adds a -Y option to xargs which does, well,
pretty much what I imagine the above would do. Here are a couple of
examples:
dima@spike% cat test# test input
this is just
a test; it has
no real purpose
in life
dima@spike% ./xargs -Y {} echo CMD LINE ARGS test
CMD LINE ARGS this is just a test; it has no real purpose in life
dima@spike% ./xargs -Y {} echo {} CMD LINE ARGS test
this is just a test; it has no real purpose in life CMD LINE ARGS
dima@spike% ./xargs -Y {} echo CMD {} LINE ARGS test
CMD this is just a test; it has no real purpose in life LINE ARGS
dima@spike% ./xargs -Y {} echo CMD LINE {} ARGS test
CMD LINE this is just a test; it has no real purpose in life ARGS
dima@spike% ./xargs -Y {} echo CMD LINE ARGS {} test
CMD LINE ARGS this is just a test; it has no real purpose in life
dima@spike% ./xargs -n 2 -Y {} echo CMD LINE {} ARGS test
CMD LINE this is ARGS
CMD LINE just a ARGS
CMD LINE test; it ARGS
CMD LINE has no ARGS
CMD LINE real purpose ARGS
CMD LINE in life ARGS
I'm not sure the patch is entirely correct. xargs seems to be overly
complicated in the way it does some of its processing, but it works
and I believe I managed to maintain most of the assumptions it makes.
Comments? Suggestions?
Thanks,
Dima Dorfman
[EMAIL PROTECTED]
Index: xargs.c
===
RCS file: /st/src/FreeBSD/src/usr.bin/xargs/xargs.c,v
retrieving revision 1.9
diff -u -r1.9 xargs.c
--- xargs.c 1999/08/28 01:07:50 1.9
+++ xargs.c 2001/04/20 22:37:15
@@ -73,6 +73,8 @@
int cnt, indouble, insingle, nargs, nflag, nline, xflag, wasquoted;
char **av, *argp, **ep = env;
long arg_max;
+ int apargs = 0;
+ char **avv, *replstr = NULL;
/*
* POSIX.2 limits the exec line length to ARG_MAX - 2K. Running that
@@ -96,7 +98,7 @@
nline -= strlen(*ep++) + 1 + sizeof(*ep);
}
nflag = xflag = wasquoted = 0;
- while ((ch = getopt(argc, argv, "0n:s:tx")) != -1)
+ while ((ch = getopt(argc, argv, "0n:s:txY:")) != -1)
switch(ch) {
case 'n':
nflag = 1;
@@ -115,6 +117,9 @@
case '0':
zflag = 1;
break;
+ case 'Y':
+ replstr = optarg;
+ break;
case '?':
default:
usage();
@@ -144,6 +149,13 @@
else {
cnt = 0;
do {
+ if (replstr strcmp(*argv, replstr) == 0) {
+ apargs = 1;
+ *argv++;
+ for (avv = argv; *avv; *avv++)
+ cnt += strlen(*avv) + 1;
+ break;
+ }
cnt += strlen(*bxp++ = *argv) + 1;
} while (*++argv);
}
@@ -211,6 +223,8 @@
if (xp == exp || p ebp || ch == EOF) {
if (xflag xp != exp p ebp)
errx(1, "insufficient space for arguments");
+ for (avv = argv; apargs *avv; *avv++)
+ strlen(*xp++ = *avv) + 1;
*xp = NULL;
run(av);
if (ch == EOF)
@@ -253,6 +267,8 @@
if (xflag)
errx(1, "insufficient space for arguments");
+ for (avv = argv; apargs *avv; *avv++)
+ strlen(*xp++ = *avv) + 1;
*xp = NULL;
run(av);
xp = bxp;
To Unsubscribe: send mail to
Re: cp -d dir patch for review (or 'xargs'?)
Folks, although there was much rejoicing, I think there's no need for a new option to cp. Just use the toolbox, it's not too hard: (cat bigfilelist; echo destdir) | xargs cp I like this version of the patch!! It's much much cleaner than hacking up cp or xargs, it even follows the unix principle of using simple tools and glueing them togeather to do bigger jobs, is unix implementation independent, and is very clear in what it does. [You even taugh this old dog a new trick he totally blanked out on!!] Or even echo destdir bigfilelist xargs cp bigfilelist should do the trick. I don't like this one, bigfilelist gets modified and may need to be used again for something else. Please don't commit this version of the patch :-) Regards, Jens -- Jens Schweikhardt http://www.schweikhardt.net/ SIGSIG -- signature too long (core dumped) -- Rod Grimes - KD7CAX @ CN85sl - (RWG25) [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
On Fri, Apr 20, 2001 at 07:26:18PM -0700, Rodney W. Grimes wrote: (cat bigfilelist; echo destdir) | xargs cp I like this version of the patch!! It's much much cleaner than hacking up cp or xargs, it even follows the unix principle of using simple tools and glueing them togeather to do bigger jobs, is unix implementation independent, and is very clear in what it does. It's clean, simple, and unfortunately, totally bogus. Try: echo 1 2 3 4 5 6 7 8 9 | xargs -n 4 echo Now consider what would happen with the above suggested construct with a very long file list. I don't see a problem with adding an option to cp to treat the first argument as the target instead of the last argument. It's a simple solution, the code change is simple, and it produces the exact desired result. What's the problem? -Brian To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
Try: echo 1 2 3 4 5 6 7 8 9 | xargs -n 4 echo Now consider what would happen with the above suggested construct with a very long file list. I don't see a problem with adding an option to cp to treat the first argument as the target instead of the last argument. It's a simple solution, the code change is simple, and it produces the exact desired result. What's the problem? Unfortunatly, cp is not alone in needing this feature. I think a more sensable approach would be to add an "append args" flag to xargs. For example "--", which could be used like so: xargs cp -- destdir EOF first_file second_file third file EOF would run cp 'first_file' 'second_file' 'third file' destdir to pass an argument of two or more dashes to the command, add an extra dash like so: xargs echo -- foo --- bar -- bar EOF first_file second_file third file EOF would run echo 'first_file' 'second_file' 'third file' foo -- bar - bar You get the idea. brad To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
On Fri, Apr 20, 2001 at 07:26:18PM -0700, Rodney W. Grimes wrote: (cat bigfilelist; echo destdir) | xargs cp I like this version of the patch!! It's much much cleaner than hacking up cp or xargs, it even follows the unix principle of using simple tools and glueing them togeather to do bigger jobs, is unix implementation independent, and is very clear in what it does. It's clean, simple, and unfortunately, totally bogus. Try: echo 1 2 3 4 5 6 7 8 9 | xargs -n 4 echo Now consider what would happen with the above suggested construct with a very long file list. bleck... try this for your sample: $ (echo 1 2 3 4 5 6 7 8 9 | xargs -n 4) | while read x; do echo -n $x; echo " dst" done 1 2 3 4 dst 5 6 7 8 dst 9 dst $ I don't see a problem with adding an option to cp to treat the first argument as the target instead of the last argument. It's a simple solution, the code change is simple, and it produces the exact desired result. What's the problem? It's yet another non-portable option. -- Rod Grimes - KD7CAX @ CN85sl - (RWG25) [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
At 3:39 PM -0400 4/19/01, John W. De Boskey wrote: I have added a -d dir option to cp. This allows the target directory to be specified at the head of the command line instead of the tail. This makes cp work much more nicely with tools like xargs... (allowing for major performance improvements over inline shell loops). The patch is at: http://www.freebsd.org/~jwd/cp-d.patch which allows: cp -d target_directory source1 source2 ... sourceN and/or cat big_file_list | xargs cp -d target_directory While I can see how this is useful for 'cp', it only fixes that how xargs works wrt that one command. It doesn't do anything for 'mv', for instance. It seems to me that the problem here is due to xargs, not cp. What other tools are there like xargs, where this new option would really be useful? What I'm wondering is if it would be better to add a new option to 'xargs' itself. Something like: cat big_file_list | xargs -last target_directory cp Or maybe something to indicate where the list of arguments should go in a command. Hrm. Let's say '-Y replstr' or '-y[replstr]' (no blank after -y). If no [replstr] is given on -y, it defaults to the two characters '[]'. Then one might do: cat big_file_list | xargs -y cp [] target_directory This is similar to the '-I' and '-i' parameters on the xargs command that I see in solaris, except that '-I' (stands for "insert mode") forces xarg to build a separate command for each line it is being fed, and for -I/-i the replstr can be specified multiple times in the command xargs will be executing. I picked -Y and -y for no other reason than they didn't seem to be used... Any other letter would be fine by me. I think both -I/-i and -Y/-y would be useful additions to xargs, and would be a more general solution to the problem you're trying to address. On the other hand, the man page for 'xargs' on FreeBSD says: The xargs utility is expected to be IEEE Std 1003.2 (``POSIX.2'') compliant. so I don't know how we go about adding options to it. On the other hand, that same issue is faced by adding options to 'cp', as there is a similar claim made in cp's man page. -- Garance Alistair Drosehn= [EMAIL PROTECTED] Senior Systems Programmer or [EMAIL PROTECTED] Rensselaer Polytechnic Instituteor [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
Garance A Drosihn [EMAIL PROTECTED] writes: Or maybe something to indicate where the list of arguments should go in a command. Hrm. Let's say '-Y replstr' or '-y[replstr]' (no blank after -y). If no [replstr] is given on -y, it defaults to the two characters '[]'. Then one might do: cat big_file_list | xargs -y cp [] target_directory This is a great idea! I'm willing to implement it if nobody else wants to. you're trying to address. On the other hand, the man page for 'xargs' on FreeBSD says: The xargs utility is expected to be IEEE Std 1003.2 (``POSIX.2'') compliant. so I don't know how we go about adding options to it. On the other hand, that same issue is faced by adding options to 'cp', as there is a similar claim made in cp's man page. I don't think it's a problem. We're adding new options here, not changing--sometimes known as breaking--what already exists. I'm pretty sure that the standards don't say anything to the effect of, "You must support this and nothing else." That'd be rather silly. Dima Dorfman [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: cp -d dir patch for review (or 'xargs'?)
At 10:08 PM -0700 4/19/01, Dima Dorfman wrote: Garance A Drosihn [EMAIL PROTECTED] writes: Or maybe something to indicate where the list of arguments should go in a command. Hrm. Let's say '-Y replstr' or '-y[replstr]' (no blank after -y). If no [replstr] is given on -y, it defaults to the two characters '[]'. Then one might do: cat big_file_list | xargs -y cp [] target_directory This is a great idea! I'm willing to implement it if nobody else wants to. Woo-hoo! Someone to do the work! Yes! you're trying to address. On the other hand, the man page for 'xargs' on FreeBSD says: The xargs utility is expected to be IEEE Std 1003.2 (``POSIX.2'') compliant. so I don't know how we go about adding options to it. On the other hand, that same issue is faced by adding options to 'cp', as there is a similar claim made in cp's man page. I don't think it's a problem. We're adding new options here, not changing--sometimes known as breaking--what already exists. I'm pretty sure that the standards don't say anything to the effect of, "You must support this and nothing else." That'd be rather silly. Actually, it's not as silly as it sounds. If you're writing scripts, and you use those extra parameters, then you'll get into trouble when running the script on some other POSIX-based OS which does not have these new options. I really do like the idea of both the -I/-i options from solaris, and the -Y/-y options that I just dreamed up, but I'm not sure what the right procedure is to introduce them (and eventually have them standard everywhere... :-). Maybe we could initially have a 'yargs' command, which is just like 'xargs' except that it adds those four options. Maybe I'm just overly pedantic. Hmm. Checking my copy of "Single Unix Specification, v2", the -I/-i parameters are defined in THAT standard, but it doesn't have anything matching my -Y/-y suggestion. Hmm, I wonder if I should be copying this "meta-question" to the mailing list for standardizing things... -- Garance Alistair Drosehn= [EMAIL PROTECTED] Senior Systems Programmer or [EMAIL PROTECTED] Rensselaer Polytechnic Instituteor [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
