I think, i hade the same error after using long time ago xargs.
So i switched to:
find foo | while read file; do foo "$file"; done
After I had a discussion about it in the serversupportforum, I tried
it again and it has worked with 20,000 files.
Xargs splits arguments and run the programm more times to work out the
whole queue, if your arguments are too long:
$ export LANG=en.EN.UTF-8
$ xargs --show-limits
Your environment variables take up 3106 bytes
POSIX upper limit on argument length (this system): 2091998
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 2088892
Size of command buffer we are actually using: 131072
Files on my system:
$ find / 2>/dev/null -print | wc -l
316764
When I do run xargs normal:
$ time find / 2>/dev/null -print0 | xargs -0 echo | wc -l
143
real 0m0.551s
user 0m0.348s
sys 0m0.324s
This means xargs runs echo 143 times
If I limit xargs:
$ time find / 2>/dev/null -print0 | xargs -s4096 -0 echo | wc -l
4612
real 0m2.279s
user 0m0.332s
sys 0m0.596s
It runs 4612 times the programm echo and takes more time.
Only for echo (read arguments, parsing options, error processing,
little bit logic, send to stdout) and redirecting to /dev/null.
2013/2/26 Jesse Molina <je...@opendreams.net>:
>
> Andre Müller noted that when find calls an external application with -exec
> as I originally wrote below, it makes that call individually for each
> argument, which is why it takes so long on 60K files.
>
> Piping the find output to xargs allows for dramatically faster results.
>
> Recommended replacement:
>
> # Fix directory permissions
> #$NICECMD find $FIXTARGET -type d -exec chmod --preserve-root
> u+rwx,g+rwx,o-rwx {} \;
> $NICECMD find $FIXTARGET -type d -print0 | xargs -0 chmod --preserve-root
> u+rwx,g+rwx,o-rwx
>
> # Fix file permissions
> #$NICECMD find $FIXTARGET -type f -exec chmod --preserve-root
> u=rw,g=rw,o-rwx {} \;
>
> $NICECMD find $FIXTARGET -type f -print0 | xargs -0 chmod --preserve-root
> u=rw,g=rw,o-rwx
>
>
>
> Beware: I seem to remember that you can run into "Too many arguments" kind
> of error on older OSes that have a limit (4096 I think) on the amount of
> input a command can accept. However, this should be safe for almost
> everyone and you can get around those kinds of limits with xargs -s or maybe
> -L.
>
> It's also noteworthy that I failed to quote my {} brackets , which isn't
> safe. Replace with '{}'.
>
> Find fails.
>
> Thanks Andre!
>
>
>
>
> Jesse Molina wrote:
>>
>>
>> This is the way that I do it within my control script.
>>
>> The real iowait pain here is iterating over the 60,000 files three times
>> over. If you can live without the chown line, that would cut down on
>> processing time.
>>
>> Note that if you symlink to a base install, that may have an effect on
>> processing time since the referrer file won't get touched.
>>
>>
>>
>> FIXTARGET=/path/to/yer/gameserver
>> SETUSER=foo
>> SETGROUP=staff
>>
>> # Make it nice
>> NICECMD="nice -n 11 ionice -c 2 -n 6"
>>
>> # Fix ownership
>> $NICECMD chown -R -h --preserve-root $SETUSER:$SETGROUP $FIXTARGET
>>
>> # Fix directory permissions
>> $NICECMD find $FIXTARGET -type d -exec chmod --preserve-root
>> u+rwx,g+rwx,o-rwx {} \;
>>
>> # Fix file permissions
>> $NICECMD find $FIXTARGET -type f -exec chmod --preserve-root
>> u=rw,g=rw,o-rwx {} \;
>>
>> # Set +x on the srcds_ binaries
>> for EACH in $(find $FIXTARGET -maxdepth 3 -type f -name "srcds_*" |
>> egrep "srcds_[amd|i386|i486|i586|i686|run|linux|osx]") ; do
>> chmod ug+x $EACH
>> done
>>
>>
>>
>>
>> Rudy Bleeker wrote:
>>>
>>> Jesse, I'm interested in your script, since bad file permissions get
>>> on my nerves.
>>
>>
>
> --
> # Jesse Molina
> # Mail = je...@opendreams.net
> # Cell = 1-602-323-7608
>
>
>
> _______________________________________________
> To unsubscribe, edit your list preferences, or view the list archives,
> please visit:
> https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux
_______________________________________________
To unsubscribe, edit your list preferences, or view the list archives, please
visit:
https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux
