Re: little cp diff

2010-02-10 Thread Ted Unangst
On Mon, Feb 8, 2010 at 9:48 AM, Mark Kettenis mark.kette...@xs4all.nl wrote: For one thing, I'd like to see some real benchmarks. Does using a larger buffer really speed up cp? You claim moving a head between reads and writes takes time, but so does moving it between reads. And modern

Re: little cp diff

2010-02-08 Thread Ted Unangst
On Sun, Feb 7, 2010 at 6:11 PM, Theo de Raadt dera...@cvs.openbsd.org wrote: I think adding big chunks of sysctl code to such specific applications is very un-unix. If choosing a buffer size is going to be a common operation, it should be an API called to ask what the buffer size should be.

Re: little cp diff

2010-02-08 Thread Ted Unangst
On Mon, Feb 8, 2010 at 9:48 AM, Mark Kettenis mark.kette...@xs4all.nl wrote: So how do we move forward? For one thing, I'd like to see some real benchmarks. Does using a larger buffer really speed up cp? You claim moving a head between reads and writes takes time, but so does moving it

Re: little cp diff

2010-02-07 Thread Anton Maksimenkov
2010/2/7 Ted Unangst ted.unan...@gmail.com: But using sysctl is insane. That is not portable. Why does cp need to be portable? Who's porting OpenBSD cp to other systems? Good question. -- antonvm

Re: little cp diff

2010-02-07 Thread Theo de Raadt
Why does cp need to be portable? Who's porting OpenBSD cp to other systems? Will the same diff be added to every io generating program in /bin? They all generate io, of course.

Re: little cp diff

2010-02-07 Thread Ted Unangst
On Sun, Feb 7, 2010 at 2:09 PM, Theo de Raadt dera...@cvs.openbsd.org wrote: Why does cp need to be portable? Who's porting OpenBSD cp to other systems? Will the same diff be added to every io generating program in /bin? They all generate io, of course. I was planning on something similar

Re: little cp diff

2010-02-07 Thread Uriel
On Mon, Feb 8, 2010 at 12:11 AM, Theo de Raadt dera...@cvs.openbsd.org wrote: On Sun, Feb 7, 2010 at 2:09 PM, Theo de Raadt dera...@cvs.openbsd.org wrote: Why does cp need to be portable? B Who's porting OpenBSD cp to other systems? Will the same diff be added to every io generating program in

Re: little cp diff

2010-02-07 Thread Ted Unangst
On Sun, Feb 7, 2010 at 6:11 PM, Theo de Raadt dera...@cvs.openbsd.org wrote: And in mv, for the cross-device case? I see others that do fast reads, like sum. diff? cmp? grep? How many others? If cp(1) is such a common operation (I bet it isn't), how soon before other programs doing

Re: little cp diff

2010-02-07 Thread Theo de Raadt
On Sun, Feb 7, 2010 at 6:11 PM, Theo de Raadt dera...@cvs.openbsd.org wro= te: And in mv, for the cross-device case? =A0I see others that do fast reads, like sum. =A0diff? =A0cmp? =A0grep? =A0How many others? =A0If cp(1= ) is such a common operation (I bet it isn't), how soon before other

Re: little cp diff

2010-02-07 Thread Anton Maksimenkov
2010/2/8 Theo de Raadt dera...@cvs.openbsd.org: For those of you who asked why cp needs to be portable, come on. You've got it all wrong. If cp isn't written in a portable fashion, then what is the point of doing anything else in a portable fashion. This is good and reasonable answer. So I

Re: little cp diff

2010-02-06 Thread Otto Moerbeek
On Fri, Feb 05, 2010 at 09:24:30PM -0500, Ted Unangst wrote: Though for a program like cp, this may qualify as a big diff. :) Continuing in my make IO suck less phase, cp would be a lot more efficient if it didn't bounce the disk heads around so much. Instead of using a tiny 64k buffer,

Re: little cp diff

2010-02-06 Thread Vadim Zhukov
On 6 February 2010 c. 17:53:09 Otto Moerbeek wrote: On Sat, Feb 06, 2010 at 09:14:53AM -0500, Ted Unangst wrote: On Sat, Feb 6, 2010 at 8:44 AM, Mark Kettenis mark.kette...@xs4all.nl wrote: Continuing in my make IO suck less phase, cp would be a lot more efficient if it didn't bounce

Re: little cp diff

2010-02-06 Thread Bob Beck
Whenever I find myself wishing for such knobs, I simply install FreeBSD. 2010/2/6 Vadim Zhukov persg...@gmail.com: On 6 February 2010 c. 17:53:09 Otto Moerbeek wrote: On Sat, Feb 06, 2010 at 09:14:53AM -0500, Ted Unangst wrote: On Sat, Feb 6, 2010 at 8:44 AM, Mark Kettenis

Re: little cp diff

2010-02-06 Thread Otto Moerbeek
On Sat, Feb 06, 2010 at 12:58:21PM -0500, Ted Unangst wrote: On Sat, Feb 6, 2010 at 9:53 AM, Otto Moerbeek o...@drijf.net wrote: The buffer cache should ne smart enough to optmize read and writes in such large chunks so that the seeking gets reduced. The problem with How? cp reads 64k.

Re: little cp diff

2010-02-06 Thread Bob Beck
On 6 February 2010 11:37, Claus Assmann ca+openbsd_t...@esmtp.org wrote: Putting that kind of optimization into programs that read/write large amount of data seems like the wrong way to go. It belongs in a central place. (That's just a comment from a clueless bystander...) It belongs in

Re: little cp diff

2010-02-06 Thread Bob Beck
Now I look at the diff and see that the memory used is reasonably small. But still, if multiple copies of your cp are run together on a machine where available physical mem is low I might end up with a machine swapping. That is certainly not going to speed up cp. If multiple copies of

Re: little cp diff

2010-02-06 Thread Ted Unangst
On Sat, Feb 6, 2010 at 1:35 PM, Otto Moerbeek o...@drijf.net wrote: Add to that that doing I/O in big chunks is also putting more pressure on the buffer cache. The buffer cache keeps a copy of whatever goes through it regardless of size. The io is broken down into 64k chunks no matter what

Re: little cp diff

2010-02-06 Thread Ted Unangst
On Sat, Feb 6, 2010 at 1:54 PM, Theo de Raadt dera...@cvs.openbsd.org wrote: How? cp reads 64k So change it to 1MB. Yeah, right, like anyone will ok a diff like that on the vax. But using sysctl is insane. That is not portable. Why does cp need to be portable? Who's porting OpenBSD cp to