Re: How to copy n bytes from stdin to stdout?
On Wed, 27 Jun 2018 10:06:40 -0700
xi wrote:
> > On Jun 25, 2018, at 16:19, Tomasz Rola wrote:
> >
> > On Sun, Jun 24, 2018 at 10:53:37PM -0400, Steve Litt wrote:
> >> On Thu, 21 Jun 2018 00:56:04 +0200
> >> Tomasz Rola wrote:
> >>
> > [...]
> >>> Craps. I have consulted OpenBSD's manpage for dd and there is no
> >>> mention of iflag. So this will not work on OpenBSD. I will have to
> >>> rethink this, sorry.
> >>>
> >>
> >> Untested...
> >>
> >> int main(int argc, char* argv[]){
> >> long l = atod(argv[1]);
> >> while(l--){
> >>if (c = getc(STDIN) != EOF)
> >>putc(c, STDOUT);
> >>else
> >>break;
> >> }
> >> return 0;
> >> }
> >>
> >> I haven't tested it so it might not be exactly right, and of course
> >> error handling would need to be added, but you know what I mean.
> >> IIRC getc() and putc() are very well buffered so it will be fast.
> >> In my youth I wrote similar functions using low level read() and
> >> write() and doing my own buffering, and those things were *really*
> >> fast, but I think that's overkill in this century.
> >>
> >> As far as finding command line tools that do it, if that's becoming
> >> hard to do, why not just write a 10 line program?
> >
> > Actually, I have written few such programs to satiate my own
> > curiosity
> > - I was dragged away from computer and in the meantime, others
> > joined thread and even wrote nice buffered version of solution in
> > C. I pitted this solution against my programs (in C, with
> > fgetc/fputc and Common Lisp, with read-sequence/write-sequence) and
> > head-c.c was many times faster (about hundred or more times) than
> > my programs.
> >
> > I am not sure if there is performance difference between fgetc/fputc
> > and getc/putc. Man says getc are macros around fgetc. Might be worth
> > checking, but I guess no difference.
> >
> > My curiosity also "wanted" to know how much of performance hit was
> > to be expected when writing best to my knowledge optimised Common
> > Lisp vs simplistic C - they were similar in performance, with CL
> > compiled by SBCL and few times slower, and head-c.c had beaten them
> > both by many lengths. I am a bit surprised that in CL, performance
> > was about the same, whether reading one byte or many at once.
> > Perhaps I will find a way to speed it up some more.
> >
> > As of finding command line tools, I had working script in about an
> > hour (and buggy one in few minutes). Buggy, because "dd | dd" is bad
> > idea, and after finding better options for using dd in my script -
> > which worked, but under Linux - I had also found out they would not
> > work in OpenBSD.
> >
> > So, I consider it a worthy lesson for myself. Next time, I might
> > just fire up Emacs and write a script in CL (mostly, because this
> > is what is comfy for me nowadays, and I will not object against
> > having compiled script for free). Or something similar, or maybe
> > even do it in C, why not.
> >
> > BTW, the version of nread.sh (improved options) was on par with
> > head-c.c, so writing a script with right things inside is very good
> > choice, too. If the script actually works :-) .
> >
> > While the speed is not big problem for input of about 1 megabyte, it
> > becomes a problem when gigabytes are copied.
> >
> > --
> > Regards,
> > Tomasz Rola
> >
> > --
> > ** A C programmer asked whether computer had Buddha's nature.
> > ** ** As the answer, master did "rm -rif" on the programmer's
> > home** ** directory. And then the C programmer became
> > enlightened... **
> > **
> > ** ** Tomasz Rola
> > mailto:tomasz_r...@bigfoot.com **
>
> If you want to do this in C, you can also simply take advantage of
> the fact that read(2) takes a number of bytes as argument and stops
> reading at EOF:
>
> #include
> #include
> #include
>
> int
> main(int argc, char *argv[])
> {
> if (argc < 2)
> return 1;
>
> size_t n;
> char *argend = argv[1] + strlen(argv[1]);
> if (!(n = strtoull(argv[1], , 10)))
> return 1;
>
> char *buf = malloc(n);
> size_t nr = read(0, buf, n);
> write(1, buf, nr);
> free(buf);
>
> return 0;
> }
>
> This is probably as fast as it gets for a task this simple, and it
> copies as many bytes as malloc is willing to allocate.
This is how I did it, both in C and in Turbo Pascal 3.0, back in the
1980's before getc() and putc() were well buffered, except I had the
read and write in a loop. If the buffer is a multiple of sector length,
it's remarkably fast. My findings were that a 64KB buffer was lightning
fast. In Turbo Pascal, a 512KB buffer was about 10% faster.
In this and successive threads people were talking about speed and
visual progress indicator. It's trivial to add nr to the total after
each write, and indicate with:
printf("%ld bytes written.\r", nr);
If that slows too much, do it on 1 out of every 100 writes.
By the mid 1990's, I noticed that
Re: How to copy n bytes from stdin to stdout?
> On Jun 25, 2018, at 16:19, Tomasz Rola wrote:
>
> On Sun, Jun 24, 2018 at 10:53:37PM -0400, Steve Litt wrote:
>> On Thu, 21 Jun 2018 00:56:04 +0200
>> Tomasz Rola wrote:
>>
> [...]
>>> Craps. I have consulted OpenBSD's manpage for dd and there is no
>>> mention of iflag. So this will not work on OpenBSD. I will have to
>>> rethink this, sorry.
>>>
>>
>> Untested...
>>
>> int main(int argc, char* argv[]){
>> long l = atod(argv[1]);
>> while(l--){
>>if (c = getc(STDIN) != EOF)
>>putc(c, STDOUT);
>>else
>>break;
>> }
>> return 0;
>> }
>>
>> I haven't tested it so it might not be exactly right, and of course
>> error handling would need to be added, but you know what I mean. IIRC
>> getc() and putc() are very well buffered so it will be fast. In my
>> youth I wrote similar functions using low level read() and write() and
>> doing my own buffering, and those things were *really* fast, but I
>> think that's overkill in this century.
>>
>> As far as finding command line tools that do it, if that's becoming
>> hard to do, why not just write a 10 line program?
>
> Actually, I have written few such programs to satiate my own curiosity
> - I was dragged away from computer and in the meantime, others joined
> thread and even wrote nice buffered version of solution in C. I pitted
> this solution against my programs (in C, with fgetc/fputc and Common
> Lisp, with read-sequence/write-sequence) and head-c.c was many times
> faster (about hundred or more times) than my programs.
>
> I am not sure if there is performance difference between fgetc/fputc
> and getc/putc. Man says getc are macros around fgetc. Might be worth
> checking, but I guess no difference.
>
> My curiosity also "wanted" to know how much of performance hit was to
> be expected when writing best to my knowledge optimised Common Lisp vs
> simplistic C - they were similar in performance, with CL compiled by
> SBCL and few times slower, and head-c.c had beaten them both by many
> lengths. I am a bit surprised that in CL, performance was about the
> same, whether reading one byte or many at once. Perhaps I will find a
> way to speed it up some more.
>
> As of finding command line tools, I had working script in about an
> hour (and buggy one in few minutes). Buggy, because "dd | dd" is bad
> idea, and after finding better options for using dd in my script -
> which worked, but under Linux - I had also found out they would not
> work in OpenBSD.
>
> So, I consider it a worthy lesson for myself. Next time, I might just
> fire up Emacs and write a script in CL (mostly, because this is what
> is comfy for me nowadays, and I will not object against having compiled
> script for free). Or something similar, or maybe even do it in C, why
> not.
>
> BTW, the version of nread.sh (improved options) was on par with
> head-c.c, so writing a script with right things inside is very good
> choice, too. If the script actually works :-) .
>
> While the speed is not big problem for input of about 1 megabyte, it
> becomes a problem when gigabytes are copied.
>
> --
> Regards,
> Tomasz Rola
>
> --
> ** A C programmer asked whether computer had Buddha's nature. **
> ** As the answer, master did "rm -rif" on the programmer's home**
> ** directory. And then the C programmer became enlightened... **
> ** **
> ** Tomasz Rola mailto:tomasz_r...@bigfoot.com **
>
If you want to do this in C, you can also simply take advantage of the fact
that read(2) takes a number of bytes as argument and stops reading at EOF:
#include
#include
#include
int
main(int argc, char *argv[])
{
if (argc < 2)
return 1;
size_t n;
char *argend = argv[1] + strlen(argv[1]);
if (!(n = strtoull(argv[1], , 10)))
return 1;
char *buf = malloc(n);
size_t nr = read(0, buf, n);
write(1, buf, nr);
free(buf);
return 0;
}
This is probably as fast as it gets for a task this simple, and it copies
as many bytes as malloc is willing to allocate.
Re: How to copy n bytes from stdin to stdout?
Tomasz Rola wrote:
> On Sun, Jun 24, 2018 at 10:53:37PM -0400, Steve Litt wrote:
> > On Thu, 21 Jun 2018 00:56:04 +0200
> > Tomasz Rola wrote:
> >
> [...]
> > > Craps. I have consulted OpenBSD's manpage for dd and there is no
> > > mention of iflag. So this will not work on OpenBSD. I will have to
> > > rethink this, sorry.
> > >
> >
> > Untested...
> >
> > int main(int argc, char* argv[]){
> > long l = atod(argv[1]);
> > while(l--){
> > if (c = getc(STDIN) != EOF)
> > putc(c, STDOUT);
> > else
> > break;
> > }
> > return 0;
> > }
> >
> > I haven't tested it so it might not be exactly right, and of course
> > error handling would need to be added, but you know what I mean. IIRC
> > getc() and putc() are very well buffered so it will be fast. In my
> > youth I wrote similar functions using low level read() and write() and
> > doing my own buffering, and those things were *really* fast, but I
> > think that's overkill in this century.
> >
> > As far as finding command line tools that do it, if that's becoming
> > hard to do, why not just write a 10 line program?
>
> Actually, I have written few such programs to satiate my own curiosity
> - I was dragged away from computer and in the meantime, others joined
> thread and even wrote nice buffered version of solution in C. I pitted
> this solution against my programs (in C, with fgetc/fputc and Common
> Lisp, with read-sequence/write-sequence) and head-c.c was many times
> faster (about hundred or more times) than my programs.
>
> I am not sure if there is performance difference between fgetc/fputc
> and getc/putc. Man says getc are macros around fgetc. Might be worth
> checking, but I guess no difference.
>
> My curiosity also "wanted" to know how much of performance hit was to
> be expected when writing best to my knowledge optimised Common Lisp vs
> simplistic C - they were similar in performance, with CL compiled by
> SBCL and few times slower, and head-c.c had beaten them both by many
> lengths. I am a bit surprised that in CL, performance was about the
> same, whether reading one byte or many at once. Perhaps I will find a
> way to speed it up some more.
>
> As of finding command line tools, I had working script in about an
> hour (and buggy one in few minutes). Buggy, because "dd | dd" is bad
> idea, and after finding better options for using dd in my script -
> which worked, but under Linux - I had also found out they would not
> work in OpenBSD.
>
> So, I consider it a worthy lesson for myself. Next time, I might just
> fire up Emacs and write a script in CL (mostly, because this is what
> is comfy for me nowadays, and I will not object against having compiled
> script for free). Or something similar, or maybe even do it in C, why
> not.
>
> BTW, the version of nread.sh (improved options) was on par with
> head-c.c, so writing a script with right things inside is very good
> choice, too. If the script actually works :-) .
>
> While the speed is not big problem for input of about 1 megabyte, it
> becomes a problem when gigabytes are copied.
Wow.
Re: How to copy n bytes from stdin to stdout?
On Sun, Jun 24, 2018 at 10:53:37PM -0400, Steve Litt wrote:
> On Thu, 21 Jun 2018 00:56:04 +0200
> Tomasz Rola wrote:
>
[...]
> > Craps. I have consulted OpenBSD's manpage for dd and there is no
> > mention of iflag. So this will not work on OpenBSD. I will have to
> > rethink this, sorry.
> >
>
> Untested...
>
> int main(int argc, char* argv[]){
> long l = atod(argv[1]);
> while(l--){
> if (c = getc(STDIN) != EOF)
> putc(c, STDOUT);
> else
> break;
> }
> return 0;
> }
>
> I haven't tested it so it might not be exactly right, and of course
> error handling would need to be added, but you know what I mean. IIRC
> getc() and putc() are very well buffered so it will be fast. In my
> youth I wrote similar functions using low level read() and write() and
> doing my own buffering, and those things were *really* fast, but I
> think that's overkill in this century.
>
> As far as finding command line tools that do it, if that's becoming
> hard to do, why not just write a 10 line program?
Actually, I have written few such programs to satiate my own curiosity
- I was dragged away from computer and in the meantime, others joined
thread and even wrote nice buffered version of solution in C. I pitted
this solution against my programs (in C, with fgetc/fputc and Common
Lisp, with read-sequence/write-sequence) and head-c.c was many times
faster (about hundred or more times) than my programs.
I am not sure if there is performance difference between fgetc/fputc
and getc/putc. Man says getc are macros around fgetc. Might be worth
checking, but I guess no difference.
My curiosity also "wanted" to know how much of performance hit was to
be expected when writing best to my knowledge optimised Common Lisp vs
simplistic C - they were similar in performance, with CL compiled by
SBCL and few times slower, and head-c.c had beaten them both by many
lengths. I am a bit surprised that in CL, performance was about the
same, whether reading one byte or many at once. Perhaps I will find a
way to speed it up some more.
As of finding command line tools, I had working script in about an
hour (and buggy one in few minutes). Buggy, because "dd | dd" is bad
idea, and after finding better options for using dd in my script -
which worked, but under Linux - I had also found out they would not
work in OpenBSD.
So, I consider it a worthy lesson for myself. Next time, I might just
fire up Emacs and write a script in CL (mostly, because this is what
is comfy for me nowadays, and I will not object against having compiled
script for free). Or something similar, or maybe even do it in C, why
not.
BTW, the version of nread.sh (improved options) was on par with
head-c.c, so writing a script with right things inside is very good
choice, too. If the script actually works :-) .
While the speed is not big problem for input of about 1 megabyte, it
becomes a problem when gigabytes are copied.
--
Regards,
Tomasz Rola
--
** A C programmer asked whether computer had Buddha's nature. **
** As the answer, master did "rm -rif" on the programmer's home**
** directory. And then the C programmer became enlightened... **
** **
** Tomasz Rola mailto:tomasz_r...@bigfoot.com **
Re: How to copy n bytes from stdin to stdout?
On Thu, 21 Jun 2018 00:56:04 +0200
Tomasz Rola wrote:
> On Thu, Jun 21, 2018 at 12:44:14AM +0200, Tomasz Rola wrote:
> [...]
> > => (591 60): cat nread
> > #!/bin/sh
> >
> > # nread n - read up to n bytes from stdio, put them on to stdout
> >
> > N=$1
> >
> > dd bs=512 count=$((N / 512)) iflag=fullblock 2>/dev/null
> > dd bs=1 count=$((N % 512)) iflag=fullblock 2>/dev/null
>
> Craps. I have consulted OpenBSD's manpage for dd and there is no
> mention of iflag. So this will not work on OpenBSD. I will have to
> rethink this, sorry.
>
Untested...
int main(int argc, char* argv[]){
long l = atod(argv[1]);
while(l--){
if (c = getc(STDIN) != EOF)
putc(c, STDOUT);
else
break;
}
return 0;
}
I haven't tested it so it might not be exactly right, and of course
error handling would need to be added, but you know what I mean. IIRC
getc() and putc() are very well buffered so it will be fast. In my
youth I wrote similar functions using low level read() and write() and
doing my own buffering, and those things were *really* fast, but I
think that's overkill in this century.
As far as finding command line tools that do it, if that's becoming
hard to do, why not just write a 10 line program?
--
SteveT
Steve Litt
June 2018 featured book: Twenty Eight Tales of Troubleshooting
http://www.troubleshooters.com/28
Re: How to copy n bytes from stdin to stdout?
On Thu, Jun 21, 2018 at 08:06:46PM +0200, Vincent Legoll wrote: > Hello, > > The man page did not say bs has to be a power of 2. > > On a very old macppc openbsd box: > > vince@mini:~$ dd count=1 bs=123456789 < /dev/zero > zero.bin > 1+0 records in > 1+0 records out > 123456789 bytes transferred in 9.833 secs (12554493 bytes/sec) > > On a much more recent core i7 linux: > > vince@dell:~$ dd count=1 bs=123456789 < /dev/zero > zero.bin > 1+0 records in > 1+0 records out > 123456789 bytes (123 MB, 118 MiB) copied, 0,0703818 s, 1,8 GB/s > > > This may not work with huge bs though... > > > -- > Vincent Legoll It was already mentioned that this does not work when short reads occur. -Otto
Re: How to copy n bytes from stdin to stdout?
Hello, The man page did not say bs has to be a power of 2. On a very old macppc openbsd box: vince@mini:~$ dd count=1 bs=123456789 < /dev/zero > zero.bin 1+0 records in 1+0 records out 123456789 bytes transferred in 9.833 secs (12554493 bytes/sec) On a much more recent core i7 linux: vince@dell:~$ dd count=1 bs=123456789 < /dev/zero > zero.bin 1+0 records in 1+0 records out 123456789 bytes (123 MB, 118 MiB) copied, 0,0703818 s, 1,8 GB/s This may not work with huge bs though... -- Vincent Legoll
Re: How to copy n bytes from stdin to stdout?
On Thu, Jun 21, 2018 at 3:15 PM, Daniel Hartmeier wrote: > I think they meant dd and just didn't care about efficiency: > > http://austingroupbugs.net/bug_view_page.php?bug_id=407 Thanks for digging this out! > Does ghead -c beat a simple buffer loop? Your head-c.c seems to have the same performance as ghead -c, presumably because they're doing the same thing. :)
Re: How to copy n bytes from stdin to stdout?
On Thu, Jun 21, 2018 at 03:08:10PM +0200, Jan Stary wrote: > On Jun 20 17:32:51, maxim.pich...@gmail.com wrote: > > dd bs=1 count=1234567 will copy 1234567 bytes and then stop, but it's slow. > > $ dd bs=1 count=1234567 < /dev/zero > /dev/null > 1234567+0 records in > 1234567+0 records out > 1234567 bytes transferred in 4.507 secs (273886 bytes/sec) > > $ dd count=1 bs=1234567 < /dev/zero > /dev/null > 1+0 records in > 1+0 records out > 1234567 bytes transferred in 0.001 secs (653970943 bytes/sec) The latter will stop on a short read. -Otto
Re: How to copy n bytes from stdin to stdout?
On Thu, Jun 21, 2018 at 03:08:10PM +0200, Jan Stary wrote: > $ dd count=1 bs=1234567 < /dev/zero > /dev/null > 1+0 records in > 1+0 records out > 1234567 bytes transferred in 0.001 secs (653970943 bytes/sec) That was my first hunch as well, but try $ printf "foo\nbar\n" | dd count=1 bs=1234567 and then $ (printf "foo\n"; printf "bar\n") | dd count=1 bs=1234567 Daniel
Re: How to copy n bytes from stdin to stdout?
On Thu, Jun 21, 2018 at 10:57:40AM +0200, Maximilian Pichler wrote:
> > dd ibs=1 count=n
>
> Nice, this is about three time as fast as bs=1. Both are much slower
> than 'ghead -c'.
I think they meant dd and just didn't care about efficiency:
http://austingroupbugs.net/bug_view_page.php?bug_id=407
Does ghead -c beat a simple buffer loop?
Daniel
#include
#include
#define BUFSIZE 65536
int main(int argc, char *argv[])
{
char buf[BUFSIZE];
size_t n, r;
if (argc != 2 || (n = atoi(argv[1])) < 1) {
fprintf(stderr, "usage: %s number\n", argv[0]);
return (1);
}
do {
r = fread(buf, 1, n > BUFSIZE ? BUFSIZE : n, stdin);
if (r > 0) {
fwrite(buf, 1, r, stdout);
n -= r;
}
} while (r > 0 && n > 0);
return (0);
}
Re: How to copy n bytes from stdin to stdout?
On Jun 20 17:32:51, maxim.pich...@gmail.com wrote: > dd bs=1 count=1234567 will copy 1234567 bytes and then stop, but it's slow. $ dd bs=1 count=1234567 < /dev/zero > /dev/null 1234567+0 records in 1234567+0 records out 1234567 bytes transferred in 4.507 secs (273886 bytes/sec) $ dd count=1 bs=1234567 < /dev/zero > /dev/null 1+0 records in 1+0 records out 1234567 bytes transferred in 0.001 secs (653970943 bytes/sec)
Re: How to copy n bytes from stdin to stdout?
On Thu, Jun 21, 2018 at 11:05 AM, Abel Abraham Camarillo Ojeda
wrote:
> $ cat file | perl -ne 'BEGIN { $/ = \1 } print if $. <= 5; exit 0 if $. == 5'
This is much slower than 'dd ibs=1'
Re: How to copy n bytes from stdin to stdout?
On Thu, Jun 21, 2018 at 3:57 AM, Maximilian Pichler
wrote:
> On Thu, Jun 21, 2018 at 9:48 AM, Otto Moerbeek wrote:
>>> On Thu, Jun 21, 2018 at 12:11:52AM +0200, Maximilian Pichler wrote:
>>> > I'm just wondering what these other utilities might be.
>>>
>>> hexdump -v -n 1234567 -e '"%c"'
>
> Speed-wise this is roughly on par with 'dd bs=1'.
>
>>> If the input doesn't contain backslashes (or something else, tr(1))
>>>
>>> vis -aoF6 | head -n 1234567 | unvis
>
> Backslashes exist. :)
>
>> Variation that buffers the writes:
>>
>> dd ibs=1 count=n
>
> Nice, this is about three time as fast as bs=1. Both are much slower
> than 'ghead -c'.
>
Reads buffered the first 5 chars from file:
$ cat file | perl -ne 'BEGIN { $/ = \1 } print if $. <= 5; exit 0 if $. == 5'
maetel$ kdump | grep read\(0
43509 perl CALL read(0,0xc9e9996f000,0x2000)
maetel$
maetel$ kdump | grep write\(1
43509 perl CALL write(1,0xc9e0c6c5000,0x5)
maetel$
Re: How to copy n bytes from stdin to stdout?
On Thu, Jun 21, 2018 at 9:48 AM, Otto Moerbeek wrote: >> On Thu, Jun 21, 2018 at 12:11:52AM +0200, Maximilian Pichler wrote: >> > I'm just wondering what these other utilities might be. >> >> hexdump -v -n 1234567 -e '"%c"' Speed-wise this is roughly on par with 'dd bs=1'. >> If the input doesn't contain backslashes (or something else, tr(1)) >> >> vis -aoF6 | head -n 1234567 | unvis Backslashes exist. :) > Variation that buffers the writes: > > dd ibs=1 count=n Nice, this is about three time as fast as bs=1. Both are much slower than 'ghead -c'.
Re: How to copy n bytes from stdin to stdout?
On Thu, Jun 21, 2018 at 07:38:49AM +0200, Daniel Hartmeier wrote: > On Thu, Jun 21, 2018 at 12:11:52AM +0200, Maximilian Pichler wrote: > > > I'm just wondering what these other utilities might be. > > hexdump -v -n 1234567 -e '"%c"' > > If the input doesn't contain backslashes (or something else, tr(1)) > > vis -aoF6 | head -n 1234567 | unvis > > Daniel Variation that buffers the writes: dd ibs=1 count=n Reads are still unbuffered though, -Otto
Re: How to copy n bytes from stdin to stdout?
On Thu, Jun 21, 2018 at 12:11:52AM +0200, Maximilian Pichler wrote: > I'm just wondering what these other utilities might be. hexdump -v -n 1234567 -e '"%c"' If the input doesn't contain backslashes (or something else, tr(1)) vis -aoF6 | head -n 1234567 | unvis Daniel
Re: How to copy n bytes from stdin to stdout?
On Thu, Jun 21, 2018 at 12:44:14AM +0200, Tomasz Rola wrote: [...] > => (591 60): cat nread > #!/bin/sh > > # nread n - read up to n bytes from stdio, put them on to stdout > > N=$1 > > dd bs=512 count=$((N / 512)) iflag=fullblock 2>/dev/null > dd bs=1 count=$((N % 512)) iflag=fullblock 2>/dev/null Craps. I have consulted OpenBSD's manpage for dd and there is no mention of iflag. So this will not work on OpenBSD. I will have to rethink this, sorry. -- Regards, Tomasz Rola -- ** A C programmer asked whether computer had Buddha's nature. ** ** As the answer, master did "rm -rif" on the programmer's home** ** directory. And then the C programmer became enlightened... ** ** ** ** Tomasz Rola mailto:tomasz_r...@bigfoot.com **
Re: How to copy n bytes from stdin to stdout?
On Thu, Jun 21, 2018 at 12:02:14AM +0200, Maximilian Pichler wrote: > Your script is incorrect. > > $ dd if=/dev/zero bs=1 | ./nread 1234567 | wc -c > 0+2411 records in > 0+2411 records out > 2411 bytes transferred in 0.038 secs (62579 bytes/sec) > 135+0 records in > 135+0 records out > 135 bytes transferred in 0.001 secs (126148 bytes/sec) > 2546 > I have slightly modified the script to ease debugging. Also, slight mod to read full blocks - I was not aware that strange things can happen when two dd's feed one another in a pipe. => (591 60): cat nread #!/bin/sh # nread n - read up to n bytes from stdio, put them on to stdout N=$1 dd bs=512 count=$((N / 512)) iflag=fullblock 2>/dev/null dd bs=1 count=$((N % 512)) iflag=fullblock 2>/dev/null => (591 61): md5sum (591 65): dd if=/dev/zero bs=1 2>/dev/null | /usr/bin/time nread 1234567 | wc -c 0.14user 3.00system 0:03.55elapsed 88%CPU (0avgtext+0avgdata 768maxresident)k 0inputs+0outputs (0major+737minor)pagefaults 0swaps 1234567 => (591 66): cat HUGE | /usr/bin/time nread 1234567 | wc -c 0.00user 0.01system 0:00.02elapsed 54%CPU (0avgtext+0avgdata 768maxresident)k 0inputs+0outputs (0major+734minor)pagefaults 0swaps 1234567 -- Regards, Tomasz Rola -- ** A C programmer asked whether computer had Buddha's nature. ** ** As the answer, master did "rm -rif" on the programmer's home** ** directory. And then the C programmer became enlightened... ** ** ** ** Tomasz Rola mailto:tomasz_r...@bigfoot.com **
Re: How to copy n bytes from stdin to stdout?
POSIX says (http://pubs.opengroup.org/onlinepubs/9699919799/utilities/head.html#tag_20_57_18) about the 'head' utility: "There is no -c option (as there is in tail) because it is not historical practice and because other utilities in this volume of POSIX.1-2017 provide similar functionality." I'm just wondering what these other utilities might be. On Wed, Jun 20, 2018 at 5:32 PM, Maximilian Pichler wrote: > dd bs=1 count=1234567 will copy 1234567 bytes and then stop, but it's slow. > > I can't seem to think of a faster command that also works in the > presence of short reads and blocking. There is ghead -c from coreutils > in ports, but this should be possible in base, no? > > Max
Re: How to copy n bytes from stdin to stdout?
Your script is incorrect. $ dd if=/dev/zero bs=1 | ./nread 1234567 | wc -c 0+2411 records in 0+2411 records out 2411 bytes transferred in 0.038 secs (62579 bytes/sec) 135+0 records in 135+0 records out 135 bytes transferred in 0.001 secs (126148 bytes/sec) 2546 $ man dd ... count=n Copy only n input blocks. ... These are input, not output blocks. On Wed, Jun 20, 2018 at 11:42 PM, Tomasz Rola wrote: > On Wed, Jun 20, 2018 at 08:20:16PM +0200, Maximilian Pichler wrote: >> On Wed, Jun 20, 2018 at 7:17 PM, Tomasz Rola wrote: >> > But seriously: man sh. >> >> Are you saying there is a shell built-in that does this? If so, which one? > > => (591 13):cat nread > #!/bin/sh > > # nread n - read up to n bytes from stdio, put them on to stdout > > N=$1 > > dd bs=512 count=$((N / 512)) > dd bs=1 count=$((N % 512)) > > => (591 14): md5sum 9c6d3e6aa2b11f6351290fc4f770bf44 - > > => (591 15): chmod a+x nread > > => (591 16): cat HUGE | /usr/bin/time ./nread 1234567 | wc -c > 2411+0 records in > 2411+0 records out > 1234432 bytes (1.2 MB) copied, 0.0122527 s, 101 MB/s > 135+0 records in > 135+0 records out > 135 bytes (135 B) copied, 0.000620305 s, 218 kB/s > 0.00user 0.01system 0:00.02elapsed 57%CPU (0avgtext+0avgdata 768maxresident)k > 0inputs+0outputs (0major+731minor)pagefaults 0swaps > 1234567 > > Total time is well below 1s. If you want faster, then you have to > write it in C or assembly. > > -- > Regards, > Tomasz Rola > > -- > ** A C programmer asked whether computer had Buddha's nature. ** > ** As the answer, master did "rm -rif" on the programmer's home** > ** directory. And then the C programmer became enlightened... ** > ** ** > ** Tomasz Rola mailto:tomasz_r...@bigfoot.com ** >
Re: How to copy n bytes from stdin to stdout?
On Wed, Jun 20, 2018 at 08:20:16PM +0200, Maximilian Pichler wrote: > On Wed, Jun 20, 2018 at 7:17 PM, Tomasz Rola wrote: > > But seriously: man sh. > > Are you saying there is a shell built-in that does this? If so, which one? => (591 13):cat nread #!/bin/sh # nread n - read up to n bytes from stdio, put them on to stdout N=$1 dd bs=512 count=$((N / 512)) dd bs=1 count=$((N % 512)) => (591 14): md5sum (591 15): chmod a+x nread => (591 16): cat HUGE | /usr/bin/time ./nread 1234567 | wc -c 2411+0 records in 2411+0 records out 1234432 bytes (1.2 MB) copied, 0.0122527 s, 101 MB/s 135+0 records in 135+0 records out 135 bytes (135 B) copied, 0.000620305 s, 218 kB/s 0.00user 0.01system 0:00.02elapsed 57%CPU (0avgtext+0avgdata 768maxresident)k 0inputs+0outputs (0major+731minor)pagefaults 0swaps 1234567 Total time is well below 1s. If you want faster, then you have to write it in C or assembly. -- Regards, Tomasz Rola -- ** A C programmer asked whether computer had Buddha's nature. ** ** As the answer, master did "rm -rif" on the programmer's home** ** directory. And then the C programmer became enlightened... ** ** ** ** Tomasz Rola mailto:tomasz_r...@bigfoot.com **
Re: How to copy n bytes from stdin to stdout?
On Wed, Jun 20, 2018 at 7:17 PM, Tomasz Rola wrote: > But seriously: man sh. Are you saying there is a shell built-in that does this? If so, which one?
Re: How to copy n bytes from stdin to stdout?
On Wed, Jun 20, 2018 at 06:50:06PM +0200, Tomasz Rola wrote: [...] > > Maybe suggest to the teacher that she sticks to numbers divisible by > 512? That is what haxorz do. But seriously: man sh. -- Regards, Tomasz Rola -- ** A C programmer asked whether computer had Buddha's nature. ** ** As the answer, master did "rm -rif" on the programmer's home** ** directory. And then the C programmer became enlightened... ** ** ** ** Tomasz Rola mailto:tomasz_r...@bigfoot.com **
Re: How to copy n bytes from stdin to stdout?
On Wed, Jun 20, 2018 at 06:35:49PM +0200, Maximilian Pichler wrote: > On Wed, Jun 20, 2018 at 6:27 PM, Tomasz Rola wrote: > > On my Linux box: > > ? > > > cat HUGE | /usr/bin/time dd bs=1 count=1234944 | wc -c > > stdin might be something much faster than your disk, in which case the > relative cost of bs=1 increases. So is this the problem with slow disk? > > cat HUGE | /usr/bin/time dd bs=1024 count=1206 | wc -c > > Doesn't work for prime numbers. ;) Maybe suggest to the teacher that she sticks to numbers divisible by 512? That is what haxorz do. -- Regards, Tomasz Rola -- ** A C programmer asked whether computer had Buddha's nature. ** ** As the answer, master did "rm -rif" on the programmer's home** ** directory. And then the C programmer became enlightened... ** ** ** ** Tomasz Rola mailto:tomasz_r...@bigfoot.com **
Re: How to copy n bytes from stdin to stdout?
On Wed, Jun 20, 2018 at 6:27 PM, Tomasz Rola wrote: > On my Linux box: ? > cat HUGE | /usr/bin/time dd bs=1 count=1234944 | wc -c stdin might be something much faster than your disk, in which case the relative cost of bs=1 increases. > cat HUGE | /usr/bin/time dd bs=1024 count=1206 | wc -c Doesn't work for prime numbers. ;)
Re: How to copy n bytes from stdin to stdout?
On Wed, Jun 20, 2018 at 06:01:21PM +0200, Maximilian Pichler wrote: > On Wed, Jun 20, 2018 at 5:50 PM, Solene Rapenne wrote: > > it's slow because it flushes the output every byte, what would you > > expect? Maybe you should do in a different manner. > > I know, my question is what such a different manner might look like. :) On my Linux box: cat HUGE | /usr/bin/time dd bs=1 count=1234944 | wc -c 1234944+0 records in 1234944+0 records out 1234944 bytes (1.2 MB) copied, 4.33782 s, 285 kB/s 0.33user 4.00system 0:04.34elapsed 99%CPU (0avgtext+0avgdata 772maxresident)k 0inputs+0outputs (0major+249minor)pagefaults 0swaps 1234944 cat HUGE | /usr/bin/time dd bs=1024 count=1206 | wc -c 1206+0 records in 1206+0 records out 1234944 bytes (1.2 MB) copied, 0.00895034 s, 138 MB/s 0.00user 0.00system 0:00.01elapsed 66%CPU (0avgtext+0avgdata 768maxresident)k 0inputs+0outputs (0major+248minor)pagefaults 0swaps 1234944 -- Regards, Tomasz Rola -- ** A C programmer asked whether computer had Buddha's nature. ** ** As the answer, master did "rm -rif" on the programmer's home** ** directory. And then the C programmer became enlightened... ** ** ** ** Tomasz Rola mailto:tomasz_r...@bigfoot.com **
Re: How to copy n bytes from stdin to stdout?
On Wed, Jun 20, 2018 at 5:50 PM, Solene Rapenne wrote: > it's slow because it flushes the output every byte, what would you > expect? Maybe you should do in a different manner. I know, my question is what such a different manner might look like. :)
Re: How to copy n bytes from stdin to stdout?
Maximilian Pichler writes: > dd bs=1 count=1234567 will copy 1234567 bytes and then stop, but it's slow. > > I can't seem to think of a faster command that also works in the > presence of short reads and blocking. There is ghead -c from coreutils > in ports, but this should be possible in base, no? > > Max it's slow because it flushes the output every byte, what would you expect? Maybe you should do in a different manner.
How to copy n bytes from stdin to stdout?
dd bs=1 count=1234567 will copy 1234567 bytes and then stop, but it's slow. I can't seem to think of a faster command that also works in the presence of short reads and blocking. There is ghead -c from coreutils in ports, but this should be possible in base, no? Max
