Re: bash sockets: printf \x0a does TCP fragmentation

2018-10-12 Thread Dirk Wetter
On 11.10.18 23:35, Bob Proulx wrote: > The smallest of details can break the largest of programs. :-) > > Good to hear you have things resolved and now working for you! Thanks all for your insights and help! Cheers, Dirk

Re: bash sockets: printf \x0a does TCP fragmentation

2018-10-11 Thread Bob Proulx
Dirk Wetter wrote: > PS + @Bob: fd 5 is not a tty in the program -- but interactively in > this PoC you want to make sure it is not taken yet. Understood. I originally mentioned the isatty() thing because that is how libc decides to line buffer or not. But Chet has confirmed that bash's

Re: bash sockets: printf \x0a does TCP fragmentation

2018-10-11 Thread Chet Ramey
On 10/11/18 12:53 PM, Dirk Wetter wrote: > This all -- including cat -- sounded reasonable. But it seems using sockets > the internal printf > as opposed to the one from coreutils is still causing fragmentation other > than expected with > strace PoC: Bash line-buffers stdout, so newlines

Re: bash sockets: printf \x0a does TCP fragmentation

2018-10-11 Thread Dirk Wetter
On 11.10.18 18:53, Dirk Wetter wrote: > bash 0$ exec 5<>/dev/tcp/81.169.199.25/443 > bash 0$ printf >

Re: bash sockets: printf \x0a does TCP fragmentation

2018-10-11 Thread Dirk Wetter
On 23.09.18 20:29, Bob Proulx wrote: > Robert Elz wrote: > $ { command printf "one\n"; sleep 1; command printf "two\n" ;} | strace -v > -o /tmp/dd.strace.out -e write,read dd status=none ibs=1M obs=1M ; head > /tmp/*.strace.out > one > two > ... > read(0, "one\n", 1048576)

Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-26 Thread Chet Ramey
On 9/26/18 2:17 AM, Dirk Wetter wrote: >> then what happens if "$data" is >> a paragraph of text with embedded newlines.  In that case, >> it sounds like bash might break apart the single printf >> output into smaller packets rather than transmitting the >> entirety of "$data" in 1 write

Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-26 Thread Dirk Wetter
On 9/25/18 2:15 PM, L A Walsh wrote: > > > On 9/24/2018 6:05 AM, Greg Wooledge wrote: >> On Sat, Sep 22, 2018 at 11:50:17AM +0200, dirk+b...@testssl.sh wrote: >>   >>> On 9/22/18 7:30 AM, Bob Proulx wrote: >>>     dirk+b...@testssl.sh wrote:   > printf -- "$data" >&5

Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-25 Thread Greg Wooledge
> On 9/25/18 9:04 AM, dirk+b...@testssl.sh wrote: > > env or exec: never thought about it (thanks!) but as both are external > > commands, that would mean upon every call one additional external program. > > (yes, I know that there is such thing as a fs buffer). Subshells also costs > > resources.

Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-25 Thread Chet Ramey
On 9/25/18 8:25 AM, Greg Wooledge wrote: > On Tue, Sep 25, 2018 at 05:15:02AM -0700, L A Walsh wrote: >> On 9/24/2018 6:05 AM, Greg Wooledge wrote: >>> On Sat, Sep 22, 2018 at 11:50:17AM +0200, dirk+b...@testssl.sh wrote: On 9/22/18 7:30 AM, Bob Proulx wrote: > dirk+b...@testssl.sh wrote:

Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-25 Thread Chet Ramey
On 9/25/18 9:04 AM, dirk+b...@testssl.sh wrote: > FYI: I ended up checking with type before whether an external printf > exists and set a variable for this and then just call this variable. > > env or exec: never thought about it (thanks!) but as both are external > commands, that would mean

Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-25 Thread dirk+bash
On 9/23/18 8:26 PM, Chet Ramey wrote: > On 9/22/18 4:22 PM, Bob Proulx wrote: > >> Note that I *did* provide you with a way to do what you wanted to do. :-) >> >> It was also noted in another message that the external standalone >> printf command line utility did buffer as you desired. That

Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-25 Thread Greg Wooledge
On Tue, Sep 25, 2018 at 05:15:02AM -0700, L A Walsh wrote: > On 9/24/2018 6:05 AM, Greg Wooledge wrote: > > On Sat, Sep 22, 2018 at 11:50:17AM +0200, dirk+b...@testssl.sh wrote: > > > On 9/22/18 7:30 AM, Bob Proulx wrote: > > > > dirk+b...@testssl.sh wrote: > > > > > printf -- "$data" >&5

Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-25 Thread L A Walsh
On 9/24/2018 6:05 AM, Greg Wooledge wrote: On Sat, Sep 22, 2018 at 11:50:17AM +0200, dirk+b...@testssl.sh wrote: On 9/22/18 7:30 AM, Bob Proulx wrote: dirk+b...@testssl.sh wrote: printf -- "$data" >&5 2>/dev/null What happens if $data contains % format strings?

Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-24 Thread Greg Wooledge
On Sat, Sep 22, 2018 at 11:50:17AM +0200, dirk+b...@testssl.sh wrote: > On 9/22/18 7:30 AM, Bob Proulx wrote: > > dirk+b...@testssl.sh wrote: > >> printf -- "$data" >&5 2>/dev/null > > > > What happens if $data contains % format strings? What happens if the > > format contains a sequence such as

Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-23 Thread Chet Ramey
On 9/23/18 2:46 PM, Bob Proulx wrote: > Chet Ramey wrote: >> It's that bash sets stdout and stderr to be line-buffered, not anything >> printf-specific. > > Shouldn't bash set stdout buffering based upon the output descriptor > being a tty or not the same as other libc stdio behavior? It's been

Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-23 Thread Bob Proulx
Chet Ramey wrote: > Bob Proulx wrote: > > It was also noted in another message that the external standalone > > printf command line utility did buffer as you desired. That seems > > another very good solution too. Simply use "command printf ..." to > > force using the external version. > > This

Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-23 Thread Bob Proulx
Robert Elz wrote: > Bob Proulx wrote: > | Using the same buffer size > | for input and output is usually most efficient. > > Yes, but as the objective seemed to be to make big packets, that is probably > not as important. The original complaint concerned flushing a data blob content upon

Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-23 Thread Chet Ramey
On 9/22/18 4:22 PM, Bob Proulx wrote: > Note that I *did* provide you with a way to do what you wanted to do. :-) > > It was also noted in another message that the external standalone > printf command line utility did buffer as you desired. That seems > another very good solution too. Simply

Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-23 Thread Chet Ramey
On 9/22/18 6:49 AM, dirk+b...@testssl.sh wrote: > > > On 9/22/18 12:38 PM, Ilkka Virta wrote: >> On 22.9. 02:34, Chet Ramey wrote: >>> Newline? It's probably that stdout is line-buffered and the newline causes >>> a flush, which results in a write(2). >> >> Mostly out of curiosity, what kind of

Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-23 Thread Robert Elz
Date:Sat, 22 Sep 2018 23:51:08 -0600 From:Bob Proulx Message-ID: <20180922231240358868...@bob.proulx.com> | Using the same buffer size | for input and output is usually most efficient. Yes, but as the objective seemed to be to make big packets, that is probably

Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-22 Thread Bob Proulx
Robert Elz wrote: > ps: f there was actually a desire to use dd to do re-buffering, the > correct usage is not to use "bs=" (which simply does read write with > a buffer that size) but "obs=" which reads (using ibs if needed, which it > would not be here), copies to an output buffer and writes

Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-22 Thread Robert Elz
Date:Sat, 22 Sep 2018 14:22:19 -0600 From:Bob Proulx Message-ID: <20180922111950901701...@bob.proulx.com> | Primarily a shell script is a command and control program. It is very | good for that purpose. It is typically used for that purpose. That | is the

Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-22 Thread Bob Proulx
I see that you have subscribed now. Awesome! If you and others would be so kind as to list-reply instead of CC'ing me directly that would be great. I read the replies on the mailing list. dirk+b...@testssl.sh wrote: > Bob Proulx wrote: > > You are doing something that is quite unusual. You

Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-22 Thread Chet Ramey
On 9/22/18 6:38 AM, Ilkka Virta wrote: > On 22.9. 02:34, Chet Ramey wrote: >> Newline? It's probably that stdout is line-buffered and the newline causes >> a flush, which results in a write(2). > > Mostly out of curiosity, what kind of buffering logic does Bash (or the > builtin printf in

Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-22 Thread Chet Ramey
On 9/22/18 4:21 AM, dirk+b...@testssl.sh wrote: > > > On 9/22/18 1:34 AM, Chet Ramey wrote: >> On 9/21/18 4:13 PM, dirk+b...@testssl.sh wrote: >>> >>> Hello there, >>> >>> we discovered a strange phenomenon in the project testssl.sh: >>> >>> After opening a TCP socket with a fd (here: 5), when

Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-22 Thread dirk+bash
On 9/22/18 12:30 PM, Ilkka Virta wrote: > The coreutils printf seems to output 'foo\nbar\n' as a single write, though > (unless > it goes to the terminal, so the usual stdio buffering), so you might be able > to use > that. thx. Might be not that portable but we'll see. > In any case, if a

Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-22 Thread dirk+bash
On 9/22/18 12:38 PM, Ilkka Virta wrote: > On 22.9. 02:34, Chet Ramey wrote: >> Newline? It's probably that stdout is line-buffered and the newline causes >> a flush, which results in a write(2). > > Mostly out of curiosity, what kind of buffering logic does Bash (or the > builtin > printf in

Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-22 Thread Ilkka Virta
On 22.9. 02:34, Chet Ramey wrote: Newline? It's probably that stdout is line-buffered and the newline causes a flush, which results in a write(2). Mostly out of curiosity, what kind of buffering logic does Bash (or the builtin printf in particular) use? It doesn't seem to be the usual stdio

Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-22 Thread Ilkka Virta
On 22.9. 12:50, dirk+b...@testssl.sh wrote: cat has a problem with binary chars, right? And: see below. No, it just loops with read() and write(), it shouldn't touch any of the bytes (except for cat -A and such). But it probably doesn't help in coalescing the write blocks, it's likely to

Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-22 Thread dirk+bash
On 9/22/18 7:30 AM, Bob Proulx wrote: > dirk+b...@testssl.sh wrote: >> we discovered a strange phenomenon in the project testssl.sh: > > You are doing something that is quite unusual. You are using a shell > script direction on a TCP socket. That isn't very common. Do you think there

Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-22 Thread dirk+bash
On 9/22/18 1:34 AM, Chet Ramey wrote: > On 9/21/18 4:13 PM, dirk+b...@testssl.sh wrote: >> >> Hello there, >> >> we discovered a strange phenomenon in the project testssl.sh: >> >> After opening a TCP socket with a fd (here: 5), when writing to it, >> it seems that >> >> printf -- "$data" >&5

Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-21 Thread Bob Proulx
dirk+b...@testssl.sh wrote: > we discovered a strange phenomenon in the project testssl.sh: You are doing something that is quite unusual. You are using a shell script direction on a TCP socket. That isn't very common. More typically one would use a C program instead. So it isn't surprising

Re: bash sockets: printf \x0a does TCP fragmentation

2018-09-21 Thread Chet Ramey
On 9/21/18 4:13 PM, dirk+b...@testssl.sh wrote: > > Hello there, > > we discovered a strange phenomenon in the project testssl.sh: > > After opening a TCP socket with a fd (here: 5), when writing to it, > it seems that > > printf -- "$data" >&5 2>/dev/null > > does not do what it is intended.

bash sockets: printf \x0a does TCP fragmentation

2018-09-21 Thread dirk+bash
Hello there, we discovered a strange phenomenon in the project testssl.sh: After opening a TCP socket with a fd (here: 5), when writing to it, it seems that printf -- "$data" >&5 2>/dev/null does not do what it is intended. "$data" is a ClientHello like