Re: bash sockets: printf \x0a does TCP fragmentation
Robert Elz wrote: > ps: f there was actually a desire to use dd to do re-buffering, the > correct usage is not to use "bs=" (which simply does read write with > a buffer that size) but "obs=" which reads (using ibs if needed, which it > would not be here), copies to an output buffer and writes only when that > buffer is full (or on EOF on the input). If the goal is to minimize writes then it won't matter as long as the buffer size picked is larger than needed. Using the same buffer size for input and output is usually most efficient. First let's set it to something small to prove that it is buffering as expected. $ printf -- "%s\n" one two | strace -o /tmp/out -e read,write dd status=none bs=2 ; cat /tmp/out one two ... read(0, "on", 2)= 2 write(1, "on", 2) = 2 read(0, "e\n", 2) = 2 write(1, "e\n", 2) = 2 read(0, "tw", 2)= 2 write(1, "tw", 2) = 2 read(0, "o\n", 2) = 2 write(1, "o\n", 2) = 2 read(0, "", 2) = 0 +++ exited with 0 +++ Lots of reads and writes but all as expected. Or set just the output buffer size large. Then the input buffer size defaults to 512 bytes on my system. $ printf -- "%s\n" one two | strace -o /tmp/out -e write,read dd status=none obs=1M ; cat /tmp/out one two ... read(0, "one\ntwo\n", 512) = 8 read(0, "", 512)= 0 write(1, "one\ntwo\n", 8) = 8 +++ exited with 0 +++ But even if ibs is much too small it still behaves okay with a small input buffer size and a large output buffer size. $ printf -- "%s\n" one two | strace -o /tmp/out -e write,read dd status=none ibs=2 obs=1M ; cat /tmp/out one two ... read(0, "on", 2)= 2 read(0, "e\n", 2) = 2 read(0, "tw", 2)= 2 read(0, "o\n", 2) = 2 read(0, "", 2) = 0 write(1, "one\ntwo\n", 8) = 8 +++ exited with 0 +++ Then set both ibs and obs to be something quite large using bs= and let it gather up all of the input and write with that buffer size. $ printf -- "%s\n" one two | strace -o /tmp/out -e write,read dd status=none bs=1M ; cat /tmp/out one two ... read(0, "one\ntwo\n", 1048576) = 8 write(1, "one\ntwo\n", 8) = 8 read(0, "", 1048576)= 0 +++ exited with 0 +++ It seems to me that using a large buffer size for both read and write would be the most efficient. It can then use the same buffer that data was read into for the output buffer directly. Bob
Re: bash sockets: printf \x0a does TCP fragmentation
Date:Sat, 22 Sep 2018 14:22:19 -0600 From:Bob Proulx Message-ID: <20180922111950901701...@bob.proulx.com> | Primarily a shell script is a command and control program. It is very | good for that purpose. It is typically used for that purpose. That | is the mainstream use and it is very unlikely one will run into | unusual situations there. | | But programming tasks that are much different from command and control | tasks, such as I completely agree with all of that - shells should be used for running other programs, which do the actual work, and only in the most simplistic of cases actually doing stuff themselves. This kind of explosion is what destroyed perl as a useful tool - as originally created by Larry Wall it was very useful, combining the power of string manipulation with regular expressions (for which sed would have been used previously) with the field splitting a floating point calculations from awk, and the i/o and program cotrol normally found in a shell. Then it was set upon by morons who seem to believe that it must be possible to write evenything in whatever is their favourits programming language, and added networking ability (including the ability to format packets of course), and others added threading, and others object orientation, and now all that's left is a giant mess. Every attempt should be made to resist shells moving in the same direction. | > I doubt you can judge on this by just looking at a single line | > of code -- the project has > 18k LoC in bash. That in itself tells me it is probably misguided.While I can imagine tasks that would need that much shell code, they are very very very rare (and most often using auto-generated, and highly repititious, code.) kre ps: f there was actually a desire to use dd to do re-buffering, the correct usage is not to use "bs=" (which simply does read write with a buffer that size) but "obs=" which reads (using ibs if needed, which it would not be here), copies to an output buffer and writes only when that buffer is full (or on EOF on the input).
Re: bash sockets: printf \x0a does TCP fragmentation
I see that you have subscribed now. Awesome! If you and others would be so kind as to list-reply instead of CC'ing me directly that would be great. I read the replies on the mailing list. dirk+b...@testssl.sh wrote: > Bob Proulx wrote: > > You are doing something that is quite unusual. You are using a shell > > script direction on a TCP socket. That isn't very common. > > Do you think there should be a paragraph NOT COMMON where bash sockets > should rather belong to? You actually had not included enough background information to know if you were using the bash built in network implementation or not. You only showed that you had set up fd 5 connected to a network socket. That can happen because, for example, a script was used to service an inetd configuration or similar. It doesn't actually need to be the built in network protocol at all. But now that you have said the above I guess I can assume that you are using the built in implementation. As to whether the documentation should say this or not that is not really practical. There are a godzillian different things that are not typically addressed by writing a shell script. As a practical matter it is impossible to list everything out explicitly. And if one tries then the complaint is that the documentation is so long and detailed that is is unusable due to it. Primarily a shell script is a command and control program. It is very good for that purpose. It is typically used for that purpose. That is the mainstream use and it is very unlikely one will run into unusual situations there. But programming tasks that are much different from command and control tasks, such as your program interacting by TCP with other devices on the network, are not as common. I don't have facts to back that up but I do believe that to be true based upon the way I have seen shell scripts being programmed and used over a long period of time. Of course if you have spent the last 20 years programming network shell scripts then your observations will bias you the other way. :-) > > More > > typically one would use a C program instead. So it isn't surprising > > that you are finding interactions that are not well known. > > Bob, my intention was not to discuss program languages and what is typical > with you or anybody else here. Hmm... Put yourself in our shoes. You stood up on the podium that is this public mailing list and spoke into the megaphone addressing all of us complaining that bash's printf was buggy. But from my perspective printf is behaving as expected. It is designed to deal with line oriented data. It will also deal with binary data if one is careful. But it appears that your application wasn't careful enough and had tripped over some problems. Should we (me!) keep silent about those very obvious problems? It feels obvious to me but apparently not to the author of the above. As has often been said many eyes make all bugs apparent. I was pointing this out to you as a public service. But in response you seem hostile by the language above and below. That isn't encouraging any help. :-( > >> printf -- "$data" >&5 2>/dev/null > > > > Why is stderr discarded? That is almost always bad because it > > discards any errors that might occur. You probably shouldn't do this.> > > What happens if $data contains % format strings? What happens if the > > format contains a sequence such as \c? This looks problematic. This > > is not a safe programming proctice. > > I doubt you can judge on this by just looking at a single line > of code -- the project has > 18k LoC in bash. That single line of code was problematic just by itself standing alone without the rest of the program around it. That is independent of anything the rest of the program might contain. However if you would like to pass sections of the rest of the program through the help-bash mailing list then I am sure the group there would help improve the quality of it. > Github is the place to discuss and do PRs for our project. No. Sorry. You came here to this mailing list. Therefore this is the place to discuss it. Please put yourself in my shoes. If the case were reversed and I came over to Github and then stated that Github was not the place for the discussion but that you needed to set up email and come over to my mailing list and discuss it there instead. How would you feel? I had come into your house, asked you for help, then wanted you to go elsewhere? How would you feel? I can tell you that I do not feel very welcome by it. Also remember that Github is a non-free service. That is free as in freedom, not free as in beer. The free in Free Software. Or in this case the opposite of it being non-free. We try not to use software that does not respect our freedoms nor ask others to do so either. It's a philosophy of life thing. I hope you will understand. > >> If there's a workaround, please let me know. (tried to add "%b" with no > >> effect). Otherwise I
Segmentation fault in restore_tilde (bashline.c)
Found via fuzzing (of `read -e') using AFL, To reproduce: In a shell with `emacs' readline mode, type: ~/ e.g. (gdb) r The program being debugged has been started already. Start it from the beginning? (y or n) y Starting program: /home/dualbus/src/gnu/bash/bash [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". bash-5.0$ ~/ Program received signal SIGSEGV, Segmentation fault. __memset_avx2_unaligned_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:206 206 ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S: No such file or directory. (gdb) bt #0 __memset_avx2_unaligned_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:206 #1 0x5575518e in internal_malloc (n=18446744073709551609, file=0x55762db7 "bashline.c", line=3015, flags=1) at malloc.c:870 #2 0x557568dc in sh_malloc (bytes=18446744073709551609, file=0x55762db7 "bashline.c", line=3015) at malloc.c:1302 #3 0x5569bf1f in sh_xmalloc (bytes=18446744073709551609, file=0x55762db7 "bashline.c", line=3015) at xmalloc.c:185 #4 0x556854f6 in restore_tilde (val=0x55990aa8 "~/.swt", directory_part=0x55990ac8 "~", '\"' ) at bashline.c:3015 #5 0x556855fe in maybe_restore_tilde (val=0x55990aa8 "~/.swt", directory_part=0x55990ac8 "~", '\"' ) at bashline.c:3031 #6 0x55682940 in command_word_completion_function (hint_text=0x55990a88 "~", '\"' , "/", state=0) at bashline.c:2157 #7 0x55718e12 in rl_completion_matches (text=0x55990a88 "~", '\"' , "/", entry_function=0x55680f3c ) at complete.c:2185 #8 0x55714eb3 in gen_completion_matches (text=0x55990a88 "~", '\"' , "/", start=0, end=18, our_func=0x55680f3c , found_quote=2, quote_char=0) at complete.c:1228 #9 0x55718352 in rl_complete_internal (what_to_do=9) at complete.c:2013 #10 0x55688187 in bash_specific_completion (what_to_do=9, generator=0x55680f3c ) at bashline.c:3812 #11 0x55687a6c in bash_complete_command_internal (what_to_do=9) at bashline.c:3690 #12 0x55687958 in bash_complete_command (ignore=1, ignore2=33) at bashline.c:3662 #13 0x55704df2 in _rl_dispatch_subseq (key=33, map=0x557cec60 , got_subseq=0) at readline.c:852 #14 0x55705793 in _rl_dispatch_subseq (key=27, map=0x557cdc40 , got_subseq=0) at readline.c:986 #15 0x557049e5 in _rl_dispatch (key=1437245440, map=0x557cdc40 ) at readline.c:798 #16 0x5570432d in readline_internal_char () at readline.c:632 #17 0x5570441a in readline_internal_charloop () at readline.c:659 #18 0x55704470 in readline_internal () at readline.c:671 #19 0x5570374f in readline (prompt=0x5598f3e8 "bash-5.0$ ") at readline.c:377 #20 0x555c5b2a in yy_readline_get () at ./parse.y:1476 #21 0x555c5945 in yy_getc () at ./parse.y:1409 #22 0x555c7648 in shell_getc (remove_quoted_newline=1) at ./parse.y:2334 #23 0x555ca34c in read_token (command=0) at ./parse.y:3230 #24 0x555c8e07 in yylex () at ./parse.y:2744 #25 0x555bfef6 in yyparse () at y.tab.c:1823 #26 0x555bf299 in parse_command () at eval.c:303 #27 0x555bf4a9 in read_command () at eval.c:347 #28 0x555bea6b in reader_loop () at eval.c:143 #29 0x555ba3e7 in main (argc=1, argv=0x7fffe7d8, env=0x7fffe7e8) at shell.c:805 (gdb) frame 4 #4 0x556854f6 in restore_tilde (val=0x55990aa8 "~/.swt", directory_part=0x55990ac8 "~", '\"' ) at bashline.c:3015 3015 ret = (char *)xmalloc (dl2 + 2 + l); (gdb) p dl2 $3 = 1 (gdb) p l $4 = -10 (gdb) p vl $5 = 6 (gdb) p xl $6 = 17 (gdb) p rl_line_buffer $8 = 0x55a40e08 '\337' , ... Due to the expansions performed in restore_tilde, xl ends up being larger than vl, and thus, we end up requesting a negative amount of memory. I kind of understand what the problem is, but I don't really know how to fix it.
Re: bash sockets: printf \x0a does TCP fragmentation
On 9/22/18 6:38 AM, Ilkka Virta wrote: > On 22.9. 02:34, Chet Ramey wrote: >> Newline? It's probably that stdout is line-buffered and the newline causes >> a flush, which results in a write(2). > > Mostly out of curiosity, what kind of buffering logic does Bash (or the > builtin printf in particular) use? Bash sets stdout and stderr to line buffering. It's done this since the early 1.x days. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: bash sockets: printf \x0a does TCP fragmentation
On 9/22/18 4:21 AM, dirk+b...@testssl.sh wrote: > > > On 9/22/18 1:34 AM, Chet Ramey wrote: >> On 9/21/18 4:13 PM, dirk+b...@testssl.sh wrote: >>> >>> Hello there, >>> >>> we discovered a strange phenomenon in the project testssl.sh: >>> >>> After opening a TCP socket with a fd (here: 5), when writing to it, >>> it seems that >>> >>> printf -- "$data" >&5 2>/dev/null >>> >>> does not do what it is intended. "$data" is a ClientHello like >>> >>> '\x16\x03\x01\x2\x00\x01\x00\x1\xfc\x03\x03\x54\x51\x1e\x7a\xde\xad\xbe\xef\x31\x33\x07\x00\x00\x00\x00\x00\xcf\xbd\x39\x04\xcc\x16\x0a\...' >>> >>> Each \x0a like the last one causes a new TCP fragment to begin which can be >>> easily >>> spotted when using wireshark while running e.g. >> >> Newline? It's probably that stdout is line-buffered and the newline causes >> a flush, which results in a write(2). > > Anything one can do on the level of bash or non-syscall land? What about > ulimit -b ? Bash sets stdout and stderr to be line buffered. It's done this forever, long before I added /dev/tcp and socket support. I'd expect this to have shown up long ago. I doubt stdio attempts to divine the buffer size of the underlying file descriptor. Changing the socket buffer size would probably not be reflected into stdio's buffering behavior. > >>> If there's a workaround, please let me know. (tried to add "%b" with no >>> effect). Otherwise I believe it's a bug. >> >> How? Does the emitted output not correspond to what's passed to printf >> in some way? > > "\x0a" is a legitimate byte which is send from time to time over the > socket. It happens if the record layer is e.g. 522 bytes (\x02\x0a), if > a standard cipher is included in the handshake like (\xc0\x0a) or DES-CBC3-SHA > (\x00\x0a) ECDHE-ECDSA-AES256-SHA or at any other occasion. That's the thing: 0x0a is a legitimate input byte and is faithfully reproduced in the output. It's a bug if the output doesn't reflect the input. We're talking about side effects you don't want, which happen at a different layer. > > Everything works as expected and like a charm for years now -- the only thing > isthe > underlying TCP fragmentation which is caused as a side effect by sending > \x0a. So the newline has been an issue all along? > As indicated a few servers under certain condition can't cope with it asif > the TCP > first segment is too small they don't treat this as ClientHello > and just drop the packet, see thread @ > https://github.com/drwetter/testssl.sh/pull/1113, specifically the hint wrt > https://support.f5.com/csp/article/K53322151 . > > My stance is simply if I use in the internal feature of bash for TCP socket > programming I didn't expect to have side effects like this. These side effects are well below bash's layer of abstraction. You can unbuffer the output by using some kind of unbuffering program (as others have noted, `dd' is probably the most portable) and avoid them. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: bash sockets: printf \x0a does TCP fragmentation
On 9/22/18 12:30 PM, Ilkka Virta wrote: > The coreutils printf seems to output 'foo\nbar\n' as a single write, though > (unless > it goes to the terminal, so the usual stdio buffering), so you might be able > to use > that. thx. Might be not that portable but we'll see. > In any case, if a TCP endpoint cares about getting full data packets within a > single > segment, I'd say it's broken. fully agree. But unfortunately it just comforts us :-) Keep in mind that the purpose of the tool is testing and if due to a bug it can't do that, were the ones being blamed or we need to do really strange workarounds to avoid '\x0a' in the first 8 bytes. Dirk
Re: bash sockets: printf \x0a does TCP fragmentation
On 9/22/18 12:38 PM, Ilkka Virta wrote: > On 22.9. 02:34, Chet Ramey wrote: >> Newline? It's probably that stdout is line-buffered and the newline causes >> a flush, which results in a write(2). > > Mostly out of curiosity, what kind of buffering logic does Bash (or the > builtin > printf in particular) use? It doesn't seem to be the usual stdio logic where > you get > line-buffering if printing to a terminal and block buffering otherwise. I get > a > distinct write per line even if the stdout of Bash itself is redirected to say > /dev/null or a pipe: > > $ strace -etrace=write bash -c 'printf "foo\nbar\n"' > /dev/null > write(1, "foo\n", 4) = 4 > write(1, "bar\n", 4) = 4 > +++ exited with 0 +++ Oh. But thanks anyway! coreutils in fact does it in one shot as you indicated. Dirk
Re: bash sockets: printf \x0a does TCP fragmentation
On 22.9. 02:34, Chet Ramey wrote: Newline? It's probably that stdout is line-buffered and the newline causes a flush, which results in a write(2). Mostly out of curiosity, what kind of buffering logic does Bash (or the builtin printf in particular) use? It doesn't seem to be the usual stdio logic where you get line-buffering if printing to a terminal and block buffering otherwise. I get a distinct write per line even if the stdout of Bash itself is redirected to say /dev/null or a pipe: $ strace -etrace=write bash -c 'printf "foo\nbar\n"' > /dev/null write(1, "foo\n", 4)= 4 write(1, "bar\n", 4)= 4 +++ exited with 0 +++ -- Ilkka Virta / itvi...@iki.fi
Re: bash sockets: printf \x0a does TCP fragmentation
On 22.9. 12:50, dirk+b...@testssl.sh wrote: cat has a problem with binary chars, right? And: see below. No, it just loops with read() and write(), it shouldn't touch any of the bytes (except for cat -A and such). But it probably doesn't help in coalescing the write blocks, it's likely to just write() whatever it gets immediately. And you can't really solve the issue at hand by piping to any intermediate program, as that program couldn't know how long to buffer the input. Unless you use something that buffers for a particular amount of time, which of course causes a unnecessary delay. The coreutils printf seems to output 'foo\nbar\n' as a single write, though (unless it goes to the terminal, so the usual stdio buffering), so you might be able to use that. In any case, if a TCP endpoint cares about getting full data packets within a single segment, I'd say it's broken. -- Ilkka Virta / itvi...@iki.fi
Re: bash sockets: printf \x0a does TCP fragmentation
On 9/22/18 7:30 AM, Bob Proulx wrote: > dirk+b...@testssl.sh wrote: >> we discovered a strange phenomenon in the project testssl.sh: > > You are doing something that is quite unusual. You are using a shell > script direction on a TCP socket. That isn't very common. Do you think there should be a paragraph NOT COMMON where bash sockets should rather belong to? > More > typically one would use a C program instead. So it isn't surprising > that you are finding interactions that are not well known. Bob, my intention was not to discuss program languages and what is typical with you or anybody else here. >> printf -- "$data" >&5 2>/dev/null > > Why is stderr discarded? That is almost always bad because it > discards any errors that might occur. You probably shouldn't do this.> > What happens if $data contains % format strings? What happens if the > format contains a sequence such as \c? This looks problematic. This > is not a safe programming proctice. I doubt you can judge on this by just looking at a single line of code -- the project has > 18k LoC in bash. Github is the place to discuss and do PRs for our project. >> If there's a workaround, please let me know. (tried to add "%b" with no >> effect). Otherwise I believe it's a bug. > > You can re-block the output stream using other tools such as 'cat' or > 'dd'. Since you are concerned about block size then perhaps dd is the > better of the two. > > | cat cat has a problem with binary chars, right? And: see below. > Or probably better: > > | dd status=none bs=1M > > Or use whatever block size you wish. The 'dd' program will read the > input into its buffer and then output that block of data all in one > write(2). That seems to be what you are wanting. We actually use dd to read from the socket. Of course we could use writing to it as well -- at a certain point of time. Still, a prerequisite would be that printf is the culprit and not how bash + libs do sockets. > P.S. You can possibly use the 'stdbuf' command to control the output > buffering depending upon the program. > > info stdbuf That could be an option, thanks. Need to check though whether a) it doesn't fragment then -- not sure while reading it b) it's per default available on every platform supported by testssl.sh. Cheers, Dirk
Re: bash sockets: printf \x0a does TCP fragmentation
On 9/22/18 1:34 AM, Chet Ramey wrote: > On 9/21/18 4:13 PM, dirk+b...@testssl.sh wrote: >> >> Hello there, >> >> we discovered a strange phenomenon in the project testssl.sh: >> >> After opening a TCP socket with a fd (here: 5), when writing to it, >> it seems that >> >> printf -- "$data" >&5 2>/dev/null >> >> does not do what it is intended. "$data" is a ClientHello like >> >> '\x16\x03\x01\x2\x00\x01\x00\x1\xfc\x03\x03\x54\x51\x1e\x7a\xde\xad\xbe\xef\x31\x33\x07\x00\x00\x00\x00\x00\xcf\xbd\x39\x04\xcc\x16\x0a\...' >> >> Each \x0a like the last one causes a new TCP fragment to begin which can be >> easily >> spotted when using wireshark while running e.g. > > Newline? It's probably that stdout is line-buffered and the newline causes > a flush, which results in a write(2). Anything one can do on the level of bash or non-syscall land? What about ulimit -b ? >> If there's a workaround, please let me know. (tried to add "%b" with no >> effect). Otherwise I believe it's a bug. > > How? Does the emitted output not correspond to what's passed to printf > in some way? "\x0a" is a legitimate byte which is send from time to time over the socket. It happens if the record layer is e.g. 522 bytes (\x02\x0a), if a standard cipher is included in the handshake like (\xc0\x0a) or DES-CBC3-SHA (\x00\x0a) ECDHE-ECDSA-AES256-SHA or at any other occasion. Everything works as expected and like a charm for years now -- the only thing isthe underlying TCP fragmentation which is caused as a side effect by sending \x0a. As indicated a few servers under certain condition can't cope with it asif the TCP first segment is too small they don't treat this as ClientHello and just drop the packet, see thread @ https://github.com/drwetter/testssl.sh/pull/1113, specifically the hint wrt https://support.f5.com/csp/article/K53322151 . My stance is simply if I use in the internal feature of bash for TCP socket programming I didn't expect to have side effects like this. Thx, Dirk
