Hello, VTE co-developer here. (VTE, a.k.a. libvte is the terminal emulation widget behind GNOME Terminal and several other terminal emulator apps.)
Note that this very bug is also filed at https://savannah.gnu.org/bugs/?36831 . --- The "background color erase" ("bce") feature of most terminals suffers from a fundamental design flaw. You cannot "just print" some piece of text containing non-default background color, and expect it to appear on the screen correctly. By "just printing" what I mean is that you don't go into the business of querying the terminal size, tracking the cursor position, knowing which characters will be double wide, computing where the line would wrap to the next one, etc. I'm only considering simple apps that print messages using printf() or similar, like grep does. The problem is that if you're at the bottom of the screen and autowrapping occurs, the entire new line is added with the currently active background color, which might not be the terminal's default background color. If that new line isn't filled with text entirely then the rest remains with that color. The workaround applied by "grep" by default is to switch back to the default background color and emit the sequence that clears to the end of the line. This, while fixes the background color issue (apart from a possible temporary flicker if the terminal doesn't process grep's output in a single step and updates its display in the middle), introduces a different bug: possibly chopping off one letter of output. For the sake of explaining it, assume 80 columns numbered from 1 to 80. The way terminals behave is: After printing 78 chars the cursor is in column 79. After printing one more char the cursor is in column 80. After printing the 80th char in the 80th column, the cursor remains in column 80 (over this newly printed character) (*), but a special flag is set, in Xterm's source I believe it's called "do_wrap", it denotes that the next character will wrap to the next line. ((*) VTE hides the cursor in this case, but it's irrelevant for this bug.) Some operations, like handling a backspace, or clearing to the end of the line unfortunately don't take this "do_wrap" flag into account. That is, after printing 80 characters and being in the "do_wrap" state, the "clear to end of line" escape sequence only checks that the cursor is in column 80, therefore clears everything in column 80 and its right (i.e. column 80 only). And it also clears the "do_wrap" flag. Perhaps about 8-10 years ago I talked about this conceptual problem with the developer of xterm and ncurses, and he wasn't interested in doing anything about the situation. I, however, still was. --- In VTE I made two modifications, i.e. at two places we intentionally deviate from the standard behavior to fix this nonsense and make grep work as expected. One of the workarounds is for when grep is asked to emit that clear-to-eol, one is for when it isn't. In https://bugzilla.gnome.org/show_bug.cgi?id=740789 (2014) I changed the behavior of the clear-to-eol escape sequence. If in that "do_wrap" state then it doesn't clear the last character, nor this "do_wrap" state, it becomes no-op. In https://bugzilla.gnome.org/show_bug.cgi?id=754596 (2015-2017) I modified the bce feature to fill the newly appearing line with the current background only if scrolling occurs due to an explicit newline or escape sequence, but not when it occurs to text autowrapping, in which case we use the terminal's default background color. Neither of these changes cause any problem in real life that we're aware of. The first iteration of the bce change in 2015 had some problem that I fixed in 2017. We haven't received any bugreport since then about either of these intentional deviations from the standard. The result is that grep's output renders correctly in VTE, no matter if you use its "ne" feature or not. However, the goal should be to fix it in all terminals out there. --- So, what should "grep" do? Its maintainers, or anyone interested in pushing this through (it's not going to be me, sorry) could talk to Xterm's and other terminals' authors to change the default, or find a solution. They could point out that these two workarounds in VTE have turned out to be fine, and suggest to change the default behavior to these. (However, I wouldn't expect everyone to agree. I'd expect that many of these folks would insist on strictly adhering to the standards, rather than acknowledging that the standard is conceptually broken and being okay with deviating from it.) As for clear-to-eol, I think urxvt kept the official behavior, but added the same sequence with numeric parameter 3, i.e. \e[3K, to perform the slightly modified (fixed) behavior. Could be a reasonable route to go down. As for the immediate future, i.e. assuming that grep maintainers don't want to or don't succeed in driving such a change across leading / reference terminals (probably most notably Xterm) and then wait for years for this to become widespread, the question is: Would you rather have a wrong background color, or a letter missing? I'm puzzled how upstream grep, for many-many years, has chosen to possibly chop off a letter as its default behavior, I firmly believe that it's by far the worse of the two. Or start thinking about brand new workarounds. Like Vincent's idea with a space + backspace. Or space + backspace + clear-to-eol. Or space + clear-to-eol. These _could_ be robust against issues at the right margin. Obviously these brainstorming ideas I'm randomly throwing in now have to be tested carefully across many popular terminals. Maybe with both possible values of Xterm's reverse wraparound, maybe with only the default because the other value causes ncurses corruptions in Xterm (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067679#10) so you can't really count on that anyway. Also, I'm not sure if currently grep emits clear-to-eol wherever the background color changes within a line, or at the end of the line after reverting to the terminal's default. Workarounds involving space / backspace chars probably only work with the latter approach. Also, a minor issue, but could be important / annoying to some users: printing space / backspace can modify how trailing spaces get copy-pasted in some terminals. It would be nice to copy-paste grep's output exactly as it appeared in the file. But if that's the price to pay to fix the other two problems, probably it's still the least bad. e.
