On Mon, Apr 3, 2023 at 11:47 PM Paul Eggert <egg...@cs.ucla.edu> wrote: > On 2023-04-03 20:30, Jim Meyering wrote: > > have you seen justification > > (other than for compatibility with some other tool or language) for > > allowing \d to match non-ASCII by default, in spite of the risks? > > In the example Ævar supplied in <https://bugs.gnu.org/60690>, my > impression was that it was better when \d matched non-ASCII digits. That > is, in a UTF-8 locale it's better when \d finds matches in these lines: > > >> > git-gui/po/ja.po:"- 第１行: 何をしたか、を１行で要約。\n" > >> > git-gui/po/ja.po:"- 第２行: 空白\n" > > because they contain the Japanese digits "１" and "２". This was the only > example I recall being given.
Before it was unintentionally enabled in grep-3.9, lines like that have never been matched by grep -P's '\d'. By relaxing \d, we'd weaken any application that uses say grep -P '^\d+$' to perform input validation intending to ensure that some input is all ASCII digits. It's not a big stretch to imagine that some downstream processor of that "verified" data is not prepared to deal with multi-byte digits. > Also, I find it odd that grep -P '^[\w\d]*$' matches lines containing > any sort of Arabic word characters, but it rejects lines containing > Arabic digits like "٣" that are perfectly reasonable in Arabic-language > text. I also find it odd that [\d] and [[:digit:]] mean different things. > > There are arguments on the other side, otherwise we wouldn't be having > this discussion. And it's true that grep -P '\d' formerly rejected > Arabic digits (though it's also true that grep -P '\w' formerly rejected > Arabic letters...). Still, the cure's oddness and incompatibility with > Git, Perl, etc. appears to me to be worse than the disease of dealing > with grep -P invocations that need to use [0-9] or LC_ALL="C" anyway if > they want to be portable to any program other than GNU grep. I'm primarily concerned about not introducing a persistent regression in how GNU grep's -P '\d' works in multibyte locales. The corner cases you mention do matter, of course, but are far less likely to matter in practice.
