bug#18892: few test failure with 'grep-2.20.72-d512'

2014-10-29 Thread Assaf Gordon 

Hello,

On 10/29/2014 02:29 PM, Jim Meyering wrote:

Thanks to many fixes and improvements by Paul Eggert and Norihiro Tanaka,
here is a pre-release snapshot:

grep snapshot:
   http://meyering.net/grep/grep-ss.tar.xz  1.2 MB
   http://meyering.net/grep/grep-ss.tar.xz.sig
   http://meyering.net/grep/grep-2.20.72-d512.tar.xz



On Debian 7.6 (amd64, gcc 4.7.2) and Ubuntu 14.04.1 (amd64, gcc 4.8.2), these 
fail:
  XFAIL: triple-backref
  FAIL: word-multibyte
Attached is the debian76.test-suite.log from Debian (Ubuntu seemed identical).

---

On OpenBSD5.5 (amd64, gcc 4.2.1) these failed:
  XFAIL: equiv-classes
  FAIL: word-multibyte
Attached is the openbsd55.test-suite.log.

---

On CentOS 7 (amd64, gcc 4.8.2),
   GNU Hurd (x86, gcc 4.8.2),
   OpenSUSE 13.1 (amd64, gcc 4.8.1)
this fails:
  XFAIL: triple-backref

Attached are the logs.

---

On FreeBSD 9.3 (amd64, gcc 4.2.1), this fails:
===
XFAIL: equiv-classes


C3A9
XFAIL equiv-classes (exit status: 1)
===

---

On NetBSD 6.1.4, these fail:
  XFAIL: equiv-classes
  FAIL: symlink
  FAIL: word-multibyte
Attached is netbsd614.test-suite.log

---

No failures on:
  FreeBSD 10 (amd64, clang 3.3)
  CentOS 6.5 (amd64, gcc 4.4.7)
  Fedora 20 (amd64, gcc 4.8.3)


Regards,
 - Assaf

=
   GNU grep 2.20.72-d512: tests/test-suite.log
=

# TOTAL: 85
# PASS:  63
# SKIP:  20
# XFAIL: 1
# FAIL:  1
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

XFAIL: triple-backref
=

+ initial_cwd_=/tmp/grep-2.20.72-d512/tests
+ fail=0
+ testdir_prefix_
+ printf gt
+ pfx_=gt
+ mktempd_ /tmp/grep-2.20.72-d512/tests gt-triple-backref.
+ destdir_=/tmp/grep-2.20.72-d512/tests
+ template_=gt-triple-backref.
+ MAX_TRIES_=4
+ unset TMPDIR
+ d=/tmp/grep-2.20.72-d512/tests/gt-triple-backref.4WEh
+ test -d /tmp/grep-2.20.72-d512/tests/gt-triple-backref.4WEh
+ ls -dgo /tmp/grep-2.20.72-d512/tests/gt-triple-backref.4WEh
+ tr S -
+ perms=drwx-- 2 4096 Oct 29 15:50 /tmp/grep-2.20.72-d512/tests/gt-triple-backref.4WEh
+ test 0 = 0
+ echo /tmp/grep-2.20.72-d512/tests/gt-triple-backref.4WEh
+ return
+ test_dir_=/tmp/grep-2.20.72-d512/tests/gt-triple-backref.4WEh
+ cd /tmp/grep-2.20.72-d512/tests/gt-triple-backref.4WEh
+ gl_init_sh_nl_=

+ IFS= 	

+ expr 1 + 128
+ eval trap 'Exit 129' 1
+ trap Exit 129 1
+ expr 2 + 128
+ eval trap 'Exit 130' 2
+ trap Exit 130 2
+ expr 3 + 128
+ eval trap 'Exit 131' 3
+ trap Exit 131 3
+ expr 13 + 128
+ eval trap 'Exit 141' 13
+ trap Exit 141 13
+ expr 15 + 128
+ eval trap 'Exit 143' 15
+ trap Exit 143 15
+ trap remove_tmp_ 0
+ path_prepend_ ../src
+ test 1 != 0
+ path_dir_=../src
+ abs_path_dir_=/tmp/grep-2.20.72-d512/tests/../src
+ PATH=/tmp/grep-2.20.72-d512/tests/../src:/tmp/grep-2.20.72-d512/src:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ create_exe_shims_ /tmp/grep-2.20.72-d512/tests/../src
+ return 0
+ shift
+ test 0 != 0
+ export PATH
+ failures=0
+ cat
+ gcc -std=gnu11 -c glibc.c
+ echo a
+ fail=0
+ grep -E (.?)(.?)(.?)\3\2\1 in
grep: regexec.c:1386: pop_fail_stack: Assertion `num = 0' failed.
Aborted
+ fail=1
+ compare out in
+ compare_dev_null_ out in
+ test 2 = 2
+ test xout = x/dev/null
+ test xin = x/dev/null
+ return 2
+ compare_ out in
+ diff -u out in
--- out	2014-10-29 15:50:47.689879001 -0400
+++ in	2014-10-29 15:50:47.689879001 -0400
@@ -0,0 +1 @@
+a
+ fail=1
+ Exit 1
+ set +e
+ exit 1
+ exit 1
+ remove_tmp_
+ __st=1
+ cleanup_
+ :
+ cd /tmp/grep-2.20.72-d512/tests
+ chmod -R u+rwx /tmp/grep-2.20.72-d512/tests/gt-triple-backref.4WEh
+ rm -rf /tmp/grep-2.20.72-d512/tests/gt-triple-backref.4WEh
+ exit 1
XFAIL triple-backref (exit status: 1)

FAIL: word-multibyte


+ initial_cwd_=/tmp/grep-2.20.72-d512/tests
+ fail=0
+ testdir_prefix_
+ printf gt
+ pfx_=gt
+ mktempd_ /tmp/grep-2.20.72-d512/tests gt-word-multibyte.
+ destdir_=/tmp/grep-2.20.72-d512/tests
+ template_=gt-word-multibyte.
+ MAX_TRIES_=4
+ unset TMPDIR
+ d=/tmp/grep-2.20.72-d512/tests/gt-word-multibyte.JgtS
+ test -d /tmp/grep-2.20.72-d512/tests/gt-word-multibyte.JgtS
+ ls -dgo /tmp/grep-2.20.72-d512/tests/gt-word-multibyte.JgtS
+ tr S -
+ perms=drwx-- 2 4096 Oct 29 15:50 /tmp/grep-2.20.72-d512/tests/gt-word-multibyte.Jgt-
+ test 0 = 0
+ echo /tmp/grep-2.20.72-d512/tests/gt-word-multibyte.JgtS
+ return
+ test_dir_=/tmp/grep-2.20.72-d512/tests/gt-word-multibyte.JgtS
+ cd /tmp/grep-2.20.72-d512/tests/gt-word-multibyte.JgtS
+ gl_init_sh_nl_=

+ IFS= 	

+ expr 1 + 128
+ eval trap 'Exit 129' 1
+ trap Exit 129 1
+ expr 2 + 128
+ eval trap 'Exit 130' 2
+ trap Exit 130 2
+ expr 3 + 128
+ eval trap 'Exit 131' 3
+ trap Exit 131 3
+ expr 13 + 128
+ eval trap 'Exit 141' 13
+ trap Exit 141 13
+ expr 15 + 128
+ eval trap 'Exit 143' 15
+ trap Exit 143 15
+ trap remove_tmp_ 0
+ path_prepend_ ../src
+ test 1 != 0
+ path_dir_=../src
+ abs_path_dir_=/tmp/grep-2.20.72-d512/tests/../src
+

bug#18892: few test failure with 'grep-2.20.72-d512'

2014-11-01 Thread Assaf Gordon 

Hello,

On 11/01/2014 07:06 PM, Jim Meyering wrote:

On NetBSD 6.1.4, these fail:
   XFAIL: equiv-classes
   FAIL: symlink
   FAIL: word-multibyte



...


However, I'll need more information to understand the symlink failure.
Normally the log file contains verbose output information (resulting
from init.sh's set -x), but on your NetBSD system, it appears not to
have been enabled, perhaps because that system's /bin/sh is
inadequate.  Can you rerun the tests using bash as your shell?  That
may be enough to evoke the log information that will highlight which
part(s) of the symlink test actually fail.


It turned out not to be directly a shell issue (I tried with forcing shell to 
bash), but a BSD make vs GNU make.
using gmake did use the correct shell, and produced a detailed log, which is 
attached.

If I understand correctly, the failure is triggered in line 177 in the attached 
log.
running grep -r '^' was supposed to return zero, but it returns 2.

I disabled the removal of the temporary directory, and so was able to reproduce 
manually:

$ uname -a
NetBSD  6.1.4 NetBSD 6.1.4 (GENERIC) amd64

$ pwd
/tmp/grep-2.20.72-d512/tests/gt-symlink.1F01
$ ls -lR
total 20
drwxr-xr-x  2 miles  wheel  512 Nov  2 00:55 dir
-rw-r--r--  1 miles  wheel   30 Nov  2 00:55 exp
-rw-r--r--  1 miles  wheel   30 Nov  2 00:55 grepout
-rw-r--r--  1 miles  wheel   30 Nov  2 00:55 out
-rw-r--r--  1 miles  wheel   30 Nov  2 00:55 out-t

./dir:
total 8
-rw-r--r--  1 miles  wheel  2 Nov  2 00:55 a
-rw-r--r--  1 miles  wheel  2 Nov  2 00:55 b
lrwxr-xr-x  1 miles  wheel  1 Nov  2 00:55 c - a
lrwxr-xr-x  1 miles  wheel  1 Nov  2 00:55 d - .
lrwxr-xr-x  1 miles  wheel  8 Nov  2 00:55 e - dangling

$ cd dir
$ pwd
/tmp/grep-2.20.72-d512/tests/gt-symlink.1F01/dir

$ ../../../src/grep -r '^' ; echo $?
a:a
b:b
../../../src/grep: c: Inappropriate file type or format
../../../src/grep: d: Inappropriate file type or format
../../../src/grep: e: Inappropriate file type or format
2


For comparison,
On Linux (Ubuntu 14.04.1 amd64), the exit code is zero, somehow c,d,e are 
ignored:

===
$ uname -a
Linux  3.13.0-37-generic #64-Ubuntu SMP Mon Sep 22 21:28:38 UTC 2014 x86_64 
GNU/Linux
$ pwd
/tmp/grep/grep-2.20.72-d512/tests/gt-symlink.3Zym
$ ls -lR
.:
total 20
drwxrwxr-x 2 gordon gordon 4096 Nov  1 21:18 dir
-rw-rw-r-- 1 gordon gordon   30 Nov  1 21:18 exp
-rw-rw-r-- 1 gordon gordon   30 Nov  1 21:18 grepout
-rw-rw-r-- 1 gordon gordon   30 Nov  1 21:18 out
-rw-rw-r-- 1 gordon gordon   30 Nov  1 21:18 out-t

./dir:
total 8
-rw-rw-r-- 1 gordon gordon 2 Nov  1 21:18 a
-rw-rw-r-- 1 gordon gordon 2 Nov  1 21:18 b
lrwxrwxrwx 1 gordon gordon 1 Nov  1 21:18 c - a
lrwxrwxrwx 1 gordon gordon 1 Nov  1 21:18 d - .
lrwxrwxrwx 1 gordon gordon 8 Nov  1 21:18 e - dangling

$ cd dir
$ pwd
/tmp/grep/grep-2.20.72-d512/tests/gt-symlink.3Zym/dir

$ ../../../src/grep -r '^' ; echo $?
b:b
a:a
0
===


 - Assaf
=
   GNU grep 2.20.72-d512: tests/test-suite.log
=

# TOTAL: 85
# PASS:  58
# SKIP:  24
# XFAIL: 1
# FAIL:  2
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: symlink
=

++ initial_cwd_=/tmp/grep-2.20.72-d512/tests
++ fail=0
+++ testdir_prefix_
+++ printf gt
++ pfx_=gt
+++ mktempd_ /tmp/grep-2.20.72-d512/tests gt-symlink.
+++ case $# in
+++ destdir_=/tmp/grep-2.20.72-d512/tests
+++ template_=gt-symlink.
+++ MAX_TRIES_=4
+++ case $destdir_ in
+++ case $template_ in
 unset TMPDIR
+++ d='/tmp/-p.009117aa
gt-symlink.117c'
+++ fail=1
+++ case $d in
+++ fail=1
+++ test -d '/tmp/-p.009117aa
gt-symlink.117c'
+++ fail=1
 ls -dgo '/tmp/-p.009117aa
gt-symlink.117c'
 tr S -
+++ perms=
+++ case $perms in
+++ fail=1
+++ test 1 = 0
 echo gt-symlink.
 sed 's/XX*$//'
+++ base_template_=gt-symlink.
 echo gt-symlink.
 wc -c
+++ template_length_='  16'
 echo gt-symlink.
 wc -c
+++ nx_='  12'
 expr 16 - 12
+++ nx_=4
+++ err_=
+++ i_=1
+++ :
 rand_bytes_ 4
 n_=4
 chars_=abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789
 dev_rand_=/dev/urandom
 test -r /dev/urandom
 dd ibs=4 count=1 if=/dev/urandom
 LC_ALL=C
 tr -c abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789 01234567abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789
 return
+++ X_=1F01
+++ candidate_dir_=/tmp/grep-2.20.72-d512/tests/gt-symlink.1F01
 mkdir -m 0700 /tmp/grep-2.20.72-d512/tests/gt-symlink.1F01
+++ err_=
+++ echo /tmp/grep-2.20.72-d512/tests/gt-symlink.1F01
+++ return
++ test_dir_=/tmp/grep-2.20.72-d512/tests/gt-symlink.1F01
++ cd /tmp/grep-2.20.72-d512/tests/gt-symlink.1F01
++ gl_init_sh_nl_='
'
++ IFS=' 	
'
++ for sig_ in 1 2 3 13 15
+++ expr 1 + 128
++ eval 'trap '\''Exit 129'\'' 1'
+++ trap 'Exit 129' 1
++ for sig_ in 1 2 3 13 15
+++

bug#18892: few test failure with 'grep-2.20.72-d512'

2014-11-02 Thread Assaf Gordon 

Hello,

On 11/01/2014 10:39 PM, Paul Eggert wrote:

Assaf Gordon wrote:



$ ../../../src/grep -r '^' ; echo $?
a:a
b:b
../../../src/grep: c: Inappropriate file type or format
../../../src/grep: d: Inappropriate file type or format
../../../src/grep: e: Inappropriate file type or format
2


What system calls is this 'grep' executing?  What's the output of ktrace and 
kdump?


Attached is the ktrace+tdump output of running the command.

The error appears on line 217 (open() returns -1 for file c),
though similar errors stemming from 'fstat' for files a,b are ignored (lines 
150,191).

- Assaf



netbsd.grp.out.txt.xz
Description: application/xz

bug#19005: [platform-testers] new snapshot available: grep-2.20.83-06900

2014-11-09 Thread Assaf Gordon 

On 11/09/2014 05:30 PM, Assaf Gordon wrote:


On 11/09/2014 04:00 PM, Jim Meyering wrote:

Testing the first snapshot exposed several problems,
and most have been addressed. The sole remaining problem
is the failure of an expensive (not run by default: big-match)
that makes most systems run out of memory. I haven't looked at it yet.


...


Please let us know if you find any problems.




No failures on the following VMs (amd64, tested without PCRE, and only 
non-expensive tests):
  freebsd 9.3
  freebsd 10
  openbsd 5.5
  netbsd 6.1.4

  debian 7.6
  gnewsense 3.1 (=debian 6)
  trisquel 6.0.1 (=ubuntu 12 LTS)
  ubuntu 14.04.1

  opensuse 13.1
  fedora 20
  centos 7

  GNU Hurd/Debian 0.5 (32-bit).



Two cases on less common OSes:

On DilOS 1.3.7, based on Illumos/OpenSolaris, multibyte-white-space fails.
log attached.
From a brief look, perhaps it has something to do with the recent hex printf 
change? not sure.
This is a weird system, as they try to use Illumos kernel with Debian user 
space,
so things could be mis-configured.
Few quick checks (relating to the hex printf implementation):
===
$ bash --version | head -n1
GNU bash, version 4.2.45(2)-release (x86_64-pc-solaris2.11)

$ printf --version

-bash: printf: --: invalid option
printf: usage: printf [-v var] format [arguments]

$ which printf
/usr/gnu/bin/printf

$ /usr/gnu/bin/printf --version | head -n1
printf (GNU coreutils) 8.21

$ which sed
/usr/gnu/bin/sed

$ $(which sed) --version | head -n1
/usr/gnu/bin/sed (GNU sed) 4.2.2
===


On MINIX R3.3.0, symlinks fails, which seems similar to the previous 
NetBSD6.1.4 issue.
Log attached.
I was not able to install Ktrace/Kdump yet.


I will sadly not be able to test more until next weekend.

Regards,
 - Assaf



==
   GNU grep 2.20.83-06900: tests/test-suite.log
==

# TOTAL: 85
# PASS:  62
# SKIP:  21
# XFAIL: 1
# FAIL:  1
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: multibyte-white-space
===

++ initial_cwd_=/tmp/grep-2.20.83-06900.9QfptX/grep-2.20.83-06900/tests
++ fail=0
+++ testdir_prefix_
+++ printf gt
++ pfx_=gt
+++ mktempd_ /tmp/grep-2.20.83-06900.9QfptX/grep-2.20.83-06900/tests gt-multibyte-white-space.
+++ case $# in
+++ destdir_=/tmp/grep-2.20.83-06900.9QfptX/grep-2.20.83-06900/tests
+++ template_=gt-multibyte-white-space.
+++ MAX_TRIES_=4
+++ case $destdir_ in
+++ case $template_ in
 unset TMPDIR
+++ d=/tmp/grep-2.20.83-06900.9QfptX/grep-2.20.83-06900/tests/gt-multibyte-white-space.Kp1p
+++ case $d in
+++ test -d /tmp/grep-2.20.83-06900.9QfptX/grep-2.20.83-06900/tests/gt-multibyte-white-space.Kp1p
 ls -dgo /tmp/grep-2.20.83-06900.9QfptX/grep-2.20.83-06900/tests/gt-multibyte-white-space.Kp1p
 tr S -
 LC_ALL=C
 env -- tr S -
+++ perms='drwx--   2 117 Nov 10 01:12 /tmp/grep-2.20.83-06900.9QfptX/grep-2.20.83-06900/tests/gt-multibyte-white-space.Kp1p'
+++ case $perms in
+++ test 0 = 0
+++ echo /tmp/grep-2.20.83-06900.9QfptX/grep-2.20.83-06900/tests/gt-multibyte-white-space.Kp1p
+++ return
++ test_dir_=/tmp/grep-2.20.83-06900.9QfptX/grep-2.20.83-06900/tests/gt-multibyte-white-space.Kp1p
++ cd /tmp/grep-2.20.83-06900.9QfptX/grep-2.20.83-06900/tests/gt-multibyte-white-space.Kp1p
++ gl_init_sh_nl_='
'
++ IFS=' 	
'
++ for sig_ in 1 2 3 13 15
+++ expr 1 + 128
++ eval 'trap '\''Exit 129'\'' 1'
+++ trap 'Exit 129' 1
++ for sig_ in 1 2 3 13 15
+++ expr 2 + 128
++ eval 'trap '\''Exit 130'\'' 2'
+++ trap 'Exit 130' 2
++ for sig_ in 1 2 3 13 15
+++ expr 3 + 128
++ eval 'trap '\''Exit 131'\'' 3'
+++ trap 'Exit 131' 3
++ for sig_ in 1 2 3 13 15
+++ expr 13 + 128
++ eval 'trap '\''Exit 141'\'' 13'
+++ trap 'Exit 141' 13
++ for sig_ in 1 2 3 13 15
+++ expr 15 + 128
++ eval 'trap '\''Exit 143'\'' 15'
+++ trap 'Exit 143' 15
++ trap remove_tmp_ 0
+ path_prepend_ ../src
+ test 1 '!=' 0
+ path_dir_=../src
+ case $path_dir_ in
+ abs_path_dir_=/tmp/grep-2.20.83-06900.9QfptX/grep-2.20.83-06900/tests/../src
+ case $abs_path_dir_ in
+ PATH=/tmp/grep-2.20.83-06900.9QfptX/grep-2.20.83-06900/tests/../src:/tmp/grep-2.20.83-06900.9QfptX/grep-2.20.83-06900/src:/usr/gnu/bin:/usr/bin:/usr/sbin:/sbin
+ create_exe_shims_ /tmp/grep-2.20.83-06900.9QfptX/grep-2.20.83-06900/tests/../src
+ case $EXEEXT in
+ return 0
+ shift
+ test 0 '!=' 0
+ export PATH
+ require_en_utf8_locale_
+ path_prepend_ .
+ test 1 '!=' 0
+ path_dir_=.
+ case $path_dir_ in
+ abs_path_dir_=/tmp/grep-2.20.83-06900.9QfptX/grep-2.20.83-06900/tests/.
+ case $abs_path_dir_ in
+ PATH=/tmp/grep-2.20.83-06900.9QfptX/grep-2.20.83-06900/tests/.:/tmp/grep-2.20.83-06900.9QfptX/grep-2.20.83-06900/tests/../src:/tmp/grep-2.20.83-06900.9QfptX/grep-2.20.83-06900/src:/usr/gnu/bin:/usr/bin:/usr/sbin:/sbin
+ create_exe_shims_ /tmp/grep-2.20.83-06900.9QfptX/grep-2.20.83-06900/tests/.
+ case $EXEEXT in
+ return 0
+ shift
+ test 0 '!=' 0
+ export PATH
+ case $(get-mb-cur-max en_US.UTF-8) in
++ get-mb-cur-max en_US.UTF-8

bug#19071: new snapshot available: grep-2.20.90-a07a4

2014-11-16 Thread Assaf Gordon 

Hello,

On 11/16/2014 12:07 PM, Jim Meyering wrote:

Here is a third pre-release snapshot.
 http://meyering.net/grep/grep-2.20.90-a07a4.tar.xz

...

No failures on:
  debian 7.6
  gnewsense 3.1 (=debian 6)
  ubuntu14.04.01
  trisquel 6.0.1 (=ubuntu 12 LTS)
  centos 6.5
  centos 7
  openSuSE 13.1
  fedora 20
  freebsd 10
  freebsd 9.3
  openbsd 5.5
  openbsd 5.6
  netbsd 6.1.4
  GNU Hurd/Debian 0.5
  Mac OS X 10.9

Tested on amd64 (except Hurd, i386), without expensive tests, without PCRE.

Minix R3.3.0 and DilOS 1.3.7 fail as described here:
  http://debbugs.gnu.org/cgi/bugreport.cgi?bug=19005#20

On Mac OS X 10.9 with 4GB RAM, the 'big-match' test brings the laptop to its knees, requiring a 
force quit - so I couldn't complete a check-expensive.


- Assaf

bug#22443: new snapshot available: grep-2.22.33-43f6

2016-02-04 Thread Assaf Gordon 

On 02/03/2016 01:07 AM, Jim Meyering wrote:

   http://meyering.net/grep/grep-2.22.33-43f6.tar.xz


No failures on:
  Mac OS X v10.11.3, v10.10.5, v10.9.5
  Ubuntu 14, aarch64
  Fedora 20, ppc64
  Fedora 21, ppc64le
  OpenSolaris 5.10/5.11 on i86pc, suv4u/sub4v

bug#22443: Subject: new snapshot available: grep-2.22.31-8b6a

2016-01-28 Thread Assaf Gordon 
hello,

> On Jan 28, 2016, at 19:36, Jim Meyering  wrote:
> 
>  http://meyering.net/grep/grep-2.22.31-8b6a.tar.xz


Testing from the above tarball, no test failures on the following systems:
   Fedora-21,ppc64le
   Fedora-20,ppc64
   Ubuntu-14,aarch64
   AIX 7.1,power7 
   OpenSolaris 5.10/5.11 on i86pc and suv4u/sub4v
   Mac OS X 10.10.4
   OpenBSD 5.8, 5.7, 5.6, amd64
   FreeBSD 10.1, 9.3, amd64
   NetBSD 6.1.4, amd64
   Debian-7/kFreeBSD-9,amd64

and also no test failures on various GNU/Linux on x86-64:
   Ubuntu-14.04 with gcc,clang,tcc
   Ubuntu-15.04 (also on i686)
   Debian 8.1, 7.6
   Fedora 23, 22, 21
   CentOS 7, 6.5
   openSUSE 42.1, 13.1

==


On NetBSD 7.0,amd64 - no grep tests failures, but one gnulib test failed to 
compile (test-localename).
log attached.



NetBSD70-gnulib-test-error.log
Description: Binary data




On Minix-3.3.0,i386, one test failed ('symlink'). log attached.


Minix330-test-suite.log
Description: Binary data





When building from GIT (not tarball), and "-Werror" is enabled,
building fails on Mac OS X 10.10.4 with:

$ make V=1
...
/Applications/Xcode.app/Contents/Developer/usr/bin/make  all-recursive
Making all in po
make[2]: Nothing to be done for `all'.
Making all in lib
/Applications/Xcode.app/Contents/Developer/usr/bin/make  all-am
depbase=`echo error.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
gcc -DHAVE_CONFIG_H -I. -I..   -I/usr/local/include -W -Wabi -Waddress 
-Wall -Wattributes -Wbad-function-cast -Wbuiltin-macro-redefined -Wcast-align 
-Wchar-subscripts -Wcomment -Wcomments -Wdate-time -Wdeprecated 
-Wdeprecated-declarations -Wdisabled-optimization -Wdiv-by-zero -Wempty-body 
-Wendif-labels -Wenum-compare -Wextra -Wformat-extra-args -Wformat-nonliteral 
-Wformat-security -Wformat-y2k -Wformat-zero-length -Wignored-qualifiers 
-Wimplicit -Wimplicit-function-declaration -Wimplicit-int 
-Wincompatible-pointer-types -Winit-self -Wint-conversion -Wint-to-pointer-cast 
-Winvalid-pch -Wlogical-not-parentheses -Wmain -Wmissing-braces 
-Wmissing-declarations -Wmissing-field-initializers -Wmissing-include-dirs 
-Wmissing-prototypes -Wmultichar -Wnarrowing -Wnested-externs -Wnonnull -Wodr 
-Wold-style-definition -Woverflow -Woverlength-strings -Wpacked -Wparentheses 
-Wpointer-arith -Wpointer-sign -Wpointer-to-int-cast -Wpragmas -Wreturn-type 
-Wsequence-point -Wshadow -Wshift-count-negative -Wshift-count-overflow 
-Wsizeof-array-argument -Wsizeof-pointer-memaccess -Wstrict-aliasing 
-Wstrict-prototypes -Wswitch -Wswitch-bool -Wtrigraphs -Wtype-limits 
-Wuninitialized -Wunknown-pragmas -Wunused -Wunused-function -Wunused-label 
-Wunused-parameter -Wunused-result -Wunused-value -Wunused-variable -Wvarargs 
-Wvariadic-macros -Wvolatile-register-var -Wwrite-strings 
-Wno-missing-field-initializers -Wno-missing-field-initializers 
-Wno-sign-compare -Wno-unused-parameter -Wno-format-nonliteral -Werror -g -O2 
-MT error.o -MD -MP -MF $depbase.Tpo -c -o error.o error.c &&\
mv -f $depbase.Tpo $depbase.Po
error.c:386:12: error: data argument not used by format string 
[-Werror,-Wformat-extra-args]
   file_name, line_number);
   ^
1 error generated.
make[3]: *** [error.o] Error 1
make[2]: *** [all] Error 2
make[1]: *** [all-recursive] Error 1
===

The 'gcc' command on this Mac is actually clang:
==
$ gcc -v
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr 
--with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 6.1.0 (clang-602.0.53) (based on LLVM 3.6.0svn)
Target: x86_64-apple-darwin14.4.0
Thread model: posix



regards,
 - assaf

bug#22443: new snapshot available: grep-2.22.30-e07b

2016-01-23 Thread Assaf Gordon 
Hello,

> On Jan 23, 2016, at 23:42, Jim Meyering  wrote:
> 
> grep snapshot:
>  http://meyering.net/grep/grep-2.22.30-e07b.tar.xz

One regression for 2.22.30 vs 2.22.29 on OpenSolaris 5.10, both x86pc and sun4v,
possibly related to this machine having an older shell:

   $ ./configure
   ...
   ...
   checking whether btowc(EOF) is correct... yes
   checking whether this system has an arbitrary file name length limit... yes
   checking for closedir... yes
   ./configure: bad substitution

Running diff on 'configure' between versions 2.22.29 and 2.22.30, this might be 
the culprit:
===
@@ -19138,6 +19140,11 @@ done
   fi
 fi
 
+if test -z "${host_os##os2*}"; then
+if test $HAVE_OPENDIR = 1; then
+  REPLACE_OPENDIR=1
+fi
+  fi
 
   if test $HAVE_CLOSEDIR = 0 || test $REPLACE_CLOSEDIR = 1; then
===

Trying to isolate it, gives (on that OpenSolaris 5.10 i86pc machine):
===
  $ cat 1.sh   
  #!/bin/sh
  test -z "${host_os##os2*}"

  $ ./1.sh
  ./1.sh: bad substitution
===

Just a guess, but perhaps the recent OS/2 patches to gnulib are the source 
(since 29 vs 30's difference is updated gnulib).

=


No failures on:
 Mac OS X 10.10.4
 Open Solaris 5.11 both i86pc and sun4v/u
 FreeBSD 10.1, 9.3 (amd64)
 OpenBSD 5.8, 5.7 (amd64)
 NetBD 7.0 (amd64)
 Various GNU/Linuxes on x86-66
  (Debian 7/8, CentOS 7,6.5, Ubuntu 14.04,15.04, Fedora 23,22,21, OpenSUSE 42.1)


regards,
 - assaf

bug#23031: reporting write errors and handling SIGPIPE

2016-03-18 Thread Assaf Gordon 

Hello,

First,
Attached is a patch that adds errno information to 'write error' messages, e.g.:
  $ grep [...] > /dev/full
  grep: write error: No space left on device
I hope it is self-explanatory enough (comments and suggestions are welcomed).


Second,
On one gnu/linux server I'm experiencing a strange behavior (or at least, not 
understandable to me):
grep does not immediately terminates on SIGPIPE, and instead exits and prints "write 
error" (for EPIPE).
which is partially why I wrote the above patch, to try understand what's going 
on.

An example, reproducible on my machine (running on real hardware), though hard 
to reproduce inside a VM and on other servers:
   seq 10 > in
   for i in $(seq 100) ; do
  ./src/grep -s --line-buffered -v '^$' < in | head -n1 > /dev/null ;
   done
for some of the runs  (out of 100) I get an error "./src/grep: write error: Broken 
pipe" .

attached are strace/ltrace logs of such cases. the key lines:

When grep is killed by SIGPIPE:

 == strace ==
   write(1, "1\n", 2)  = 2
   write(1, "2\n", 2)  = -1 EPIPE (Broken pipe)
   --- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=2960, si_uid=1004} ---
   +++ killed by SIGPIPE +++

 == ltrace ==
   fwrite_unlocked("1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14"..., 1, 2, 
0x7efc5dab9400) = 2
__errno_location()  
  = 0x7efc5dee46a0
   fflush_unlocked(0x7efc5dab9400, 0x18da000, 10, 4096) 
 = 0
   memchr("2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n1"..., '\n', 32766)  
 = 0x18da003
   fwrite_unlocked("2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n1"..., 1, 2, 
0x7efc5dab9400) = 2
   __errno_location()   
 = 0x7efc5dee46a0
   fflush_unlocked(0x7efc5dab9400, 0x18da004, 0, 2610 
   --- SIGPIPE (Broken pipe) ---
   +++ killed by SIGPIPE +++


When grep is not killed by SIGPIPE:

 == strace ==
   write(1, "1\n", 2)  = 2
   write(1, "2\n", 2)  = -1 EPIPE (Broken pipe)
   --- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=2893, si_uid=1004} ---
   write(2, "./src/grep: ", 12)= 12
   write(2, "write error", 11) = 11
   write(2, ": Broken pipe", 13)   = 13
   write(2, "\n", 1)   = 1
   exit_group(2)   = ?

 == ltrace ==
   fwrite_unlocked("1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14"..., 1, 2, 
0x7f9e62f50400)   = 2
   __errno_location()   
   = 0x7f9e6337b6a0
   fflush_unlocked(0x7f9e62f50400, 0x25af000, 10, 4096) 
   = 0
   memchr("2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n1"..., '\n', 32766)  
   = 0x25af003
   fwrite_unlocked("2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n1"..., 1, 2, 
0x7f9e62f50400)   = 2
   __errno_location()   
   = 0x7f9e6337b6a0
   fflush_unlocked(0x7f9e62f50400, 0x25af004, 0, 2610 
   --- SIGPIPE (Broken pipe) ---
   <... fflush_unlocked resumed> )  
   = 0x
   __errno_location()   
   = 0x7f9e6337b6a0
   error(2, 32, 0x41caa0, 32 



The server is:
  $ uname -a
  Linux x 3.13.0-77-generic #121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016 x86_64 
GNU/Linux

  $ gcc -v
  Using built-in specs.
  COLLECT_GCC=gcc
  
COLLECT_LTO_WRAPPER=/usr/local/libexec/gcc/x86_64-unknown-linux-gnu/5.2.0/lto-wrapper
  Target: x86_64-unknown-linux-gnu
  Configured with: ../gcc-5.2.0/configure --enable-languages=c,c++
  Thread model: posix
  gcc version 5.2.0 (GCC)


Thanks for any feedback,
regards,
 - assaf


grep-write-error-msg.patch.xz
Description: application/xz


grep-SIGPIPE-killed.ltrace.log.xz
Description: application/xz


grep-SIGPIPE-killed.strace.log.xz
Description: application/xz


grep-SIGPIPE-not-killed.ltrace.log.xz
Description: application/xz


grep-SIGPIPE-not-killed.strace.log.xz
Description: application/xz

bug#23267: suggestion: silently ignore EPIPE errors when SIGPIPE is set to 'ignore'

2016-04-11 Thread Assaf Gordon 
Hello,

This is a small addition to a recent change:
   1cec27a7f0e5c7bdc44a88ef51af09f87a8cbc24 
   grep: use errno consistently in write diagnostics

With the above change, if SIGPIPE is set to 'ignore' (e.g. by the shell or 
python),
then grep would exit with "write error: Broken Pipe".
Based on recent suggestion from Pádraig Brady
( http://lists.gnu.org/archive/html/coreutils/2016-04/msg4.html ) perhaps 
it would be better
to have special case for EPIPE and exit silently (as if SIGPIPE was not ignored 
and terminated the program?).

Patch attached for such modification.

One issue of which I'm not certain: I assumed that if 'prline' was called, a 
match was found.
Thus, EPIPE will result in exit(EXIT_SUCCESS) indicating a match.


Comments welcomed,
 - Assaf



0001-grep-silently-ignore-EPIPE-errors.patch
Description: Binary data

bug#23031: reporting write errors and handling SIGPIPE

2016-03-19 Thread Assaf Gordon 

Hello,

On 03/17/2016 02:49 PM, Paul Eggert wrote:

On 03/16/2016 09:14 AM, Assaf Gordon wrote:

Attached is a patch that adds errno information to 'write error' messages


Thanks, I installed the attached somewhat-tweaked version of that. Among other things I 
renamed the new static functions to avoid confusion with existing "safer" 
functions.

This doesn't address the SIGPIPE mystery so I'll leave the bug report open, but 
my guess is that this part isn't really a grep bug.


Thanks for the inclusion and the additional information.
This is indeed not a bug in grep (and thus closed).

I found that on my weird setup, programs start with SIGPIPE set to SIG_IGN by 
default (not sure how I got myself into such situation).

regards,
 - assaf

bug#22899: new snapshot available: grep-2.23.7-9e28

2016-03-04 Thread Assaf Gordon 

Hello,

On 03/03/2016 08:57 PM, Jim Meyering wrote:

grep snapshot:
   http://meyering.net/grep/grep-2.23.7-9e28.tar.xz


No failures (with minimal build, no pcre) on:
  CentOS 7 + 6.5,   amd64
  Ubuntu 14.04,   amd64
  Ubuntu 15.04   amd64 + i686
  openSUSE 42.1, 13.1,   amd64
  Fedora 23, 22, 21,   amd64
  Debian 8.1, 7.6,   amd64
  Trisquel 7.0, 6.0.1,   amd64
 
  Fedora-21, ppc64le

  Fedora-20, ppc64
  Ubuntu-14, aarch64

  OpenBSD 5.8, 5.7, 5.6, amd64
  FreeBSD 10.1, 9.3,   amd64
  NetBSD 6.1.4,   amd64
  NetBSD 7.0,   amd64 (though one gnulib test fails to compile, similar to this:
 
http://lists.gnu.org/archive/html/bug-gnulib/2016-01/msg00083.html ).
  AIX-7.1,power7
  GNU Hurd 0.7,   i686
  Debian-7/kFreeBSD-9,  amd64
  OpenSolaris 5.10/5.11 on i86pc and suv4u/sub4v

On Minix 3.3.0, one test fails ('symlink'), same as mentioned here (with 
decision not to fix):
  http://lists.gnu.org/archive/html/bug-grep/2016-01/msg00050.html


regards,
 - assaf

bug#24505: getprogname vs. AIX [was: new snapshot available: grep-2.25.92-f3e9

2016-09-22 Thread Assaf Gordon 

Hello Jim,

On 09/22/2016 12:25 AM, Jim Meyering wrote:

Gordon reported this off-list:


On AIX-7.1 32bit, compilation fails due to gnulib's new 'getprogname'
module:
 CC   getprogname.o
 getprogname.c: In function 'getprogname':
 getprogname.c:45:4: error: #error "getprogname module not ported to this
OS"
 #  error "getprogname module not ported to this OS"


Thanks again for that report.
Here is a tentative patch (let's call it "pragmatic" -- it tests
explicitly for _AIX rather than a feature-test macro like
HAVE_GETPROCS64 and an additional macro from an autoconf test for the
existence of the procinfo.h header).

Can someone let me know if this solves the problem?


Sorry for not following up on that...

The patch does not apply cleanly, I suspect you have a newer gnulib version 
than what's in
grep's git repo (the patch's ChangeLog has an entry from Sep 16):

  $ git id
  v2.25-93-gdd6936c
  $ cd gnulib
  $ git id
  v0.1-880-ga512e04
  $ git am < ~/Downloads/gnulib-AIX-getprogname.diff
  Applying: getprogname: port to AIX
  error: patch failed: ChangeLog:1
  error: ChangeLog: patch does not apply
  Patch failed at 0001 getprogname: port to AIX
  The copy of the patch that failed is found in: 
/home/gordon/projects/grep/.git/modules/gnulib/rebase-apply/patch
  When you have resolved this problem, run "git am --continue".
  If you prefer to skip this patch, run "git am --skip" instead.

I patched 'lib/getprogname.c' directly, and compilation (on AIX) fails with:

CC   getprogname.o
  getprogname.c:74:1: error: expected identifier or '(' before '}' token
   }
   ^
  make: 1254-004 The error code from the last command is 1.

Indeed there's an extra closing braces in line 74.
After removing it, compilation succeeds with AIX 32bit (haven't been able to 
compile in 64bit yet, but that's not due to grep's code. I'll try again later 
tonight).

regards,
 - assaf

bug#24602: grep-2.26 make check fails on i686

2016-10-03 Thread Assaf Gordon 
Hello,

> On Oct 3, 2016, at 17:48, Jaroslav Skarvada  wrote:
> 
> Reproduced on Fedora Rawhide, gcc-6.2.1:
> 
> [...]
> Full build log:
> https://kojipkgs.fedoraproject.org/work/tasks/8044/15928044/build.log
> x86_64 seems to work OK

As a side note,
these are gnulib tests which fail to compile, grep-2.26 is built successfully 
and its test do not fail (based on the log above).

I tested with it Ubuntu 15.04-i686 with gcc 4.9.2 and there were no failures - 
I'll create a newer i686 test VM to catch these potential failures...

-assaf

bug#26576: -v when used with -C

2017-04-21 Thread Assaf Gordon 

On Thu, Apr 20, 2017 at 02:34:47PM -0500, Eric Blake wrote:

On 04/20/2017 11:51 AM, Assaf Gordon wrote:


If I may suggest the following sed program:

 $ sed -n ':x 1,2{N;bx} ; /UGLY/{ N;N;z;bx }; /./P;N;D' file


Works as long as lines 1 and 2 do not contain UGLY. But misbehaves if
UGLY appears early:

[...]

Also misbehaves if two occurrences of UGLY appear with overlapping context:


[...]

May be fixable with even more magic, perhaps by using the hold buffer to
track the status of the last three lines, and suppressing output if any
of the last three inputs were UGLY.  But more complicated than I want to
spend time on for the sake of this email.



Good catch, thanks for pointing this out.

Indeed, that was an ad-hoc script, suitible for some limited scenarios
but not robust as a general solution.

-assaf

bug#26576: -v when used with -C

2017-04-20 Thread Assaf Gordon 

Hello,

On Thu, Apr 20, 2017 at 11:26:47AM -0500, Eric Blake wrote:

On 04/20/2017 10:37 AM, 積丹尼 Dan Jacobson wrote:

I want to do
$ cat file|some_program
but I must must exclude the UGLY line and its two neighbors.

OK I have found the UGLY line, and its two neighbors
$ grep -C 2 UGLY file
bla
bla
UGLY
bla
bla

but I have no way to exclude them before piping to some_program.


It's very corner case, so I'm not sure it's worth burning an option and
complicating grep to do this, plus waiting for a future version of grep
with the proposed new option to percolate to your machines, when you
already accomplish the same task using existing tools (admittedly with
more complexity).




If I may suggest the following sed program:

 $ cat file
 a
 b
 c
 bla1
 bla2
 UGLY
 bla3
 bla4
 e
 f
 g

 $ sed -n ':x 1,2{N;bx} ; /UGLY/{ N;N;z;bx }; /./P;N;D' file
 a
 b
 c
 e
 f
 g


The combination of N/P/D commands use sed's pattern space
as a fifo buffer (N appends a new line, P prints the last line,
D deletes the last line).
In between, if the pattern space contains the marker UGLY,
the entire buffer is deleted and the cycle is restarted.

Specifically:

1. ':x 1,2{N;bx}' => Load the buffer with the first two lines.

2. '/UGLY/ {N;N;z;bx}' => If the marker is found in the pattern
  space (which should contain 3 lines now),
  consume two more lines (N;N), clear the buffer (z) and
  jump to the beginning.
  'z' is GNU extension. It can be replaced with 's/.*//'.

3. '/./P' => If the pattern space isn't empty, print up to
  the first line;

4. 'N;D' => Read the next line from the input file and append
  it to the pattern space, Delete the last line from the
  pattern space (the same line that was printed with 'P').



The following program can be used to learn a bit more about how 
the N/P/D commands work. It uses 'l' to the print content

of the pattern space, and you can see it behaves like a FIFO:

 $ sed -n ':x 1,2{N;bx} ; l;P;N;D' file
 a\nb\nc$
 a
 b\nc\nbla1$
 b
 c\nbla1\nbla2$
 c
 bla1\nbla2\nUGLY$
 bla1
 bla2\nUGLY\nbla3$
 bla2
 UGLY\nbla3\nbla4$
 UGLY
 bla3\nbla4\ne$
 bla3
 bla4\ne\nf$
 bla4
 e\nf\ng$
 e


More information about sed's buffers can be found here:
https://www.gnu.org/software/sed/manual/sed.html#advanced-sed

hope this helps,
regards,
- assaf

bug#28081: way to change line number separator from colon to space?

2017-08-13 Thread Assaf Gordon 
Hello,

On 13/08/17 12:45 PM, Marcel Partap wrote:
>> One way:
>> grep  | sed "s/:/ /"
> Thanks, obvious approach, but loses colour 

Few things:

First,
you can force grep to output color with:

   grep --color=always  | sed 's/:/ /'


Second,
If you grep on a shell-glob (e.g. "*.txt"),
you might want to add "-H -n" to force grep
to always output the filename and the line number.
Otherwise, if the glob matches a single file,
grep by default will not output the filename/line number -
and the 'sed' will not do what you wanted.

   grep --color=always -Hn  *.txt


Third,
This assumes filenames do not contain the character ':'
(which is probably a valid assumptions most of the time, but not always).
To be more robust, you can use GNU grep's "-Z" option, which outputs
a NUL after the filename, the GNU sed to replace the NUL with something
else.
Example:

  $ grep --color=always -HZn Deb /etc/motd | sed 's/\x00/ === /'
  /etc/motd === 2:The programs included with the Debian GNU/Linux
  /etc/motd === 6:Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY



Fourth,
To easily automate all of that, you can create
a tiny shell function (put it in ~/.bashrc or ~/.bash_aliases, etc):

  nicegrep() { grep --color=always -HZn "$@" | sed 's/\x00/ === /' ; }

Then run:

  nicegrep Deb /etc/motd




regards,
 - assaf

bug#27532: getprogname: support for qemu

2017-06-29 Thread Assaf Gordon 
Hello Bruno and all,

> On Jun 29, 2017, at 12:26, Bruno Haible  wrote:
> 
> When running the testsuite of grep-3.0 with qemu user mode, some tests fail.
> [...]
> *** 1 
> ! grep: g:4: Unmatched [...
> --- 1 
> ! /tmp/grep-3.0/build-arm64/src/grep: g:4: Unmatched [...
> [...]
> argv[0] = absolute file name of qemu-aarch64
> argv[1] = absolute file name of grep (it must be absolute, since it's not
>   qemu's job to search for 'grep' in $PATH).

Luckily, many GNU test scripts already use "$prog" perl variable
for the program's name in error message, so perhaps it would be possible
to accomodate qemu without code changes.

I've encountered the same for my program (datamash).
and as an ugly hack added an undocumented option to print the program's name
to set '$prog' accordingly:
https://git.savannah.gnu.org/cgit/datamash.git/tree/tests/datamash-tests.pl#n37
  ## Cross-Compiling portability hack:
  ##  under qemu/binfmt, argv[0] (which is used to report errors) will contain
  ##  the full path of the binary, if the binary is on the $PATH.
  ##  So we try to detect what is the actual returned value of the program
  ##  in case of an error.
  my $prog = `$prog_bin ---print-progname`;
  $prog = $prog_bin unless $prog;


But this hack can be avoided, if we just run 'grep' with invalid
arguments, triggering an error, then extracting the program name from STDERR.

E.g. for 
https://git.savannah.gnu.org/cgit/grep.git/tree/tests/filename-lineno.pl
change the following on line 26 from:
my $prog = 'grep';
to
my $prog = 'grep';
my $full_prog_name = `$prog --invalid-option-for-testing 2>&1 | head -n1 | 
cut -f1 -d:`;
$prog = $full_prog_name if $full_prog_name;

This is untested code, but should work more-or-less.
Once "$prog" is updated, all the tests should pass.

Hope this helps,
regards,
 - assaf

bug#30525: Unexpected matches for input data from a patch file

2018-03-01 Thread Assaf Gordon 

Hello Markus,

I believe there are actually several different issues here,
perhaps it's worth stating them explicitly to ensure we're
on the same page.


On 2018-03-01 12:52 AM, SF Markus Elfring wrote:

* Does the tool “grep” output any extra colour information also for
    matched tab characters?

[...]

grep's --color option was not used.


The matched characters are marked in red (for other search patterns)
on my test system even if this command parameter is not passed explicitly.

[...]
> There are further challenges to consider for special characters.
> Would it make sense to replace them by printable variants?

First,

grep will print color information in the following situations:
1. If you use "--color=always"
2. If you use "--color=auto" and the output is a terminal
3. If you don't specify "--color" at all, and environment
variable GREP_OPTIONS is empty, and the output is a terminal
(then "--color=auto" is assumed).

Observe the following:

If you type this on the terminal, the letter "A" should be colored:

  $ printf "AB\n" | grep A
  AB

With this command, grep's output is a pipe (not a terminal),
and by default there will be no color:

  $ printf "AB\n" | grep A | cat
  AB

You can force color output with:

  $ printf "AB\n" | grep --color=always A | cat
  AB

And you can examine the color escape sequences with:

  $ printf "AB\n" | grep --color=always A | od -An -c
   033   [   0   1   ;   3   1   m 033   [   K   A 033   [   m 033
 [   K   B  \n

The characters "\033[01;31m" are the sequence to change color,
and "\033[m" is the sequence to reset the colors.
These are technically called "ansi terminal control escape sequences", 
more here: https://en.wikipedia.org/wiki/ANSI_escape_code .


Therefore, when discussing grep's coloring options
it's important to say if the output is a terminal or not,
and whether coloring is on or off (and when troubleshooting,
it is best to explicitly use --color=XXX).



Second,
When coloring is enabled (e.g. with "--color=always"), grep will
surround the TAB characters with the color escape sequences.
Observe the following:

  $ printf "\t\t\n" | grep -E --color=always '\s+' | od -An -c
   033   [   0   1   ;   3   1   m 033   [   K  \t  \t 033   [   m
   033   [   K  \n

Notice there is an ansi color escape sequence, followed by two tabs 
(\t), followed by "reset color" sequence, followed by "\n".





Third,
Grep's default coloring is red text and default background.
But TAB (and space) are empty characters - they do not have text.
Because the default background color is not changed, you will not
see them highlighted with a color.
You can change the default color with GREP_COLORS environment variables.

For example:

Here you should see "A" and "B" in color,
with empty (non-colored) spaces between them:

  $ printf "A \tB\n" | grep --color=always '.*'
  A   B

You can force the background color to be something else
like so:

  $ printf "A \tB\n" | GREP_COLORS="mt=41" grep --color=always '.*'
  A   B

The above command should print "A" and "B" with red background
and default text color (See the next item regarding the white-space 
colors). To learn more about using GREP_COLORS, read "man grep".




Fourth,
This is an unexpected "gotcha" - some terminal programs
do NOT color tab characters at all! they just move the cursor,
while others print multiple spaces which are colored.

(by terminal programs I mean "xterm" or "gnome-terminal" or "konsole",
and this is also affected by using tmux or screen, etc.).

To test this, try the following on several terminals:

  printf "\033[41m A B\tC\n\033[m"

This sequences means:
background red color, space, A, space, B, tab, C, new line, reset-color.


On gnome-terminal, I get the entire line in red background.
On xterm, I get black space between B and C - meaning TAB is just moving 
the cursor.
You might see different results on your terminal - which means "grep" is 
not the problem at all here.




Fifth and last,
You asked about replacing non-printable characters.
This is easy enough to do with existing programs,
so not likely to be added as a new option to grep.

If you just want to replace TAB characters:

  printf "\t\tif\n" | grep --color=always -E '^\s+[^i]' | cat -T
or
  printf "\t\tif\n" | grep --color=always -E '^\s+[^i]' | tr '\t' x

To replace other characters, add more characters to 'tr', or something 
more complicated, use sed:


  printf "\t\tif\n" | grep --color=always -E '^\s+[^i]' \
  | sed 's/\t//g ; s/ //g'



Hope this helps,
regards,
 - assaf

bug#32704: Can grep search for a line feed and a null character at the same time?

2018-09-15 Thread Assaf Gordon 

Hello,

On 15/09/18 11:57 AM, 21na...@gmail.com wrote:

Le 15/09/2018 à 19:06, Eric Blake a écrit :

On 9/15/18 11:43 AM, 21na...@gmail.com wrote:

But is it at least possible to find “\x0A\x00” with grep?


If you bend the rules by throwing -P into the mix, yes :)

So it is possible to find “\x0A\x00” alone, but for example 
“\x74\x00\x0D\x00\x0A\x00\x74\x00\x65\00” is impossible to find with the 
“-P” option?


If I may suggest a different tool, GNU sed can handle such regexes more 
easily than grep.

The 'trick' is to accumulate multiple lines into memory, then run the
regex on the entire buffer.

1.
If you input is small enough to fit in memory,
you can load the entire file into memory,
and run the regex on the buffer:

$ printf 
'\xFF\xFE\x0D\x00\x0A\x00\x74\x00\x65\x00\x73\x00\x74\x00\x0D\x00\x0A\x00\x74\x00\x65\x00\x73\x00\x74\x00\x5F\x00\x74\x00\x77\x00\x6F\x00\x0D\x00\x0A\x00' 
\

 | LC_ALL=C sed -n 'H;$!d ; x ; /\x0a\x00/q0 ; q1' \
   && echo MATCH || echo NO-MATCH

The "H;$!d" commands accumulate lines into the hold buffer.
The "x" command copies the hold buffer into the pattern buffer.
Then the regex "/\x0a\x00/" searches in the buffer.
If there was a match, sed quits with exit code 0 ("q0").
Otherwise, sed quits with exit code 1 ("q1").


2.
If the file is too big to fit in memory,
you can process it line-by-line like so:

$ printf 
'\xFF\xFE\x0D\x00\x0A\x00\x74\x00\x65\x00\x73\x00\x74\x00\x0D\x00\x0A\x00\x74\x00\x65\x00\x73\x00\x74\x00\x5F\x00\x74\x00\x77\x00\x6F\x00\x0D\x00\x0A\x00' 
\

 | LC_ALL=C sed -n 'N;/\x00\x0a/q0;$q1;D;' \
 && echo MATCH || echo NO-MATCH

The N,D commands work in tandem to append the next line into the
buffer, then delete the last line from the buffer (think FIFO).
The regex then operates on the buffer which contains the last two lines.



More details are in the manual:
 https://www.gnu.org/software/sed/manual/sed.html#Multiline-techniques
https://www.gnu.org/software/sed/manual/sed.html#Text-search-across-multiple-lines



regards,
 - assaf

bug#36764: grep -q throwing up No such file or directory error

2019-07-22 Thread Assaf Gordon 
tag 36764 notabug
close 36764
stop

Hello,

(answering out-of-order)

On Mon, Jul 22, 2019 at 06:12:10PM +0100, Lewis Farnworth wrote:

> To reiterate:
> 
> if grep -q "thing_i'm_looking_for" $variable_i'm_looking_in; then
> 
> doesn't work, on CentOS 7 or Debian 9.

This is incorrect usage, and has never worked with gnu grep.
The correct usage is:

   grep -q "thing_i'm_looking_for" "FILE to look in"

Or, to check the content of a variable, use:

   echo "$variable_i'm_looking_in" | grep -q "$thing_i'm_looking_for"

or the more robust:

   printf "%s" "$variable_i'm_looking_in" | grep -q "$thing_i'm_looking_for"

> I previously had a working script, that used -q in this context here:
> 
> if grep -q "RewriteCond %{HTTP_REFERER} !.*example.com*" $line; then
> 
> -
> That syntax was perfectly functioning as a conditional inside of a while
> loop, to detect a colleague's error inside of a bunch of .htaccess rules.

As for the above, I would guess that the "while" loop read a FILENAME
into the variable $line (e.g. a path of an .htaccess file),
and then grep would search for the pattern in that file.

If you have access to the machine where the above works,
add "echo LINE=$line" to print the content of the variable,
and you'll see it contains a file name.


> This time, I'm trying to basically ascertain which servers on a given
> domain are using Gmail. I've ran some dig, grep, sed & awk to ascertain a
> list of mail servers the domain is using. One of the outputs which I'm
> looking for looks something like this:
> 
> $   echo $mailServ:
> alt3.aspmx.l.google.com.
> alt4.aspmx.l.google.com.
> alt1.aspmx.l.google.com.
> alt2.aspmx.l.google.com.
> aspmx.l.google.com.
> 
> *The script I'm running:*
> 
> if grep -q "google.com" $mailServ; then
> printf "%s uses google for mail \n" $mailServ
> fi
> 
> I've run into a few weird error messages when trying to use the -q
> option... And I'm 100% certain that this is not a syntax issue (but there's
> a part of me wishing it is).
> 
> *The error I get is: *
> 
> grep: aspmx.l.google.com.: No such file or directory
> grep: alt1.aspmx.l.google.com.: No such file or directory
> grep: alt4.aspmx.l.google.com.: No such file or directory
> grep: alt2.aspmx.l.google.com.: No such file or directory
> grep: alt3.aspmx.l.google.com.: No such file or directory

This is expected, as explained above.

> When in actual fact, all I need is the  printf command output. I tried to
> run the exact same script on Debian 9, to ensure that was not the issue...
> Apparently that doesn't work either.

Use something like:

  if echo "$mailServ" | grep -q "google.com"; then
 printf "%s uses google for mail \n" $mailServ
  fi

Note that this operates on the entire content of "$mailServ" (all lines
together), not line-by-line.


> I tried to use the first script I wrote again, for the sake of a sanity
> check, and that script is now broken too.

If an unmodified script used to work on CentOS before an upgrade,
and stops working after the upgrade (again, unmodified),
and you've isolated the issue to grep, please provide more details
(e.g. the entire script).

As such, I'm closing this as "not a bug", but discussion can continue
by replying to this thread.

regards,
 - assaf

P.S.
Note that in your regex patterns you use "." as a literal dot,
but it is in fact a regex special character meaning "any character".
So:

 grep "google.com" $FILE

would also match a file containing "googleXcom".

To match a literal dot, use "\.".

bug#41687: regex search for indexed files

2020-06-06 Thread Assaf Gordon 

tag 41687 notabug
close 41687
stop

Hello,

On 2020-06-03 8:27 a.m., Peng Yu wrote:

grep can do regex search but it needs to scan each file. When the
number of files are large, it can be slow.

Is there an alternative tool that can do regex search in the indexed
files (including .docx .pdf and other commonly used file formats that
can be converted to text) so that the search can be fast?


It seems you are mixing several questions together.

1. If you want "grep" to search only specific set of files,
use the "--include" or "--exclude" options.
Or better yet, use find+xargs+grep .

2. If you want to search in non-text files, use appropriate programs
that understand the file format (e.g. "pdfgrep")
or programs that can convert the custom format to text (e.g. "antiword" 
and "wv").


3. You've mentioned "indexed files" - if you're looking for a program
that scans files and indexes them, and then allows you to search the 
index, look for "Desktop search" programs, e.g. 
https://en.wikipedia.org/wiki/List_of_search_engines#Desktop_search_engines

https://en.wikipedia.org/wiki/Recoll
https://en.wikipedia.org/wiki/Tracker_(search_software)

---

Lastly,
For all of these topics, a simple internet search would have given you
the above results. PLEASE respect everyone's time by first doing 
searching for answers yourself, before posting questions on a public 
mailing list.


---

Since this is not a bug in grep, I'm marking this as "closed".

regards,
 - assaf