[Bug 195763] bsdgrep, empty matches and -o

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195763

Ed Maste  changed:

   What|Removed |Added

   Assignee|freebsd-bugs@FreeBSD.org|kev...@freebsd.org

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"

[Bug 195763] bsdgrep, empty matches and -o

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195763

Ed Maste  changed:

   What|Removed |Added

 Status|Open|In Progress

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"

[Bug 195763] bsdgrep, empty matches and -o

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195763

--- Comment #8 from commit-h...@freebsd.org ---
A commit references this bug:

Author: emaste
Date: Wed Apr  5 18:41:46 UTC 2017
New revision: 316536
URL: https://svnweb.freebsd.org/changeset/base/316536

Log:
  bsdgrep: create additional tests for coverage on recent fixes

  Create additional tests to cover regressions that were discovered by
  PRs linked to reviews D10098, D10102, and D10104.

  It is worth noting that neither bsdgrep(1) nor gnugrep(1) in the base
  system currently pass all of these tests, and gnugrep(1) not quite being
  up to snuff was also noted in at least one of the PRs.

  PR:   175314 202022 195763 180990 197555 197531 181263 209116
  Submitted by: Kyle Evans 
  Reviewed by:  cem, ngie, emaste
  MFC after:1 month
  Differential Revision:https://reviews.freebsd.org/D10112

Changes:
  head/contrib/netbsd-tests/usr.bin/grep/d_color_a.in
  head/contrib/netbsd-tests/usr.bin/grep/d_color_a.out
  head/contrib/netbsd-tests/usr.bin/grep/d_color_b.in
  head/contrib/netbsd-tests/usr.bin/grep/d_color_b.out
  head/contrib/netbsd-tests/usr.bin/grep/d_color_c.out
  head/contrib/netbsd-tests/usr.bin/grep/d_escmap.in
  head/contrib/netbsd-tests/usr.bin/grep/d_f_file_empty.in
  head/contrib/netbsd-tests/usr.bin/grep/d_oflag_zerolen_a.in
  head/contrib/netbsd-tests/usr.bin/grep/d_oflag_zerolen_a.out
  head/contrib/netbsd-tests/usr.bin/grep/d_oflag_zerolen_b.in
  head/contrib/netbsd-tests/usr.bin/grep/d_oflag_zerolen_b.out
  head/contrib/netbsd-tests/usr.bin/grep/d_oflag_zerolen_c.in
  head/contrib/netbsd-tests/usr.bin/grep/d_oflag_zerolen_c.out
  head/contrib/netbsd-tests/usr.bin/grep/d_oflag_zerolen_d.in
  head/contrib/netbsd-tests/usr.bin/grep/d_oflag_zerolen_e.in
  head/contrib/netbsd-tests/usr.bin/grep/d_oflag_zerolen_e.out
  head/contrib/netbsd-tests/usr.bin/grep/t_grep.sh
  head/usr.bin/grep/tests/Makefile

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"

[Bug 195763] bsdgrep, empty matches and -o

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195763

Ed Maste  changed:

   What|Removed |Added

   See Also||https://bugs.freebsd.org/bu
   ||gzilla/show_bug.cgi?id=2091
   ||16

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"

[Bug 195763] bsdgrep, empty matches and -o

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195763

--- Comment #7 from commit-h...@freebsd.org ---
A commit references this bug:

Author: emaste
Date: Mon Apr  3 23:16:51 UTC 2017
New revision: 316477
URL: https://svnweb.freebsd.org/changeset/base/316477

Log:
  bsdgrep: fix matching behaviour

  - Set REG_NOTBOL if we've already matched beginning of line and we're
examining later parts

  - For each pattern we examine, apply it to the remaining bits of the
line rather than (potentially) smaller subsets

  - Check for REG_NOSUB after we've looked at all patterns initially
matching the line

  - Keep track of the last match we made to later determine if we're
simply not matching any longer or if we need to proceed another byte
because we hit a zero-length match

  - Match the earliest and longest bit of each line before moving the
beginning of what we match to further in the line, past the end of the
longest match; this generally matches how gnugrep(1) seems to behave,
and seems like pretty good behavior to me

  - Finally, bail out of printing any matches if we were set to print all
(empty pattern) but -o (output matches) was set

  PR:   195763, 180990, 197555, 197531, 181263, 209116
  Submitted by: "Kyle Evans" 
  Reviewed by:  cem
  MFC after:1 month
  Relnotes: Yes
  Differential Revision:https://reviews.freebsd.org/D10104

Changes:
  head/usr.bin/grep/util.c

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"

[Bug 195763] bsdgrep, empty matches and -o

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195763

Ed Maste  changed:

   What|Removed |Added

   See Also||https://bugs.freebsd.org/bu
   ||gzilla/show_bug.cgi?id=1975
   ||31

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"

[Bug 195763] bsdgrep, empty matches and -o

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195763

Ed Maste  changed:

   What|Removed |Added

   See Also||https://bugs.freebsd.org/bu
   ||gzilla/show_bug.cgi?id=1975
   ||55

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"

[Bug 195763] bsdgrep, empty matches and -o

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195763

Ed Maste  changed:

   What|Removed |Added

   See Also||https://bugs.freebsd.org/bu
   ||gzilla/show_bug.cgi?id=1809
   ||90

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"

[Bug 195763] bsdgrep, empty matches and -o

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195763

Kyle Evans  changed:

   What|Removed |Added

 Attachment #179175|0   |1
is obsolete||

--- Comment #6 from Kyle Evans  ---
Created attachment 179227
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=179227=edit
Proposed patch to address line matching issues

Address one last nit that I found in some more extensive testing -- if we're
going to replace the last match made (due to overlap), it should be an earlier
or strictly longer match than the current one.

Test case: echo "abcdef" | grep -o -e "ab" -e "bc"
Swap the order of -e arguments, "ab" and "bc" would previously yield different
results when they should both yield "ab".

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"

[Bug 195763] bsdgrep, empty matches and -o

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195763

Mark Linimon  changed:

   What|Removed |Added

   Keywords||patch

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"

[Bug 195763] bsdgrep, empty matches and -o

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195763

Kyle Evans  changed:

   What|Removed |Added

 Attachment #179173|0   |1
is obsolete||

--- Comment #5 from Kyle Evans  ---
Created attachment 179175
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=179175=edit
Proposed patch to address line matching issues

Ugh, sorry -- this really should be the final version. Shortly after the last
update, I caught a mailing list entry that mentioned broken beginning of line
matching behavior and figured it would be wise to catch this as well.

REG_NOTBOL needs to be manually set if we're past the beginning of the string
because we also use REG_STARTEND to limit the scope of our matching, which
makes the regex functions think we're beginning of line all the time.

We do not need to do the same with EOL since we limit the scope by adjusting
rm_so, resetting rm_eo to the end of the line. It might not be a bad idea to
later set it anyways, just in case we change our mind, but for now that's out
of scope.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"

[Bug 195763] bsdgrep, empty matches and -o

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195763

Kyle Evans  changed:

   What|Removed |Added

 Attachment #179171|0   |1
is obsolete||

--- Comment #4 from Kyle Evans  ---
Created attachment 179173
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=179173=edit
Proposed patch to address line matching issues

The failure in bug #181263 case 1 also ended up being fairly straightforward to
address -- pmatch gets clobbered by a non-matching case and ended up with bogus
values, so now we look at the last *actual match* (if st isn't advancing) and
see if that ends up being a zero-length match.

This catches corner cases like the one found where .* can infinitely match but
the 'a' pattern comes second. It shouldn't affect any legitimate cases, since
those will cause an advancement in nst prior to this.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"

[Bug 195763] bsdgrep, empty matches and -o

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195763

Kyle Evans  changed:

   What|Removed |Added

 Attachment #179126|0   |1
is obsolete||

--- Comment #3 from Kyle Evans  ---
Created attachment 179171
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=179171=edit
Proposed patch to address line matching issues

emaste@ - I believe this to be my final iteration of this patch. Because this
and bug #209116 are both somewhat major issues with procline(), this patch
addresses both PRs and obsoletes both previous patches.

I've added to printline() a bail-out if -o and matchall (empty pattern) are set
-- this coincides with GNU grep's behavior.

Assuming I ran the tests correctly (make && make install && kyua test -k
/usr/tests/usr.bin/grep/Kyuafile), I haven't yet introduced any regressions
there. One of my next steps might be to review the unit test coverage and make
sure it's sufficient, though, since it was passing before despite having all
kinds of problems.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"

[Bug 195763] bsdgrep, empty matches and -o

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195763

--- Comment #2 from Kyle Evans  ---
(In reply to Kyle Evans from comment #1)

The 'ignore invalid patterns' change my previous patch makes seems definitively
wrong -- I'll have to revise this part and I'm just going to leave this here:

$ echo 'test' | grep ''
test
$ echo 'test' | grep -o ''
$ echo 'test' | bsdgrep ''
test
$ echo 'test' | bsdgrep -o ''

The regex bits declare this as a 4-character match, not sure off-hand how to
treat this but I don't believe ignoring the pattern is the right thing to do.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"

[Bug 195763] bsdgrep, empty matches and -o

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195763

Kyle Evans  changed:

   What|Removed |Added

 CC||bsdpo...@kyle-evans.net

--- Comment #1 from Kyle Evans  ---
Created attachment 179126
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=179126=edit
Proposed patch to address zero length match issues with bsdgrep

Hi,

Attached is a patch that seems to fix the behavior of zero-length matches.
procline() was previously using something along the lines of 'next start (st)
== match start' to indicate failure, but this isn't right when zero-length
matches are possible. Therefore, we make a couple of critical changes:

1.) Keep track of the number of total matches we came across
2.) Don't add zero-length matches to our output
3.) If we had any matches, keep going. If st didn't advance beyond our first
match, we only had zero-length matches

To fix the empty expression case, I changed the argv pattern matching bits in
main() to actually skip invalid expressions. 

Your test expressions now yield the following results:
root@ghost:/usr/obj/usr/src/usr.bin/grep# echo '01:1:01' | ./bsdgrep -Eo
'(^|:)0*'
0
:
:0
# Good
root@ghost:/usr/obj/usr/src/usr.bin/grep# echo '1:1:01' | ./bsdgrep -Eo
'(^|:)0*'
:
:0
# Better
root@ghost:/usr/obj/usr/src/usr.bin/grep# echo 'bla bla' | ./bsdgrep -Eo
'[[:alnum:]]*'
bla
bla
# Good
root@ghost:/usr/obj/usr/src/usr.bin/grep# printf 'bla\nbla\n' | ./bsdgrep -Eo
''
# Good

I think this approach is mostly right, except for perhaps the bits in
grep.c:main() to make sure we don't recognize invalid patterns as input
patterns.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"

[Bug 195763] bsdgrep, empty matches and -o

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195763

Ed Maste  changed:

   What|Removed |Added

 Status|New |Open
 CC||ema...@freebsd.org

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"