Re: [vim/vim] Automatic regular expression engine selection matches incorrectly on long line (#867)

[Bram Moolenaar]

Christian Brabandt wrote:

> On Do, 16 Jun 2016, fritzophrenic wrote:
> 
> > > Can you please set verbose before searching?
> > 
> > I did "gvim -N -u NONE -i NONE" then ":set verbose=15". Searching for 
> > "\%(|\u.*\)\@<=[^|\t]\+$" in a long line with no uppercase characters still 
> > matches, and I get no messages at all. 
> > 
> > > I wonder if this happens, because the NFA engine stops because it returns 
> > > NFA_TOO_EXPENSIVE and then the retry with the old engine is somehow 
> > > confused?
> > 
> > I don't think so (and I guess I proved it with the verbose setting, unless 
> > I did that wrong), because I can force the new engine with \%#=2 or by 
> > setting 'regexpengine', and the pattern works as expected.
> 
> Okay, I checked the problem. My suspicion was correct, that patch 
> 7.4.494 causes this problem, because after recursive_regmatch() returns 
> NFA_TOO_EXPENSIVE nfa_listid will be restored from save_nfa_listid and 
> then the exit condition in nfa_regmatch does not match anymore.
> 
> Attached is a patch, including a test.
> (attachment won't be visible in github, see the thread 
> https://groups.google.com/d/msg/vim_dev/BE0ISqcD3aI/ZGeOMrFQJgAJ for the 
> patch)

Glad you could locate and fix it!

-- 
hundred-and-one symptoms of being an internet addict:
163. You go outside for the fresh air (at -30 degrees) but open the
 window first to hear new mail arrive.

 /// Bram Moolenaar -- b...@moolenaar.net -- http://www.Moolenaar.net   \\\
///sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\  an exciting new programming language -- http://www.Zimbu.org///
 \\\help me help AIDS victims -- http://ICCF-Holland.org///

-- 
-- 
You received this message from the "vim_dev" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

--- 
You received this message because you are subscribed to the Google Groups 
"vim_dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to vim_dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [vim/vim] Automatic regular expression engine selection matches incorrectly on long line (#867)

[Christian Brabandt]

On Do, 16 Jun 2016, fritzophrenic wrote:

> > Can you please set verbose before searching?
> 
> I did "gvim -N -u NONE -i NONE" then ":set verbose=15". Searching for 
> "\%(|\u.*\)\@<=[^|\t]\+$" in a long line with no uppercase characters still 
> matches, and I get no messages at all. 
> 
> > I wonder if this happens, because the NFA engine stops because it returns 
> > NFA_TOO_EXPENSIVE and then the retry with the old engine is somehow 
> > confused?
> 
> I don't think so (and I guess I proved it with the verbose setting, unless I 
> did that wrong), because I can force the new engine with \%#=2 or by setting 
> 'regexpengine', and the pattern works as expected.

Okay, I checked the problem. My suspicion was correct, that patch 
7.4.494 causes this problem, because after recursive_regmatch() returns 
NFA_TOO_EXPENSIVE nfa_listid will be restored from save_nfa_listid and 
then the exit condition in nfa_regmatch does not match anymore.

Attached is a patch, including a test.
(attachment won't be visible in github, see the thread 
https://groups.google.com/d/msg/vim_dev/BE0ISqcD3aI/ZGeOMrFQJgAJ for the 
patch)

Best,
Christian
-- 
Die beste kurzfristige Erfolgsmeldung ist das Mienenspiel deines
Bankdirektors.
-- Helmar Nahr

-- 
-- 
You received this message from the "vim_dev" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

--- 
You received this message because you are subscribed to the Google Groups 
"vim_dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to vim_dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
From e1580984bd0299422be8e9a5fc4edfc87154dd5a Mon Sep 17 00:00:00 2001
From: Christian Brabandt 
Date: Tue, 28 Jun 2016 19:54:53 +0200
Subject: [PATCH] do not restore nfa_listid state, when pattern too expensive

Currently, nfa_listid is restored after a call to recursive_regmatch,
which may cause the exit condition in nfa_regmatch()

if (prog->re_engine == AUTOMATIC_ENGINE && nfa_listid >= NFA_MAX_STATES)

to fail. In that case, the automatic engine might find an wrong match.

fixes #867

Add a test for that bug.
---
 src/regexp_nfa.c  | 7 +--
 src/testdir/test64.in | 3 ++-
 src/testdir/test64.ok | 3 +++
 3 files changed, 10 insertions(+), 3 deletions(-)

diff --git a/src/regexp_nfa.c b/src/regexp_nfa.c
index 6b83699..87efef3 100644
--- a/src/regexp_nfa.c
+++ b/src/regexp_nfa.c
@@ -5210,9 +5210,12 @@ recursive_regmatch(
 if (REG_MULTI)
 	regline = reg_getline(reglnum);
 reginput = regline + save_reginput_col;
-nfa_match = save_nfa_match;
+if (result != NFA_TOO_EXPENSIVE)
+{
+	nfa_match = save_nfa_match;
+	nfa_listid = save_nfa_listid;
+}
 nfa_endp = save_nfa_endp;
-nfa_listid = save_nfa_listid;
 
 #ifdef ENABLE_LOG
 log_fd = fopen(NFA_REGEXP_RUN_LOG, "a");
diff --git a/src/testdir/test64.in b/src/testdir/test64.in
index fd19d3a..360418c 100644
--- a/src/testdir/test64.in
+++ b/src/testdir/test64.in
@@ -578,7 +578,7 @@ Gop:"
 :" Check patterns matching cursor position.
 :func! Postest()
  new
- call setline(1, ['ffo', 'bob', 'zoo', 'koo', 'moo', "\t\t\tfoo", 'abababababababfoo', 'bababababababafoo', '_'])
+ call setline(1, ['ffo', 'bob', 'zoo', 'koo', 'moo', "\t\t\tfoo", 'abababababababfoo', 'bababababababafoo', '_', ' xx xxx x x xx xx xx x xxx xx    xxx xxx xxx x xx xx xx xxx  x  xxx  x  xxx  xxx xxx x   x xxx xx x xxx  xx xxx xxx x xxx x x xx xx xxx  xx xxx xxx xxx xxx  xxx   xxx xx x x xx xx xxx xxx   x xx xx x xxx xxx  x  xx  xx  xxx xxx x'])
  call setpos('.', [0, 1, 0, 0])
  s/\%>3c.//g
  call setpos('.', [0, 2, 4, 0])
@@ -590,6 +590,7 @@ Gop:"
  %s/\%>6l\%3c./!/g
  %s/\%>7l\%12c./?/g
  %s/\%>7l\%<9l\%>5v\%<8v./#/g
+ $s/\%(|\u.*\)\@<=[^|\t]\+$//ge
  1,$yank
  quit!
 endfunc
diff --git a/src/testdir/test64.ok b/src/testdir/test64.ok
index 92f06ea..c218f8e 100644
--- a/src/testdir/test64.ok
+++ b/src/testdir/test64.ok
@@ -1076,6 +1076,7 @@ moo
 ab!babababababfoo
 ba!ab##abab?bafoo
 **!*_
+  ! xxx? xx xxx x x xx xx xx x xxx xx    xxx xxx xxx x xx xx xx xxx  x  xxx  x  xxx  xxx xxx x   x xxx xx x xxx  xx xxx xxx x xxx x x xx xx xxx  xx xxx xxx xxx xxx 

Re: [vim/vim] Automatic regular expression engine selection matches incorrectly on long line (#867)

[Christian Brabandt]

Am 2016-06-15 23:39, schrieb fritzophrenic:

Confirmed, this also affects 7.4.1940, both on 64-bit Windows 7, and
on Solaris. Setting the 'regexpengine' to either 1 or 2 also causes
the pattern to correctly not match.



Can you please set verbose before searching?

I wonder if this happens, because the NFA engine stops because it 
returns NFA_TOO_EXPENSIVE
and then the retry with the old engine is somehow confused? There should 
be a message

about switching the engine, if verbose is set.

Best,
Christian

--
--
You received this message from the "vim_dev" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

--- 
You received this message because you are subscribed to the Google Groups "vim_dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email 
to vim_dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[bug] with regular expression and SEARCH_START flag

[Christian Brabandt]

Bram,
I think I've found a bug with the regular expression engine:

~$ vim -u NONE -N
:$put ='foobar'
:$put =''
:$put =''
:/foobar/
:call search('foobar', 'c')
 Note how the cursor /does not/ move
:+
:call search('^$', 'c')
 Note, that the cursor /moves/ one line down, although the 'c' flag is
 given and so the cursor should stay in its actual position
:call search('^$', 'bc')
 Note, that when searching backwards, the cursor /does not/ move as
 expected.

Sorry for this brief explanation. I hope, you see the problem.
I am kind of busy currently, so I can't provide a detailed bug report or 
a patch.

Mit freundlichen Grüßen
Christian
-- 

-- 
-- 
You received this message from the vim_dev maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

--- 
You received this message because you are subscribed to the Google Groups 
vim_dev group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to vim_dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [bug] with regular expression and SEARCH_START flag

[Bram Moolenaar]

Christian Brabandt wrote:

 I think I've found a bug with the regular expression engine:
 
 ~$ vim -u NONE -N
 :$put ='foobar'
 :$put =''
 :$put =''
 :/foobar/
 :call search('foobar', 'c')
  Note how the cursor /does not/ move
 :+
 :call search('^$', 'c')
  Note, that the cursor /moves/ one line down, although the 'c' flag is
  given and so the cursor should stay in its actual position
 :call search('^$', 'bc')
  Note, that when searching backwards, the cursor /does not/ move as
  expected.
 
 Sorry for this brief explanation. I hope, you see the problem.
 I am kind of busy currently, so I can't provide a detailed bug report or 
 a patch.

I can reproduce it.  I hope you can find a fix when you are not busy.

-- 
Mynd you, m00se bites Kan be pretty nasti ...
 Monty Python and the Holy Grail PYTHON (MONTY) PICTURES LTD

 /// Bram Moolenaar -- b...@moolenaar.net -- http://www.Moolenaar.net   \\\
///sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\  an exciting new programming language -- http://www.Zimbu.org///
 \\\help me help AIDS victims -- http://ICCF-Holland.org///

-- 
-- 
You received this message from the vim_dev maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

--- 
You received this message because you are subscribed to the Google Groups 
vim_dev group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to vim_dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [bug] with regular expression and SEARCH_START flag

[Christian Brabandt]

Hi Bram!

On Sa, 16 Feb 2013, Bram Moolenaar wrote:

 
 Christian Brabandt wrote:
 
  I think I've found a bug with the regular expression engine:
  
  ~$ vim -u NONE -N
  :$put ='foobar'
  :$put =''
  :$put =''
  :/foobar/
  :call search('foobar', 'c')
   Note how the cursor /does not/ move
  :+
  :call search('^$', 'c')
   Note, that the cursor /moves/ one line down, although the 'c' flag is
   given and so the cursor should stay in its actual position
  :call search('^$', 'bc')
   Note, that when searching backwards, the cursor /does not/ move as
   expected.
  
  Sorry for this brief explanation. I hope, you see the problem.
  I am kind of busy currently, so I can't provide a detailed bug report or 
  a patch.
 
 I can reproduce it.  I hope you can find a fix when you are not busy.

This patch fixes it for me:


diff --git a/src/search.c b/src/search.c
--- a/src/search.c
+++ b/src/search.c
@@ -727,6 +727,8 @@
++matchcol;
}
}
+   if (options  SEARCH_START)
+   break;
if (ptr[matchcol] == NUL
|| (nmatched = vim_regexec_multi(regmatch,
  win, buf, lnum + matchpos.lnum,


Mit freundlichen Grüßen
Christian
-- 
Ich brauche das Alleinsein, um den Weg nach innen zu suchen.
-- Petter Moen

-- 
-- 
You received this message from the vim_dev maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

--- 
You received this message because you are subscribed to the Google Groups 
vim_dev group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to vim_dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Regular expression: **

[Tom Link]

Hi,

I have just noticed that vim allows to use ** as regular expression.
Shouldn't this throw a Nested * exception like .** does?

Examples:

echo match(abc, **)
= 0

echo match(abc, .**)
= E61: Nested *

Regards
Tom

-- 
You received this message from the vim_dev maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

Re: Regular expression: **

[Christian Brabandt]

Hi Tom!

On Mo, 12 Apr 2010, Tom Link wrote:

 Hi,
 
 I have just noticed that vim allows to use ** as regular expression.
 Shouldn't this throw a Nested * exception like .** does?
 
 Examples:
 
 echo match(abc, **)
 = 0
 
 echo match(abc, .**)

I believe this is a special case, that is mentioned at :h  /star
,
|  Exception: When * is used at the start of the pattern or just 
|  after ^ it matches the star character.
`

which means you have been searching for the equivalence of /\** (zero or 
any number of stars) and matches everywhere.

regards,
Christian
-- 

-- 
You received this message from the vim_dev maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

To unsubscribe, reply using remove me as the subject.

Re: Regular expression: **

[Matt Wozniski]

On Mon, Apr 12, 2010 at 10:36 AM, Tom Link micat...@gmail.com wrote:
 I believe this is a special case, that is mentioned at :h  /star

 Thanks. I missed that paragraph.

 Regards,
 Tom

O_o I never knew that one, either.  Huh.

~Matt

-- 
You received this message from the vim_dev maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

To unsubscribe, reply using remove me as the subject.