[issue2636] Adding a new regex module (compatible with re)
Jeffrey C. Jacobs added the comment: If I recall, I started this thread with a plan to update re itself with implementations of various features listed in the top post. If you look at the list of files uploaded by me there are seme complete patches for Re to add various features like Atomic Grouping. If we wish to therefore bring re to regex standard we could start with those features. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1693050] \w not helpful for non-Roman scripts
Jeffrey C. Jacobs added the comment: Thanks Matthew and sorry to put you through more work; I just wanted to verify exactly which unicode (UTF-16 I take it) were being used to verify if the UNICODE standard expected them to be treated as unique words or single letters within a word. Sanskrit is an alphabet, not an ideograph so each symbol is considered a letter. So I believe your implementation is correct and yes, you are right, re is at fault. There are just accenting characters and letters in that sequence so they should be interpreted as a single word of 6 letters, as you determine, and not one of the first letter. Mind you, I misinterpreted msg190100 in that I thought you were using findall in which case the answer should be 1, but as far as length of extraction, yes, 6, I totally agree. Sorry for the misunderstanding. http://www.unicode.org/charts/PDF/U0900.pdf contains the code chart for Hindi. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1693050 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1693050] \w not helpful for non-Roman scripts
Jeffrey C. Jacobs added the comment: Matthew, I think that is considered a single word in Sanscrit or Thai so Python 3.x is correct. In this case you've written the Sanscrit word for Hindi. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1693050 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1693050] \w not helpful for non-Roman scripts
Jeffrey C. Jacobs added the comment: Maybe you could show us the byte-for-byte hex of the string you're testing so we can examine if it's really a code point intending word boundary or just a code point for the sake of beginning a new character. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1693050 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17980] CVE-2013-2099 ssl.match_hostname() trips over crafted wildcard names
Changes by Jeffrey C. Jacobs timeho...@users.sourceforge.net: -- nosy: +timehorse ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17980 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Adding a new regex module (compatible with re)
Jeffrey C. Jacobs timeho...@users.sourceforge.net added the comment: Although V1, V2 is less wordy, technically the current behavior is version 2.2.2, so logically this should be re.VERSION222 vs. re.VERSION3 vs. re.VERSIONn, with corresponding (?V222), (?V3) and future (?Vn). But that said, I think 2.2.2 can be shorthanded to 2, so basically start counting from there. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Adding a new regex module (compatible with re)
Jeffrey C. Jacobs timeho...@users.sourceforge.net added the comment: On 1 September 2011 16:12, Matthew Barnett rep...@bugs.python.org wrote: Matthew Barnett pyt...@mrabarnett.plus.com added the comment: I think I need a show of hands. For my part, I recommend literal flags, i.e. re.VERSION222, re.VERSION300, etc. Then you know exactly what you're getting and although it may be confusing, we can then slowly deprecate re.VERSION222 so that people can get used to the new syntax. Returning to lurking on my own issue. :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.7 (modifications to current re 2.2.2)
Jeffrey C. Jacobs timeho...@users.sourceforge.net added the comment: +1 on VC -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.7 (modifications to current re 2.2.2)
Jeffrey C. Jacobs timeho...@users.sourceforge.net added the comment: What about a regex flag? Like regex.W or (?w)? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.7 (modifications to current re 2.2.2)
Jeffrey C. Jacobs timeho...@users.sourceforge.net added the comment: My only addition opinion is that re is very much used in deployed python applications and was written not just for correctness but also speed. As such, regex should be benchmarked fairly to show that it is commensurately speedy. I wouldn't not personally object to a slightly slower module, though not one that is noticeably slower and if it can be proven faster in the average case, it's one more check in the box for favorable inclusion. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.7 (modifications to current re 2.2.2)
Jeffrey C. Jacobs timeho...@users.sourceforge.net added the comment: /lurk Re: timings Thanks for the info, John. First of all, I really like those tests and could you please submit a patch or other document so that we could combine them into the python test suite. The python test suite, which can be run as part of 'make test' or IIRC there is a way to run JUST the 2 re test suites which I seem to have senior moment'd, includes a built-in timing output over some of the tests, though I don't recall which ones were being timed: standard cases or pathological (rare) ones. Either way, we should include some timings that are of a standard nature in the test suite to make Matthew's and any other developer's work easier. So, John, if you are not familiar with the test suite, I can look into adding the specific cases you've developed into the test suite so we can have a more representative timing of things. Remember, though, that when run as a single instance, at least in the existing engine, the re compiler caches recent compiles, so repeatedly compiling an expression flattens the overhead in a single run to a single compile and lookup, where as your tests recompile at each test (though I'm not sure what timeit is doing: if it invokes a new instance of python each time, it is recompiling each time, if it is reusing the instance, it is only compiling once). Having not looked at Matthew's regex code recently (nice name, BTW), I don't know if it also contains the compiled expression cache, in which case, adding it in might help timings. Originally, the cache worked by storing ~100 entries and cleared itself when full; I have a modification which increases this to 256 (IIRC) and only removes the 128 oldest to prevent thrashing at the boundary which I think is better if only for a particular pathological case. In any case, don't despair at these numbers, Matthew: you have a lot of time and potentially a lot of ways to make your engine faster by the time 1.7 alpha is coined. But also be forewarned, because, knowing what I know about the current re engine and what it is further capable of, I don't think your regex will be replacing re in 1.7 if it isn't at least as fast as the existing engine for some standard set of agreed upon tests, no matter how many features you can add. I have no doubt, with a little extra monkey grease, we could implement all new features in the existing engine. I don't want to have to reinvent the wheel, of course, and if Matthew's engine can pick up some speed everybody wins! So, keep up the good work Matthew, as it's greatly appreciated! Thanks all! Jeffrey. lurk -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.7 (modifications to current re 2.2.2)
Jeffrey C. Jacobs timeho...@users.sourceforge.net added the comment: Mea culpa et mes apologies, The '-s' option to John's expressions are indeed executed only once -- they are one-time setup lines. The final quoted expression is what's run multiple times. In other words, improving caching in regex will not help. sigh Merci, Antoine! Jeffrey. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.7 (modifications to current re 2.2.2)
Jeffrey C. Jacobs timeho...@users.sourceforge.net added the comment:
Okay, as I said, Atomic Grouping, etc., off a recent 2.6 is already
available and I can do any cleanups requested to those already
mentioned, I just don't want to start any new items at the moment. As
it is, we are still over a year from any of this seeing the light of day
as it's not going to be merged until we start 2.7 / 3.1 alpha.
Fortunately, I think Matthew here DOES have a lot of potential to have
everything wrapped up by then, but I think to summarize everyone's
concern, we really would like to be able to examine each change
incrementally, rather than as a whole. So, for the purposes of this, I
would recommend that you, Matthew, make a version of your new engine
WITHOUT any Atomic Group, variable length look behind / ahead
assertions, reverse string scanning, positional, negated or scoped
inline flags, group key indexing or any other feature described in the
various issues, and that we then evaluate purely on the merits of the
engine itself whether it is worth moving to that engine, and having made
that decision officially move all work to that design if warranted.
Personally, I'd like to see that 'pure' engine for myself and maybe we
can all develop an appropriate benchmark suite to test it fairly against
the existing engine. I also think we should consider things like
presentation (are all lines terminated by column 80), number of
comments, and general readability. IMHO, the current code is conformant
in the line length, but VERY deficient WRT comments and readability, the
later of which it sacrifices for speed (as well as being retrofitted for
iteration rather than recursion). I'm no fan of switch-case, but I
found that by turning the various case statements into bite-sized
functions and adding many, MANY comments, the code became MUCH more
readable at the minor cost of speed. As I think speed trumps
readability (though not blindly), I abandoned my work on the engines,
but do feel that if we are going to keep the old engine, I should try
and adapt my comments to the old framework to make the current code a
bit easier to understand since the framework is more or less the same
code as in the existing engine, just re-arranged.
I think all of the things you've added to your engine, Matthew, can,
with varying levels of difficulty be implemented in the existing Regexp
Engine, though I'm not suggesting that we start that effort. Simply,
let's evaluate fairly whether your engine is worth the switch over.
Personally, I think the engine has some potential -- though not much
better than current WRT readability -- but we've only heard anecdotal
evidence of it's superior speed. Even if the engine isn't faster,
developing speed benchmarks that fairly gage any potential new engine
would be handy for the next person to have a great idea for a rewrite,
so perhaps while you peruse the stripped down version of your engine,
the rest of us can work on modifying regex_tests.py, test_re.py and
re_tests.py in Lib/test specifically for the purpose of benchmarking.
If we can focus on just these two issues ('pure' engine and fair
benchmarks) I think I can devote some time to the later as I've dealt a
lot with benchmarking (WRT the compiler-cache) and test cases and hope
to be a bit more active here.
--
message_count: 69.0 - 70.0
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue2636
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.7 (modifications to current re 2.2.2)
Jeffrey C. Jacobs timeho...@users.sourceforge.net added the comment: Thanks, Antione! Then I think for the most part any changes to Regexp will have to wait for 3.2 / 2.7. -- message_count: 71.0 - 72.0 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue214033] re incompatibility in sre
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment:
The duplicate zero-or-one repeat operator bug described in this issue
originally no longer exists in python 2.6.
However, Trent Mick brings up a fair point in that expressions of the
form (x*)? generate an error (issue 1456280) when internally the '?'
should be passively stripped from the expression by the Python Regular
Expression Compiler because it is redundant. The same goes for
expressions of the form (x*)* (issue 2537). Also, there is a problem
with expressions of the form (x*){n,m} (issue 1633953), since the x*
matches as much as it can, and thus it sees the range repeat operation
as redundant -- in this case I think the range repeat should have the
effect of matching (x*)(x*)(x*)... n to m times, but since the first
time matches everything, the subsequent matches all match zero-width
expressions following the first one. I am tracking all of these issues
under Item 33 of Issue 2636.
The are the 3 known redundant repeat issues, but this one, the zero-or-
one followed by zero-or-one is AFAICT fixed in python 2.6 as the
expression originally listed now passes compile.
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue214033
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1456280] Traceback error when compiling Regex
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment:
This is another version of the redundant repeat issue defined in issues
2537 and 1633953 and although not described by the original report for
issue 214033, the comments further down that issue also describe a
similar situation.
In this case, the problem arises from the '[(](?Pkey[^)]*)?[)]'
portion of your regexp code because you have a zero-or-more repeat
repeated zero-or-one times, which in the current version of python
causes this error. Technically, the outer zero-or-one operator ('?') is
redundant and you can eliminate it, but this IMHO should not cause the
error listed below and I will look into a solution in issue 2636.
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1456280
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1633953] re.compile((.*$){1,4}, re.MULTILINE) fails
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment:
On first blush, this issue sounds quite similar to issue 2537, but I
have been looking at different scenarios and found that there is a
subtle difference because, grammatically:
(?m)(?:.*$)(.*$)
is the same as:
(?m)(.*$){2}
Yet the former compiles while the later raises the exception you list
below. Thus, I think the issue YOU raise is indeed related to the
redundant repeat operator issue numbered 2537, BUT, when I match an
expression with the alternate form, I get an empty string in my capture
group, since in a range repeat over a capture group, only the last group
is captured, while the entire expression matches only the first line,
without the end-line character. Thus, the other thing to remember is
that ^ and $ are zero-width matches, so when you write .*$, you are
saying match up to, but not including, the end of the line. If you
immediately follow that with another .*$, you will start from the point
up to, but not including, the end of the line, which means the next
character is an end of line. Thus, when you reach the second .*$, you
capture nothing because the .* is allowed to be zero-length and you
still haven't advanced PAST the end of the line.
As a working alternative, you could write r'(?m)(?:(.*$)[\r\n]*){1,4}' ,
since this would give you your 1-4 lines, but also consume the carriage
return and line feed characters to get you to the next line.
Since we don't want to change the meaning of $ and ^ to make them
capturing (custom POSIX character classes may make 'capturing' a new
line character easier), and the 'redundant repeat operator' is already
listed as a bug (your expression is essentially saying (.*){1,4}$
because it does not capture the new-line character(s) and thus has a
redundant repeat operation in the range repeat expression), I'm willing
to call this a repeat (technically repeated by as this issue is older)
of issue 2537.
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1633953
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.7 (modifications to current re 2.2.2)
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: Good work, Matthew. Now, another bazaar hint, IMHO, is once of my favourite commands: switch. I generally develop all in one directory, rather than getting a new directory for each branch. Once does have to be VERY careful to type bzr info to make sure the branch you're editing is the one you think it is! but with bzr switch, you do a differential branch switch that allows you to change your development branch quickly and painlessly. This assumes you did a bzr checkout and not a bzr pull. If you did a pull, you can still turn this into a checkout, where all VCS actions are mirrored on the server, by using the 'bind' command. Make sure you push your branch first. You don't need to worry about all this binding, pushing and pulling if you choose checkout, but OTOH, if your connection is over-all very slow, you may still be better off with a pulled branch rather than a checkouted one. Anyway, good catch on those 4 lines and I'll see if I can get your earlier branches up to date. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.7 (modifications to current re 2.2.2)
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: Matthew, I've traced down the patch failures in my merges and now each of the 4 versions of code on Launchpad should compile, though the first 2 do not pass all the negative look-behind tests, though your later 2 do. Any chance you could back-port that fix to the lp:~pythonregexp2.7/python/issue2636-01+09-02+17 branch? If you can, I can propagate that fix to the higher levels pretty quickly. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3255] [proposal] alternative for re.sub
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: Implementing Issue 3482 should solve this problem, and I will try to add it to issue 2636 so that it is captured in the general Regexp 2.7 redesign. -- nosy: +timehorse versions: +Python 2.7 ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3255 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2650] re.escape should not escape underscore
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: -- versions: +Python 2.7, Python 3.1 -Python 2.6, Python 3.0 ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2650 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2650] re.escape should not escape underscore
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: -- nosy: +timehorse ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2650 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1721518] Small case which hangs
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: -- nosy: +timehorse versions: +Python 2.7 -Python 2.4 ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1721518 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1721518] Small case which hangs
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: Tested on 2.6rc2 and slow but successful. Issue 1662851 may be related. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1721518 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1693050] \w not helpful for non-Roman scripts
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: -- nosy: +timehorse versions: +Python 2.7 -Python 2.4 ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1693050 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2537] re.compile(r'((x|y+)*)*') should fail
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: -- nosy: +timehorse versions: +Python 2.7 -Python 2.6 ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2537 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1633953] re.compile((.*$){1,4}, re.MULTILINE) fails
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: -- nosy: +timehorse versions: +Python 2.7 -Python 2.5 ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1633953 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1282] re module needs to support bytes / memoryview well
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: -- nosy: +timehorse ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1282 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue214033] re incompatibility in sre
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: -- nosy: +timehorse ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue214033 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1708652] Exact matching
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: -- nosy: +timehorse versions: +Python 2.7 ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1708652 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1456280] Traceback error when compiling Regex
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: -- nosy: +timehorse ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1456280 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3665] Support \u and \U escapes in regexes
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: -- nosy: +timehorse ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3665 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3482] re.split, re.sub and re.subn should support flags
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: -- nosy: +timehorse ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3482 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3482] re.split, re.sub and re.subn should support flags
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: -- versions: +Python 2.7, Python 3.1 -Python 2.6, Python 3.0 ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3482 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3299] invalid object destruction in re.finditer()
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: -- versions: +Python 2.7 -Python 2.6 ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3299 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3665] Support \u and \U escapes in regexes
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: -- versions: +Python 2.7, Python 3.1 -Python 3.0 ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3665 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1519638] Unmatched Group issue - workaround
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: -- nosy: +timehorse ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1519638 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1519638] Unmatched Group issue - workaround
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: -- versions: +Python 2.7 -Python 2.5 ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1519638 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1662581] the re module can perform poorly: O(2**n) versus O(n**2)
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: -- nosy: +timehorse versions: +Python 2.7 ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1662581 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.7 (modifications to current re 2.2.2)
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: Matthew, Did you upload a public SSH key to your Launchpad account? You're on MS Windows, right? I can try and do an install on an MS Windows XP box or 2 I have lying around and see how that works, but we should try and solve this vexing thing I've noticed about Windows development, which is that Windows cannot understand Unix-style file permissions, and so when I check out Python on Windows and then check it back in, I've noticed that EVERY python and C file is changed by virtue of its permissions having changed. I would hope there's some way to tell Bazaar to ignore 'permissions' changes because I know our edits really have nothing to do with that. Anyway, I'll try a few things visa-vi Windows to see if I get a similar problem; there's also the https://answers.launchpad.net/bazaar forum where you can post your Bazaar issues and see if the community can help. Search previous questions or click the Ask a question button and type your subject. Launchpad's UI is even smart enough to scan your question title for similar ones so you may be able to find a solution right away that way. I use the Launchpad Answers section all the time and have found it usually is a great way of getting help. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.7 (modifications to current re 2.2.2)
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: Great, Matthew!! Now, I'm still in the process of setting up branches related to your work; generally they should be created from a core and set of features implemented for example: To get from Version 2 to Version 3 of your Engine, I had to first check out lp:~pythonregexp2.7/python/issue2636-01+09-02+17 and then push it back onto launchpad as lp:~pythonregexp2.7/python/issue2636-01+09-02+17+26. This way the check-in logs become coherent. So, please hold off on checking your code in until I have your current patch-set checked in, which I should finish by today; I also need to rename some of the projects based on the fact that you also implemented item 26 in most of your patches. Actually, I keep a general To-Do list of what I am up to on the https://code.launchpad.net/~pythonregexp2.7/python/issue2636 whiteboard, which you can also edit, if you want to see what I'm up to. But I'll try to have that list complete by today, fingers crossed! In the mean time, would you mind seeing if you are getting the file permissions issue by doing a checkout or pull or branch and then calling bzr stat to see if this caused Bazaar to add your entire project for checkin because the permissions changed. Thanks and congratulations again! ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.7 (modifications to current re 2.2.2)
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: Thanks, Matthew. My reading of that Answer is that you should be okay because you, I assume, installed the Windows-Native package rather than the cygwin that I first tested. I think the problem is specific to Cygwin as well as the circumstances described in the article. Still, it should be quite easy to verify if you just check out python and then do a stat, as this will show all files whose permissions have changed as well as general changes. Unfortunately, I am still working on setting up those branches, but once I finish documenting each of the branches, I should proceed more rapidly. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.7 (modifications to current re 2.2.2)
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: Phew! Okay, all you patches have been applied as I said in a previous message, and you should now be able to check out lp:~pythonregexp2.7/python/issue2636+01+09-02+17+18+19+20+21+24+26 where you can then apply your latest known patch (rc2+7) to add a fix for the findall / finditer bug. However, please review my changes to: a) lp:~pythonregexp2.7/python/issue2636-01+09-02+17 b) lp:~pythonregexp2.7/python/issue2636-01+09-02+17+26 c) lp:~pythonregexp2.7/python/issue2636-01+09-02+17+18+26 d) lp:~pythonregexp2.7/python/issue2636-01+09-02+17+18+19+20+21+26 To make sure my mergers are what your code snapshots should be. I did get one conflict with patch 5 IIRC where a reverse attribute was added to the SRE_STATE struct, and get a weird grouping error when running the tests for (a) and (b), which I think is a typo; a compile error regarding the afore mentioned missing reverse attribute from patch 3 or 4 in (c) and the SRE_FLAG_REVERSE seems to have been lost in (d) for some reason. Also, if you feel like tackling any other issues, whether they have numbers or not, and implementing them in your current development line, please let me know so I can get all the documentation and development branches set up. Thanks and good luck! ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue433029] SRE: posix classes aren't supported
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: To clarify, you mean named character sets as found in Perl and Emacs, which are normally written, for example, like '[:ALPHANUM:]', right? We are working on that as Item 8 of Issue 2636: Regexp 2.7. If not, please clarify so I nknow what needs to be added. Thanks! -- nosy: +timehorse versions: +Python 2.7 ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue433029 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3299] invalid object destruction in re.finditer()
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: -- nosy: +timehorse ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3299 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.7 (modifications to current re 2.2.2)
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: I've enumerated the current list of Item Numbers at the official Launchpad page for this issue: https://launchpad.net/~pythonregexp2.7 There you will find links to each development branch associated with each item, where a broader description of each issue may be found. I will no longer enumerate the entire list here as it has grown too long to keep repeating; please consult that web page for the most up-to-date list of items we will try to tackle in the Python Regexp 2.7 update. Also, anyone wanting to join the development team who already has a Launchpad account can just go to the Python Regexp 2.7 web site above and request to join. You will need Bazaar to check out, pull or branch code from the repository, which is available at www.bazaar-vcs.org. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1160] Medium size regexp crashes python
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: -- nosy: +timehorse ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1160 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1160] Medium size regexp crashes python
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: It seems that changing the size type of the Regular Expression Byte-code is a nice quick-fix, even though it doubles the size of a pattern. It may have the added benefit that most machine architectures available today are at least partially, if not fully, 32-bit oriented so that retrieving op codes may in fact be faster if we make this change. OTOH, it implies something interesting IMHO with the repeat count limits we currently have. Repeat counts can be explicitly set up to 65534 times because 65535, being the largest number you can express in a 16-bit unsigned integer, is currently reserved to mean Infinite. It seems to me this is a great opportunity to set that limit to (unsigned long)-1, since that repeat count is incredibly large. OTOH, if size is an issue, we could change the way sizes are expressed in the Regexp Op Codes (typically in skip counts) to be 15-bit, with the Most Significant Bit being reserved for 'extended' expressions. In this way, a value of 0x could be expressed as: 0x 0x 0x0003 Of course, parsing number in this form is a pain, to say the least, and unlike in Python, the C-library would not play nicely if someone tried to express a number that could not fit into what the architecture defined an int to be. Plus, there is the problem of how you express Infinite with this scheme. The advantage though would be we don't have to change the op-code size and these 'extended' counts would be very rare indeed. Over all, I'm more of an Occam's Razor fan in that the simplest solution is probably the best: just change the op-code size to unsigned long (which, on SOME architectures would actually make it 64-bits!) and define the 'Infinite' constant as (unsigned long)-1. Mind you, I prefer defining the constant in Python, not C, and it would be hard for Python to determine that particular value being that Python is meant to be 'the same' regardless of the underlying architecture, but that's another issue. Anyway, as 2.6 is in Beta, this will have to wait for Python 2.7 / 3.1, and so I will add an item to Issue 2636 with respect to it. -- versions: +Python 2.7 -Python 2.5 ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1160 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.7 (modifications to current re 2.2.2)
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: Good catch, Matthew, and if you spot any other outstanding Regular Expression issues feel free to mention them here. I'll give issue 1160 an item number of 25 and think all we need to do here is change SRE_CODE to be typedefed to an unsigned long and change the repeat count constants (which would be easier if we assume item 10: shared constants). ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1647489] zero-length match confuses re.finditer()
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: Perl gives this result for your new expression: ,undef,undef undef,undef,abc undef,,undef I think it has to do with not thinking of a string as a sequence of characters, but as a sequence of characters separated by null-space. Null-space is can be captured, but ONLY if it is part of a zero-width match, and once captured, it can no longer be captured by another zero-width expression. This is in keeping which what I see as Perl's behaviour, namely that the (q*) group never participates in the first match because, initially the (^z*) captures it. OTOH, when it gets to the null-space AFTER the 'abc' capture, the (^z*) cannot participate because it has a at-beginning restriction. The evaluator then moves on to the (q*), which has no such restriction and this time it matches, consuming the final null-space. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1647489 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.7 (modifications to current re 2.2.2)
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: Hmmm. Well, some of those are already covered: #2636: self #1160: Item 25 #1647489 : Item 24 #3511: Item 23 #3825: Item 9-2 #433028 : Item 21 #433027 : Item 20 #433024 : Item 19 #3262: Item 22 #3299: TBD #3665: TBD #3482: TBD #1519638 : TBD #1662581 : TBD #3255: TBD #2650: TBD #433030 : Item 1 #1721518 : TBD #1693050 : TBD #2537: TBD #1633953 : TBD #1282: TBD #814253 : TBD (but I think you implemented this, didn't you Matthew?) #214033 : TBD #1708652 : TBD #694374 : TBD #433029 : Item 8 I'll have to get nosy and go over the rest of these to see if any of them have already been solved, like the duplicate test case issue from a while ago, but someone forgot to close them. I'm thinking specifically the '\u' escape sequence one. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue433024] SRE: (?flag) isn't properly scoped
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: -- nosy: +timehorse ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue433024 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue433027] SRE: (?-flag) is not supported.
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: -- nosy: +timehorse ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue433027 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue433028] SRE: (?flag:...) is not supported
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: -- nosy: +timehorse ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue433028 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3825] Major reworking of Python 2.5.2 re module
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: Matthew, I am really happy that you are making such progress on your engine, but can I PLEASE ask you to slow down for a moment? We have a lot of issues already listed in issue 2636 that is a catch-all for any Python 2.7 Regexp improvements, including your new engine, and I have been working frantically to try and document all the changes YOU are making here into the general Regexp 2.7 modification thread and setting up development trees in my Bazaar VCS repository for your work. There is also a recommended process for patching which makes it easier for the moderators to accept your patches which is to provide dis-entangled functionality and letting each improvement stand on its own two feet. In other words, let your engine stand ONLY on it's 2x speed improvements. We already have an implementation of Atomic Grouping / Possessive Qualifiers in issue 2636 but you have a version of your engine with both. We have no such 'feature-only' implementation for Variable-Length Look-Behind, for a Reverse flag, for Positionally Dependent modifier flags or modifier negation flags, as well as the zero-width Regular Expression split feature, though you and I completely agree these would all be great things to have! The more features you add to your engine as an all-or-nothing proposition, the less likely the moderators are going to be to adapt it because it's harder for them to examine the merits of each individual piece. That is why issue 2636 is broken up into items (currently 1 - 18, with your proposals bringing that up toward 22) and where alternate, combined features are provided if implementing 1 features would affect the implementation of another. Please understand that I personally have no problem with you redesigning large swaths of the Python Regular Expression engine. I would personally, like to see one of the design goals of any new engine not only be speed but better source comments because my main beef with the current engine is that it took me a month to understand and part of my redesign in issue 2636 9-1 was to add copious comments to the engine so that future developers would understand what was going on and be able to pick up from my work. I am not proposing we use my 9-1 engine because it is 8% slower than the current engine and I don't intend to propose anything slower. But it would be nice if you could add lots of comments to your engine so that others could help develop features against it. None the less, I will fully support your engine if it does indeed perform substantially and measurably faster and am happy to see all the Regexp issues you are finding are finally being implemented, all be it entangled with your engine. But let's return to the fundamentals of what you propose IN THIS THREAD, which simply to propose a new Regexp Engine which is 2x faster than the existing engine (Which I have allocated item 9-2 in the issue 2636 thread). I am not trying to put more work on your hands -- in fact, what I am trying to do is get us to co-operate on a better python Regexp Engine so that I can help you to achieve your goals. Please read issue 2636 and join the discussion there; feel free to add any new items you feel are missing from my existing list. And remember, each new feature needs tests and documentation changes. I have been doing each for any feature I undertake and would be happy to share those skills with you. Let's work together to see your engine be the new model, okay? Thanks. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3825 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3511] Incorrect charset range handling with ignore case flag?
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: -- nosy: +timehorse ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3511 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3511] Incorrect charset range handling with ignore case flag?
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: I think this is even more complicated when you consider that localization my be an issue. Consider Á: is this grammatically before A or after a? From a character set point of view, it is typically after a but when Locale is taken into account, all that is done is there is a change to relative ordering, so Á appears somewhere before A and B. But when this is done, does that mean that [9-Á] is going to cover ALL uppercase and ALL lowercase and ALL characters with ord from 91 to 96 and 123 to 127 and all kinds of other UNICODE symbols? And how will this effect case-insensitivity. In a sense, I think it may only be safe to say that character class ranges are ONLY appropriate over Alphabetic character ranges or numeric character ranges, since the order of the ASCII symbols between 0 and 47, 56 and 64, 91 adn 96 and 123 and 127, though well-defined, are none the less implementation dependent. When we bring UNICODE into this, things get even more befuddled with some Latin characters in Latin-1, some in Latin-2, Cyrillic, Hebrew, Arabic, Chinese, Japanese and Korean character sets just to name a few of the most common! And how does a total ordering of characters apply to them? In the end, I think it's just dangerous to define character group ranges that span the gap BETWEEN numbers and alphabetics. Instead, I think a better solution is simply to implement Emacs / Perl style named character classes as in issue 2636 sub-item 8. I do agree this is a problem, but as I see it, the solution may not be that simple, especially in a UNICODE world. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3511 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.7 (modifications to current re 2.2.2)
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: Thanks for weighing in Matthew! Yeah, I do get some flack for item 2 because originally item 3 wasn't supposed to cover named groups but on investigation it made sense that it should. I still prefer 2 over-all but the nice thing about them being separate items is that we can accept 2 or 3 or both or neither, and for the most part development for the first phase of 2 is complete though there is still IMHO the issue of UNICODE name groups (visa-vi item 14) and the name collision problem which I propose fixing with an Attribute / re.A flag. So, I think it may end up that we could support both 3 by default and 2 via a flag or maybe 3 and 2 both but with 2 as is, with name collisions hidden (i.e. if you have r'(?Pstring...)' as your capture group, typing m.string will still give you the original comparison string, as per the current python documentation) but have collision-checking via the Attribute flag so that with r'(?A)(?Pstring...)' would not compile because string is a reserved word. Your interpretation of 4 matches mine, though, and I would definitely suggest using Perl's \g-n notation for relative back-references, but further, I was thinking, if not part of 4, part of the catch-all item 11 to add support for Perl's (?name...) as a synonym for Python's (?Pname...) and Perl's \kname for Python's (?P=name) notation. The evolution of Perl's name group is actually interesting. Years ago, Guido had a conversation with Larry Wall about using the (?P...) capture sequence for python-specific Regular Expression blocks. So Python went ahead and implemented named capture groups. Years later, the Perl folks thought named capture groups were a neat idea and adapted them in the (?..) form because Python had restricted the (?P...) notation to themselves so they couldn't use our even if they wanted to. Now, though, with Perl adapting (?..), I think it inevitable that Java and even C++ may see this as the defacto standard. So I 100% agree, we should consider supporting (?name...) in the parser. Oh, and as I suggested in Issue 3825, I have these new item proposals: Item 18: Add a re.REVERSE, re.R (?r) flag for reversing the direction of the String Evaluation against a given Regular Expression pattern. See issue 516762, as implemented in Issue 3825. Item 19: Make various in-line flags positionally dependant, for example (?i) makes the pattern before this case-sensitive but after it case-insensitive. See Issue 433024, as implemented in Issue 3825. Item 20: All the negation of in-line flags to cancel their effect in conditionally flagged expressions for example (?-i). See Issue 433027, as implemented in Issue 3825. Item 21: Allow for scoped flagged expressions, i.e. (?i:...), where the flag(s) is applied to the expression within the parenthesis. See Issue 433028, as implemented in Issue 3825. Item 22: Zero-width regular expression split: when splitting via a regular expression of Zero-length, this should return an expression equivalent to splitting at each character boundary, with a null string at the beginning and end representing the space before the first and after the last character. See issue 3262. Item 23: Character class ranges over case-insensitive matches, i.e. does (?i)[9-A] contain '_' , whose ord is greater than the ord of 'A' and less than the ord of 'a'. See issue 5311. And I shall create a bazaar repository for your current development line with the unfortunately unwieldy name of lp:~timehorse/python/issue2636-01+09-02+17+18+19+20+21 as that would, AFAICT, cover all the items you've fixed in your latest patch. Anyway, great work Matthew and I look forward to working with you on Regexp 2.7 as you do great work! ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1647489] zero-length match confuses re.finditer()
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: -- nosy: +timehorse ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1647489 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1647489] zero-length match confuses re.finditer()
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: Hmmm. This strikes me as a bug, beyond the realm of Issue 3262. The two items may be related, but the dropping of the 'a' seems like unexpected behaviour that I doubt any current code is expecting to occur. Clearly, what is going on is that the Engine starts scanning at the 'a', finds the Zero-Width match and, having found a match, increments its pointer within the input string, thus skipping the 'a' when it matches 'bc'. If it is indeed a bug, I think this should be considered for inclusion in Python 2.6 rather than being part of the new Engine Design in Issue 3626. I think the solution would simply be to not increment the ptr (which points to the input string) when findall / finditer encounters a Zero-Width match. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1647489 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1647489] zero-length match confuses re.finditer()
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: Never mind inclusion in 2.6 as no-one has repeated this bug in re-world examples yet so it's going to have to wait for the Regexp 2.7 engine in issue 2636. -- versions: +Python 2.7 -Python 2.5 ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1647489 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1647489] zero-length match confuses re.finditer()
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: Ah, I see the problem, if ptr is not incremented, then it will keep matching the first expression, (^z*), so it would have to both 'skip' the 'a' and NOT skip the 'a'. Hmm. You're right, Matthew, this is pretty complicated. Now, for your expression, Matthew, r'(z*)|(^q*)|(\w+)', Perl gives: ,undef,undef undef,undef,abc ,undef,undef Meaning it doesn't even bother matching the ^q* since the ^z* matches first. This seems the logical behaviour and fits with the idea that a Zero-Width match would both only match once and NOT consume any characters. An internal flag would just have to be created to tell the 2 find functions whether the current value of ptr would allow for a No Zero-Width Match option on second go-around. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1647489 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.7 (modifications to current re 2.2.2)
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: Good catch on issue 1647489 Matthew; it looks like this is where that bug fix will end up going. But, I am unsure if the solution for this issue is going to be the same as for 3262. I think the solution here is to add an internal flag that will keep track of whether the current character had previously participated in a Zero-Width match and thus not allow any subsequent zero-width matches associated beyond the first, and at the same time not consuming any characters in a Zero-width match. Thus, I have allocated this fix as Item 24, but it may be later merged with 22 if the solutions turn out to be more or less the same, likely via a 22+24 thread. The main difference, though, as I see it, is that the change in 24 may be considered a bug where the general consensus of 22 is that it is more of a feature request and given Guido's acceptance of a flag-based approach, I suggest we allocate re.ZEROWIDTH, re.Z and (?z) flags to turn on the behaviour you and I expect, but still think that be best as a 2.7 / 3.1 solution. I would also like to add a from __futurue__ import ZeroWidthRegularExpressions or some such to make this the default behaviour so that by version 3.2 it may indeed be considered the default. Anyway, I've allocated all the new items in the launchpad repository so feel free to go to http://www.bazaar-vcs.org/ and install Bazaar for windows so you can download any of the individual item development threads and try them out for yourself. Also, please consider setting up a free launchpad account of your very own so that I can perhaps create a group that would allow us to better share development. Thanks again Matthew for all your greatly appreciated contributions! ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.7 (modifications to current re 2.2.2)
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: I've moved all the development branches to the ~pythonregexp2.7 team so that we can work collaboratively. You just need to install Bazaar, join www.launchpad.net, upload your public SSH key and then request to be added to the pythonregexp2.7 team. At that point, you can check out any code via: bzr co lp:~pythonregexp2.7/python/issue2636-* This should make co-operative development easier. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3262] re.split doesn't split with zero-width regex
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: I think Mike Coleman proposal of enabling this behaviour via flag is probably best and IMHO we should consider it under these circumstances. Intuitively, I think you're interpretation of what re.split should do under zero-width conditions is logical, and I almost think this should be a 2-minor number transition à la from __future__ import zeroWidthRegexpSplit if we are to consider it as the long-term 'right thing to do'. 3000 (3.0) seems a good place to also consider it for true overhaul / reexamination, especially as we are writing 'upgrade' scripts for many of the other Python features. However, I would say this, Guido has spoken and it may be too late for the pebbles to vote. I would like to add this patch as a new item to the general Regexp Enhancements thread of issue 2636 though, as I think it is an idea worth considering when overhauling Regexp. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3262 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3262] re.split doesn't split with zero-width regex
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: -- nosy: +timehorse ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3262 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3654] Duplicated test name in regex test script
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: -- nosy: +timehorse ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3654 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue516762] have a way to search backwards for re
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: -- nosy: +timehorse ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue516762 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.7 (modifications to current re 2.2.2)
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment:
Update 16 Sep 2008:
Based on the work for issue #3825, I would like to simply update the
item list as follows:
1) Atomic Grouping / Possessive Qualifiers (See also Issue #433030)
[Complete]
2) Match group names as attributes (e.g. match.foo) [Complete save
issues outlined above]
3) Match group indexing (e.g. match['foo'], match[3])
4) Perl-style back-references (e.g. compile(r'(a)\g{-1}'), and possibly
adding the r'\k' escape sequence for keywords.
5) Parenthesis-Aware Python Comment (e.g. r'(?P#...)') [Complete]
6) Expose support for Template expressions (expressions without repeat
operators), adding test cases and documentation for existing code.
7) Larger compiled Regexp cache (256 vs. 100) and reduced thrashing
risk. [Complete]
8) Character Classes (e.g. r'[:alphanum:]')
9) Proposed Engine redesigns and cleanups (core item only contains
cleanups and comments to the current design but does not modify the design).
9-1) Single-loop Engine redesign that runs 8% slower than current.
[Complete]
9-1-1) 3-loop Engine redesign that runs 10% slower than current. [Complete]
9-2) Matthew Bernett's Engine redesign as per issue #3825
10) Have all C-Python shared constants stored in 1 place
(sre_constants.py) and generated by that into C constants
(sre_constants.h). [Complete AFAICT]
11) Scan Perl 5.10.0 for other potential additions that could be
implemented for Python.
12) Documentation suggestions by Jim J. Jewett [Complete]
13) Add grouptuples method to the Match object (i.e. match.grouptuples()
returns (index, name or None, value) ) suitable for iteration.
14) UNICODE match group names, as per PEP-3131.
15) Add __doc__ strings and other Python niceties to the Pattern_Type,
Match_Type and Scanner_Type (experimental).
16) Implement any remaining TODOs and FIXMEs in the Regexp modules.
16-1) Allow for the disassociation of a source string from a Match_Type,
assuming this will still leave the object in a reasonable state.
17) Variable-length [Positive and Negative] Look-behind assertions, as
described and implemented in Issue #3825.
---
Now, we have a combination of Items 1, 9-2 and 17 available in issue
#3825, so for now, refer to that issue for the 01+09-02+17 combined
solution. Eventually, I hope to merge the work between this and that issue.
I sadly admit I have made not progress on this since June because
managing 30 some lines of development, some of which having complex
diamond branching, e.g.:
01 is the child of Issue2636
09 is the child of Issue2636
10 is the child of Issue2636
09-01 is the child of 09
09-01-01 is the child of 09-01
01+09 is the child of 01 and 09
01+10 is the child of 01 and 10
09+10 is the child of 09 and 10
01+09-01 is the child of 01 and 09-01
01+09-01-01 is the child of 01 and 09-01-01
09-01+10 is the child of 09-01 and 10
09-01-01+10 is the child of 09-01-01 and 10
Which all seems rather simple until you wrap your head around:
01+09+10 is the child of 01, 09, 10, 01+09, 01+10 AND 09+10!
Keep in mind the reason for all this complex numbering is because many
issues cannot be implemented in a vacuum: If you want Atomic Grouping,
that's 1 implementation, if you want Shared Constants, that's a
different implementation. but if you want BOTH Atomic Grouping and
Shared Constants, that is a wholly other implementation because each
implementation affects the other. Thus, I end up with a plethora of
branches and a nightmare when it comes to merging which is why I've been
so slow in making progress. Bazaar seems to be very confused when it
comes to a merge in 6 parts between, for example 01, 09, 10, 01+09,
01+10 and 09+10, as above. It gets confused when it sees the same
changes applied in a previous merge applied again, instead of simply
realizing that the change in one since last merge is EXACTLY the same
change in the other since last merge so effectively there is nothing to
do; instead, Bazaar gets confused and starts treating code that did NOT
change since last merge as if it was changed and thus tries to role back
the 01+09+10-specific changes rather than doing nothing and generates a
conflict. Oh, that I could only have a version control system that
understood the kind of complex branching that I require!
Anyway, that's the state of things; this is me, signing out!
--
title: Regexp 2.6 (modifications to current re 2.2.2) - Regexp 2.7
(modifications to current re 2.2.2)
versions: +Python 2.7 -Python 2.6
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2636
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3825] Major reworking of Python 2.5.2 re module
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: Well, I implemented this months ago, but have been busy with other things so I haven't updated in a while. I noticed that the current version is missing my patches for Atomic Grouping / Possessive Qualifiers and a number of other patches I added in #2636 , but I do have working test cases and documentation updates for all the features I've so far implemented as well as splitting my work into separate sub-issues to make individual selection easier -- though with a number of my modifications, I found that there are SO MANY co-dependencies between, say, an engine modification (item 9) and adding Atomic Grouping / Possessive Qualifiers (item 1) and using shared Engine Constants (item 10) that I need a branch for Atomic, a branch for Atomic + Engine Mod 1, Atomic + Engine Mod 2, Atomic + Shared Constants, Atomic + Engine Mod 1 + Shared Constants AND Atomic + Engine Mod 2 + Shared Constants, and those were just THREE item co-dependencies. My code is all off of the trunk line for 2.6 and is currently available via my Bazaar repository under https://code.launchpad.net/~timehorse, where you can access any source tree via the bazaar version control client. The main reason I got stumped in my development which might otherwise have implemented ALL the issues I intended by now is that very situation I just described where development of new features is NOT mutually independent. You can see by all my branches that the multiplicity of A or B or C is just nightmarish, but what had to be done to keep issues independent. Anyway, I'm looking forward to having a look at your suggestions and think we may take best advantage with combining our work visa vi these things; after all, there's no point re-inventing the wheel. Thanks again for your contribution, Matthew! -- nosy: +timehorse ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3825 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3825] Major reworking of Python 2.5.2 re module
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: I have uploaded my test cases for Atomic Grouping / Possessive Qualifier, which is the common code we seem to have developed, as this may be of use to you. I also have documentation, but for now, would you mind running these tests against your code to see what the test outputs and also, how did you come up with the 2x result? Was that running the test suite? Usually, the regexp module is benchmarked against its test suite and there are timings built into that, so it may be useful if you could run the unmodified Lib/test/test_re.py you got from the trunk against the original code before modification and your modification, and do so a few times to get a good average result on multi-tasking systems, and post the results here so we can get a good statistical feel for how your new engine improves efficiency. Certainly, I support any Engine that works faster, as I myself have tried to make it faster but ended up with something 8% slower instead, alas. Also, good thinking on fixing the Negative Look-behind variable-width issue; I wish I'd thought of that, but I am curious about something: did you remove the optimization for fixed-width look-behind? The old code only allowed fixed with because that test can be done quickly; I noticed your code adds a lot of new REV opcodes to handle back-tracking and I assume look-behind logic for variable-width look-behind. It would be handy if the compiler and engine would be able to differentiate between fixed-width look-behind (optimized as was originally) and variable-width (using your advanced code). Thanks to AMK for some of these suggestions. Your changes are quite radical though so I am still trying to wade through them all and I still don't have a full-picture of how you've changed things, but there are some good ideas here, IMHO, especially if you do indeed get 2x speedup. Added file: http://bugs.python.org/file11499/test_re.py ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3825 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.6 (modifications to current re 2.2.2)
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: Thanks for weighing in Mark! Actually, your point is valid and quite fair, though I would not assume that Item 3 would be included just because Item 2 isn't. I will do my best to develop both, but I do not make the final decision as to what python includes. That having been said, 3 seems very likely at this point so we may be okay, but let me give this one more try as I think I have a better solution to make Item 2 more palatable. Let's say we have 5 choices here: a) Simply disallow the exposure of match group name attributes if the names collide with an existing member of the basic Match Object interface. b) Expose the reserved names through a special prefix notation, and for forward compatibility, expose all names via this prefix notation. In other words, if the prefix was 'k', match.kpos could be used to access pos; if it was '_', match._pos would be used. If Item 3 is implemented, it may be sufficient to allow access via match['pos'] as the canonical way of handling match group names using reserved words. c) Don't expose the names directly; only expose them through a prefixed name, e.g. match._pos or match.kpos. d) (As Mark suggested) we drop Item 2 completely. I have not invested much work in this so that would not bother me, but IMHO I actually prefer Item 2 to 3 so I would really like to see it preserved in some form. e) Add an option, re.MATCH_ATTRIBUTES, that is used as a Match Creation flag. When the re.MATCH_ATTRIBUTES or re.A flag is included in the compile, or (?a) is included in the pattern, it will do 2 things. First, it will raise an exception if either a) there exists an unnamed capture group or b) the capture group name is a reserved keyword. In addition to this, I would put in a hook to support a from __future__ so that any post 2.6 changes to the match object type can be smoothly integrated a version early to allow programmers to change when any future changes come. Secondly, I would *conditionally* allow arbitrary capture group name via the __getattr__ handler IFF that flag was present; otherwise you could not access Capture Groups by name via match.foo. I really like the idea of e) so I'm taking Item 2 out of the ready for merge category and going to put it in the queue for the modifications spelled out above. I'm not too worried about our flags differing from Perl too much as we did base our first 4 on Perl (x, s, m, i), but subsequently added Unicode and Locale, which Perl does not have, and never implemented o (since our caching semantic already pretty much gives every expression that), e (which is specific to Perl syntax AFAICT) and g (which can be simulated via re.split). So I propose we take A and implement it as I've specified and that is the current goal of Item 2. Once this is done and working, we can decide whether it should be included in the python trunk. How does that sound to you, Mark and anyone else who wishes to weigh in? ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.6 (modifications to current re 2.2.2)
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: Well, it's time for another update on my progress... Some good news first: Atomic Grouping is now completed, tested and documented, and as stated above, is classified as issue2636-01 and related patches. Secondly, with caveats listed below, Named Match Group Attributes on a match object (item 2) is also more or less complete at issue2636-02 -- it only lacks documentation. Now, I want to also update my list of items. We left off at 11: Other Perl-specific modifications. Since that time, I have spawned a number of other branches, the first of which (issue2636-12) I am happy to announce is also complete! 12) Implement the changes to the documentation of re as per Jim J. Jewett suggestion from 2008-04-24 14:09. Again, this has been done. 13) Implement a grouptuples(...) method as per Mark Summerfield's suggest on 2008-05-28 09:38. grouptuples would take the same filtering parameters as the other group* functions, and would return a list of 3- tuples (unless only 1 group was requested). It should default to all match groups (1..n, not group 0, the matching string). 14) As per PEP-3131 and the move to Python 3.0, python will begin to allow full UNICODE-compliant identifier names. Correspondingly, it would be the responsibility of this item to allow UNICODE names for match groups. This would allow retrieval of UNICODE names via the group* functions or when combined with Item 3, the getitem handler (m[u'...']) (03+14) and the attribute name itself (e.g. getattr(m, u'...')) when combined with item 2 (02+14). 15) Change the Pattern_Type, Match_Type and Scanner_Type (experimental) to become richer Python Types. Specifically, add __doc__ strings to each of these types' methods and members. 16) Implement various FIXMEs. 16-1) Implement the FIXME such that if m is a MatchObject, del m.string will disassociate the original matched string from the match object; string would be the only member that would allow modification or deletion and you will not be able to modify the m.string value, only delete it. - Finally, I want to say a couple notes about Item 2: Firstly, as noted in Item 14, I wish to add support for UNICODE match group names, and the current version of the C-code would not allow that; it would only make sense to add UNICODE support if 14 is implemented, so adding support for UNICODE match object attributes would depend on both items 2 and 14. Thus, that would be implemented in issue2636-02+14. Secondly, there is a FIXME which I discussed in Item 16; I gave that problem it's own item and branch. Also, as stated in Item 15, I would like to add more robust help code to the Match object and bind __doc__ strings to the fixed attributes. Although this would not directly effect the Item 2 implementation, it would probably involve moving some code around in its vicinity. Finally, I would like suggestions on how to handle name collisions when match group names are provided as attributes. For instance, an expression like '(?Ppos.*)' would match more or less any string and assign it to the name pos. But pos is already an attribute of the Match object, and therefore pos cannot be exposed as a named match group attribute, since match.pos will return the usual meaning of pos for a match object, not the value of the capture group names pos. I have 3 proposals as to how to handle this: a) Simply disallow the exposure of match group name attributes if the names collide with an existing member of the basic Match Object interface. b) Expose the reserved names through a special prefix notation, and for forward compatibility, expose all names via this prefix notation. In other words, if the prefix was 'k', match.kpos could be used to access pos; if it was '_', match._pos would be used. If Item 3 is implemented, it may be sufficient to allow access via match['pos'] as the canonical way of handling match group names using reserved words. c) Don't expose the names directly; only expose them through a prefixed name, e.g. match._pos or match.kpos. Personally, I like a because if Item 3 is implemented, it makes a fairly useful shorthand for retrieving keyword names when a keyword is used for a name. Also, we could put a deprecation warning in to inform users that eventually match groups names that are keywords in the Match Object will eventually be disallowed. However, I don't support restricting the match group names any more than they already are (they must be a valid python identifier only) so again I would go with a) and nothing more and that's what's implemented in issue2636-02.patch. - Now, rather than posting umteen patch files I am posting one bz2- compressed tar of ALL patch files for all threads, where each file is of the form: issue2636(-\d\d|+\d\d)*(-only)?.patch For instance, issue2636-01.patch is the p1 patch that is a difference between the current Python trunk
[issue2636] Regexp 2.6 (modifications to current re 2.2.2)
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: Removed file: http://bugs.python.org/file10052/issue2636-09.patch ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.6 (modifications to current re 2.2.2)
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: Removed file: http://bugs.python.org/file10467/issue2636.diff ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.6 (modifications to current re 2.2.2)
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: Removed file: http://bugs.python.org/file10428/issue2636-05-only.diff ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.6 (modifications to current re 2.2.2)
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: Removed file: http://bugs.python.org/file10468/issue2636-05.diff ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.6 (modifications to current re 2.2.2)
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: Removed file: http://bugs.python.org/file10469/issue2636-07.diff ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.6 (modifications to current re 2.2.2)
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: Removed file: http://bugs.python.org/file10470/issue2636-07-only.diff ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue433030] SRE: Atomic Grouping (?...) is not supported
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: I have finished work on the Atomic Grouping / Possessive Qualifiers support and am including a patch to achieve this; however, http://bugs.python.org/issue2636 should be consulted for the complete list of changes in the works for the Regexp engine. -- keywords: +patch Added file: http://bugs.python.org/file10646/issue2636-01.patch ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue433030 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue433030] SRE: Atomic Grouping (?...) is not supported
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: Removed file: http://bugs.python.org/file9897/PyLibDiffs.txt ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue433030 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.6 (modifications to current re 2.2.2)
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: Sorry, as I stated in the last post, I generated the patches then realized that I was missing the documentation for Item 2, so I have updated the issue2636-02.patch file and am attaching that separately until the next release of the patch tarball. issue2636-02-only.patch should be ignored and I will only regenerate it with the correct documentation in the next tarball release so I can move on to either Character Classes or Relative Back-references. I wanna pause Item 3 for the moment because 2, 3, 13, 14, 15 and 16 all seem closely related and I need a break to allow my mind to wrap around the big picture before I try and tackle each one. Added file: http://bugs.python.org/file10647/issue2636-02.patch ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.6 (modifications to current re 2.2.2)
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: Added file: http://bugs.python.org/file10467/issue2636.diff ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.6 (modifications to current re 2.2.2)
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: Removed file: http://bugs.python.org/file10427/issue2636.diff ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.6 (modifications to current re 2.2.2)
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: Added file: http://bugs.python.org/file10468/issue2636-05.diff ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.6 (modifications to current re 2.2.2)
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: Removed file: http://bugs.python.org/file10429/issue2636-05.diff ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.6 (modifications to current re 2.2.2)
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: Mark scribbled: One possible solution would be a grouptuples() function that returned a tuple of 3-tuples (index, name, captured_text) with the name being None for unnamed groups. Hmm. Well, that's not a bad idea at all IMHO and would, AFAICT probably be easier to do than (2) but I would still do (2) but will try to add that to one of the existing items or spawn another item for it since it is kind of a distinct feature. My preference right now is to finish off the test cases for (7) because it is already coded, then finish the work on (1) as that was the original reason for modification then on to (2) then (3) as they are related and then I don't mind tackling (8) because I think that one shouldn't be too hard. Interestingly, the existing engine code (sre_parse.py) has a place-holder, commented out, for character classes but it was never properly implemented. And I will warn that with Unicode, I THINK all the character classes exist as unicode functions or can be implemented as multiple unicode functions, but I'm not 100% sure so if I run into that problem, some character classes may initially be left out while I work on another item. Anyway, thanks for the input, Mark! ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.6 (modifications to current re 2.2.2)
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: Added file: http://bugs.python.org/file10428/issue2636-05-only.diff __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.6 (modifications to current re 2.2.2)
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: I am finally making progress again, after a month of changing my patches from my local svn repository to bazaar hosted on launchpad.net, as stated in my last update. I also have more or less finished the probably easiest item, #5, so I have a full patch for that available now. First, though, I want to update my No matter what patch, which is to say these are the changes I want to make if any changes are made to the Regexp code. Added file: http://bugs.python.org/file10427/issue2636.diff __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.6 (modifications to current re 2.2.2)
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: Added file: http://bugs.python.org/file10429/issue2636-05.diff __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.6 (modifications to current re 2.2.2)
Changes by Jeffrey C. Jacobs [EMAIL PROTECTED]: Removed file: http://bugs.python.org/file10056/issue2636-05.patch __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.6 (modifications to current re 2.2.2)
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: I am making my changes in a Bazaar branch hosted on Launchpad. It took me quite a while to get things set up more-or-less logically but there they are and I'm currently trying to re-apply my local changes up to today into the various branches I have. Each of the 11 issues I outlined originally has its own branch, with a root branch from which all these branches are derived to serve as a place for a) merging in python 2.6 alpha concurrent development (merges) and to apply any additional re changes that don't fall into any of the other categories, of which I have so far found only 2 small ones. Anyway, if anyone is interested in monitoring my progress, it is available at: https://code.launchpad.net/~timehorse/ I will still post major milestones here, but one can monitory day-to-day progress on Launchpad. Also on launchpad you will find more detail on the plans for each of the 11 modifications, for the curious. Thanks again for all the advice! __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.6 (modifications to current re 2.2.2)
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: Thank you and Merci Antoine! That is a good point. It is clearly specific to the compiler whether a switch-case will be turned into a series of conditional branches or simply creating an internal jump table with lookup. And it is true that most compilers, if I understand correctly, use the jump-table approach for any switch-case over 2 or 3 entries when the cases are tightly grouped and near 0. That is probably why the original code worked so fast. I'll see if I can combine the best of both approaches. Thanks again! __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue433030] SRE: Atomic Grouping (?...) is not supported
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: Quick update (see thread 2636 for other updates to the Regex functionality): I do have a design in my mind for how to accomplish both Atomic Grouping and Possessive Qualifiers and it would work with either the existing Engine design or the new design I have been playing with. It's just a matter of getting around to implementing it, testing it and documenting it. I currently am having problem with Bazaar / Launchpad which are beyond the scope of this issue but are effecting my work. Those issues can be found on the Launchpad site if you look for Answers / Bugs originating from timehorse. Tracker [EMAIL PROTECTED] http://bugs.python.org/issue433030 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.6 (modifications to current re 2.2.2)
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment:
Thanks Jim for your thoughts!
Armaury has already explained about Perl 5.10.0. I suppose it's like
Macintosh version numbering, since Mac Tiger went from version 10.4.9 to
10.4.10 and 10.4.11 a few years ago. Maybe we should call Python 2.6
Python 2.06 just in case. But 2.6 is the known last in the 2 series so
it's not a problem for us! :)
as well as add a few python-specific
because this also adds to the scope.
At this point the only python-specific changes I am proposing would be
items 2, 3 (discussed below), 5 (discussed below), 6 and 7. 6 is only a
documentation change, the code is already implemented. 7 is just a
better behavior. I think it is RARE one compiles more than 100 unique
regular expressions, but you never know as projects tend to grow over
time, and in the old code the 101st would be recompiled even if it was
just compiled 2 minutes ago. The patch is available so I leave it to
the community to judge for themselves whether it is worth it, but as you
can see, it's not a very large change.
2) Make named matches direct attributes
of the match object; i.e. instead of m.group('foo'),
one will be able to write simply m.foo.
3) (maybe) make Match objects subscriptable, such
that m[n] is equivalent to m.group(n) and allow slicing.
(2) and (3) would both be nice, but I'm not sure it makes sense to do
*both* instead of picking one.
Well, I think named matches are better than numbered ones, so I'd
definitely go with 2. The problem with 2, though, is that it still
leaves the rather typographically intense m.group(n), since I cannot
write m.3. However, since capture groups are always numbered
sequentially, it models a list very nicely. So I think for indexing by
group number, the subscripting operator makes sense. I was not
originally suggesting m['foo'] be supported, but I can see how that may
come out of 3. But there is a restriction on python named matches that
they have to be valid python and that strikes me as 2 more than 3
because 3 would not require such a restriction but 2 would. So at least
I want 2, but it seems IMHO m[1] is better than m.group(1) and not in
the least hard or a confusing way of retrieving the given group. Mind
you, the Match object is a C-struct with python binding and I'm not
exactly sure how to add either feature to it, but I'm sure the C-API
manual will help with that.
5) Add a well-formed, python-specific comment modifier,
e.g. (?P#...);
[handles parens in comments without turning on verbose, but is slower]
Why? It adds another incompatibility, so it has to be very useful or
clear. What exactly is the advantage over just turning on verbose?
Well, Larry Wall and Guido agreed long ago that we, the python
community, own all expressions of the form (?P...) and although I'd be
my preference to make (?#...) more in conformance with understanding
parenthesis nesting, changing the logic behind THAT would make python
non-standard. So as far as any conflicting design, we needn't worry.
As for speed, the this all occurs in the parser and does not effect the
compiler or engine. It occurs only after a (?P has been read and then
only as the last check before failure, so it should not be much slower
except when the expression is invalid. The actual execution time to
find the closing brace of (?P#...) is a bit slower than that for (?#...)
but not by much.
Verbose is generally a good idea for anything more than a trivial
Regular Expression. However, it can have overhead if not included as
the first flag: an expression is always checked for verbose
post-compilation and if it is encountered, the expression is compiled a
second time, which is somewhat wasteful. But the reason I like the
(?P#...) over (?#...) is because I think people would more tend to assume:
r'He(?# 2 (TWO) ls)llo' should match Hello but it doesn't.
That expression only matches He ls)llo, so I created the (?P#...) to
make the comment match type more intuitive:
r'He(?P# 2 (TWO) ls)llo' matches Hello.
9) C-Engine speed-ups. ...
a number of Macros are being eliminated where appropriate.
Be careful on those, particular on str/unicode and different
compile options.
Will do; thanks for the advice! I have only observed the UNICODE flag
controlling whether certain code is used (besides the ones I've added)
and have tried to stay true to that when I encounter it. Mind you,
unless I can get my extra 10% it's unlikely I'd actually go with item 9
here, even if it is easier to read IMHO. However, I want to run the new
engine proposal through gprof to see if I can track down some bottlenecks.
At some point, I hope to get my current changes on Launchpad if I can
get that working. If I do, I'll give a link to how people can check out
my working code here as well.
__
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2636
[issue2636] Regexp 2.6 (modifications to current re 2.2.2)
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: This simple patch adds (?P#...)-style comment support. Added file: http://bugs.python.org/file10056/issue2636-05.patch __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.6 (modifications to current re 2.2.2)
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: I am very sorry to report (at least for me) that as of this moment, item 9), although not yet complete, is stable and able to pass all the existing python regexp tests. Because these tests are timed, I am using the timings from the first suite of tests to perform a benchmark of performance between old and new code. Based on discussion with Andrew Kuchling, I have decided for the sake of simplicity, the timing of each version is to be calculated by the absolute minimum time to execute observed because it is believed this execution would have had the most continuous CPU cycles and thus most closely represents the true execution time. It is this current conclusion that greatly saddens me, not that the effort has not been valuable in understanding the current engine. Indeed, I understand the current engine now well enough that I could proceed with the other modifications as-is rather than implementing them with the new engine. Mind you, I will likely not bring over the copious comments that the new engine received when I translated it to a form without C_Macros and gotos, as that would require too much effort IMHO. Anyway, all that being said, and keeping in mind that I am not 100% satisfied with the new engine and may still be able to wring some timing out of it -- not that I will spend much more time on this -- here is where we currently stand: Old Engine: 6.574s New Engine: 7.239s This makes the old Engine 665ms faster over the entire first test_re.py suite, or 9% faster than the New Engine. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue2636 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Regexp 2.6 (modifications to current re 2.2.2)
New submission from Jeffrey C. Jacobs [EMAIL PROTECTED]:
I am working on adding features to the current Regexp implementation,
which is now set to 2.2.2. These features are to bring the Regexp code
closer in line with Perl 5.10 as well as add a few python-specific
niceties and potential speed-ups and clean-ups.
I will be posting regular patch updates to this thread when major
milestones have been reach with a description of the feature(s) added.
Currently, the list of proposed changes are (in no particular order):
1) Fix a href=http://bugs.python.org/issue433030;issue 433030/a by
adding support for Atomic Grouping and Possessive Qualifiers
2) Make named matches direct attributes of the match object; i.e.
instead of m.group('foo'), one will be able to write simply m.foo.
3) (maybe) make Match objects subscriptable, such that m[n] is
equivalent to m.group(n) and allow slicing.
4) Implement Perl-style back-references including relative back-references.
5) Add a well-formed, python-specific comment modifier, e.g. (?P#...);
the difference between (?P#...) and Perl/Python's (?#...) is that the
former will allow nested parentheses as well as parenthetical escaping,
so that patterns of the form '(?P# Evaluate (the following) expression,
3\) using some other technique)'. The (?P#...) will interpret this
entire expression as a comment, where as with (?#...) only, everything
following ' expression...' would be considered part of the match.
(?P#...) will necessarily be slower than (?#...) and so only should be
used if richer commenting style is required but the verbose mode is not
desired.
6) Add official support for fast, non-repeating capture groups with the
Template option. Template is unofficially supported and disables all
repeat operators (*, + and ?). This would mainly consist of documenting
its behavior.
7) Modify the re compiled expression cache to better handle the
thrashing condition. Currently, when regular expressions are compiled,
the result is cached so that if the same expression is compiled again,
it is retrieved from the cache and no extra work has to be done. This
cache supports up to 100 entries. Once the 100th entry is reached, the
cache is cleared and a new compile must occur. The danger, all be it
rare, is that one may compile the 100th expression only to find that one
recompiles it and has to do the same work all over again when it may
have been done 3 expressions ago. By modifying this logic slightly, it
is possible to establish an arbitrary counter that gives a time stamp to
each compiled entry and instead of clearing the entire cache when it
reaches capacity, only eliminate the oldest half of the cache, keeping
the half that is more recent. This should limit the possibility of
thrashing to cases where a very large number of Regular Expressions are
continually recompiled. In addition to this, I will update the limit to
256 entries, meaning that the 128 most recent are kept.
8) Emacs/Perl style character classes, e.g. [:alphanum:]. For instance,
:alphanum: would not include the '_' in the character class.
9) C-Engine speed-ups. I commenting and cleaning up the _sre.c Regexp
engine to make it flow more linearly, rather than with all the current
gotos and replace the switch-case statements with lookup tables, which
in tests have shown to be faster. This will also include adding many
more comments to the C code in order to make it easier for future
developers to follow. These changes are subject to testing and some
modifications may not be included in the final release if they are shown
to be slower than the existing code. Also, a number of Macros are being
eliminated where appropriate.
10) Export any (not already) shared value between the Python Code and
the C code, e.g. the default Maximum Repeat count (65536); this will
allow those constants to be changed in 1 central place.
11) Various other Perl 5.10 conformance modifications, TBD.
More items may come and suggestions are welcome.
-
Currently, I have code which implements 5) and 7), have done some work
on 10) and am almost 9). When 9) is complete, I will work on 1), some
of which, such as parsing, is already done, then probably 8) and 4)
because they should not require too much work -- 4) is parser-only
AFAICT. Then, I will attempt 2) and 3), though those will require
changes at the C-Code level. Then I will investigate what additional
elements of 11) I can easily implement. Finally, I will write
documentation for all of these features, including 6).
In a few days, I will provide a patch with my interim results and will
update the patches with regular updates when Milestones are reached.
--
components: Library (Lib)
messages: 65513
nosy: timehorse
severity: normal
status: open
title: Regexp 2.6 (modifications to current re 2.2.2)
type: feature request
versions: Python 2.6
__
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2636
[issue433030] SRE: Atomic Grouping (?...) is not supported
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: I'm digging into the sre_parse.py at the moment and this I have all the changes I need for that now. The rest of the changes I believe are in either sre_compile.py or more likely directly in _sre.c, so I will examine those files next. I am attaching a single diff for expedience. This is not an official patch, just a sample to see the progress I am making. I forgot the correct format for patch files but I promise to get it right when I have made more progress. -- components: +Library (Lib) versions: +Python 2.6 Added file: http://bugs.python.org/file9897/PyLibDiffs.txt Tracker [EMAIL PROTECTED] http://bugs.python.org/issue433030 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue433030] SRE: (?...) is not supported
Jeffrey C. Jacobs [EMAIL PROTECTED] added the comment: Fredrik, If you're still listening, I am gonna try and tackle this one but I would like to know why you or the famous Jeffrey of the Regexp world claims that there is already code in the Regexp Engine for Atomic Grouping? Adding a hook for (?...) should be trivial but I don't wanna re-invent the wheel if the proper stack-unwind code already exists. Thanks. Of course, Andrew (a.k.a. A.M. Kuchling) already asked this question and you did not answer it, so I guess you're not reading this, but if you are, please respond. Thanks! Tracker [EMAIL PROTECTED] http://bugs.python.org/issue433030 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
