[J.B. Langston <jblangs...@datastax.com>]
> And unfortunately it does appear that my app took an almost a 20%
> performance hit from using regex instead of re, unfortunately.
> Processing time for a test dataset with 700MB of logs went from
> 77 seconds with the standard library re to 92 seconds with regex.
> Profiling confirms that the time spent in the groupdict method went
> from 3.27% to 9.41% and the time spent in match went from
> 5.19 to 10.33% of the execution time.
I'm mildly surprised! Most times people report that regex is at least
modestly faster than re.
For peak performance at the expense of flexibility (e.g., no
backreferences supported), perhaps you'd get a major speed gain from
an entirely different regexp engine approach. Google built such an
engine, which has worst-case linear matching runtime (but, depending
on details, _may_ require space exponential in the regexp to compile a
regexp object). A Python binding for that is here ("pip install
pyre2"):
https://pypi.org/project/pyre2/
I haven't used it - this isn't a particular area of interest for me.
Note that in its "Performance" section, it shows examples where it
blows re out of the water. But, in all those cases, regex was modestly
to very significantly faster than re too.
YMMV.
> So this means switching to regex is probably a no go.
The difference between 77 and 92 seconds doesn't, on the face of it,
scream "disaster" to me - but suit yourself.
> If hanging regexes become a common occurrence for my app I might decide
> it's worth the performance hit in the name of safety, but at this point I
> would
> rather not.
If they were destined to become a common occurrence, they already
would have done so. You blamed your bad case on unexpected data, but
the _actual_ cause was an unintended typo in one of your regexps.
So it goes. You're using a tool with a hyper-concise notation, where a
correct expression is pretty much indistinguishable from line noise,
and a typo is rarely detectable as a syntax error.
So you're learning the hard way that you have to be on crisis-level
alert when writing regexps: they're extremely touchy and unforgiving,
pyre2 would spare you from all match-time timing disasters, but
"touchy and unforgiving" applies all the same. Instead of a typo
causing exponential runtime, it may instead cause the regexp to match
(or fail to match) in unintended ways. About which no clue of any kind
will be left behind, unless you stare at the inputs and outputs and
check them yourself. But, in that case, 92 seconds wouldn't even get
you through 1000 bytes ;-)
_______________________________________________
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/JXHY26ABJZX7CM5LXDZJLHPKNGYKND5E/
Code of Conduct: http://python.org/psf/codeofconduct/
