On 2014-08-29 06:43:45 -0700, Paul Eggert wrote: > Thanks, but that patch seems to depend on libpcre internals, in that it > "knows" that pcre_exec cannot possibly succeed without first checking its > entire input buffer for invalid UTF-8 bytes. Even if that's true now, it > reflects a performance bug that might be fixed in a future libpcre version.
If I understand correctly, I don't think that's an internal. The pcreapi(3) man page says about PCRE_NO_UTF8_CHECK: [...] Note that this option can also be passed to pcre_exec() and pcre_dfa_exec(), to suppress the validity checking of subject strings only. If the same string is being matched many times, the option can be safely set for the second and subsequent matchings to improve performance. The last sentence would imply that the UTF8 checking is done on the whole input buffer before matching is done. > Also, I don't see why grep needs to copy the buffer when there's an encoding > error. Why not simply rerun the matcher on the initial prefix that doesn't > have an encoding-error byte, and then (if that doesn't find a match), try > matching the suffix after the encoding-error byte? This approach would not > only avoid the buffer copy, it would avoid knowledge of libpcre internals. If there are many invalid UTF8 bytes, this would be slow, IMHO (it could be worth a try, though). But is the copy of the buffer really needed? Couldn't the invalid UTF8 sequences just be replaced by null bytes? Note that in case of invalid UTF8 bytes, in some (many?) cases, the cause is a binary file (possibly with some text in it), where lines can be very long. So, wouldn't it mean that it can take significantly more memory? -- Vincent Lefèvre <vinc...@vinc17.net> - Web: <https://www.vinc17.net/> 100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/> Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon) -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org