Greg Hurrell wrote:
> I've been porting Ferret (a plug-in that wraps ag/ack/grep)[0] to use the new
> job/channel functionality. Implementation up on GitHub[1].
>
> I've found that reading very long lines in NL mode locks up Vim. Discovered
> this while searching for a string "foobar", which basically runs `ag
> --vimgrep foobar` as a job. In this particular codebase, there is a massive
> source map file which is 150MB in size and is minified such that it is a
> single line, and it happens to match "foobar" . This is pathological, I know,
> but Vim appears to be exhibiting degenerate performance in this case. It pegs
> the CPU at 100% and runs for as long as I will let it (20 minutes or more),
> making me suspect some sort of quadratic growth.
>
> Minimal repro case:
>
> # Create pathological file
> mkdir example
> cd example
> ruby -e "File.open('sample.txt', 'w').puts(('x' * 150_000_000) + 'foobar')"
>
> # Inside Vim now:
> func! CloseHandler(channel)
> while ch_status(a:channel) == 'buffered'
> echomsg ch_read(a:channel)
> endwhile
> endfunc
> job_start('ag --vimgrep foobar',{'close_cb':'CloseHandler'})
>
> At this point Vim locks up. Without the close handler, the command runs and
> returns just fine.
>
> For now my workaround is to set up an `~/.agignore` file so that ag doesn't
> look at the dastardly sourcemap. Another thing I could try to do, but haven't
> yet, is switch to "raw" mode and implement my own buffering.
>
> I know this is not a "reasonable" workload by any means, but still seems like
> it would be nice to address the degenerate perf here. I'd expect this kind of
> thing to be slow, but scale basically linearly with the length of the line
> being read.
>
> Thoughts?
My first reaction is "well, don't do that then".
The code isn't really prepared for extremely long lines. It reads some
parts at a time (8K I believe). Then channel_collapse() concatenates
each part to the previous, thus you get lots of allocations of
increasing size. You could have a go at making this more efficient.
Also using strlen() many times is a large amount of overhead here.
Anyway, what would you do with a 100 Mbyte response from a channel?
You probably just want to avoid that.
--
"Lisp has all the visual appeal of oatmeal with nail clippings thrown in."
-- Larry Wall
/// Bram Moolenaar -- [email protected] -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///
--
--
You received this message from the "vim_dev" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php
---
You received this message because you are subscribed to the Google Groups
"vim_dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
