On Thursday, June 9, 2016 at 6:53:06 PM UTC+3, Bram Moolenaar wrote: > Ramel Eshed wrote: > > > > > 1) Actually, there is a bug here which is not exactly what I've > > > > described earlier. The issue is that :call setqflist([]), instead of > > > > adding one more list after the last list, will clear the next list and > > > > delete the ones after. For example: let's say I used :grep 4 times so > > > > I have now 4 lists. Now, if I do :colder 3, the first list becomes the > > > > current list. :call setqflist([]) will empty list 2, and delete lists > > > > 3 and 4. > > > > > > We do remove newer lists when adding a new list before the end, but in > > > this case it should not happen. > > I'm not sure I understand. Is removing all the newer lists are the > > expected behavior? why? and why is it different here? > > Removing newer lists is what happens when inserting a list. But you are > not inserting here, you are clearing it. I'm using setqflist([]) with no 'r' flag. Doesn't it suppose to insert a list? Anyway, is there a reason to delete the newer lists? > > > > > 2) Although quickfix performance got much better, I'm afraid there is > > > > more to do: > > > > a) It takes me about 9 seconds(!) to add 68,505 entries to the qf > > > > list (using simple :grep command. Time was measured from after the > > > > grep command output finished, of course). Yegappan's test takes > > > > only 2 seconds because all the results are from the same file. Try > > > > to add the loop variable to the file name in his test and see what > > > > happens... > > > > > > > > This is the profiler log of the search I did: > > > > > > > > % cumulative self self total > > > > time seconds seconds calls ms/call ms/call name > > > > 40.63 1.69 1.69 68510 0.02 0.02 buf_valid > > > > 21.88 2.60 0.91 68505 0.01 0.02 > > > > buflist_findname_stat > > > > 12.38 3.12 0.52 127664549 0.00 0.00 otherfile_buf > > > > 4.81 3.32 0.20 4345 0.05 0.05 buflist_findnr > > > > > > Well, when adding an entry a buffer is created, so that we keep track of > > > the actual name when doing ":cd". And it avoids duplicates. > > > buf_valid() is often needed to check if an autocommand didn't delete a > > > buffer. Both of these are hard to get rid of. > > This can really slow down Vim. I'm not sure if my test cases here are > > realistic, but I'm sure that if you keep Vim session active for enough > > time you'll probably get thousands of unlisted buffers which will slow > > down everything. > > In this case, if there is nothing else can be done (like using another > > data structure for qf entries) I think that command for deleting > > quickfix list is necessary. As I mentioned earlier, :call > > setqflist([], 'r') is not good enough and it also doesn't free the > > associated buffers. > > Maybe a simple mechanism for search, which does not use hidden marks, > > should be considered. When you're starting a search command you don't > > really know how many results you'll get so the last thing you want to > > do is to add 100000 results to the qf list. You need to be able to > > review the results, maybe filter them, and then add them to the > > quickfix if needed (in many cases it is not needed, like when you only > > want to review the code without changing anything). Actually, this is > > what I've done in the Agrep plugin.. > > The only way to avoid creating buffers is by intentionally dropping some > functionality and only keeping the file name. Perhaps we can still > expand it to a full path (that's also not a cheap operation, but at > least it's only once per entry). I'm not sure, but maybe something like file inode index number can be used for that and will be faster?
> > The file name would be turned into a buffer only when jumping to the > location then. > > I suppose this is a property of the quickfix list. Perhaps with a > function like "setqfflag()"? Would not work with commands like :cfile > though. Sounds good to me, although I didn't understand the :cfile problem you mentioned. > > > > > b) I remember you did 2 things in order to avoid the line > > > > adjustments when it's not necessary: check first if the buffer has a > > > > quickfix entry and don't call line adjustment when adding a line at > > > > the end. > > > > > > > > I still see that adding to a buffer when there are many qf entries > > > > is very slow. After I had the 68505 results of the :grep command, > > > > adding ~84000 lines to a buffer (from channel output) became really > > > > slow. This is the profiler log: > > > > > > How are the lines added? > > The lines are added by using 'out_io': 'buffer'. Without having many > > (4000) unlisted buffers adding 80000 lines takes about 1.2 seconds. > > The unlisted buffers slow this down to ~6 seconds. If, in addition, I > > add an out_cb to this job which calls setbufvar() it can take even 15 > > seconds. > > OK, so it's not adding the lines that's slow, but checking every time > whether the buffer pointer we have is still valid. > Sorry, my mistake. I don't know what was wrong in my previous test (maybe I was testing it with the profiler build..) but the only noticed difference I see is when calling to setbufvar(). No real difference when only adding lines. > > > > Each sample counts as 0.01 seconds. > > > > % cumulative self self total > > > > time seconds seconds calls s/call s/call name > > > > 31.46 6.98 6.98 149100 0.00 0.00 buf_valid > > > > 29.20 13.46 6.48 80590 0.00 0.00 buflist_findpat > > > > 26.66 19.38 5.92 84950 0.00 0.00 buflist_findnr > > > > 3.56 20.17 0.79 68507 0.00 0.00 > > > > buflist_findname_stat > > > > 2.28 20.67 0.51 127673186 0.00 0.00 otherfile_buf > > > > > > > > It seems like checking the buffer each time alone takes a lot of time. > > > > Is there any way to optimize the above functions? > > > > Also, why I still see all these buffer functions even though all the > > > > lines were added at the end of the buffer? > > > > > > If you use 'errorformat' it will locate the file name and find out what > > > buffer has that file name. > > > > > I'm not using this in this case. > > Ehm, you do get those file names added as buffers, thus that must happen > somewhere. > > Anyway, it seems that your problem is that you use thousands of buffers > and that's something Vim wasn't prepared for. I know that this is not the typical use case but, as I said, during a long Vim session there might be thousands of unlisted buffers eventually. I think that the solution you've proposed should make quickfix much more robust. -- -- You received this message from the "vim_dev" maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to the Google Groups "vim_dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
