On Friday, June 2, 2023 at 3:13:12 PM UTC-4 Bram Moolenaar wrote:
Brett Stahlman wrote: > Some time ago, while testing various configurations of a plugin that makes > very heavy use of syntax regions, I noticed that in certain configurations, > redraws were timing out after the default 2 second 'redrawtime'. In an > attempt to understand why, I profiled Vim and discovered that much of the > time was spent in the in_id_list() function, which was called over 115 > million times in the space of a few seconds. Shocked that any function > could need to be called so frequently, I began digging around in the > Vim source to understand what was happening. What I discovered was > that in_id_list() is used in a very inefficient strategy for finding > all syntax groups in the current group's 'contains' or 'nextgroup' > list. The calls are made within syn_current_attr() in a loop over > *all* syntax items, and because in_id_list() performs a linear search > of the 'contains' or 'nextgroup' list for each iteration of this outer > loop, the cost of the algorithm is O(M x N), where M is the number of > defined syntax items, and N is the size of the current item's > 'contains' or 'nextgroup' list. Moreover, this nested loop is executed > each time the current syntax item changes, so buffers with lots of > syntax regions in a small screen area will be hit especially hard. [...] Thanks for looking into this. Although your use case appears to be very specific. I was planning on taking another look at syntax highlighting mechanisms, but several other things will come first, thus it might be a while before I get to it. That is going to be in the context of ideas to use treesitter and TextMate. After making changes for that performance probably has to be tuned again. No problem. Understood. The syntax engine is very complex. Making changes always has the risk of introducing bugs. There is not that much testing at the moment. Ideally every syntax plugin comes with at least a minimal test. I know some authors have their own tests, which is fine to check the plugin before sending out a new version. But it is not useful for checking the implementation. Also, every author has to reinvent the wheel. It would be very helpful to start by adding a few syntax plugin tests, with a generic way to do this. Once we have the setup and a few examples, we can ask others to create more tests. Using screendump tests would be the simplest and most comprehensive. But executing them would be a bit slow. We can run them separately from "make test" to avoid slowing everything down. The indent tests can function as a rough example, have a "syntax.ext" file as input and a "syntax_ext.dump" file as a reference output. Something has to be added for variants with settings. I think this would be extremely useful. I've definitely wanted this sort of capability to validate changes made to my own plugin. When I wrote the initial version years ago, I went a bit crazy with configuration options. Now, with so many option permutations, it's easy to add a feature that works properly for the most common configurations but breaks one of the corner cases. I searched the Vim scripts site some time ago looking for some sort of test framework with a descriptive syntax for test specification. I wasn't even aware of the screendump API before today. I'm assuming I could use this mechanism to test my own filetype and syntax plugins, provided I maintained my own 'testdir/dumps' locally. Thanks, Brett S. Once we have some number of tests like this I would feel more confident in changing the code to optimize for performance. The tests could also output timing to be able to compare before/after performance. Comparing times only works locally, since every setup is different. -- hundred-and-one symptoms of being an internet addict: 96. On Super Bowl Sunday, you followed the score by going to the Yahoo main page instead of turning on the TV. /// Bram Moolenaar -- [email protected] -- http://www.Moolenaar.net \\\ /// \\\ \\\ sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ /// \\\ help me help AIDS victims -- http://ICCF-Holland.org /// -- -- You received this message from the "vim_dev" maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to the Google Groups "vim_dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/vim_dev/73a911f6-74a7-4a2f-9eb9-68f0a4b9d702n%40googlegroups.com.
