netrw plugin bug?
Hi All, In Vim 7, there seems to be a bug in netrw plugin behavior. It looks like it wipes out clipboard register (mine is set to unnamed). That is, I yank something in one file, then I do, say, :e ., pick up a new file, open it with Enter and try to paste the text. A space will be pasted instead of previously yanked text. -- Alexei Alexandrov
A couple of notes on VIM performance
Hi All! I'd like to whine about VIM 7 performance a little bit. I use VIM both on Windows and Linux and really love it. But recently I found myself using another editor (embedded editor in FAR file manager, if you know what it is) more often than I'd like to. I analyzed myself and found that the reason is VIM startup time. My VIM configuration became more comples, I moved to VIM 7 and suddenly my the delay of VIM startup became too long for my brain so the brain chooses to start another editor sometimes. This applied to Windows only, because Linux startup is OK (so far - I'm on 6.3 there). Timed with gvim -c :q .vimrc it takes 0.91 seconds for VIM to start up and shut down. I didn't like this thing and decided to do some investigations on VIM startup time. Since the start up time usually has to do with reading various configuration files, I profiled the VIM startup with filemon (from sysinternals.com) and it showed several interesting (maybe to me only) things. 1. Enabling spell checking in .vimrc file is pretty expensive. When I comment out line set spell spelllang=en,ru the startup time reduces to 0.6 seconds. 2. Taglist plugin does a search for ctags command which becames pretty expensive once you have a lot of directories in your PATH. I was able to make it to avoid the search by adding line let g:Tlist_Ctags_Cmd = 'ctags' to my .vimrc. This saved about 0.06 seconds. Not that big but better than it was. 3. I found that I pay about 0.2 seconds on every VIM startup for Perl, TCL, Python and Scheme interfaces. This is very approximate number - I got it from filemon logs and didn't try to recompile VIM without those features. As far as I can see, VIM locates (and loads?) Perl, TCL, Python and Scheme libraries and reads in / parses perl.vim, tcl.vim, python.vim and scheme.vim syntax files. From the source code it seems to be that loading of these interfaces should be on demand, so I'm not sure why this happens. Any hints? 4. There are other places which contribute to the start-up time (not so big, though). For example, I found a lot of reading from $VIMRUNTIME/rgb.txt file. It turned out that I use desert color scheme, which refers the colors by their names. These names are read from the abovementioned file along with their RGB values (gui_mch_get_color). What is not very good is that reading is done for every gui_mch_get_color call (well, if the name of the color is not in pre-defined table) and the result is not kept in the memory. The number 1 above concerns me most of all. I really like spell-checking but paying 0.3 seconds on each startup seems to be slow. It would be nice if VIM could do the loading of spl files in background. -- Alexei Alexandrov
Patch for python syntax
Hi All! Here is a small patch for python.vim syntax file which enables spell checking in comments and strings only. It would be nice to include it into VIM 7. P.S. It seems that very few syntax files support @Spell/@NoSpell correctly. -- Alexei Alexandrov python.vim.patch Description: Binary data
Re: Bug with --remote-tab-silent
Hi Bram Moolenaar, you wrote: I can't reproduce it. It could be caused by a plugin. Does adding -V10 show some context of the error? Does anyone else see this? Not sure if my previous message reached the list: the bug is reproducible only if the extension of the file is in 'wildignore' -- Alexei Alexandrov
Re: Bug with --remote-tab-silent
Hi A.J.Mechelynck, you wrote: Exe or dll files are usually binaries (which shouldn't cause E479 however); and unlike cmd.exe when executing them, Vim doesn't lookup the PATH for editfiles. 1) Are you sure the directory containing the editfile was current? If it wasn't, you should mention the path to the file on the command-line. 2) If the file's dir was current, try using gvim --remote-tab-silent +set bin bjam.exe Note: The Vim help is not very explicit: I'm not sure if the additional parameter should be +set bin +set bin or +set\ bin If one of them doesn't set 'binary', try another variant. The key in the bug report is that error is reproducible only with --remote-tab-silent. I'm not sure why the advices above can have different behavior with this option. -- Alexei Alexandrov
Re: BOF Vim 8 - Suggestions
Hi John Beckett, you wrote: The best way to expand Vim usage IMHO would be to work out better default settings to improve the first hour of contact. In addition, perform necessary fixups, but resist new features. I also think that Vim is feature-enough. What affects me most of all every day is Vim performance. Especially on Windows. Startup is slower than it should be and redraw of the screen is also rather slow - and almost unusable with TTF fonts with Antialiasing on. If would need to choose _one_ thing I'd like to have in Vim improved I'd choose performance... -- Alexei Alexandrov
Re: Odp: BOF Vim 8 - Suggestions
Hi Stefano Zacchiroli, you wrote: The solution of being compatible when invoked as 'vi' and being nocompatible when invoked in a different way made everybody happy. The (trivial) patch we are using to implement this behaviour is available at: http://svn.debian.org/wsvn/pkg-vim/trunk/packages/vim/debian/tiny/vimrc.tiny.diff?op=filerev=0sc=0 FWIW we also set a lot of other default values when not invoked as 'vi'. This really seems to be a good option. -- Alexei Alexandrov
Re: BOF Vim 8 - Suggestions
Hi Bram Moolenaar, you wrote: To finish, I'm not suggesting that the optimum one-size-fits-all set of mappings be determined. Any set of well-planned mappings would be better than the current blank slate. The best set of mappings is what you make yourself. That's like: don't give a hungry person food, teach him how to grow food. Sort of OT but still: before teaching it's better to give them some food - otherwise they will die before they even learn anything... -- Alexei Alexandrov
Vim 7 performance notes
Hi Bram et al.,
I'm doing some performance investigations of Vim code trying to understand
whether there are any possibilities to improve it.
Currently I've made the following observations (all investigations are done on
Windows):
Redundant work is done during regexp operations in syntax highlighting. On some
files it is very noticable. The stack of the hotspot is ... syn_current_attr
syn_regexec vim_regexec_multi vim_regexec_both regtry regmatch
ga_grow alloc_clear memset. So alloc_clear spends quite a few clockticks in
lalloc() and memset().
The reason for this is pessimistically big grow size for regset growing array:
ga_init2(regstack, 1, 1);
This is not very good: many regexp operations don't go deep - non-match is
detected very quickly. But even one element on the stack will lead to
allocating at least 1 bytes (which should be fast with good CRT memory
allocator) and (worse) initializing these 1 bytes with zeros (won't be that
fast).
One possible solution would be to keep regstack alive across calls to
vim_regexec_both, but I'm not sure if it's can be done safely. What I did was
replacing the grow size with smaller number and making the grow size for
growing arrays dynamic with increase of 25%:
--- regexp.c (revision 136)
+++ regexp.c (working copy)
@@ -3350,7 +3350,7 @@
/* Init the regstack empty. Use an item size of 1 byte, since we push
* different things onto it. Use a large grow size to avoid reallocating
* it too often. */
-ga_init2(regstack, 1, 1);
+ga_init2(regstack, 1, 64);
/* Init the backpos table empty. */
ga_init2(backpos, sizeof(backpos_T), 10);
--- misc2.c (revision 136)
+++ misc2.c (working copy)
@@ -1905,6 +1905,7 @@
{
if (n gap-ga_growsize)
n = gap-ga_growsize;
+gap-ga_growsize += (gap-ga_growsize 2);
len = gap-ga_itemsize * (gap-ga_len + n);
pp = alloc_clear((unsigned)len);
if (pp == NULL)
With this change I can see serious performance improvements, but I'm not sure
if they are safe.
Bram, does it look making any sense?
--
Alexei Alexandrov
Compilation error with -D EXITFREE
Hi All! Do I understand it correct that EXITFREE define in Vim source code can be used to make sure all resources are cleaned up on exit (and those which are not should be treated as leaks)? To be able to use it on Windows I had to do the following changes: Index: misc2.c === --- misc2.c (revision 205) +++ misc2.c (working copy) @@ -1075,7 +1075,7 @@ ResetRedobuff(); ResetRedobuff(); -#ifdef FEAT_CLIENTSERVER +#if defined(FEAT_CLIENTSERVER) defined(FEAT_X11) vim_free(serverDelayedStartName); #endif Index: os_mswin.c === --- os_mswin.c (revision 205) +++ os_mswin.c (working copy) @@ -239,6 +239,11 @@ if (gui.in_use) gui_exit(r); + +#ifdef EXITFREE +free_all_mem(); +#endif + exit(r); } -- Alexei Alexandrov
Re: Vim 7 performance notes
Hi Bram Moolenaar, you wrote: Speed should be OK this way, but it does keep up to 32 Kbyte allocated. That may not seem much, but if we do this in many places it quickly adds up. Any keep limit greater than initial size (e.g. 16384 bytes) will give the same effect in many cases. By many cases here I mean cases when the stack just doesn't grow more than 512 bytes (very usual for syntax highlighting). Is 16384 still too big? Regexp matching and syntax highlighting is one of the most important things in VIM so maybe it's tolerable? Also, as far as I understand, previosly (version 6) VIM used program stack as regstack, right? Compared with program stack solution current solution with the change proposed is much better because program stack never shrinks - once it grows to 1M, for example, the memory won't be ever given back. At least, this is so on Windows, Linux and I guess most Unixes. Can you show the benchmarks you used to see the performance and the stack space that is being used? Otherwise we keep guessing the best numbers. OK. Platform used for investigations: x86, Windows XP SP 2. Pentium 4 Northwood, 2.4 GHz, 512M RAM. I did 2 things: understanding stack usage and performance measurement. To understand the stack usage I added some simple logging to regexp.c: printing ga_maxlen before regstack and backpos clearing and forced the arrays to have the grow size 1 (so that ga_maxlen will be high watermark in bytes). Of course, for performance investigations I used version with normal grow size and without logging. The version with logging was used to perform the following: 1. With syntax highlighting on, the following files were viewed from gg to G (with PgDn) and the following high watermark of stack size was observed: spell.c (444 bytes), $VIMRUNTIME/filetype.vim (820 bytes), big text file with a lot of syntax errors (252 bytes) 2. Command gvim -c vimgrep /a\(.\)*z/ *.c | q was executed in VIM 7 source directory. Stack watermark - 31008 bytes. This is example of non-optimal regexp which tends to take a lot of stack space. Similar other test cases were tried leading to the following conclusions: 1) there is a lot of vim_regexec_both calls during syntax highlighting which work in very shallow stacks (1K); 2) when user searches for something with regexp there are cases when regular expression can require big amount of memory (10K). The performance measurements were done against original version (7.0.188) and modified regexp.c (initial: 8192, keep limit: 16384). Each measurement was performed 3 times, minimal time was picked up. First, I test the syntax highlighting speed: Command: gvim.exe -f $VIMRUNTIME/filetype.vim -c for i in range(199) | redraw! | endfor | q Original version: 10.6 seconds Modified version: 8.5 seconds The difference is about 25%. Second, I did some grepping through Vim sources again: Command: gvim.exe -c vimgrep /a.*z/ *.c | q Original version: 6.6 seconds Modified version: 5.6 seconds The difference is about 15%. Coding detail: please don't use if (!number), use if (number == 0), that is so much easier to read. Checking if ga_data is NULL would be simpler. Got it - no problem. -- Alexei Alexandrov
Re: Vim 7 performance notes
Hi Mikolaj Machowski, you wrote: Nice work. Could you send or place somewhere patches? I'd like to test them on more complex regexps. Here it is. Note that the biggest speed-up is observed when regexp is matched a lot of times. The regexp mechanism itself is not affected at all here - so if you have one regexp which runs very long you won't probably notice any major difference. -- Alexei Alexandrov regexp.c.diff Description: Binary data
Re: Vim 7 performance notes
Hi Alexei Alexandrov, you wrote: Hi Bram et al., I'm doing some performance investigations of Vim code trying to understand whether there are any possibilities to improve it. I've also noticed that Vim spends somewhat significant time on startup loading spell files (I have 2 languages in my .vimrc: set spelllang=en,ru). The time is mostly spent in EnterCriticalSection/LeaveCriticalSection with getc() upper the stack. The reason for this is CRT blocking since the runtime is multithreaded. It's Windows, but on Linux it should be similar. As far as I understand, Vim doesn't access the spell file from multiple threads. Thus, the situation can be improved a lot: on Linux by using getc_unlocked. On Windows, starting VS 2005 there is a function _getc_nolock. Before VS 2005 this function can be emulated by macro: #define _getc_nolock(_stream) (--(_stream)-_cnt = 0 ? \ 0xff *(_stream)-_ptr++ : _filbuf(_stream)) By switching to non-blocking getc() in spell.c I was able to reduce Vim startup time from about 0.9 seconds to about 0.7 seconds. -- Alexei Alexandrov
Re: Vim 7 performance notes
Hi George V. Reilly, you wrote: How did you measure the time in EnterCriticalSection and LeaveCriticalSection? I discovered the problem by using a performance tuning tool which uses sampling approach to get statistical profile data. The intrusivity of this class of tools is very low. I verified that the problem exists by compiling Vim with unlocked getc() and measuring the difference (without any tool, just by $ time ...). If there's no lock contention, these routines are little more than InterlockedIncrement and InterlockedDecrement, without a kernel transition or blocking. You're absolutely right, it doesn't need to switch the context when the CS is free. But InterlockedIncrement/Decrement is not that cheap. On uniprocessor machine it takes about 200 cycles of CPU cycles per atomic operation. Thus, EnterCS/LeaveCS pair will take about 400 cycles of CPU. The program which confirms these numbers is attached. So it means that to read 1 Mbyte of data with locking getc (this is roughly the size of Russian + English spl files) we need to pay 400e6 cycles for these useless attempts to syncronize. Given the frequency of my machine 2.4 GHz = 2400e6 we get that 400e6 means 1/6 of second which is 0.17 seconds - exactly the speedup I observed. And remember that you need to pay this price on every Vim startup. In other words, if you're seeing significant time in Enter/LeaveCS, I can think of two causes. Either your measurement tool has perturbed the results, or there really is some multithreaded lock contention. The former seems more likely, as Vim is single-threaded, but who knows what some DLLs in the Vim process might be doing. No, there isn't any contention. The critical section in Microsoft multi-threaded CRT is per-FILE* so it's impossible that any guy competes with you unless you give them the FILE *. As far as I can say, descriptor to opened spell file is absolutetly private inside spell.c Also, the numbers above show that the overhead is exactly this without any contention. If there were competition, the overhead would be much bigger. I would be vary wary of using the _getc_nolock macro until we understand why you are seeing those results. -- Alexei Alexandrov test_cs.c_ Description: Binary data
Re: Vim 7 performance notes
Hi Mikolaj Machowski, you wrote: When testing it with VST it gave 3.4% speed improvements (the same metodology - 3 tests before and after, choose the best results). Well, it's not that much but it's still positive result. :-) -- Alexei Alexandrov
Re: Vim 7 performance notes
Hi Bram Moolenaar, you wrote: It's Windows, but on Linux it should be similar. I would not assume it's similar until there is proof. Of course. I'm going to investigate it there. This sounds like a bug in getc(). It should know that only one thread is reading the file and skip the locking. I don't think we should fix library bugs in Vim, unless it's crashing or another significant problem. It can't be a bug. I might be missing what you mean, but I can't see how it can know that only one thread is reading a file. It doesn't have a clue whether you gave this FILE * to other threads or not. It tries to be lightweight - as I described in a separate mail it uses InterlockedIncrement/Decrement but they are not that lightweight - they don't require switching to kernel mode but still take about 200 cycles of CPU each. The only optimization that I see could be avoiding blocking (and even trying to block) in case if there is only one thread in current process and if there is guarantee that this particular call is guaranteed not to create any threads. But 1) it still may be expensive and 2) Vim has some background threads anyway, probably. Perhaps you can already get a big increase by not compiling for debugging? With MSVC this usually has a big impact. Also largely defeats profiling with debugging enabled. I do _all_ performance measurements using optimized version of binary with symbols. This is just a must for performance tuning. -- Alexei Alexandrov
Re: patch 7.0.193
Hi Bram Moolenaar, you wrote: Yes, well, there is a bit of a conflict about what Vim should do. You told Vim to ignore files with a certain extension, and then you tell it to edit such a file. In a new Vim this results in editing the file anyway, since there is nothing else to do. In an existing Vim you get an error message, and keep the file you were working on. I would think this is OK. Is there a specific situation where you need Vim to edit the file you wanted ignored? I absolutely agree with what Tony said in sibling message. I expect that 'wildignore' only affects file name completion. If I ask Vim to open this file explicitly, I know what I'm doing. Also, note that the error message occurs even when I say gvim.exe --remote-tab-silent some file matching 'wildignore' and there isn't any gvim opened. So In an existing Vim you get an error message, and keep the file you were working on. doesn't apply here well. There wasn't any file I was working on. -- Alexei Alexandrov
Re: copyindent/preserveindent question
Hi Gary Johnson, you wrote: From :help preserveindent: Normally the indent is replaced by a series of tabs followed by spaces as required (unless |'expandtab'| is enabled, in which case only spaces are used). I believe this sentence is about how VIM operates when preserveindent/copyindent is NOT set. You should also have quoted the next one: Enabling this option means the indent will preserve as many existing characters as possible for indenting, and only add additional tabs or spaces as required. -- Alexei Alexandrov
Spell checking in cwindow
Hi All! When spell checking is on by default, quick fix window is full of spell checking errors. Probably it should have off always, shouldn't it? -- Alexei Alexandrov
