Hi drlatex!
On Mo, 27 Jul 2009, drlatex wrote:
[quoting repaired]
> John Beckett wrote:
>> :setlocal noswapfile undolevels=-1
>> :%join!
>
>
> I understand that %join joins all lines, but what does the exclamation mark
> do ??\
See :h :join
> Also, what does " :setlocal noswapfile undolevels=-1 " do ?? Is this
> necessary ?? I tried just %join on a small file and it seems to do what I
> want.
This tries basically to speed up things a little bit. See :h 'swf' and
:h 'ul'.
>
> Finally, this method is still soo slow for my data file! Is there no way to
> just remove the last character of each line ?? Or even better, to highlight
I have been running the following test:
#v+
$ seq 1 $((1024*1024)) > large_file
$ time gvim -u NONE -N -c 'set noswapfile ei=all syntax=off ul=-1 fdm=manual
|%join!|w file_join|q' large_file
real 53m31.454s
user 0m0.105s
sys 0m0.076s
#v-
I basically aborted vim after 50 Minutes. It had joined about 480,000
rows. Yes, it really was that bad. Memory was not an issue, though.
Looks like, the join algorithm grows exponential with the number of
lines to join. I think, there has even been a patch suggested to
improve this behaviour¹
#v+
$ time gvim -u NONE -N -c 'set noswapfile ei=all syntax=off ul=-1 fdm=manual
|call setline(1, join(getline(1,line("$")), ""))|2,$d_|w large_append|q!'
large_file
real 7m1.085s
user 0m0.105s
sys 0m0.183s
#v-
This looks better, though in this case memory usage was a problem. So
I thought, the following function may work even faster and consume
lower memory:
,----
| fu! EOL()
| let limit=10000
| let lines=line('$')
| let runs=lines/limit+1
|
| for i in range(0,runs)
| let start=i*limit+1
| let end =i*limit+limit
| call setline(i+1, join(getline(start,end), ''))
| endfor
|
| exe ":" . (runs+1) . ",$d_"
| %join!
| endfu
`----
Running vim using this function, I get this result:
#v+
$ time gvim -u NONE -N -c 'set noswapfile ei=all syntax=off ul=-1 fdm=manual
|so eol.vim|call EOL()|w large_eol1|q!' large_file
real 0m6.153s
user 0m0.105s
sys 0m0.061s
#v-
which seems reasonable and memory consumption was not an issue.
But the fastest way, as both Johns already stated, would by to use an
external tool like tr, which you could even use as a filter:
#v+
$ time gvim -u NONE -N -c "set noswapfile ei=all syntax=off ul=-1 fdm=manual
|%! tr -d '\n'" large_file -c ':w large_tr|:q!' large_file
2 files to edit
real 0m2.477s
user 0m0.105s
sys 0m0.076s
#v-
For reference here are the timings for running :%s/\n//
#v+
$ time gvim -u NONE -N -c 'set noswapfile ei=all syntax=off ul=-1 fdm=manual
|%s/\n//|w large_subst|q!' large_file
real 23m23.303s
user 0m0.121s
sys 0m0.045s
#v-
Here again, I aborted pressing Ctrl-C which showed, that about 248,000
lines have been joined. So a good fourth of all records have been
processed.
regards,
Christian
__
¹) Here is the thread:
http://thread.gmane.org/gmane.editors.vim.devel/22065
and you can find a patch at the vim_extended repository:
http://repo.or.cz/w/vim_extended.git
--
:wq!
--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_use" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---
