hello,
you almost convinced me with the disk time...
but that is not true. Reading the same file with a command like grep,
takes 567msecs.
try
time grep XXXXXXXXXXX ./all3.c #this will not find anything, but will
process all the file.. and no mmap is used!!!!!
the result is:
real 0m0.466s
user 0m0.232s
sys 0m0.236s
so grep spends some more time in userspace as well but the kernel time
is the same as of my small test program I wrote to read the same file
into memory..
The result is
real 0m0.253s
user 0m0.000s
sys 0m0.252s
This means that reading 600mb of memory takes 0.252 secs in kernel
(that is the concrete IO...).
So reading the 600mb file takes 252 time, and processing 13 seconds...
bellow is the little c program...
so.. again I would make positive critic. Just wondering if sthg. could
be tuned.
#include <unistd.h>
#include <fcntl.h>
int main(int argc, char ** argv)
{
if(argc < 1)
{
printf("file name not provided..\n");
return -1;
}
int fd = open(argv[1], O_RDONLY);
if(fd < 0)
{
printf("cannot open file %s", argv[1]);
return -1;
}
int chunk_size = 100000;
if(argc > 2)
{
chunk_size = atoi(argv[2]);
}
char *buffer = (char * )malloc(chunk_size);
int res;
int count = 0;
while( (res = read(fd, buffer, chunk_size) ) > 0)
{
count ++;
}
printf("total read %d*%d=%d", count, chunk_size, count*chunk_size);
//sorrry but deallocation in next version...
}
On Jun 16, 7:52 am, Ben Schmidt <[EMAIL PROTECTED]> wrote:
> misi wrote:
> > Hello,
>
> > would like to use vim to edit large files (600mb). Unfortunatelly, it
> > looks like opening such large files even on a Pentium 3.4Ghz with PATA
> > disk takes quite some time (ubuntu, kernel 2.6.22-14). I know that vim
> > creates some internal tree representation of the file, but I wonder if
> > something could be tuned, so that such loading times decrease, even if
> > memory usage would increase?
> [...]
> > time vim -u ./.vimrc --noplugin -c":q" ../all3.c
>
> > the result is:
>
> > real 0m17.670s
> > user 0m15.457s
> > sys 0m1.428s
>
> I think what is taking the time is actually reading the file from disk.
> Vim reads the entire file into memory, and getting 600 MB off disk takes
> time.
>
> [...]
>
> > I profiled vim, and opening the same file I got the file below....
> [...]
> > Flat profile:
>
> > Each sample counts as 0.01 seconds.
> > % cumulative self self total
> > time seconds seconds calls s/call s/call name
> > 46.12 1.96 1.96 1 1.96 4.22 readfile
> > 23.53 2.96 1.00 31025600 0.00 0.00 ml_append_int
> > 12.94 3.51 0.55 31025601 0.00 0.00 ml_updatechunk
> > 8.24 3.86 0.35 31361959 0.00 0.00 ml_find_line
> > 4.94 4.07 0.21 31025600 0.00 0.00 ml_append
> [...]
> > 0.00 4.25 0.00 11709 0.00 0.00 mch_breakcheck
>
> The profile only shows an execution time of four and a quarter seconds.
> Is this really how long it took? If so, it's probably because most of
> the file was cached in memory by the OS from your previous reading of
> it. I don't think that kind of time is unreasonable by any means, and
> although you could probably scrape a bit off it, I doubt it would be
> worth it, as it would be likely to impair code readability somewhat,
> and/or cause errors, and these are worse than slowness! If it didn't
> take that long, I wonder what doesn't show up in the profile. Perhaps
> the overhead of the function calls themselves. If that's the case,
> optimisation would again be possible, but harder than optimising the
> time that *does* show up, so again probably not worth it or unwise for
> the same reasons as above.
>
> I may look into it a little more, particularly if you send back some
> more details, but I doubt much can be done without a pretty big
> overhaul--i.e. making Vim read in a background thread or lazily or such.
>
> Ben.
--~--~---------~--~----~------------~-------~--~----~
You received this message from the "vim_dev" maillist.
For more information, visit http://www.vim.org/maillist.php
-~----------~----~----~----~------~----~------~--~---
