FILE I/O Performance

[George Gatling (Contractor)]

I am still working on our new file i/o library and I have a 
performance/reliability question.  The specification of the file format 
requires the data be sequential in the file... so if I want to add data to 
a table at the beginning of the file I am required to shift all of the 
following tables to make room for the new data.  Therefore, if a program 
was writing to two tables in the file in a loop, there would be a lot of 
data shuffling... and it would get worse the longer the loop ran.  Does 
anyone have any experience with this sort of thing?  Am I worried about 
something that I wouldn't even be able to notice in the end?  Our files 
routinely reach several hundred meg... and gigabyte files are not unheard 
of... what would be the performance hit on trying to insert data at the 
beginning of a several hundred meg file?

An alternative I am considering is to maintain each table in the file in a 
separate temporary file while the file is open... then when the file is 
closed LabVIEW would concatenate together the temporary files to produce 
the output file according to the format specification.  The downside here 
is that I am now managing who-knows-how-many temporary file refnums while 
the file is open... this is not a huge deal from the library programming 
point of view, but are there issues in labview with having many files open 
at the same time?  Are there other disadvantages with this approach I have 
not thought of?

Finally, does anyone have some other suggestion for how I could cleanly 
write the sorts of files I am describing here?

Thanks
George Gatling
Applied Technology Division, SFA Inc.
Space Physics Simulation Chamber
US Naval Research Laboratory
202-404-5405 (phone)
202-767-3553 (fax)
If trees could scream, would we be so cavalier about cutting them down?
We might, if they screamed all the time, for no good reason.  --Jack Handy