On 27/11/00 4:03 pm, Gregory Lypny <[EMAIL PROTECTED]> wrote:
> I'm writing a simple stack to filter data from big text files (more
> than 150 MB), and export tab-delimited subsets to text files that can be
> used in spreadsheets and statistical packages.
> Here's some questions I hope you can help me with.
>
> 1. Because of the size of the raw data files, I want to read in
> one record at a time into a temporary variable, process it, and then
> empty it before proceeding to the next record.
Given that your files are 150MB, this is the only way to go.
> I've used
>
> read dataFile until "//"
>
> successfully to read in exactly one record. But how can I repeat this
> one-record-at-a-time process until the end of the file? Can I nest the
> read command within a repeat loop, something like the following?
>
> repeat until eof
> read dataFile until "//"
> (process data)
> end repeat
JFYI (I had a similar example to hand that I was doing just now anyway),
*if* the file had been smaller (say only a few MB) the fastest way to
process it would have been to read it all in at once, and then parse the
entire file into an array. Then you can work on each element in the array
as you like. Something like:
put URL ("file:"&tDataPath) into tData
put true into tNewIndex
set the itemDel to tab --assuming the gaps between name and record are tabs
repeat for each line l in tData
if l is "//" then
put true into tNewIndex
next repeat
end if
if tNewIndex then
put false into tNewIndex
put item 2 of l into tIndex --assumes a unique ID on line 1
end if
put l & cr after tDataRecord[tIndex]
end repeat
set the itemDel to comma
To see the elements created, put keys(tDataRecord).
However, in your case, to read through the file in a repeat loop:
open file tDatapath
repeat until it is eof
read from file tDataPath until "//"
put it into tProcessData
--either call a function, or for speed do the processing
--given the size of your file, do it here
end repeat
close file tDataPath
> 2. About filtering or picking off specific records: I need to
> find records with a specific ID or a specific CYTOBAND. I'm currently
> using the lineOffset function to confirm in each record that an ID line
> or a CYTOBAND line exists, and then I'm using the Match function to grab
> the information in those lines. Is this reasonable? Is there a better
> way?
That seems reasonable. You probably don't even need to use matchText to
extract text: using lineOffset should be enough to extract the lines you
need - I think this is likely to be faster than using matchText.
> 3. Processing a whole data file may take some time. Is there a
> way that I can allow the user to abort the read should they have to?
You could add something like this to the end of the loop:
if the mouseClick then
answer warning "Really abort?" with "No" or "Yes"
if it is "Yes" then exit repeat
end if
Checking the mouseClick function takes time though, so to be most efficient
only check it every 100th or 1000th time (or however often you want to).
put 0 into tCounter
open file tDatapath
repeat until it is eof
read from file tDataPath until "//"
put it into tProcessData
--either call a function, or for speed do the processing
--given the size of your file, do it here
end repeat
add 1 to tCounter
if tCounter is 1000 then --insert number other than 1000 here if req.
put 0 into tCounter
if the mouseClick then
answer warning "Really abort?" with "No" or "Yes"
if it is "Yes" then exit repeat
end if
end if
close file tDataPath
Regards,
Kevin
Kevin Miller <[EMAIL PROTECTED]> <http://www.runrev.com/>
Runtime Revolution Limited (formerly Cross Worlds Computing).
Tel: +44 (0)131 672 2909. Fax: +44 (0)1639 830 707.
Archives: http://www.mail-archive.com/[email protected]/
Info: http://www.xworlds.com/metacard/mailinglist.htm
Please send bug reports to <[EMAIL PROTECTED]>, not this list.
