Hi Everyone,
I'm writing a simple stack to filter data from big text files (more
than 150 MB), and export tab-delimited subsets to text files that can be
used in spreadsheets and statistical packages.
Below is a sample record from the file. Each record begins with the
variable ID. Each variable is on its own line, with the variable name in
caps, and each line is ended with a hard return. The number of lines per
record varies; that is, some variables are missing from some records.
The end of each record is a double forward slash ("//").
My questions follow the sample record.
----------------Sample record----------------------------------
ID Hs.30
TITLE membrane-spanning 4-domains, subfamily A, member 1
GENE MS4A1
CYTOBAND 11q13
LOCUSLINK 2206
CHROMOSOME 11
TXMAP D11S1357-D11S913; MARKER=SGC35231; RHPANEL=GB4
PROTSIM ORG=Homo sapiens; PROTGI=232084; PROTID=SP:Q01362; PCT=100;
ALN=243
PROTSIM ORG=Mus musculus; PROTGI=119867; PROTID=SP:P20490; PCT=58;
ALN=242
PROTSIM ORG=Rattus norvegicus; PROTGI=119868; PROTID=SP:P13386;
PCT=55; ALN=241
SCOUNT 4
SEQUENCE ACC=M89796; NID=g337417; PID=g337418
SEQUENCE ACC=D10583; NID=g219881; PID=g219882
SEQUENCE ACC=NM_000139; NID=g4503676; PID=g4503677
SEQUENCE ACC=AV742479; NID=g10860060; CLONE=CBTAME03; END=5'; LID=4714
//
----------------End of sample record----------------------------------
Here's some questions I hope you can help me with.
1. Because of the size of the raw data files, I want to read in
one record at a time into a temporary variable, process it, and then
empty it before proceeding to the next record.
I've used
read dataFile until "//"
successfully to read in exactly one record. But how can I repeat this
one-record-at-a-time process until the end of the file? Can I nest the
read command within a repeat loop, something like the following?
repeat until eof
read dataFile until "//"
(process data)
end repeat
2. About filtering or picking off specific records: I need to
find records with a specific ID or a specific CYTOBAND. I'm currently
using the lineOffset function to confirm in each record that an ID line
or a CYTOBAND line exists, and then I'm using the Match function to grab
the information in those lines. Is this reasonable? Is there a better
way?
3. Processing a whole data file may take some time. Is there a
way that I can allow the user to abort the read should they have to?
Your thoughts on this would be greatly appreciated.
Regards,
Greg
_________________________________________
Gregory Lypny
Associate Professor of Finance
Concordia University
_________________________________________
"Take chances, make mistakes!"
- Ms Frizzle, The Magic School Bus
Archives: http://www.mail-archive.com/[email protected]/
Info: http://www.xworlds.com/metacard/mailinglist.htm
Please send bug reports to <[EMAIL PROTECTED]>, not this list.
