Re: [Numpy-discussion] Reading footer lines
On Tue, Aug 13, 2013 at 6:01 AM, Daπid wrote: > Alternatively, you could also use seek to put the pointer a certain > distance from the end of the file and start from there, That's what I'd do if the file(s) may be too large to simply dump into memory. > but this could cause problems in Windows. what problems? this is pretty straightforward stuff! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Reading footer lines
On 13 August 2013 14:20, Resmi wrote: > As a workaround, I've tried using os.system along with grep. And I get the > following output : > os.system("grep -e 'tx' 'data.dat' ") > ## tx =2023.06 > 0 > > Why is there a 0 in the output? The file has no blank lines. That 0 corresponds to the exit status of os.system, as there were no errors, it returns a 0. My idea would be to read the file and store it into a StringIO. That can be fed to np.loadtxt to extract the numbers, and look for your number. My (untested) attempt: import StringIO with datafile open as f: ... raw_data = StringIO.StringIO() ... raw_data.write(f.read()) data = np.loadfromtxt(raw_data) numbers = [float(line.split['='][1].strip()) for line in raw_data.readlines() if line[0] == '#'] If there is only one number there and it is after all the data, you could skip them all. From the shape of data you know there are N lines of data: for _ in range(N): ... raw_data.readline() while True: ... line = raw_data.readline() ... if line[0] == '#': .. number = float(line.split['='][1].strip()) .. break Alternatively, you could also use seek to put the pointer a certain distance from the end of the file and start from there, but this could cause problems in Windows. David. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Reading footer lines
Hi Resmi On Tue, Aug 13, 2013 at 2:20 PM, Resmi wrote: > I've a list of long files of numerical data ending with footer lines > (beginning with #). I am using numpy.loadtxt to read the numbers, and > loadtxt ignores these footer lines. I want the numpy code to read one of the > footer lines and extract words from it. Is there a way to use loadtxt for > this? If there weren't many files I could have used the line number (which > keep varying between the files) of the footer line along with linecache. > Nevertheless there should be a generic way to do this in numpy? If the data-set is not too big, you can read it into memory with "f.readlines()" and then do a bit of preparsing on the resulting array before sending the rest of it to np.loadtxt (which allows an array of strings as input). Stéfan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Reading footer lines
On Tue, Aug 13, 2013 at 1:20 PM, Resmi wrote: > Hi, > > I've a list of long files of numerical data ending with footer lines > (beginning with #). I am using numpy.loadtxt to read the numbers, and > loadtxt ignores these footer lines. I want the numpy code to read one of the > footer lines and extract words from it. Is there a way to use loadtxt for > this? If there weren't many files I could have used the line number (which > keep varying between the files) of the footer line along with linecache. > Nevertheless there should be a generic way to do this in numpy? > > As a workaround, I've tried using os.system along with grep. And I get the > following output : > os.system("grep -e 'tx' 'data.dat' ") > ## tx =2023.06 > 0 > > Why is there a 0 in the output? The file has no blank lines. The os.system function call returns the integer return value (error level) of the command invoked, by convention zero for success but the value is set by the tool (here grep). > Since I want the value 2023.06 in the numpy code for later use I tried to > pipe the output to a variable:- > test = os.system(command) > But that isn't working as test is getting assigned the value 0. > Tried subprocess.call(['grep','-e','tx','data.dat']) which is also ending up > in the same fashion. > > It'll be great if I can get to know (i) a way to read the footer lines (ii) > to sort out the operation of os.system and subprocess.call output > processing. Don't use os.system, instead you should use subprocess: http://docs.python.org/2/library/subprocess.html The standard library module commands would also work but is not cross platform, and is also deprecated in favour of subprocess: http://docs.python.org/2/library/commands.html Peter ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Reading footer lines
Hi, I've a list of long files of numerical data ending with footer lines (beginning with #). I am using numpy.loadtxt to read the numbers, and loadtxt ignores these footer lines. I want the numpy code to read one of the footer lines and extract words from it. Is there a way to use loadtxt for this? If there weren't many files I could have used the line number (which keep varying between the files) of the footer line along with *linecache.* Nevertheless there should be a generic way to do this in numpy? As a workaround, I've tried using os.system along with grep. And I get the following output : *>>> os.system("grep -e 'tx' 'data.dat' ")* * ## tx =2023.06 * *0* Why is there a 0 in the output? The file has no blank lines. Since I want the value 2023.06 in the numpy code for later use I tried to pipe the output to a variable:- *test = os.system(command)* But that isn't working as *test* is getting assigned the value 0. Tried *subprocess.call(['grep','-e','tx','data.dat'])* which is also ending up in the same fashion. It'll be great if I can get to know (i) a way to read the footer lines (ii) to sort out the operation of os.system and subprocess.call output processing. Thanks, Resmi ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion