[Numpy-discussion] How to import input data to make ndarray for batch processing?
Hi All, I am new to Numpy (also Scipy). I am trying to reshape my text data which is in one single column (10,000 rows). I want the data to be in 100x100 array form. I have many files to convert like this. All of them are having file names like 0, 1, 2, 500. with out any extension. Actually, I renamed actual files so that I can import them in Matlab for batch processing. Since Matlab also new for me, I thought I will try Numpy first. Can any body help me in writing the script to do this for making batch processing. Thanks in advance, Venkat -- *** D.Venkat Research Scholar Dept of Physics IISc, Bangalore India-560 012 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] How to import input data to make ndarray for batch processing?
Do you want to save the file to disk as 100x100 matrices, or just to read them into the memory? Are the files in ascii or binary format? Nadav From: numpy-discussion-boun...@scipy.org [numpy-discussion-boun...@scipy.org] On Behalf Of Venkat [dvr...@gmail.com] Sent: 18 November 2010 16:49 To: Discussion of Numerical Python Subject: [Numpy-discussion] How to import input data to make ndarray for batch processing? Hi All, I am new to Numpy (also Scipy). I am trying to reshape my text data which is in one single column (10,000 rows). I want the data to be in 100x100 array form. I have many files to convert like this. All of them are having file names like 0, 1, 2, 500. with out any extension. Actually, I renamed actual files so that I can import them in Matlab for batch processing. Since Matlab also new for me, I thought I will try Numpy first. Can any body help me in writing the script to do this for making batch processing. Thanks in advance, Venkat -- *** D.Venkat Research Scholar Dept of Physics IISc, Bangalore India-560 012 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] How to import input data to make ndarray for batch processing?
El jeu., 18-11-2010 a las 20:19 +0530, Venkat escribió: I have many files to convert like this. All of them are having file names like 0, 1, 2, 500. with out any extension. Actually, I renamed actual files so that I can import them in Matlab for batch processing. Since Matlab also new for me, I thought I will try Numpy first. Can any body help me in writing the script to do this for making batch processing. One point that others did not answer is the 'batch' part. If your files are named sequentially, you can 'template' the argument you pass to the loader function. For example, if you load with numpy.loadtxt your data that is stored in files named 'mydata0', 'mydata1', 'mydata511', your batch processing may look like that for ind in xrange(512): filename = 'mydata%d' % ind data = numpy.loadtxt(filename, ... ) #... your processing on single file with adapted range of indices (see xrange doc), string formatting (see string doc) and arguments to loader function -- Fabrice Silva ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] How to import input data to make ndarray for batch processing?
On 11/18/10 7:40 AM, Dave Hirschfeld wrote:
In [7]: data = np.loadtxt('dummy_data.txt')
or, faster:
data = np.fromfile('dummy_data.txt', dtype=np.float64, sep = ' ')
fromfile() is not very flexible, and doesn't have good error handling,
but it's a lot faster than loadtxt for the simple cases like this.
-Chris
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/ORR(206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] How to import input data to make ndarray for batch processing?
On Nov 18, 2010, at 6:49 AM, Venkat wrote: I am trying to reshape my text data which is in one single column (10,000 rows). I want the data to be in 100x100 array form. If all you want to do is converting the actual files, and you are using a unix-ish operating system, you don't even need python: paste - - - - - - - - - - filename newfilename should do the trick, without any assumptions on the type of data or change in precision due to reading/writing. Hope this helps, Lutz ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
