[Numpy-discussion] How to import input data to make ndarray for batch processing?
Hi All, I am new to Numpy (also Scipy). I am trying to reshape my text data which is in one single column (10,000 rows). I want the data to be in 100x100 array form. I have many files to convert like this. All of them are having file names like 0, 1, 2, 500. with out any extension. Actually, I renamed actual files so that I can import them in Matlab for batch processing. Since Matlab also new for me, I thought I will try Numpy first. Can any body help me in writing the script to do this for making batch processing. Thanks in advance, Venkat -- *** D.Venkat Research Scholar Dept of Physics IISc, Bangalore India-560 012 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] How to import input data to make ndarray for batch processing?
Do you want to save the file to disk as 100x100 matrices, or just to read them into the memory? Are the files in ascii or binary format? Nadav From: numpy-discussion-boun...@scipy.org [numpy-discussion-boun...@scipy.org] On Behalf Of Venkat [dvr...@gmail.com] Sent: 18 November 2010 16:49 To: Discussion of Numerical Python Subject: [Numpy-discussion] How to import input data to make ndarray for batch processing? Hi All, I am new to Numpy (also Scipy). I am trying to reshape my text data which is in one single column (10,000 rows). I want the data to be in 100x100 array form. I have many files to convert like this. All of them are having file names like 0, 1, 2, 500. with out any extension. Actually, I renamed actual files so that I can import them in Matlab for batch processing. Since Matlab also new for me, I thought I will try Numpy first. Can any body help me in writing the script to do this for making batch processing. Thanks in advance, Venkat -- *** D.Venkat Research Scholar Dept of Physics IISc, Bangalore India-560 012 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] How to import input data to make nda rray for batch processing?
Venkat dvr002 at gmail.com writes:
Hi All,I am new to Numpy (also Scipy).I am trying to reshape my text data
which is in one single column (10,000 rows).I want the data to be in 100x100
array form.I have many files to convert like this. All of them are having file
names like 0, 1, 2, 500. with out any extension.
Actually, I renamed actual files so that I can import them in Matlab for batch
processing.Since Matlab also new for me, I thought I will try Numpy first. Can
any body help me in writing the script to do this for making batch processing.
Thanks in advance,Venkat
In [2]: dummy_data = np.random.randn(100,100)
In [3]: dummy_data.shape
Out[3]: (100, 100)
In [4]: dummy_data.flatten().shape
Out[4]: (1,)
In [5]: np.savetxt('dummy_data.txt', dummy_data.flatten())
In [6]: !less dummy_data.txt
2.571031186906808100e-01
1.566790681796508500e+00
-6.846267829937818800e-01
3.271332705287631200e-01
-7.482409829656505600e-02
1.429095432454441600e-01
-6.41694801869400e-01
-5.319842186383831900e-01
-4.047786844569442600e-01
-6.696045994533519300e-01
-4.895085917712171400e-01
6.969419814656118200e-01
6.656815445278644300e-01
7.455135053686292600e-01
...
In [7]: data = np.loadtxt('dummy_data.txt')
In [8]: data.shape
Out[8]: (1,)
In [9]: data = data.reshape(100, 100)
In [10]: data.shape
Out[10]: (100, 100)
In [11]: np.allclose(dummy_data, data)
Out[11]: True
HTH,
Dave
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] How to import input data to make ndarray for batch processing?
El jeu., 18-11-2010 a las 20:19 +0530, Venkat escribió: I have many files to convert like this. All of them are having file names like 0, 1, 2, 500. with out any extension. Actually, I renamed actual files so that I can import them in Matlab for batch processing. Since Matlab also new for me, I thought I will try Numpy first. Can any body help me in writing the script to do this for making batch processing. One point that others did not answer is the 'batch' part. If your files are named sequentially, you can 'template' the argument you pass to the loader function. For example, if you load with numpy.loadtxt your data that is stored in files named 'mydata0', 'mydata1', 'mydata511', your batch processing may look like that for ind in xrange(512): filename = 'mydata%d' % ind data = numpy.loadtxt(filename, ... ) #... your processing on single file with adapted range of indices (see xrange doc), string formatting (see string doc) and arguments to loader function -- Fabrice Silva ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy + amdlibm?
Anyone tried building numpy with amdlibm? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Returning numpy scalars in cython functions
The cython function below returns a long int: @cython.boundscheck(False) def mysum(np.ndarray[np.int64_t, ndim=1] a): sum of 1d numpy array with dtype=np.int64. cdef Py_ssize_t i cdef int asize = a.shape[0] cdef np.int64_t asum = 0 for i in range(asize): asum += a[i] return asum What's the best way to make it return a numpy long int, or whatever it is called, that has dtype, ndim, size, etc. class methods? The only thing I could come up with is changing the last line to return np.array(asum)[()] It works. And adds some overhead: a = np.arange(10) timeit mysum(a) 1000 loops, best of 3: 167 ns per loop timeit mysum2(a) 100 loops, best of 3: 984 ns per loop And for scale: timeit np.sum(a) 10 loops, best of 3: 3.3 us per loop I'm new to cython. Did I miss any optimizations in the mysum function above? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Returning numpy scalars in cython functions
A Thursday 18 November 2010 18:51:04 Keith Goodman escrigué: The cython function below returns a long int: @cython.boundscheck(False) def mysum(np.ndarray[np.int64_t, ndim=1] a): sum of 1d numpy array with dtype=np.int64. cdef Py_ssize_t i cdef int asize = a.shape[0] cdef np.int64_t asum = 0 for i in range(asize): asum += a[i] return asum What's the best way to make it return a numpy long int, or whatever it is called, that has dtype, ndim, size, etc. class methods? The only thing I could come up with is changing the last line to return np.array(asum)[()] It works. And adds some overhead: a = np.arange(10) timeit mysum(a) 1000 loops, best of 3: 167 ns per loop timeit mysum2(a) 100 loops, best of 3: 984 ns per loop And for scale: timeit np.sum(a) 10 loops, best of 3: 3.3 us per loop Perhaps the scalar constructor is your best bet: type(np.array(2)[()]) type 'numpy.int64' type(np.int_(2)) type 'numpy.int64' timeit np.array(2)[()] 100 loops, best of 3: 791 ns per loop timeit np.int_(2) 100 loops, best of 3: 234 ns per loop -- Francesc Alted ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Returning numpy scalars in cython functions
A Thursday 18 November 2010 19:08:00 Francesc Alted escrigué: type(np.int_(2)) Err, for maximum portability you can use the int64 constructor: type(np.int64(2)) type 'numpy.int64' Cheers, -- Francesc Alted ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Returning numpy scalars in cython functions
On Thu, Nov 18, 2010 at 10:08 AM, Francesc Alted fal...@pytables.org wrote: A Thursday 18 November 2010 18:51:04 Keith Goodman escrigué: What's the best way to make it return a numpy long int, or whatever it is called, that has dtype, ndim, size, etc. class methods? The only thing I could come up with is changing the last line to return np.array(asum)[()] Perhaps the scalar constructor is your best bet: type(np.array(2)[()]) type 'numpy.int64' type(np.int_(2)) type 'numpy.int64' timeit np.array(2)[()] 100 loops, best of 3: 791 ns per loop timeit np.int_(2) 100 loops, best of 3: 234 ns per loop Perfect! Thank you. a = np.arange(10) timeit mysum2(a) 100 loops, best of 3: 1.16 us per loop timeit mysum2_francesc(a) 100 loops, best of 3: 451 ns per loop I also added @cython.wraparound(False). ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] How to import input data to make ndarray for batch processing?
On 11/18/10 7:40 AM, Dave Hirschfeld wrote:
In [7]: data = np.loadtxt('dummy_data.txt')
or, faster:
data = np.fromfile('dummy_data.txt', dtype=np.float64, sep = ' ')
fromfile() is not very flexible, and doesn't have good error handling,
but it's a lot faster than loadtxt for the simple cases like this.
-Chris
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/ORR(206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] How to import input data to make ndarray for batch processing?
On Nov 18, 2010, at 6:49 AM, Venkat wrote: I am trying to reshape my text data which is in one single column (10,000 rows). I want the data to be in 100x100 array form. If all you want to do is converting the actual files, and you are using a unix-ish operating system, you don't even need python: paste - - - - - - - - - - filename newfilename should do the trick, without any assumptions on the type of data or change in precision due to reading/writing. Hope this helps, Lutz ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Test failures on 2.6
I'm having almost exactly the same problem, but with Python 2.6.1, Numpy 1.2.1, and Nose 0.11.3. Nobody responded to TJ the first time around, so any advice would be greatly appreciated. Thanks, Brad -- From: T J tjhnson at gmail.com Subject: Test failures on 2.6 Newsgroups: gmane.comp.python.numeric.general Date: 2008-10-05 20:53:22 GMT (2 years, 6 weeks, 1 day, 13 hours and 32 minutes ago) Hi, I'm getting a couple of test failures with Python 2.6, Numpy 1.2.0, Nose 0.10.4: nose version 0.10.4 ..FK .. .../share/home/me/usr/lib/python2.6/site-packages/numpy/lib/tests/test_io.py:68: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert(c.readlines(), ./share/home/me/usr/lib/python2.6/site-packages/numpy/ma/tests/test_core.py:1315: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert(store._mask, True) /home/me/usr/lib/python2.6/site-packages/numpy/ma/tests/test_core.py:1322: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert(store._mask, True) /home/me/usr/lib/python2.6/site-packages/numpy/ma/tests/test_core.py:1989: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert(test.mask, [0,1,0,0,0,0,0,0,0,0]) ...E == ERROR: Tests the min/max functions with explicit outputs -- Traceback (most recent call last): File /home/me/usr/lib/python2.6/site-packages/numpy/ma/tests/test_core.py, line 653, in test_minmax_funcs_with_output result = npfunc(xm,axis=0,out=nout) File /home/me/usr/lib/python2.6/site-packages/numpy/core/fromnumeric.py, line 1525, in amin return amin(axis, out) File /home/me/usr/lib/python2.6/site-packages/numpy/ma/core.py, line 2978, in min np.putmask(out, newmask, np.nan) ValueError: cannot convert float NaN to integer == FAIL: test_umath.TestComplexFunctions.test_against_cmath -- Traceback (most recent call last): File /home/me/usr/lib/python2.6/site-packages/nose-0.10.4-py2.6.egg/nose/case.py, line 182, in runTest self.test(*self.arg) File /home/me/usr/lib/python2.6/site-packages/numpy/core/tests/test_umath.py, line 268, in test_against_cmath assert abs(a - b) atol, %s %s: %s; cmath: %s%(fname,p,a,b) AssertionError: arcsin 2: (1.57079632679-1.31695789692j); cmath: (1.57079632679+1.31695789692j) -- Ran 1726 tests in 8.856s FAILED (KNOWNFAIL=1, errors=1, failures=1) nose.result.TextTestResult run=1726 errors=1 failures=1 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Test failures on 2.6
On Oct 5, 2008, at 10:53 PM, T J wrote: Hi, I'm getting a couple of test failures with Python 2.6, Numpy 1.2.0, Nose 0.10.4: Wow, 1.2.0 ? That's fairly ancient. I gather the bugs in numpy.ma have been corrected since (they don't really look familiar, though). And with a more recent numpy ? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Advise for numerical programming content (New python user)
Users, I am an average Fortran user. I am new to python and I am currently evaluating options and functionalities of numerical programming and related 2d and 3d graphic outputs with python. Kindly share your experience in scientific programming with python like how do you like it, comparison with Fortran and C++. Which version of python + numpy+scipy are compatible with each other or if any other numerical analysis package is available (I am working on windows environment.) Does graphic output like maps, histogram, crossplot, tornado charts is good enough with basic installation or needs some additional packages? Your feedback is valuable for me to start. Thanks Regards Sachin Sachin Kumar Sharma Senior Geomodeler - Samarang Project (IPM) Field Development Production Services (DCS) Schlumberger Sdn. Bhd., 7th Floor, West Wing, Rohas Perkasa, No. 8 Jalan Perak, Kuala Lumpur, 50450, Malaysia Mobile: +60 12 2196443 * Email: ssharm...@exchange.slb.commailto:ssharm...@bombay.oilfield.slb.com sachin_sha...@petronas.com.my ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Advise for numerical programming content (New python user)
On 11/18/2010 9:48 PM, Sachin Kumar Sharma wrote: Does graphic output like maps, histogram, crossplot, tornado charts is good enough with basic installation or needs some additional packages? For the graphics, you should probably first consider Matplotlib. For your other questions, perhaps look at Python Scripting for Computational Science by Hans Petter Langtangen. Alan Isaac ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Advise for numerical programming content (New python user)
Thanks Alan, Best regards Sachin Sachin Kumar Sharma Senior Geomodeler -Original Message- From: numpy-discussion-boun...@scipy.org [mailto:numpy-discussion-boun...@scipy.org] On Behalf Of Alan G Isaac Sent: Friday, November 19, 2010 10:55 AM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Advise for numerical programming content (New python user) On 11/18/2010 9:48 PM, Sachin Kumar Sharma wrote: Does graphic output like maps, histogram, crossplot, tornado charts is good enough with basic installation or needs some additional packages? For the graphics, you should probably first consider Matplotlib. For your other questions, perhaps look at Python Scripting for Computational Science by Hans Petter Langtangen. Alan Isaac ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Advise for numerical programming content (New python user)
For a beginner, I think pythonxy is a good option. Then you don't have to worry about the compatibility issue. http://www.pythonxy.com/ and as for the plotting, you can use the following packages: 2D - Matplotlib or Gnuplot (both are good ... but, if you want Matlab kind of environment, try Matplotlib) 3D - Mayavi or Gnuplot (I think Gnuplot has some limitations in 3D plotting) regards zinka On Fri, Nov 19, 2010 at 12:02 PM, Sachin Kumar Sharma ssharm...@slb.comwrote: Thanks Alan, Best regards Sachin Sachin Kumar Sharma Senior Geomodeler -Original Message- From: numpy-discussion-boun...@scipy.org [mailto: numpy-discussion-boun...@scipy.org] On Behalf Of Alan G Isaac Sent: Friday, November 19, 2010 10:55 AM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Advise for numerical programming content (New python user) On 11/18/2010 9:48 PM, Sachin Kumar Sharma wrote: Does graphic output like maps, histogram, crossplot, tornado charts is good enough with basic installation or needs some additional packages? For the graphics, you should probably first consider Matplotlib. For your other questions, perhaps look at Python Scripting for Computational Science by Hans Petter Langtangen. Alan Isaac ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
