On Wed, Sep 8, 2010 at 9:21 AM, Rui DaCosta <[email protected]> wrote: > Thanks again, > > I was trying to > follow http://www.scipy.org/Numpy_Example_List#head-779283aaa3cc20a3786ad33c2ee1fee9d68a4a53 > but with the key difference being, that instead of the data for the three > arrays being constructed from literals, that i'd be sourcing from a > relational database. The data is quite simple, in that I have 199 recorded > days for 8 different variables, but am just trying to do a proof-of-concept > first with three, when I had this problem. > The functionality I am seeking to reproduce is in Excel as a 3x3 grid: > =CORREL(rf1,rf1) > =CORREL(rf1,rf2) > =CORREL(rf1,rf3) > =CORREL(rf2,rf1) > =CORREL(rf2,rf2) > =CORREL(rf2,rf3) > =CORREL(rf3,rf1) > =CORREL(rf3,rf2) > =CORREL(rf3,rf3) > I tried to simplify and isolate the problem, and hence the confusion. > So In sourcing from the DB, I have 3 different cursors (taking a subset of 3 > of the 8), each 199 rows in length. > From reading the lists and blogs, the way to put these into numpy seemed to > be: > rf1 = np.asarray(cursor1.fetchall(), dtype=np.float64) > rf2 = np.asarray(cursor2.fetchall(), dtype=np.float64) > rf3 = np.asarray(cursor3.fetchall(), dtype=np.float64) > Then to try and produce a correlation matrix: > print (np.corrcoef([rf1, rf2, rf3])) > But this is where I get the original error. > From looking at column_stack, this doesn't seem like what I want to achieve, > or am I looking at this the wrong way somehow?.
column_stack and rowvar=0 should work, but it might be easier to get rid of the extra dimension that the cursor gives you e.g. rf1 = np.asarray(cursor1.fetchall(), dtype=np.float64).ravel() depends on whether you want to work with 2d or 1d data arrays Josef > Thanks again, > Rui > > ________________________________ > From: "[email protected]" <[email protected]> > To: Discussion of Numerical Python <[email protected]> > Sent: Wed, September 8, 2010 2:34:42 PM > Subject: Re: [Numpy-discussion] ValueError: objects are not aligned > > On Wed, Sep 8, 2010 at 8:08 AM, Rui DaCosta <[email protected]> wrote: >> Thanks for your assistance, >> I was following this example: >> >> http://www.scipy.org/Numpy_Example_List#head-779283aaa3cc20a3786ad33c2ee1fee9d68a4a53 > > I didn't know, > If arrays are 1d, then numpy does the stacking for you., but it > doesn't work with column-vectors or 2d arrays. >>>> np.corrcoef([np.arange(3), np.arange(3)], rowvar=1) > array([[ 1., 1.], > [ 1., 1.]]) >>>> np.corrcoef([np.arange(3), np.arange(3)], rowvar=0) > array([[ NaN, NaN, NaN], > [ NaN, NaN, NaN], > [ NaN, NaN, NaN]]) >> >> I intend to use different arrays once I have resolved this problem. >> Trying your suggestion: >> ... >> print (rf1) >> print (np.corrcoef(rf1, rf1, rowvar=0)) >> results in: >> [[-0.00641625] >> [-0.00498411] >> [-0.0038015 ]] >> [[ 1. 1.] >> [ 1. 1.]] >> >> which should have been a 3x3 matrix > > I'm not sure what you want here, you only have three values > variance would also be zero, and corrcoef wouldn't be defined if you > only have 1 observation per variable > >>>> np.corrcoef(np.arange(3)[:,None]) > array([[ NaN, NaN, NaN], > [ NaN, NaN, NaN], > [ NaN, NaN, NaN]]) > >>>> np.corrcoef(np.column_stack([np.arange(5), np.arange(5)**2, >>>> np.arange(5)**3]), rowvar=0) > array([[ 1. , 0.9589266 , 0.90588235], > [ 0.9589266 , 1. , 0.98713033], > [ 0.90588235, 0.98713033, 1. ]]) > > examples are in docstring for np.cov > > > > >> as for "stacking rf1 rf1 into a single array first" how would I do this? > >>>> np.column_stack([np.arange(3)[:,None], np.arange(3)[:,None]]) > array([[0, 0], > [1, 1], > [2, 2]]) >>>> np.column_stack([np.arange(3)[:,None], np.arange(3)]) #doesn't required >>>> 2d > array([[0, 0], > [1, 1], > [2, 2]]) > >>>> np.c_[np.arange(3)[:,None], np.arange(3)[:,None]] > array([[0, 0], > [1, 1], > [2, 2]]) >>>> np.hstack([np.arange(3)[:,None], np.arange(3)[:,None]]) > array([[0, 0], > [1, 1], > [2, 2]]) > > > Josef > >> Regards, >> Rui >> ________________________________ >> From: "[email protected]" <[email protected]> >> To: Discussion of Numerical Python <[email protected]> >> Sent: Wed, September 8, 2010 1:09:46 PM >> Subject: Re: [Numpy-discussion] ValueError: objects are not aligned >> >> On Wed, Sep 8, 2010 at 5:42 AM, RuiDC <[email protected]> wrote: >>> >>> I'm getting this error, which must be a simple error in getting the >>> result >>> from the cursor in the right shape, how do I get the cursor into the >>> array >>> as a single dimension?: >>> >>> import numpy as np >>> import sqlite3 >>> conn = sqlite3.connect("simpledb") >>> cursor1 = conn.execute("select LogReturn from tblReturns limit 3") >>> rf1 = np.asarray(cursor1.fetchall(), dtype=np.float64) >>> print (rf1) >>> print (np.corrcoef([rf1, rf1])) >> >> numpy.corrcoef(x, y=None, rowvar=1, bias=0) >> >> try >> np.corrcoef(rf1, rf1, rowvar=0) no [ ] >> or >> stacking rf1 rf1 into a single array first >> >> Josef >> >>> >>> result: >>> [[-0.00641625] >>> [-0.00498411] >>> [-0.0038015 ]] >>> Traceback (most recent call last): >>> File "C:\Documents and >>> Settings\rui\workspace\numpytest\src\numpytest.py", >>> line 38, in <module> >>> print (np.corrcoef([rf1, rf1])) >>> File "C:\Python26\lib\site-packages\numpy\lib\function_base.py", line >>> 2003, in corrcoef >>> c = cov(x, y, rowvar, bias, ddof) >>> File "C:\Python26\lib\site-packages\numpy\lib\function_base.py", line >>> 1953, in cov >>> return (dot(X, X.T.conj()) / fact).squeeze() >>> -- >>> View this message in context: >>> >>> http://old.nabble.com/ValueError%3A-objects-are-not-aligned-tp29641661p29641661.html >>> Sent from the Numpy-discussion mailing list archive at Nabble.com. >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> [email protected] >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> [email protected] >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> [email protected] >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > _______________________________________________ > NumPy-Discussion mailing list > [email protected] > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > [email protected] > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ NumPy-Discussion mailing list [email protected] http://mail.scipy.org/mailman/listinfo/numpy-discussion
