Re: [Numpy-discussion] matrix inversion
Hi, > i am trying to invert matrices like this: > [[ 0.01643777 -0.13539939 0.11946689] > [ 0.12479926 0.01210898 -0.09217618] > [-0.13050087 0.07575163 0.01144993]] > > in perl using Math::MatrixReal; > and in various online calculators i get > [ 2.472715991745 3.680743681735 -3.831392002314 ] > [ -4.673105249083 -5.348238625096 -5.703193038649 ] > [ 2.733966489601 -6.567940452290 -5.936617926811 ] well, inverting latter matrix, I get >>> n=np.mat([[ 2.472715991745 , 3.680743681735 ,-3.831392002314 ], [ -4.673105249083, -5.348238625096, -5.703193038649 ], [ 2.733966489601, -6.567940452290, -5.936617926811 ]]) >>> n.I matrix([[ 0.01643777, -0.13539939, 0.11946689], [ 0.12479926, 0.01210898, -0.09217618], [-0.13050087, -0.07575163, -0.01144993]]) Which is nearly the same matrix as the matrix you started with, but quite. There are some minus signs more in the last two values... sure you didn't forget these? Adding them by hand gives nearly the same value as your Perl result, but better, the residua on the off diagonals are significantly lower. Greetings Martin ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] matrix inversion
The svs are 1.1695e-01, 1.1682e-01, 6.84719250e-10 so if you try >>> np.linalg.pinv(a,1e-5) array([[ 0.41097834, 3.12024106, -3.26279309], [-3.38526587, 0.30274957, 1.89394811], [ 2.98692033, -2.30459609, 0.28627222]]) you at least get an answer that's not near-random. w On Wed, 10 Aug 2011, Nadav Horesh wrote: > The matrix in singular, so you can not expect a stable inverse. > > Nadav. > > > From: numpy-discussion-boun...@scipy.org [numpy-discussion-boun...@scipy.org] > On Behalf Of jp d [yo...@yahoo.com] > Sent: 11 August 2011 03:50 > To: numpy-discussion@scipy.org > Subject: [Numpy-discussion] matrix inversion > > hi, > i am trying to invert matrices like this: > [[ 0.01643777 -0.13539939 0.11946689] > [ 0.12479926 0.01210898 -0.09217618] > [-0.13050087 0.07575163 0.01144993]] > > in perl using Math::MatrixReal; > and in various online calculators i get > [ 2.472715991745 3.680743681735 -3.831392002314 ] > [ -4.673105249083 -5.348238625096 -5.703193038649 ] > [ 2.733966489601 -6.567940452290 -5.936617926811 ] > > using python , numpy and linalg.inv (or linalg.pinv) i get a divergent answer > [[ 6.79611151e+07 1.01163031e+08 1.05303510e+08] > [ 1.01163057e+08 1.50585545e+08 1.56748838e+08] > [ 1.05303548e+08 1.56748831e+08 1.63164381e+08]] > > any suggestions? > > thanks > jpd > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] matrix inversion
The matrix in singular, so you can not expect a stable inverse. Nadav. From: numpy-discussion-boun...@scipy.org [numpy-discussion-boun...@scipy.org] On Behalf Of jp d [yo...@yahoo.com] Sent: 11 August 2011 03:50 To: numpy-discussion@scipy.org Subject: [Numpy-discussion] matrix inversion hi, i am trying to invert matrices like this: [[ 0.01643777 -0.13539939 0.11946689] [ 0.12479926 0.01210898 -0.09217618] [-0.13050087 0.07575163 0.01144993]] in perl using Math::MatrixReal; and in various online calculators i get [ 2.472715991745 3.680743681735 -3.831392002314 ] [ -4.673105249083 -5.348238625096 -5.703193038649 ] [ 2.733966489601 -6.567940452290 -5.936617926811 ] using python , numpy and linalg.inv (or linalg.pinv) i get a divergent answer [[ 6.79611151e+07 1.01163031e+08 1.05303510e+08] [ 1.01163057e+08 1.50585545e+08 1.56748838e+08] [ 1.05303548e+08 1.56748831e+08 1.63164381e+08]] any suggestions? thanks jpd ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] matrix inversion
On 8/10/2011 8:50 PM, jp d wrote: > i am trying to invert matrices like this: > [[ 0.01643777 -0.13539939 0.11946689] > [ 0.12479926 0.01210898 -0.09217618] > [-0.13050087 0.07575163 0.01144993]] > in perl using Math::MatrixReal; > and in various online calculators i get > [ 2.472715991745 3.680743681735 -3.831392002314 ] > [ -4.673105249083 -5.348238625096 -5.703193038649 ] > [ 2.733966489601 -6.567940452290 -5.936617926811 ] > using python , numpy and linalg.inv (or linalg.pinv) i get a divergent answer > [[ 6.79611151e+07 1.01163031e+08 1.05303510e+08] > [ 1.01163057e+08 1.50585545e+08 1.56748838e+08] > [ 1.05303548e+08 1.56748831e+08 1.63164381e+08]] Please demonstrate with code:: >>> m = np.mat([[ 0.01643777,-0.13539939, 0.11946689],[ 0.12479926, 0.01210898,-0.09217618 ],[-0.13050087, 0.07575163, 0.01144993]]) >>> m.I matrix([[ -2.60023901e+08, -3.87056678e+08, -4.02898472e+08], [ -3.87056814e+08, -5.76150592e+08, -5.99731775e+08], [ -4.02898597e+08, -5.99731775e+08, -6.24278108e+08]]) Thank you, Alan Isaac ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] matrix inversion
hi, i am trying to invert matrices like this: [[ 0.01643777 -0.13539939 0.11946689] [ 0.12479926 0.01210898 -0.09217618] [-0.13050087 0.07575163 0.01144993]] in perl using Math::MatrixReal; and in various online calculators i get [ 2.472715991745 3.680743681735 -3.831392002314 ] [ -4.673105249083 -5.348238625096 -5.703193038649 ] [ 2.733966489601 -6.567940452290 -5.936617926811 ] using python , numpy and linalg.inv (or linalg.pinv) i get a divergent answer [[ 6.79611151e+07 1.01163031e+08 1.05303510e+08] [ 1.01163057e+08 1.50585545e+08 1.56748838e+08] [ 1.05303548e+08 1.56748831e+08 1.63164381e+08]] any suggestions? thanks jpd ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpydoc - latex longtables error
Hi,
On Wed, Aug 10, 2011 at 5:03 PM, wrote:
> On Wed, Aug 10, 2011 at 6:17 PM, Matthew Brett
> wrote:
>> Hi,
>>
>> On Wed, Aug 10, 2011 at 12:38 PM, Skipper Seabold
>> wrote:
>>> On Wed, Aug 10, 2011 at 3:28 PM, Matthew Brett
>>> wrote:
Hi,
I think this one might be for Pauli.
I've run into an odd problem that seems to be an interaction of
numpydoc and autosummary and large classes.
In summary, large classes and numpydoc lead to large tables of class
methods, and there seems to be an error in the creation of the large
tables in latex.
Specifically, if I run 'make latexpdf' with the attached minimal
sphinx setup, I get a pdflatex error ending thus:
...
l.118 \begin{longtable}{LL}
and this is because longtable does not accept LL as an argument, but
needs '|l|l|' (bar - el - bar - el - bar).
I see in sphinx.writers.latex.py, around line 657, that sphinx knows
about this in general, and long tables in standard ReST work fine with
the el-bar arguments passed to longtable.
if self.table.colspec:
self.body.append(self.table.colspec)
else:
if self.table.has_problematic:
colwidth = 0.95 / self.table.colcount
colspec = ('p{%.3f\\linewidth}|' % colwidth) * \
self.table.colcount
self.body.append('{|' + colspec + '}\n')
elif self.table.longtable:
self.body.append('{|' + ('l|' * self.table.colcount) +
'}\n')
else:
self.body.append('{|' + ('L|' * self.table.colcount) +
'}\n')
However, using numpydoc and autosummary (see the conf.py file), what
seems to happen is that, when we reach the self.table.colspec test at
the beginning of the snippet above, 'self.table.colspec' is defined:
In [1]: self.table.colspec
Out[1]: '{LL}\n'
and thus the LL gets written as the arg to longtable:
\begin{longtable}{LL}
and the pdf build breaks.
I'm using the numpydoc out of the current numpy source tree.
At that point I wasn't sure how to proceed with debugging. Can you
give any hints?
>>>
>>> It's not a proper fix, but our workaround is to edit the Makefile for
>>> latex (and latexpdf) to
>>>
>>> https://github.com/statsmodels/statsmodels/blob/master/scikits/statsmodels/docs/Makefile#L94
>>> https://github.com/statsmodels/statsmodels/blob/master/scikits/statsmodels/docs/make.bat#L121
>>>
>>> to call the script to replace the longtable arguments
>>>
>>> https://github.com/statsmodels/statsmodels/blob/master/scikits/statsmodels/docs/fix_longtable.py
>>>
>>> The workaround itself probably isn't optimal, and I'd be happy to hear
>>> of a proper fix.
>>
>> Thanks - yes - I found your workaround in my explorations, I put in a
>> version in our tree too:
>>
>> https://github.com/matthew-brett/nipy/blob/latex_build_fixes/tools/fix_longtable.py
>>
>> - but I agree it seems much better to get to the root cause.
>
> When I tried to figure this out, I never found out why the correct
> sphinx longtable code path never gets reached, or which code
> (numpydoc, autosummary or sphinx) is filling in the colspec.
No - it looked hard to debug. I established that it required numpydoc
and autosummary to be enabled.
See you,
Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpydoc - latex longtables error
On Wed, Aug 10, 2011 at 6:17 PM, Matthew Brett wrote:
> Hi,
>
> On Wed, Aug 10, 2011 at 12:38 PM, Skipper Seabold wrote:
>> On Wed, Aug 10, 2011 at 3:28 PM, Matthew Brett
>> wrote:
>>> Hi,
>>>
>>> I think this one might be for Pauli.
>>>
>>> I've run into an odd problem that seems to be an interaction of
>>> numpydoc and autosummary and large classes.
>>>
>>> In summary, large classes and numpydoc lead to large tables of class
>>> methods, and there seems to be an error in the creation of the large
>>> tables in latex.
>>>
>>> Specifically, if I run 'make latexpdf' with the attached minimal
>>> sphinx setup, I get a pdflatex error ending thus:
>>>
>>> ...
>>> l.118 \begin{longtable}{LL}
>>>
>>> and this is because longtable does not accept LL as an argument, but
>>> needs '|l|l|' (bar - el - bar - el - bar).
>>>
>>> I see in sphinx.writers.latex.py, around line 657, that sphinx knows
>>> about this in general, and long tables in standard ReST work fine with
>>> the el-bar arguments passed to longtable.
>>>
>>> if self.table.colspec:
>>> self.body.append(self.table.colspec)
>>> else:
>>> if self.table.has_problematic:
>>> colwidth = 0.95 / self.table.colcount
>>> colspec = ('p{%.3f\\linewidth}|' % colwidth) * \
>>> self.table.colcount
>>> self.body.append('{|' + colspec + '}\n')
>>> elif self.table.longtable:
>>> self.body.append('{|' + ('l|' * self.table.colcount) + '}\n')
>>> else:
>>> self.body.append('{|' + ('L|' * self.table.colcount) + '}\n')
>>>
>>> However, using numpydoc and autosummary (see the conf.py file), what
>>> seems to happen is that, when we reach the self.table.colspec test at
>>> the beginning of the snippet above, 'self.table.colspec' is defined:
>>>
>>> In [1]: self.table.colspec
>>> Out[1]: '{LL}\n'
>>>
>>> and thus the LL gets written as the arg to longtable:
>>>
>>> \begin{longtable}{LL}
>>>
>>> and the pdf build breaks.
>>>
>>> I'm using the numpydoc out of the current numpy source tree.
>>>
>>> At that point I wasn't sure how to proceed with debugging. Can you
>>> give any hints?
>>>
>>
>> It's not a proper fix, but our workaround is to edit the Makefile for
>> latex (and latexpdf) to
>>
>> https://github.com/statsmodels/statsmodels/blob/master/scikits/statsmodels/docs/Makefile#L94
>> https://github.com/statsmodels/statsmodels/blob/master/scikits/statsmodels/docs/make.bat#L121
>>
>> to call the script to replace the longtable arguments
>>
>> https://github.com/statsmodels/statsmodels/blob/master/scikits/statsmodels/docs/fix_longtable.py
>>
>> The workaround itself probably isn't optimal, and I'd be happy to hear
>> of a proper fix.
>
> Thanks - yes - I found your workaround in my explorations, I put in a
> version in our tree too:
>
> https://github.com/matthew-brett/nipy/blob/latex_build_fixes/tools/fix_longtable.py
>
> - but I agree it seems much better to get to the root cause.
When I tried to figure this out, I never found out why the correct
sphinx longtable code path never gets reached, or which code
(numpydoc, autosummary or sphinx) is filling in the colspec.
Josef
>
> See you,
>
> Matthew
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpydoc - latex longtables error
Hi,
On Wed, Aug 10, 2011 at 12:38 PM, Skipper Seabold wrote:
> On Wed, Aug 10, 2011 at 3:28 PM, Matthew Brett
> wrote:
>> Hi,
>>
>> I think this one might be for Pauli.
>>
>> I've run into an odd problem that seems to be an interaction of
>> numpydoc and autosummary and large classes.
>>
>> In summary, large classes and numpydoc lead to large tables of class
>> methods, and there seems to be an error in the creation of the large
>> tables in latex.
>>
>> Specifically, if I run 'make latexpdf' with the attached minimal
>> sphinx setup, I get a pdflatex error ending thus:
>>
>> ...
>> l.118 \begin{longtable}{LL}
>>
>> and this is because longtable does not accept LL as an argument, but
>> needs '|l|l|' (bar - el - bar - el - bar).
>>
>> I see in sphinx.writers.latex.py, around line 657, that sphinx knows
>> about this in general, and long tables in standard ReST work fine with
>> the el-bar arguments passed to longtable.
>>
>> if self.table.colspec:
>> self.body.append(self.table.colspec)
>> else:
>> if self.table.has_problematic:
>> colwidth = 0.95 / self.table.colcount
>> colspec = ('p{%.3f\\linewidth}|' % colwidth) * \
>> self.table.colcount
>> self.body.append('{|' + colspec + '}\n')
>> elif self.table.longtable:
>> self.body.append('{|' + ('l|' * self.table.colcount) + '}\n')
>> else:
>> self.body.append('{|' + ('L|' * self.table.colcount) + '}\n')
>>
>> However, using numpydoc and autosummary (see the conf.py file), what
>> seems to happen is that, when we reach the self.table.colspec test at
>> the beginning of the snippet above, 'self.table.colspec' is defined:
>>
>> In [1]: self.table.colspec
>> Out[1]: '{LL}\n'
>>
>> and thus the LL gets written as the arg to longtable:
>>
>> \begin{longtable}{LL}
>>
>> and the pdf build breaks.
>>
>> I'm using the numpydoc out of the current numpy source tree.
>>
>> At that point I wasn't sure how to proceed with debugging. Can you
>> give any hints?
>>
>
> It's not a proper fix, but our workaround is to edit the Makefile for
> latex (and latexpdf) to
>
> https://github.com/statsmodels/statsmodels/blob/master/scikits/statsmodels/docs/Makefile#L94
> https://github.com/statsmodels/statsmodels/blob/master/scikits/statsmodels/docs/make.bat#L121
>
> to call the script to replace the longtable arguments
>
> https://github.com/statsmodels/statsmodels/blob/master/scikits/statsmodels/docs/fix_longtable.py
>
> The workaround itself probably isn't optimal, and I'd be happy to hear
> of a proper fix.
Thanks - yes - I found your workaround in my explorations, I put in a
version in our tree too:
https://github.com/matthew-brett/nipy/blob/latex_build_fixes/tools/fix_longtable.py
- but I agree it seems much better to get to the root cause.
See you,
Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] bug with assignment into an indexed array?
Came across this today when trying to determine what was wrong with my code: import numpy as np matched_to = np.array([-1] * 5) in_ellipse = np.array([False, True, True, True, False]) match = np.array([False, True, True]) matched_to[in_ellipse][match] = 3 I would expect matched_to to now be "array([-1, -1, 3, 3, -1])", but instead, it is still all -1. It would seem that unless the view was created by a slice, then the assignment into the indexed view would not work as expected. This works: >>> matched_to[:3][match] = 3 but not: >>> matched_to[np.array([0, 1, 2])][match] = 3 Note that the following does work: >>> matched_to[np.array([0, 1, 2])] = 3 Is this a bug, or was I wrong to expect this to work this way? Thanks, Ben Root ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Efficient way to load a 1Gb file?
On 10. aug. 2011, at 21.03, Gael Varoquaux wrote: > On Wed, Aug 10, 2011 at 04:01:37PM -0400, Anne Archibald wrote: >> A 1 Gb text file is a miserable object anyway, so it might be desirable >> to convert to (say) HDF5 and then throw away the text file. > > +1 > > G +1 and a very warm recommendation of h5py. Paul ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Efficient way to load a 1Gb file?
On 10 Aug 2011, at 22:03, Gael Varoquaux wrote: > On Wed, Aug 10, 2011 at 04:01:37PM -0400, Anne Archibald wrote: >> A 1 Gb text file is a miserable object anyway, so it might be desirable >> to convert to (say) HDF5 and then throw away the text file. > > +1 There might be concerns about ensuring data accessibility agains throwing the text file away, but converting to HDF5 would be an elegant for reading in without the memory issues, too (I must confess though, I've regularly read ~ 1GB ASCII files into memory - with decent virtual memory management it did not turn out too bad...) Cheers, Derek ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Efficient way to load a 1Gb file?
On Wed, Aug 10, 2011 at 04:01:37PM -0400, Anne Archibald wrote: > A 1 Gb text file is a miserable object anyway, so it might be desirable > to convert to (say) HDF5 and then throw away the text file. +1 G ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Efficient way to load a 1Gb file?
There was also some work on a semi-mutable array type that allowed appending along one axis, then 'freezing' to yield a normal numpy array (unfortunately I'm not sure how to find it in the mailing list archives). One could write such a setup by hand, using mmap() or realloc(), but I'd be inclined to simply write a filter that converted the text file to some sort of binary file on the fly, value by value. Then the file can be loaded in or mmap()ed. A 1 Gb text file is a miserable object anyway, so it might be desirable to convert to (say) HDF5 and then throw away the text file. Anne On 10 August 2011 15:43, Derek Homeier wrote: > On 10 Aug 2011, at 19:22, Russell E. Owen wrote: > >> A coworker is trying to load a 1Gb text data file into a numpy array >> using numpy.loadtxt, but he says it is using up all of his machine's 6Gb >> of RAM. Is there a more efficient way to read such text data files? > > The npyio routines (loadtxt as well as genfromtxt) first read in the entire > data as lists, which creates of course significant overhead, but is not easy > to circumvent, since numpy arrays are immutable - so you have to first store > the numbers in some kind of mutable object. One could write a custom parser > that tries to be somewhat more efficient, e.g. first reading in sub-arrays > from a smaller buffer. Concatenating those sub-arrays would still require > about twice the memory of the final array. I don't know if using the > array.array type (which is mutable) is much more efficient than a list... > To really avoid any excess memory usage you'd have to know the total data > size in advance - either by reading in the file in a first pass to count the > rows, or explicitly specifying it to a custom reader. Basically, assuming a > completely regular file without missing values etc., you could then read in > the data like > > X = np.zeros((n_lines, n_columns), dtype=float) > delimiter = ' ' > for n, line in enumerate(file(fname, 'r')): > X[n] = np.array(line.split(delimiter), dtype=float) > > (adjust delimiter and dtype as needed...) > > HTH, > Derek > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Efficient way to load a 1Gb file?
On 10 Aug 2011, at 19:22, Russell E. Owen wrote: > A coworker is trying to load a 1Gb text data file into a numpy array > using numpy.loadtxt, but he says it is using up all of his machine's 6Gb > of RAM. Is there a more efficient way to read such text data files? The npyio routines (loadtxt as well as genfromtxt) first read in the entire data as lists, which creates of course significant overhead, but is not easy to circumvent, since numpy arrays are immutable - so you have to first store the numbers in some kind of mutable object. One could write a custom parser that tries to be somewhat more efficient, e.g. first reading in sub-arrays from a smaller buffer. Concatenating those sub-arrays would still require about twice the memory of the final array. I don't know if using the array.array type (which is mutable) is much more efficient than a list... To really avoid any excess memory usage you'd have to know the total data size in advance - either by reading in the file in a first pass to count the rows, or explicitly specifying it to a custom reader. Basically, assuming a completely regular file without missing values etc., you could then read in the data like X = np.zeros((n_lines, n_columns), dtype=float) delimiter = ' ' for n, line in enumerate(file(fname, 'r')): X[n] = np.array(line.split(delimiter), dtype=float) (adjust delimiter and dtype as needed...) HTH, Derek ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpydoc - latex longtables error
On Wed, Aug 10, 2011 at 3:28 PM, Matthew Brett wrote:
> Hi,
>
> I think this one might be for Pauli.
>
> I've run into an odd problem that seems to be an interaction of
> numpydoc and autosummary and large classes.
>
> In summary, large classes and numpydoc lead to large tables of class
> methods, and there seems to be an error in the creation of the large
> tables in latex.
>
> Specifically, if I run 'make latexpdf' with the attached minimal
> sphinx setup, I get a pdflatex error ending thus:
>
> ...
> l.118 \begin{longtable}{LL}
>
> and this is because longtable does not accept LL as an argument, but
> needs '|l|l|' (bar - el - bar - el - bar).
>
> I see in sphinx.writers.latex.py, around line 657, that sphinx knows
> about this in general, and long tables in standard ReST work fine with
> the el-bar arguments passed to longtable.
>
> if self.table.colspec:
> self.body.append(self.table.colspec)
> else:
> if self.table.has_problematic:
> colwidth = 0.95 / self.table.colcount
> colspec = ('p{%.3f\\linewidth}|' % colwidth) * \
> self.table.colcount
> self.body.append('{|' + colspec + '}\n')
> elif self.table.longtable:
> self.body.append('{|' + ('l|' * self.table.colcount) + '}\n')
> else:
> self.body.append('{|' + ('L|' * self.table.colcount) + '}\n')
>
> However, using numpydoc and autosummary (see the conf.py file), what
> seems to happen is that, when we reach the self.table.colspec test at
> the beginning of the snippet above, 'self.table.colspec' is defined:
>
> In [1]: self.table.colspec
> Out[1]: '{LL}\n'
>
> and thus the LL gets written as the arg to longtable:
>
> \begin{longtable}{LL}
>
> and the pdf build breaks.
>
> I'm using the numpydoc out of the current numpy source tree.
>
> At that point I wasn't sure how to proceed with debugging. Can you
> give any hints?
>
It's not a proper fix, but our workaround is to edit the Makefile for
latex (and latexpdf) to
https://github.com/statsmodels/statsmodels/blob/master/scikits/statsmodels/docs/Makefile#L94
https://github.com/statsmodels/statsmodels/blob/master/scikits/statsmodels/docs/make.bat#L121
to call the script to replace the longtable arguments
https://github.com/statsmodels/statsmodels/blob/master/scikits/statsmodels/docs/fix_longtable.py
The workaround itself probably isn't optimal, and I'd be happy to hear
of a proper fix.
Cheers,
Skipper
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpydoc - latex longtables error
Hi,
I think this one might be for Pauli.
I've run into an odd problem that seems to be an interaction of
numpydoc and autosummary and large classes.
In summary, large classes and numpydoc lead to large tables of class
methods, and there seems to be an error in the creation of the large
tables in latex.
Specifically, if I run 'make latexpdf' with the attached minimal
sphinx setup, I get a pdflatex error ending thus:
...
l.118 \begin{longtable}{LL}
and this is because longtable does not accept LL as an argument, but
needs '|l|l|' (bar - el - bar - el - bar).
I see in sphinx.writers.latex.py, around line 657, that sphinx knows
about this in general, and long tables in standard ReST work fine with
the el-bar arguments passed to longtable.
if self.table.colspec:
self.body.append(self.table.colspec)
else:
if self.table.has_problematic:
colwidth = 0.95 / self.table.colcount
colspec = ('p{%.3f\\linewidth}|' % colwidth) * \
self.table.colcount
self.body.append('{|' + colspec + '}\n')
elif self.table.longtable:
self.body.append('{|' + ('l|' * self.table.colcount) + '}\n')
else:
self.body.append('{|' + ('L|' * self.table.colcount) + '}\n')
However, using numpydoc and autosummary (see the conf.py file), what
seems to happen is that, when we reach the self.table.colspec test at
the beginning of the snippet above, 'self.table.colspec' is defined:
In [1]: self.table.colspec
Out[1]: '{LL}\n'
and thus the LL gets written as the arg to longtable:
\begin{longtable}{LL}
and the pdf build breaks.
I'm using the numpydoc out of the current numpy source tree.
At that point I wasn't sure how to proceed with debugging. Can you
give any hints?
Thanks a lot,
Matthew
long_test.tgz
Description: GNU Zip compressed data
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Efficient way to load a 1Gb file?
A coworker is trying to load a 1Gb text data file into a numpy array using numpy.loadtxt, but he says it is using up all of his machine's 6Gb of RAM. Is there a more efficient way to read such text data files? -- Russell ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] problems with multiple outputs with numpy.nditer
Works fine with the [...]s. Thanks very much. --George On 10 August 2011 17:15, Mark Wiebe wrote: > On Wed, Aug 10, 2011 at 3:45 AM, George Nurser wrote: >> >> Hi, >> I'm running numpy 1.6.1rc2 + python 2.7.1 64-bit from python.org on OSX >> 10.6.8. >> >> I have a f2py'd fortran routine that inputs and outputs fortran real*8 >> scalars, and I normally call it like >> >> tu,tv,E,El,IF,HF,HFI = LW.rotate2u(u,v,NN,ff,0) >> >> I now want to call it over 2D arrays UT,VT,N,f >> >> Using steam-age indexing works fine: >> >> mflux_east,mflux_north,IWE,IWE_lin,InvFr,HFroude = >> np.empty([6,ny-1,nx],dtype=np.float64) >> for j in range(ny-1): >> for i in range(nx): >> u,v,NN,ff = [x[j,i] for x in UT,VT,N,f] >> >> mflux_east[j,i],mflux_north[j,i],IWE[j,i],IWE_lin[j,i],InvFr[j,i],HFroude[j,i],HFI >> = LW.rotate2u(u,v,NN,ff,0) >> >> >> >> I decided to try the new nditer option, with >> >> it = np.nditer([UT,VT,N,f,None,None,None,None,None,None,None] >> ,op_flags=4*[['readonly']]+7*[['writeonly','allocate']] >> ,op_dtypes=np.float64) >> for (u,v,NN,ff,tu,tv,E,El,IF,HF,HFI) in it: >> tu,tv,E,El,IF,HF,HFI = LW.rotate2u(u,v,NN,ff,0) >> >> >> Unfortunately this doesn't seem to work. Writing >> aa,bb,cc,dd,ee,ff,gg = it.operands[4:] > > One problem here is that the assignment needs to assign into the view the > iterator gives, something a direct assignment doesn't actually do. Instead > of > a, b = f(c,d) > you need to write it like > a[...], b[...] = f(c,d) > so that the actual values being iterated get modified. Here's what I get: > In [7]: a = np.arange(5.) > In [8]: b, c, d = a + 1, a + 2, a + 3 > In [9]: it = np.nditer([a,b,c,d] + [None]*7, > ...: op_flags=4*[['readonly']]+7*[['writeonly','allocate']], > ...: op_dtypes=np.float64) > In [10]: for (x,y,z,w,A,B,C,D,E,F,G) in it: > : A[...], B[...], C[...], D[...], E[...], F[...], G[...] = x, y, > z, w, x+y, y+z, z+w > : > In [11]: it.operands[4] > Out[11]: array([ 0., 1., 2., 3., 4.]) > In [12]: it.operands[5] > Out[12]: array([ 1., 2., 3., 4., 5.]) > In [13]: it.operands[6] > Out[13]: array([ 2., 3., 4., 5., 6.]) > In [14]: it.operands[7] > Out[14]: array([ 3., 4., 5., 6., 7.]) > In [15]: it.operands[8] > Out[15]: array([ 1., 3., 5., 7., 9.]) > In [16]: it.operands[9] > Out[16]: array([ 3., 5., 7., 9., 11.]) > In [17]: it.operands[10] > Out[17]: array([ 5., 7., 9., 11., 13.]) > > -Mark > >> >> aa seems to contain the contents of UT (bizarrely rescaled to lie >> between 0 and 1), while bb,cc etc are all zero. >> >> >> I'm not sure whether I've just called it incorrectly, or whether >> perhaps it's only supposed to work with one output array. >> >> >> --George Nurser. >> ___ >> NumPy-Discussion mailing list >> NumPy-Discussion@scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy/ctypes segfault [was: PEP 3118 array size check]
On 10 August 2011 04:01, Pauli Virtanen wrote:
> Mon, 08 Aug 2011 11:27:14 -0400, Angus McMorland wrote:
>> I've just upgraded to the latest numpy from git along with upgrading
>> Ubuntu to natty. Now some of my code, which relies on ctypes-wrapping of
>> data structures from a messaging system, fails with the error message:
>>
>> "RuntimeWarning: Item size computed from the PEP 3118 buffer format
>> string does not match the actual item size."
>>
>> Can anyone tell me if this was a change that has been added into the git
>> version recently, in which case I can checkout a previous version of
>> numpy, or if I've got to try downgrading the whole system (ergh.)
>
> Python's ctypes module implements its PEP 3118 support incorrectly
> in recent Python versions. There's a patch in waiting:
>
> http://bugs.python.org/issue10744
>
> In the meantime, you can just silence the warnings using the warnings
> module,
>
> warnings.simplefilter("ignore", RuntimeWarning)
Thanks Pauli.
I was seeing a segfault everytime I saw the error message, and since
both started happening at the same time, I was guilty of mixing
correlation and causation. After rebuilding numpy about 10 times, I
have identified the first git commit after which the segfault appears
(feb8079070b8a659d7ee) , and a small piece of code that triggers it:
from ctypes import Structure, c_double
#-- copied out of an xml2py generated file
class S(Structure):
pass
S._pack_ = 4
S._fields_ = [
('field', c_double * 2),
]
#--
import numpy as np
print np.version.version
s = S()
print "S", np.asarray(s.field)
Can anyone confirm this, in which case it's probably a bug?
Thanks,
Angus
--
AJC McMorland
Post-doctoral research fellow
Neurobiology, University of Pittsburgh
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] problems with multiple outputs with numpy.nditer
On Wed, Aug 10, 2011 at 3:45 AM, George Nurser wrote: > Hi, > I'm running numpy 1.6.1rc2 + python 2.7.1 64-bit from python.org on OSX > 10.6.8. > > I have a f2py'd fortran routine that inputs and outputs fortran real*8 > scalars, and I normally call it like > > tu,tv,E,El,IF,HF,HFI = LW.rotate2u(u,v,NN,ff,0) > > I now want to call it over 2D arrays UT,VT,N,f > > Using steam-age indexing works fine: > > mflux_east,mflux_north,IWE,IWE_lin,InvFr,HFroude = > np.empty([6,ny-1,nx],dtype=np.float64) > for j in range(ny-1): > for i in range(nx): > u,v,NN,ff = [x[j,i] for x in UT,VT,N,f] > > mflux_east[j,i],mflux_north[j,i],IWE[j,i],IWE_lin[j,i],InvFr[j,i],HFroude[j,i],HFI > = LW.rotate2u(u,v,NN,ff,0) > > > > I decided to try the new nditer option, with > > it = np.nditer([UT,VT,N,f,None,None,None,None,None,None,None] > ,op_flags=4*[['readonly']]+7*[['writeonly','allocate']] > ,op_dtypes=np.float64) > for (u,v,NN,ff,tu,tv,E,El,IF,HF,HFI) in it: > tu,tv,E,El,IF,HF,HFI = LW.rotate2u(u,v,NN,ff,0) > > > Unfortunately this doesn't seem to work. Writing > aa,bb,cc,dd,ee,ff,gg = it.operands[4:] > One problem here is that the assignment needs to assign into the view the iterator gives, something a direct assignment doesn't actually do. Instead of a, b = f(c,d) you need to write it like a[...], b[...] = f(c,d) so that the actual values being iterated get modified. Here's what I get: In [7]: a = np.arange(5.) In [8]: b, c, d = a + 1, a + 2, a + 3 In [9]: it = np.nditer([a,b,c,d] + [None]*7, ...: op_flags=4*[['readonly']]+7*[['writeonly','allocate']], ...: op_dtypes=np.float64) In [10]: for (x,y,z,w,A,B,C,D,E,F,G) in it: : A[...], B[...], C[...], D[...], E[...], F[...], G[...] = x, y, z, w, x+y, y+z, z+w : In [11]: it.operands[4] Out[11]: array([ 0., 1., 2., 3., 4.]) In [12]: it.operands[5] Out[12]: array([ 1., 2., 3., 4., 5.]) In [13]: it.operands[6] Out[13]: array([ 2., 3., 4., 5., 6.]) In [14]: it.operands[7] Out[14]: array([ 3., 4., 5., 6., 7.]) In [15]: it.operands[8] Out[15]: array([ 1., 3., 5., 7., 9.]) In [16]: it.operands[9] Out[16]: array([ 3., 5., 7., 9., 11.]) In [17]: it.operands[10] Out[17]: array([ 5., 7., 9., 11., 13.]) -Mark > > aa seems to contain the contents of UT (bizarrely rescaled to lie > between 0 and 1), while bb,cc etc are all zero. > > > I'm not sure whether I've just called it incorrectly, or whether > perhaps it's only supposed to work with one output array. > > > --George Nurser. > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] problems with multiple outputs with numpy.nditer
Hi, I'm running numpy 1.6.1rc2 + python 2.7.1 64-bit from python.org on OSX 10.6.8. I have a f2py'd fortran routine that inputs and outputs fortran real*8 scalars, and I normally call it like tu,tv,E,El,IF,HF,HFI = LW.rotate2u(u,v,NN,ff,0) I now want to call it over 2D arrays UT,VT,N,f Using steam-age indexing works fine: mflux_east,mflux_north,IWE,IWE_lin,InvFr,HFroude = np.empty([6,ny-1,nx],dtype=np.float64) for j in range(ny-1): for i in range(nx): u,v,NN,ff = [x[j,i] for x in UT,VT,N,f] mflux_east[j,i],mflux_north[j,i],IWE[j,i],IWE_lin[j,i],InvFr[j,i],HFroude[j,i],HFI = LW.rotate2u(u,v,NN,ff,0) I decided to try the new nditer option, with it = np.nditer([UT,VT,N,f,None,None,None,None,None,None,None] ,op_flags=4*[['readonly']]+7*[['writeonly','allocate']] ,op_dtypes=np.float64) for (u,v,NN,ff,tu,tv,E,El,IF,HF,HFI) in it: tu,tv,E,El,IF,HF,HFI = LW.rotate2u(u,v,NN,ff,0) Unfortunately this doesn't seem to work. Writing aa,bb,cc,dd,ee,ff,gg = it.operands[4:] aa seems to contain the contents of UT (bizarrely rescaled to lie between 0 and 1), while bb,cc etc are all zero. I'm not sure whether I've just called it incorrectly, or whether perhaps it's only supposed to work with one output array. --George Nurser. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] PEP 3118 array size check
Mon, 08 Aug 2011 11:27:14 -0400, Angus McMorland wrote:
> I've just upgraded to the latest numpy from git along with upgrading
> Ubuntu to natty. Now some of my code, which relies on ctypes-wrapping of
> data structures from a messaging system, fails with the error message:
>
> "RuntimeWarning: Item size computed from the PEP 3118 buffer format
> string does not match the actual item size."
>
> Can anyone tell me if this was a change that has been added into the git
> version recently, in which case I can checkout a previous version of
> numpy, or if I've got to try downgrading the whole system (ergh.)
Python's ctypes module implements its PEP 3118 support incorrectly
in recent Python versions. There's a patch in waiting:
http://bugs.python.org/issue10744
In the meantime, you can just silence the warnings using the warnings
module,
warnings.simplefilter("ignore", RuntimeWarning)
--
Pauli Virtanen
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
