Re: [Numpy-discussion] problems with numdifftools
On Wed, Oct 27, 2010 at 10:50 PM, Nicolai Heitz nicolaihe...@gmx.de wrote: m 27.10.2010 02:02, schrieb Sebastian Walter: On Wed, Oct 27, 2010 at 12:59 AM, Pauli Virtanenp...@iki.fi wrote: Tue, 26 Oct 2010 14:24:39 -0700, Nicolai Heitz wrote: http://mail.scipy.org/mailman/listinfo/scipy-user I contacted them already but they didn't responded so far and I was forwarded to that list which was supposed to be more appropriated. I think you are thinking here about some other list -- scipy-user is the correct place for this discussion (and I don't remember seeing your mail there). I was pretty sure that I put it there. Unfortunately it had a different name there: Errors in calculation of the Hessian using numdifftools. But I can't find the post by myself at the moment so maybe something went wrong. [clip] 1) Can I make it run/fix it, so that it is also going to work for the SI scaling? Based on a brief look, it seems that uniform scaling will not help you, as you have two very different length scales in the problem, 1/sqrt(m w^2) C If you go to CM+relative coordinates you might be able to scale them separately, but that's fiddly and might not work for larger N. In your problem, things go wrong when the ratio between the length scales approaches 1e-15 which happens to be the machine epsilon. This implies that the algorithm runs into some problems caused by the finite precision of floating-point numbers. What exactly goes wrong and how to fix it, no idea --- I didn't look into how Numdifftools is implemented. Probably you are right. I converted it to natural units and it worked out in the 2 ion case. Increasing the number of ions leads to problems again and I have no idea where those problems come from. 2) How can I be sure that increasing the number of ions or adding a somehow more complicated term to the potential energy is not causing the same problems even in natural units? 3) In which range is numdifftools working properly. That depends on the algorithm and the problem. Personally, I wouldn't trust numerical differentiation if the problem has significantly different length scales, it is important to capture all of them accurately, and it is not clear how to scale them to the same size. Writing ND software that works as expected all the time is probably not possible even in theory. Numerical differentiation is not the only game in the town. I'd look into automatic differentiation (AD) -- there are libraries available for Python also for that, and it is numerically stable. E.g. http://en.wikipedia.org/wiki/Automatic_differentiation#Software has a list of Python libraries. I don't know which of them would be the best ones, though. they all have their pro's and con's. Being (co-)author of some of these tools, my personal and very biased advice is: if you are on Linux, I would go for PYADOLC. it provides bindings to a feature-rich and well-tested C++ library. However, the installation is a little tricker than a setup.py build since you will need to compile ADOL-C and get Boost::Python to work. PYADOLC can also differentiate much more complicated code than your example in a relatively efficient manner. Is there by chance any possibility to make PYADOLC run on a (lame) windows xp engine. If not what else would u recommend (besides switching to Linux, what I am going to do soon). 1) PYADOLC A windows version has been requested several times now. But until recently ADOL-C wasn't available as windows version. So yes, in principle it should be possible to get it to work on windows: You will need 1) boost:python http://www.boost.org/doc/libs/1_44_0/libs/python/doc/index.html 2) ADOL-C sources http://www.coin-or.org/projects/ADOL-C.xml 3) scons http://www.scons.org/ on windows. If you want to give it a try I could help to get it to work. 2) Alternatives: You can also try the ALGOPY which is pure Python and is known to work on Linux and Windows. The installation is also very easy (setup.py build or setup.py install) I have added your problem to the ALGOPY documentation: http://packages.python.org/algopy/examples/hessian_of_potential_function.html The catch is that ALGOPY is not as mature as PYADOLC. However, if you are careful to write clean code it should work reliably. Sebastian For your example the code looks like: - code --- c=classicalHamiltonian() xopt = optimize.fmin(c.potential, c.initialposition(), xtol = 1e-10) import adolc; import numpy # trace the computation adolc.trace_on(0) x = adolc.adouble(c.initialposition()) adolc.independent(x) y = c.potential(x) adolc.dependent(y) adolc.trace_off() hessian = adolc.hessian(0, xopt) eigenvalues = numpy.linalg.eigh(hessian)[0] normal_modes = c.normal_modes(eigenvalues) print 'hessian=\n',hessian print 'eigenvalues=\n',eigenvalues print
[Numpy-discussion] quadratic function
I have used both linear least squares and radial basis functions as a proxy equation, calculated from the results of computer simulations which are calculating some objective function value based on a number of varied input parameters. As an alternative option I want to add a quadratic function so if there are parameters/variables x,y,z then rather than just having a linear function f=a+bx+cy+dz I'll have f=a+bx+cx**2 + dxy + I'd like to have the option not to include all the different second order terms. Where should I start looking? Thanks Brennan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Stacking a 2d array onto a 3d array
On 26 October 2010 21:02, Dewald Pieterse dewald.piete...@gmail.com wrote: I see my slicing was the problem, np.vstack((test[:1], test)) works perfectly. Yes and no. np.newaxis (or None for short) is a very useful tool; you just stick it in an index expression and it adds an axis of length one there. If what you really wanted to do was pull out one plane of the array, then indexing with a number was the right thing to do. If you want to stack that plane back on the array, just add an axis of length one to it. S = A[1,...] A = np.vstack((S[np.newaxis,...],A)) As a side note, np.newaxis is actually just None, but I find the longer name much clearer, so I try to use it in my own code, and I always use it in examples I'm showing other people. I suppose a third option would be import numpy.newaxis as na. Anne On Wed, Oct 27, 2010 at 12:55 AM, josef.p...@gmail.com wrote: On Tue, Oct 26, 2010 at 8:15 PM, Dewald Pieterse dewald.piete...@gmail.com wrote: Starting with: In [93]: test = numpy.array([[[1,1,1],[1,1,1]],[[2,2,2],[2,2,2]],[[3,3,3],[3,3,3]]]) In [94]: test Out[94]: array([[[1, 1, 1], [1, 1, 1]], [[2, 2, 2], [2, 2, 2]], [[3, 3, 3], [3, 3, 3]]]) Slicing the complete first row: In [95]: firstrow = test[0,:,:] In [96]: firstrow Out[96]: array([[1, 1, 1], [1, 1, 1]]) I want to stack firstrow onto test to end up with: ([[[1, 1, 1], [1, 1, 1]], [[1, 1, 1], [1, 1, 1]], [[2, 2, 2], [2, 2, 2]], [[3, 3, 3], [3, 3, 3]]] vstack wants the array dimensions to be the same, is this possible without doing 1 dimensional reshape, the actual data I want to do this on is some what larger. numpy.vstack((firstrow,test)) --- ValueError Traceback (most recent call last) /mnt/home/home/bmeagle/M/programme/analiseerverwerkteprent.py in module() 1 2 3 4 5 /usr/lib64/python2.6/site-packages/numpy/core/shape_base.py in vstack(tup) 212 213 -- 214 return _nx.concatenate(map(atleast_2d,tup),0) 215 216 def hstack(tup): ValueError: arrays must have same number of dimensions What is the correct python way to do this? keep the first dimension or add it back in test = np.array([[[1,1,1],[1,1,1]],[[2,2,2],[2,2,2]],[[3,3,3],[3,3,3]]]) np.vstack((test[:1], test)) array([[[1, 1, 1], [1, 1, 1]], [[1, 1, 1], [1, 1, 1]], [[2, 2, 2], [2, 2, 2]], [[3, 3, 3], [3, 3, 3]]]) np.vstack((test[0][None,...], test)) array([[[1, 1, 1], [1, 1, 1]], [[1, 1, 1], [1, 1, 1]], [[2, 2, 2], [2, 2, 2]], [[3, 3, 3], [3, 3, 3]]]) np.vstack((test[0][None,:,:], test)) array([[[1, 1, 1], [1, 1, 1]], [[1, 1, 1], [1, 1, 1]], [[2, 2, 2], [2, 2, 2]], [[3, 3, 3], [3, 3, 3]]]) I like expand_dims for arbitrary axis, e.g. ax=1 np.concatenate((np.expand_dims(test[:,0,:],ax), test), ax) array([[[1, 1, 1], [1, 1, 1], [1, 1, 1]], [[2, 2, 2], [2, 2, 2], [2, 2, 2]], [[3, 3, 3], [3, 3, 3], [3, 3, 3]]]) Josef -- Dewald Pieterse ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Dewald Pieterse A democracy is nothing more than mob rule, where fifty-one percent of the people take away the rights of the other forty-nine. ~ Thomas Jefferson ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] quadratic function
On Thu, Oct 28, 2010 at 06:38, Brennan Williams brennan.willi...@visualreservoir.com wrote: I have used both linear least squares and radial basis functions as a proxy equation, calculated from the results of computer simulations which are calculating some objective function value based on a number of varied input parameters. As an alternative option I want to add a quadratic function so if there are parameters/variables x,y,z then rather than just having a linear function f=a+bx+cy+dz I'll have f=a+bx+cx**2 + dxy + I'd like to have the option not to include all the different second order terms. A = np.column_stack([ np.ones_like(x), x, y, z, x*x, y*y, z*z, x*y, y*z, x*z, ]) x, res, rank, s = np.linalg.lstsq(A, f) -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] whitespace in git repo
Hi Chuck, On Wed, Oct 27, 2010 at 1:30 PM, Charles R Harris charlesr.har...@gmail.com wrote: I'd like to do something here, but I'm waiting for a consensus and for someone to test things out, maybe with a test repo, to make sure things operate correctly. The documentation isn't that clear... I am getting ready to test on windows and mac. In the process of upgrading git on windows to 1.7.3.1, The following dialog appeared: Configuring line ending conversions How should Git treat line endings in text files? x Checkout Windows-style, commit Unix-style line endings Git will convert LF to CRLF when checking out text files. When committing text files, CRLF will be converted to LF. For cross-platform projects, this is the recommended setting on Windows (core.autocrlf is set to true) o Checkout as-is, commit Unix-style line endings Git will not perform any conversion when checking out text files. When committing text files, CRLF will be converted to LF. For cross-platform projects this is the recommended setting on Unix (core.autocrlf is set to input). o Checkout as-is, commit as-is Git will not perform any conversions when checking out or committing text files. Choosing this option is not recommended for cross-platform projects (core.autocrlf is set to false) This might warrant a very brief mention in the docs, for helping people set up their environment. Its too bad core.autocrlf cannot be set on a per-project basis in a file that gets committed to the repository. As far as I can tell, it can only be set in ~/.gitconfig or numpy/.git/config. Which is why I suggested adding .gitattributes, which can be committed to the repository, and the line * text=auto ensures that EOLs in text files are committed as LF, so we don't have to worry about somebody's config settings having unwanted impact on the repository. And now the bad news: I have not been able to verify that Git respects the autocrlf setting or the eol setting in .gitattributes on my windows 7 computer: I made a new clone and the line endings are LF in the working directory, both on master and in my whitespace-cleanup branch (even the nsi.in file!). (git config -l confirms that core.autocrlf is true.) To check my sanity, I tried writing files using wordpad and notepad to confirm that they are at least using CRLF, and they are *not*, according to both python's open() and grep \r\n. If it were after noon where I live, I would be looking for a bottle of whiskey. But its not, so I'll just beat my head against my desk until I've forgotten about this whole episode. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] whitespace in git repo
On Thu, Oct 28, 2010 at 9:23 AM, Darren Dale dsdal...@gmail.com wrote: Hi Chuck, On Wed, Oct 27, 2010 at 1:30 PM, Charles R Harris charlesr.har...@gmail.com wrote: I'd like to do something here, but I'm waiting for a consensus and for someone to test things out, maybe with a test repo, to make sure things operate correctly. The documentation isn't that clear... I am getting ready to test on windows and mac. In the process of upgrading git on windows to 1.7.3.1, The following dialog appeared: Configuring line ending conversions How should Git treat line endings in text files? x Checkout Windows-style, commit Unix-style line endings Git will convert LF to CRLF when checking out text files. When committing text files, CRLF will be converted to LF. For cross-platform projects, this is the recommended setting on Windows (core.autocrlf is set to true) o Checkout as-is, commit Unix-style line endings Git will not perform any conversion when checking out text files. When committing text files, CRLF will be converted to LF. For cross-platform projects this is the recommended setting on Unix (core.autocrlf is set to input). o Checkout as-is, commit as-is Git will not perform any conversions when checking out or committing text files. Choosing this option is not recommended for cross-platform projects (core.autocrlf is set to false) This might warrant a very brief mention in the docs, for helping people set up their environment. Its too bad core.autocrlf cannot be set on a per-project basis in a file that gets committed to the Yes, this would be good information to have in the notes. repository. As far as I can tell, it can only be set in ~/.gitconfig or numpy/.git/config. Which is why I suggested adding .gitattributes, which can be committed to the repository, and the line * text=auto ensures that EOLs in text files are committed as LF, so we don't have to worry about somebody's config settings having unwanted impact on the repository. Might be worth trying in a numpy/.gitconfig just to see what happens. Documentation isn't always complete. And now the bad news: I have not been able to verify that Git respects the autocrlf setting or the eol setting in .gitattributes on my windows 7 computer: I made a new clone and the line endings are LF in the working directory, both on master and in my whitespace-cleanup branch (even the nsi.in file!). (git config -l confirms that core.autocrlf is true.) To check my sanity, I tried writing files using wordpad and notepad to confirm that they are at least using CRLF, and they are *not*, according to both python's open() and grep \r\n. If it were after noon where I live, I would be looking for a Grepping for CR is tricky. The straightforward way is grep ctrl-v ctrl-m. I'm pretty sure notepad uses CRLF since it doesn't do line breaks for unix text. bottle of whiskey. But its not, so I'll just beat my head against my desk until I've forgotten about this whole episode. _ Oh, don't do that. Someone's got to explore the territory and, as an official old fart, I'm volunteering the younguns. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] whitespace in git repo
On Thu, Oct 28, 2010 at 12:11 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Thu, Oct 28, 2010 at 9:23 AM, Darren Dale dsdal...@gmail.com wrote: Hi Chuck, On Wed, Oct 27, 2010 at 1:30 PM, Charles R Harris charlesr.har...@gmail.com wrote: I'd like to do something here, but I'm waiting for a consensus and for someone to test things out, maybe with a test repo, to make sure things operate correctly. The documentation isn't that clear... I am getting ready to test on windows and mac. In the process of upgrading git on windows to 1.7.3.1, The following dialog appeared: Configuring line ending conversions How should Git treat line endings in text files? x Checkout Windows-style, commit Unix-style line endings Git will convert LF to CRLF when checking out text files. When committing text files, CRLF will be converted to LF. For cross-platform projects, this is the recommended setting on Windows (core.autocrlf is set to true) o Checkout as-is, commit Unix-style line endings Git will not perform any conversion when checking out text files. When committing text files, CRLF will be converted to LF. For cross-platform projects this is the recommended setting on Unix (core.autocrlf is set to input). o Checkout as-is, commit as-is Git will not perform any conversions when checking out or committing text files. Choosing this option is not recommended for cross-platform projects (core.autocrlf is set to false) This might warrant a very brief mention in the docs, for helping people set up their environment. Its too bad core.autocrlf cannot be set on a per-project basis in a file that gets committed to the Yes, this would be good information to have in the notes. repository. As far as I can tell, it can only be set in ~/.gitconfig or numpy/.git/config. Which is why I suggested adding .gitattributes, which can be committed to the repository, and the line * text=auto ensures that EOLs in text files are committed as LF, so we don't have to worry about somebody's config settings having unwanted impact on the repository. Might be worth trying in a numpy/.gitconfig just to see what happens. Documentation isn't always complete. And now the bad news: I have not been able to verify that Git respects the autocrlf setting or the eol setting in .gitattributes on my windows 7 computer: I made a new clone and the line endings are LF in the working directory, both on master and in my whitespace-cleanup branch (even the nsi.in file!). (git config -l confirms that core.autocrlf is true.) To check my sanity, I tried writing files using wordpad and notepad to confirm that they are at least using CRLF, and they are *not*, according to both python's open() and grep \r\n. If it were after noon where I live, I would be looking for a maybe just something obvious: Did you read the files in python as binary 'rb' ? I checked two old git checkouts (with a one and a half year old git version), pymvpa and scikits.talkbox and both have files with \r\n as line endings on my windowsXP. I don't think I did anything special, but maybe I had used a GUI interface for those. Josef Grepping for CR is tricky. The straightforward way is grep ctrl-v ctrl-m. I'm pretty sure notepad uses CRLF since it doesn't do line breaks for unix text. bottle of whiskey. But its not, so I'll just beat my head against my desk until I've forgotten about this whole episode. _ Oh, don't do that. Someone's got to explore the territory and, as an official old fart, I'm volunteering the younguns. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] quadratic function
On 29/10/2010 2:34 a.m., Robert Kern wrote: On Thu, Oct 28, 2010 at 06:38, Brennan Williams brennan.willi...@visualreservoir.com wrote: I have used both linear least squares and radial basis functions as a proxy equation, calculated from the results of computer simulations which are calculating some objective function value based on a number of varied input parameters. As an alternative option I want to add a quadratic function so if there are parameters/variables x,y,z then rather than just having a linear function f=a+bx+cy+dz I'll have f=a+bx+cx**2 + dxy + I'd like to have the option not to include all the different second order terms. A = np.column_stack([ np.ones_like(x), x, y, z, x*x, y*y, z*z, x*y, y*z, x*z, ]) x, res, rank, s = np.linalg.lstsq(A, f) OK, so in other words, you can use linalg.lstsq for whatever higher order terms you want to include or exclude. Very nice. Thanks. On a related topic I also use the Rbf radial basis function as a proxy equation. I have one set of data that it fails to return an Rbf for and I've just realised that in my set of simulations that are used to build the proxy equation I have some duplicate equations. I'm wondering if Rbf doesn't like duplicate points? It obviously doesn't affect linalg.lstsq. Brennan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] quadratic function
On Thu, Oct 28, 2010 at 12:33, Brennan Williams brennan.willi...@visualreservoir.com wrote: On 29/10/2010 2:34 a.m., Robert Kern wrote: On Thu, Oct 28, 2010 at 06:38, Brennan Williams brennan.willi...@visualreservoir.com wrote: I have used both linear least squares and radial basis functions as a proxy equation, calculated from the results of computer simulations which are calculating some objective function value based on a number of varied input parameters. As an alternative option I want to add a quadratic function so if there are parameters/variables x,y,z then rather than just having a linear function f=a+bx+cy+dz I'll have f=a+bx+cx**2 + dxy + I'd like to have the option not to include all the different second order terms. A = np.column_stack([ np.ones_like(x), x, y, z, x*x, y*y, z*z, x*y, y*z, x*z, ]) x, res, rank, s = np.linalg.lstsq(A, f) OK, so in other words, you can use linalg.lstsq for whatever higher order terms you want to include or exclude. Very nice. Thanks. Right. Just as long as the problem is linear in the coefficients, the design matrix can be derived however you like. On a related topic I also use the Rbf radial basis function as a proxy equation. I have one set of data that it fails to return an Rbf for and I've just realised that in my set of simulations that are used to build the proxy equation I have some duplicate equations. I'm wondering if Rbf doesn't like duplicate points? It obviously doesn't affect linalg.lstsq. Rbf doesn't like duplicate points. :-) -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] quadratic function
On 29/10/2010 6:35 a.m., Robert Kern wrote: On Thu, Oct 28, 2010 at 12:33, Brennan Williams brennan.willi...@visualreservoir.com wrote: On 29/10/2010 2:34 a.m., Robert Kern wrote: On Thu, Oct 28, 2010 at 06:38, Brennan Williams brennan.willi...@visualreservoir.comwrote: I have used both linear least squares and radial basis functions as a proxy equation, calculated from the results of computer simulations which are calculating some objective function value based on a number of varied input parameters. As an alternative option I want to add a quadratic function so if there are parameters/variables x,y,z then rather than just having a linear function f=a+bx+cy+dz I'll have f=a+bx+cx**2 + dxy + I'd like to have the option not to include all the different second order terms. A = np.column_stack([ np.ones_like(x), x, y, z, x*x, y*y, z*z, x*y, y*z, x*z, ]) x, res, rank, s = np.linalg.lstsq(A, f) OK, so in other words, you can use linalg.lstsq for whatever higher order terms you want to include or exclude. Very nice. Thanks. Right. Just as long as the problem is linear in the coefficients, the design matrix can be derived however you like. So I could optionally put log terms in if I thought it was linear in log(x) for example? On a related topic I also use the Rbf radial basis function as a proxy equation. I have one set of data that it fails to return an Rbf for and I've just realised that in my set of simulations that are used to build the proxy equation I have some duplicate equations. I'm wondering if Rbf doesn't like duplicate points? It obviously doesn't affect linalg.lstsq. Rbf doesn't like duplicate points. :-) OK, fair enough, I just need to add a bit of housekeeping to remove duplicate simulations. Thanks for the confirmation. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] quadratic function
On Thu, Oct 28, 2010 at 12:47, Brennan Williams brennan.willi...@visualreservoir.com wrote: On 29/10/2010 6:35 a.m., Robert Kern wrote: On Thu, Oct 28, 2010 at 12:33, Brennan Williams brennan.willi...@visualreservoir.com wrote: On 29/10/2010 2:34 a.m., Robert Kern wrote: On Thu, Oct 28, 2010 at 06:38, Brennan Williams brennan.willi...@visualreservoir.com wrote: I have used both linear least squares and radial basis functions as a proxy equation, calculated from the results of computer simulations which are calculating some objective function value based on a number of varied input parameters. As an alternative option I want to add a quadratic function so if there are parameters/variables x,y,z then rather than just having a linear function f=a+bx+cy+dz I'll have f=a+bx+cx**2 + dxy + I'd like to have the option not to include all the different second order terms. A = np.column_stack([ np.ones_like(x), x, y, z, x*x, y*y, z*z, x*y, y*z, x*z, ]) x, res, rank, s = np.linalg.lstsq(A, f) OK, so in other words, you can use linalg.lstsq for whatever higher order terms you want to include or exclude. Very nice. Thanks. Right. Just as long as the problem is linear in the coefficients, the design matrix can be derived however you like. So I could optionally put log terms in if I thought it was linear in log(x) for example? Yup! -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] whitespace in git repo
On Thu, Oct 28, 2010 at 12:23 PM, josef.p...@gmail.com wrote: On Thu, Oct 28, 2010 at 12:11 PM, Charles R Harris On Thu, Oct 28, 2010 at 9:23 AM, Darren Dale dsdal...@gmail.com wrote: And now the bad news: I have not been able to verify that Git respects the autocrlf setting or the eol setting in .gitattributes on my windows 7 computer: I made a new clone and the line endings are LF in the working directory, both on master and in my whitespace-cleanup branch (even the nsi.in file!). (git config -l confirms that core.autocrlf is true.) To check my sanity, I tried writing files using wordpad and notepad to confirm that they are at least using CRLF, and they are *not*, according to both python's open() and grep \r\n. If it were after noon where I live, I would be looking for a maybe just something obvious: Did you read the files in python as binary 'rb' ? No, I did not. You are right, this shows \r\n. Why is it necessary to open them as binary? IIUC (OIDUC), one should use 'rU' to unify line endings. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] whitespace in git repo
On Thu, Oct 28, 2010 at 2:40 PM, Darren Dale dsdal...@gmail.com wrote: On Thu, Oct 28, 2010 at 12:23 PM, josef.p...@gmail.com wrote: On Thu, Oct 28, 2010 at 12:11 PM, Charles R Harris On Thu, Oct 28, 2010 at 9:23 AM, Darren Dale dsdal...@gmail.com wrote: And now the bad news: I have not been able to verify that Git respects the autocrlf setting or the eol setting in .gitattributes on my windows 7 computer: I made a new clone and the line endings are LF in the working directory, both on master and in my whitespace-cleanup branch (even the nsi.in file!). (git config -l confirms that core.autocrlf is true.) To check my sanity, I tried writing files using wordpad and notepad to confirm that they are at least using CRLF, and they are *not*, according to both python's open() and grep \r\n. If it were after noon where I live, I would be looking for a maybe just something obvious: Did you read the files in python as binary 'rb' ? No, I did not. You are right, this shows \r\n. Why is it necessary to open them as binary? IIUC (OIDUC), one should use 'rU' to unify line endings. The python default for open(filename).read() or open(filename, 'r').read() is to standardize line endings to \n. Josef ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] problems with numdifftools
hmm, I have just realized that I forgot to upload the new version to pypi: it is now available on http://pypi.python.org/pypi/algopy On Thu, Oct 28, 2010 at 10:47 AM, Sebastian Walter sebastian.wal...@gmail.com wrote: On Wed, Oct 27, 2010 at 10:50 PM, Nicolai Heitz nicolaihe...@gmx.de wrote: m 27.10.2010 02:02, schrieb Sebastian Walter: On Wed, Oct 27, 2010 at 12:59 AM, Pauli Virtanenp...@iki.fi wrote: Tue, 26 Oct 2010 14:24:39 -0700, Nicolai Heitz wrote: http://mail.scipy.org/mailman/listinfo/scipy-user I contacted them already but they didn't responded so far and I was forwarded to that list which was supposed to be more appropriated. I think you are thinking here about some other list -- scipy-user is the correct place for this discussion (and I don't remember seeing your mail there). I was pretty sure that I put it there. Unfortunately it had a different name there: Errors in calculation of the Hessian using numdifftools. But I can't find the post by myself at the moment so maybe something went wrong. [clip] 1) Can I make it run/fix it, so that it is also going to work for the SI scaling? Based on a brief look, it seems that uniform scaling will not help you, as you have two very different length scales in the problem, 1/sqrt(m w^2) C If you go to CM+relative coordinates you might be able to scale them separately, but that's fiddly and might not work for larger N. In your problem, things go wrong when the ratio between the length scales approaches 1e-15 which happens to be the machine epsilon. This implies that the algorithm runs into some problems caused by the finite precision of floating-point numbers. What exactly goes wrong and how to fix it, no idea --- I didn't look into how Numdifftools is implemented. Probably you are right. I converted it to natural units and it worked out in the 2 ion case. Increasing the number of ions leads to problems again and I have no idea where those problems come from. 2) How can I be sure that increasing the number of ions or adding a somehow more complicated term to the potential energy is not causing the same problems even in natural units? 3) In which range is numdifftools working properly. That depends on the algorithm and the problem. Personally, I wouldn't trust numerical differentiation if the problem has significantly different length scales, it is important to capture all of them accurately, and it is not clear how to scale them to the same size. Writing ND software that works as expected all the time is probably not possible even in theory. Numerical differentiation is not the only game in the town. I'd look into automatic differentiation (AD) -- there are libraries available for Python also for that, and it is numerically stable. E.g. http://en.wikipedia.org/wiki/Automatic_differentiation#Software has a list of Python libraries. I don't know which of them would be the best ones, though. they all have their pro's and con's. Being (co-)author of some of these tools, my personal and very biased advice is: if you are on Linux, I would go for PYADOLC. it provides bindings to a feature-rich and well-tested C++ library. However, the installation is a little tricker than a setup.py build since you will need to compile ADOL-C and get Boost::Python to work. PYADOLC can also differentiate much more complicated code than your example in a relatively efficient manner. Is there by chance any possibility to make PYADOLC run on a (lame) windows xp engine. If not what else would u recommend (besides switching to Linux, what I am going to do soon). 1) PYADOLC A windows version has been requested several times now. But until recently ADOL-C wasn't available as windows version. So yes, in principle it should be possible to get it to work on windows: You will need 1) boost:python http://www.boost.org/doc/libs/1_44_0/libs/python/doc/index.html 2) ADOL-C sources http://www.coin-or.org/projects/ADOL-C.xml 3) scons http://www.scons.org/ on windows. If you want to give it a try I could help to get it to work. 2) Alternatives: You can also try the ALGOPY which is pure Python and is known to work on Linux and Windows. The installation is also very easy (setup.py build or setup.py install) I have added your problem to the ALGOPY documentation: http://packages.python.org/algopy/examples/hessian_of_potential_function.html The catch is that ALGOPY is not as mature as PYADOLC. However, if you are careful to write clean code it should work reliably. Sebastian For your example the code looks like: - code --- c=classicalHamiltonian() xopt = optimize.fmin(c.potential, c.initialposition(), xtol = 1e-10) import adolc; import numpy # trace the computation adolc.trace_on(0) x = adolc.adouble(c.initialposition()) adolc.independent(x) y = c.potential(x)
Re: [Numpy-discussion] whitespace in git repo
On Thu, Oct 28, 2010 at 3:23 PM, josef.p...@gmail.com wrote:
On Thu, Oct 28, 2010 at 2:40 PM, Darren Dale dsdal...@gmail.com wrote:
On Thu, Oct 28, 2010 at 12:23 PM, josef.p...@gmail.com wrote:
On Thu, Oct 28, 2010 at 12:11 PM, Charles R Harris
On Thu, Oct 28, 2010 at 9:23 AM, Darren Dale dsdal...@gmail.com wrote:
And now the bad news: I have not been able to verify that Git respects
the autocrlf setting or the eol setting in .gitattributes on my
windows 7 computer: I made a new clone and the line endings are LF in
the working directory, both on master and in my whitespace-cleanup
branch (even the nsi.in file!). (git config -l confirms that
core.autocrlf is true.) To check my sanity, I tried writing files
using wordpad and notepad to confirm that they are at least using
CRLF, and they are *not*, according to both python's open() and grep
\r\n. If it were after noon where I live, I would be looking for a
maybe just something obvious: Did you read the files in python as binary
'rb' ?
No, I did not. You are right, this shows \r\n. Why is it necessary to
open them as binary? IIUC (OIDUC), one should use 'rU' to unify line
endings.
The python default for open(filename).read() or open(filename,
'r').read() is to standardize line endings to \n.
Although, on a mac:
In [1]:
open('tools/win32build/nsis_scripts/numpy-superinstaller.nsi.in').readlines()[0]
Out[1]: ';\r\n'
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] ndarray __getattr__ to perform __getitem__
I have an ndarray with named dimensions. I find myself writing some
fairly laborious code with lots of square brackets and quotes. It seems
like it wouldn't be such a big deal to overload __getattribute__ so
instead of doing:
r = genfromtxt('results.dat',dtype=[('a','int'), ('b', 'f8'),
('c','int'), ('d', 'a20')])
scatter(r[r['d'] == 'OK']['a'], r[r['d'] == 'OK']['b'])
I could do:
scatter(r[r.d == 'OK'].a, r[r.d == 'OK'].b)
which is really a lot clearer. Is something like this already possible
somehow?
Is there some reason not to map __getattr__ to __getitem__?
class ndarray():
...
__getattr__ = __getitem__
Or something somewhat more sophisticated like:
def __getattr__(self, attr):
if self.hasitem(attr):
return self[attr]
else:
raise AttributeError(attr)
TIA for any comments or thoughts on this.
Ian
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] whitespace in git repo
On 10/28/10 1:25 PM, Darren Dale wrote:
No, I did not. You are right, this shows \r\n. Why is it necessary to
open them as binary? IIUC (OIDUC), one should use 'rU' to unify line
endings.
Although, on a mac:
In [1]:
open('tools/win32build/nsis_scripts/numpy-superinstaller.nsi.in').readlines()[0]
Out[1]: ';\r\n'
that's what the 'U' is for. try:
open('tools/win32build/nsis_scripts/numpy-superinstaller.nsi.in','rU').readlines()[0]
-Chris
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/ORR(206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ndarray __getattr__ to perform __getitem__
On Thu, Oct 28, 2010 at 15:17, Ian Stokes-Rees
ijsto...@hkl.hms.harvard.edu wrote:
I have an ndarray with named dimensions. I find myself writing some
fairly laborious code with lots of square brackets and quotes. It seems
like it wouldn't be such a big deal to overload __getattribute__ so
instead of doing:
r = genfromtxt('results.dat',dtype=[('a','int'), ('b', 'f8'),
('c','int'), ('d', 'a20')])
scatter(r[r['d'] == 'OK']['a'], r[r['d'] == 'OK']['b'])
I could do:
scatter(r[r.d == 'OK'].a, r[r.d == 'OK'].b)
which is really a lot clearer. Is something like this already possible
somehow?
See recarray which uses __getattribute__.
Is there some reason not to map __getattr__ to __getitem__?
Using __getattribute__ tends to slow down almost all operations on the
array substantially. Perhaps __getattr__ would work better, but all of
the methods and attributes would mask the fields. If you can find a
better solution that doesn't have such an impact on normal
performance, we'd be happy to hear it.
--
Robert Kern
I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth.
-- Umberto Eco
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ndarray __getattr__ to perform __getitem__
On 10/28/10 5:29 PM, Robert Kern wrote:
On Thu, Oct 28, 2010 at 15:17, Ian Stokes-Rees
ijsto...@hkl.hms.harvard.edu wrote:
I have an ndarray with named dimensions. I find myself writing some
fairly laborious code with lots of square brackets and quotes. It seems
like it wouldn't be such a big deal to overload __getattribute__ so
instead of doing:
r = genfromtxt('results.dat',dtype=[('a','int'), ('b', 'f8'),
('c','int'), ('d', 'a20')])
scatter(r[r['d'] == 'OK']['a'], r[r['d'] == 'OK']['b'])
I could do:
scatter(r[r.d == 'OK'].a, r[r.d == 'OK'].b)
which is really a lot clearer. Is something like this already possible
somehow?
See recarray which uses __getattribute__.
Thanks -- I'll look into it.
Is there some reason not to map __getattr__ to __getitem__?
Using __getattribute__ tends to slow down almost all operations on the
array substantially. Perhaps __getattr__ would work better, but all of
the methods and attributes would mask the fields. If you can find a
better solution that doesn't have such an impact on normal
performance, we'd be happy to hear it.
But wouldn't the performance hit only come when I use it in this way?
__getattr__ is only called if the named attribute is *not* found (I
guess it falls off the end of the case statement, or is the result of
the attribute hash table miss).
So the proviso is this shortcut only works if the field names are
distinct from any methods or attributes on the ndarray object (or its
sub-classes).
You've gotta admit that the readability of the code goes up *a lot* with
the alternative I'm proposing.
Ian
--
Ian Stokes-Rees, PhDW: http://portal.nebiogrid.org
ijsto...@hkl.hms.harvard.eduT: +1.617.432.5608 x75
NEBioGrid, Harvard Medical School C: +1.617.331.5993
attachment: ijstokes.vcf___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ndarray __getattr__ to perform __getitem__
On Thu, Oct 28, 2010 at 16:37, Ian Stokes-Rees
ijsto...@hkl.hms.harvard.edu wrote:
On 10/28/10 5:29 PM, Robert Kern wrote:
On Thu, Oct 28, 2010 at 15:17, Ian Stokes-Rees
ijsto...@hkl.hms.harvard.edu wrote:
I have an ndarray with named dimensions. I find myself writing some
fairly laborious code with lots of square brackets and quotes. It seems
like it wouldn't be such a big deal to overload __getattribute__ so
instead of doing:
r = genfromtxt('results.dat',dtype=[('a','int'), ('b', 'f8'),
('c','int'), ('d', 'a20')])
scatter(r[r['d'] == 'OK']['a'], r[r['d'] == 'OK']['b'])
I could do:
scatter(r[r.d == 'OK'].a, r[r.d == 'OK'].b)
which is really a lot clearer. Is something like this already possible
somehow?
See recarray which uses __getattribute__.
Thanks -- I'll look into it.
Is there some reason not to map __getattr__ to __getitem__?
Using __getattribute__ tends to slow down almost all operations on the
array substantially. Perhaps __getattr__ would work better, but all of
the methods and attributes would mask the fields. If you can find a
better solution that doesn't have such an impact on normal
performance, we'd be happy to hear it.
But wouldn't the performance hit only come when I use it in this way?
__getattr__ is only called if the named attribute is *not* found (I
guess it falls off the end of the case statement, or is the result of
the attribute hash table miss).
That's why I said that __getattr__ would perhaps work better.
So the proviso is this shortcut only works if the field names are
distinct from any methods or attributes on the ndarray object (or its
sub-classes).
Right.
--
Robert Kern
I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth.
-- Umberto Eco
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Inconsistency with __index__() for rank-1 arrays?
The __index__ method returns an integer from an array. The current behavior follows the idea of return an integer if there is 1-element in the array Your suggestion is to only return an integer if it is a rank-0 array, otherwise raise an error. This could potentially be changed in NumPy 2.0.I'm +0 on the suggestion. -Travis On Oct 27, 2010, at 9:34 AM, Francesc Alted wrote: Hi, I find this a bit misleading: a = np.arange(10) a[np.array(0)] 0 a[np.array([0])] array([0]) a[[0]] array([0]) But, for regular python lists we have: l = a.tolist() l[np.array(0)] 0 l[np.array([0])] 0 i.e. indexing with a rank-0 array and a rank-1 array with one single element return the same result, which I find inconsistent with the expected behaviour for this case, i.e.: l[[0]] --- TypeError Traceback (most recent call last) /tmp/tables-2.2/ipython console in module() TypeError: list indices must be integers, not list The ultimate reason for this behaviour is this: np.array(0).__index__() 0 np.array([0]).__index__() 0 But I wonder why NumPy needs the latter behaviour, instead of the more logical: np.array([0]).__index__() --- TypeError Traceback (most recent call last) /tmp/tables-2.2/ipython console in module() TypeError: only rank-0 integer arrays can be converted to an index This inconsistency has indeed introduced a bug in my application and for solving this I'd need something like: def is_idx(index): Check if an object can work as an index or not. if hasattr(index, __index__): # Only works on Python 2.5 on if (hasattr(index, shape) and index.shape == (1,)): return False try: # (as per PEP 357) idx = index.__index__() return True except TypeError: return False return False i.e. for determining if an object can be an index or not, I need to explicitly check for a shape different from (1,), which is unnecessarily complicated. So I find the current behaviour prone to introduce errors in apps and I'm wondering why exactly np.array([1]) should work as an index at all. It would not be better if that would raise a ``TypeError``? Thanks, -- Francesc Alted ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion --- Travis Oliphant Enthought, Inc. oliph...@enthought.com 1-512-536-1057 http://www.enthought.com ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
