Re: [Numpy-discussion] SVD error in Numpy. NumPy Update reversed?
Travis, Does that mean it's not worth starting a ticket? Sounds like nothing can be done, *except* to put this in the documentation and the FAQ. It has bitten several people. --- Travis E. Oliphant [EMAIL PROTECTED] wrote: Stéfan van der Walt wrote: Lou Pecora wrote: Thanks, Matthieu, that's a good step. But when the SVD function throws an exception is it clear that the user can redefine niter and recompile? Otherwise, the fix remains well hidden. Most user will be left puzzled. I think a comment in the raise statement would be good. Just point to the solution or where the user could find it. That's a valid concern. We could maybe pass down the iteration limit as a keyword? This won't work without significant re-design. This limit is in the low-level code which is an f2c'd version of some BLAS which is NumPy's default SVD implementation if it can't find a vendor BLAS. -Travis O. -- Lou Pecora, my views are my own. Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] SVD error in Numpy. NumPy Update reversed?
Hi all,
I looked at line 21902 of dlapack_lite.c, it is,
for (niter = iter; niter = 20; ++niter) {
Indeed the upper limit for iterations in the
linalg.svd code is set for 20. For now I will go with
my method (on earlier post) of squaring the matrix and
then doing svd when the original try on the original
matrix throws the linalg.linalg.LinAlgError. I do not
claim that this is a cure-all. But it seems to work
fast and avoids the original code from thrashing
around in a long iteration.
I would suggest this be made explicit in the NumPy
documentation and then the user be given the option to
reset the limit on the number of iterations.
Well, it certainly shouldn't be hardwired in as 20. At minimum it
should be a #define, and ideally it should be passed in with the
function call, but I don't know if the interface allows that.
I just wanted to mention that this particular issue has bitten me in
the past too. It would be nice to be able to have a bit more control
over the SVD iteration limit either at compile-time, or run-time.
Zach
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] SVD error in Numpy. NumPy Update reversed?
It was added as a compile-time #define on the SVN some days ago ;)
Matthieu
2008/3/24, Zachary Pincus [EMAIL PROTECTED]:
Hi all,
I looked at line 21902 of dlapack_lite.c, it is,
for (niter = iter; niter = 20; ++niter) {
Indeed the upper limit for iterations in the
linalg.svd code is set for 20. For now I will go with
my method (on earlier post) of squaring the matrix and
then doing svd when the original try on the original
matrix throws the linalg.linalg.LinAlgError. I do not
claim that this is a cure-all. But it seems to work
fast and avoids the original code from thrashing
around in a long iteration.
I would suggest this be made explicit in the NumPy
documentation and then the user be given the option to
reset the limit on the number of iterations.
Well, it certainly shouldn't be hardwired in as 20. At minimum it
should be a #define, and ideally it should be passed in with the
function call, but I don't know if the interface allows that.
I just wanted to mention that this particular issue has bitten me in
the past too. It would be nice to be able to have a bit more control
over the SVD iteration limit either at compile-time, or run-time.
Zach
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
--
French PhD student
Website : http://matthieu-brucher.developpez.com/
Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn : http://www.linkedin.com/in/matthieubrucher
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] SVD error in Numpy. NumPy Update reversed?
On Mon, Mar 24, 2008 at 6:04 PM, Lou Pecora [EMAIL PROTECTED] wrote: --- Matthieu Brucher [EMAIL PROTECTED] wrote: It was added as a compile-time #define on the SVN some days ago ;) Matthieu Thanks, Matthieu, that's a good step. But when the SVD function throws an exception is it clear that the user can redefine niter and recompile? Otherwise, the fix remains well hidden. Most user will be left puzzled. I think a comment in the raise statement would be good. Just point to the solution or where the user could find it. That's a valid concern. We could maybe pass down the iteration limit as a keyword? Lou, would you create a ticket for this as a feature enhancement, and refer to http://projects.scipy.org/scipy/numpy/ticket/706 please? Thank you. Stéfan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] SVD error in Numpy. NumPy Update reversed?
Stéfan van der Walt wrote: On Mon, Mar 24, 2008 at 6:04 PM, Lou Pecora [EMAIL PROTECTED] wrote: --- Matthieu Brucher [EMAIL PROTECTED] wrote: It was added as a compile-time #define on the SVN some days ago ;) Matthieu Thanks, Matthieu, that's a good step. But when the SVD function throws an exception is it clear that the user can redefine niter and recompile? Otherwise, the fix remains well hidden. Most user will be left puzzled. I think a comment in the raise statement would be good. Just point to the solution or where the user could find it. That's a valid concern. We could maybe pass down the iteration limit as a keyword? This won't work without significant re-design. This limit is in the low-level code which is an f2c'd version of some BLAS which is NumPy's default SVD implementation if it can't find a vendor BLAS. -Travis O. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] SVD error in Numpy. NumPy Update reversed?
I recently had a personal email reply from Damian
Menscher who originally found the error in 2002. He
states:
--
I explained the solution in a followup to my own post:
http://mail.python.org/pipermail/python-list/2002-August/161395.html
-- in short, find the dlasd4_ routine (for the current
1.0.4 version
it's at numpy/linalg/dlapack_lite.c:21902) and change
the max
iteration count from 20 to 100 or higher.
The basic problem was that they use an iterative
method to converge on
the solution, and they had a cutoff of the max number
of iterations
before giving up (to guard against an infinite loop or
cases where an
unlucky matrix would require an excessive number of
iterations and
therefore CPU). The fix I used was simply to increase
the max
iteration count (from 20 to 100 -- 50 was enough to
solve my problem
but I went for overkill just to be sure I wouldn't see
it again). It
*may* be reasonable to just leave this as an infinite
loop, or to
increase the count to 1000 or higher. A lot depends
on your preferred
failure mode:
- count too low - low cpu usage, but SVD did not
converge errors
somewhat common
- very high count - some matrices will result in
high cpu usage,
non-convergence still possible
- infinite loop - it will always converge, but may
take forever
NumPy was supposedly updated also (from 20 to 100, but
you may want to
go higher) in bug 601052. They said the fix made it
into CVS, but
apparently it got lost or reverted when they did a
release (the oldest
release I can find is v1.0 from 2006 and has it set to
20). I just
filed another bug (copy/paste of the previous one) in
hopes they'll
fix it for real this time:
http://scipy.org/scipy/numpy/ticket/706
Damian
I looked at line 21902 of dlapack_lite.c, it is,
for (niter = iter; niter = 20; ++niter) {
Indeed the upper limit for iterations in the
linalg.svd code is set for 20. For now I will go with
my method (on earlier post) of squaring the matrix and
then doing svd when the original try on the original
matrix throws the linalg.linalg.LinAlgError. I do not
claim that this is a cure-all. But it seems to work
fast and avoids the original code from thrashing
around in a long iteration.
I would suggest this be made explicit in the NumPy
documentation and then the user be given the option to
reset the limit on the number of iterations.
-- Lou Pecora, my views are my own.
Never miss a thing. Make Yahoo your home page.
http://www.yahoo.com/r/hs
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] SVD error in Numpy. NumPy Update reversed?
On Wed, Mar 19, 2008 at 11:30 AM, Lou Pecora [EMAIL PROTECTED] wrote:
I recently had a personal email reply from Damian
Menscher who originally found the error in 2002. He
states:
--
I explained the solution in a followup to my own post:
http://mail.python.org/pipermail/python-list/2002-August/161395.html
-- in short, find the dlasd4_ routine (for the current
1.0.4 version
it's at numpy/linalg/dlapack_lite.c:21902) and change
the max
iteration count from 20 to 100 or higher.
The basic problem was that they use an iterative
method to converge on
the solution, and they had a cutoff of the max number
of iterations
before giving up (to guard against an infinite loop or
cases where an
unlucky matrix would require an excessive number of
iterations and
therefore CPU). The fix I used was simply to increase
the max
iteration count (from 20 to 100 -- 50 was enough to
solve my problem
but I went for overkill just to be sure I wouldn't see
it again). It
*may* be reasonable to just leave this as an infinite
loop, or to
increase the count to 1000 or higher. A lot depends
on your preferred
failure mode:
- count too low - low cpu usage, but SVD did not
converge errors
somewhat common
- very high count - some matrices will result in
high cpu usage,
non-convergence still possible
- infinite loop - it will always converge, but may
take forever
NumPy was supposedly updated also (from 20 to 100, but
you may want to
go higher) in bug 601052. They said the fix made it
into CVS, but
apparently it got lost or reverted when they did a
release (the oldest
release I can find is v1.0 from 2006 and has it set to
20). I just
filed another bug (copy/paste of the previous one) in
hopes they'll
fix it for real this time:
http://scipy.org/scipy/numpy/ticket/706
Damian
I looked at line 21902 of dlapack_lite.c, it is,
for (niter = iter; niter = 20; ++niter) {
Indeed the upper limit for iterations in the
linalg.svd code is set for 20. For now I will go with
my method (on earlier post) of squaring the matrix and
then doing svd when the original try on the original
matrix throws the linalg.linalg.LinAlgError. I do not
claim that this is a cure-all. But it seems to work
fast and avoids the original code from thrashing
around in a long iteration.
I would suggest this be made explicit in the NumPy
documentation and then the user be given the option to
reset the limit on the number of iterations.
Well, it certainly shouldn't be hardwired in as 20. At minimum it should
be a #define, and ideally it should be passed in with the function call, but
I don't know if the interface allows that.
Chuck
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
