Re: [Numpy-discussion] MKL and OpenBLAS
Concerning numpy-MKL licence I refer to this question on the Intel Forum: http://software.intel.com/en-us/forums/topic/328344 Question on Redistribution related to numpy/scipy In the case of numpy-MKL the MKL binaries are statically linked to the pyd-files. Given the usefulness, performance and robustness of the MKL-based binaries a definite answer to this question would be desirable. Say: Can I use and re-redistribute a product with a precompiled numpy-MKL in a commercial enviroment without the need to by a Intel licence? Carl 2014-01-26 Dinesh Vadhia dineshbvad...@hotmail.com: This conversation gets discussed often with Numpy developers but since the requirement for optimized Blas is pretty common these days, how about distributing Numpy with OpenBlas by default? People who don't want optimized BLAS or OpenBLAS can then edit the site.cfg file to add/remove. I can never remember if Numpy comes with Atlas by default but either way, if using MKL is not feasible because of its licensing issues then Numpy has to be re-compiled with OpenBLAS (for example). Why not make it easier for developers to use Numpy with an in-built optimized Blas. Btw, just in case some folks from Intel are listening: how about releasing MKL binaries for all platforms for developers to do with it what they want ie. free. You know it makes sense! ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] MKL and OpenBLAS
Carl Kleffner cmkleff...@gmail.com wrote: In the case of numpy-MKL the MKL binaries are statically linked to the pyd-files. Given the usefulness, performance and robustness of the MKL-based binaries a definite answer to this question would be desirable. Say: Can I use and re-redistribute a product with a precompiled numpy-MKL in a commercial enviroment without the need to by a Intel licence? I don't see why this is relevant. If you make commercial software chances are you can afford a commercial license for Intel's C++ or Fortran compiler. If you don't, you don't charge your customers enough. Also consider this: Can software packed and linked with MKL be sold in a store? That is also redistribution, and the store is likely not to own an MKL license. There is thus only one reasonable answer. And besides, if you make commercial software, chances are your solicitor verifies the license rights. If you don't consult a solicitor, that would be at your own risk. Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Indexing changes in 1.9
Sebastian has done a lot of work to refactor/rationalize numpy indexing. The changes are extensive enough that it would be good to have more public review, so here is the release note. The NumPy indexing has seen a complete rewrite in this version. This makes most advanced integer indexing operations much faster and should have no other implications. However some subtle changes and deprecations were introduced in advanced indexing operations: * Boolean indexing into scalar arrays will always return a new 1-d array. This means that ``array(1)[array(True)]`` gives ``array([1])`` and not the original array. * Advanced indexing into one dimensional arrays used to have (undocumented) special handling regarding repeating the value array in assignments when the shape of the value array was too small or did not match. Code using this will raise an error. For compatibility you can use ``arr.flat[index] = values``, which uses the old code branch. * The iteration order over advanced indexes used to be always C-order. In NumPy 1.9. the iteration order adapts to the inputs and is not guaranteed (with the exception of a *single* advanced index which is never reversed for compatibility reasons). This means that the result is undefined if multiple values are assigned to the same element. An example for this is ``arr[[0, 0], [1, 1]] = [1, 2]``, which may set ``arr[0, 1]`` to either 1 or 2. * Equivalent to the iteration order, the memory layout of the advanced indexing result is adapted for faster indexing and cannot be predicted. * All indexing operations return a view or a copy. No indexing operation will return the original array object. * In the future Boolean array-likes (such as lists of python bools) will always be treated as Boolean indexes and Boolean scalars (including python `True`) will be a legal *boolean* index. At this time, this is already the case for scalar arrays to allow the general ``positive = a[a 0]`` to work when ``a`` is zero dimensional. * In NumPy 1.8 it was possible to use `array(True)` and `array(False)` equivalent to 1 and 0 if the result of the operation was a scalar. This will raise an error in NumPy 1.9 and, as noted above, treated as a boolean index in the future. * All non-integer array-likes are deprecated, object arrays of custom integer like objects may have to be cast explicitly. * The error reporting for advanced indexing is more informative, however the error type has changed in some cases. (Broadcasting errors of indexing arrays are reported as `IndexError`) * Indexing with more then one ellipsis (`...`) is deprecated. Thoughts? Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Indexing changes in 1.9
On Sun, Feb 2, 2014 at 10:06 AM, Charles R Harris charlesr.har...@gmail.com wrote: Sebastian has done a lot of work to refactor/rationalize numpy indexing. The changes are extensive enough that it would be good to have more public review, so here is the release note. The NumPy indexing has seen a complete rewrite in this version. This makes most advanced integer indexing operations much faster and should have no other implications. However some subtle changes and deprecations were introduced in advanced indexing operations: * Boolean indexing into scalar arrays will always return a new 1-d array. This means that ``array(1)[array(True)]`` gives ``array([1])`` and not the original array. * Advanced indexing into one dimensional arrays used to have (undocumented) special handling regarding repeating the value array in assignments when the shape of the value array was too small or did not match. Code using this will raise an error. For compatibility you can use ``arr.flat[index] = values``, which uses the old code branch. * The iteration order over advanced indexes used to be always C-order. In NumPy 1.9. the iteration order adapts to the inputs and is not guaranteed (with the exception of a *single* advanced index which is never reversed for compatibility reasons). This means that the result is undefined if multiple values are assigned to the same element. An example for this is ``arr[[0, 0], [1, 1]] = [1, 2]``, which may set ``arr[0, 1]`` to either 1 or 2. * Equivalent to the iteration order, the memory layout of the advanced indexing result is adapted for faster indexing and cannot be predicted. * All indexing operations return a view or a copy. No indexing operation will return the original array object. * In the future Boolean array-likes (such as lists of python bools) will always be treated as Boolean indexes and Boolean scalars (including python `True`) will be a legal *boolean* index. At this time, this is already the case for scalar arrays to allow the general ``positive = a[a 0]`` to work when ``a`` is zero dimensional. * In NumPy 1.8 it was possible to use `array(True)` and `array(False)` equivalent to 1 and 0 if the result of the operation was a scalar. This will raise an error in NumPy 1.9 and, as noted above, treated as a boolean index in the future. * All non-integer array-likes are deprecated, object arrays of custom integer like objects may have to be cast explicitly. * The error reporting for advanced indexing is more informative, however the error type has changed in some cases. (Broadcasting errors of indexing arrays are reported as `IndexError`) * Indexing with more then one ellipsis (`...`) is deprecated. Thoughts? The PR is #3798 https://github.com/numpy/numpy/pull/3798 if you want to test it. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] MKL and OpenBLAS
If you work in an academia world it can be relevant once third parties are involved in a bigger project. A situation may be reached, where you just have to prove the license situation of all of your software components. Numpy and scipy is 'selled' as BSD or MIT based foundation for scientific software without components with copyleft licences. For the MKL part a clear statement would be welcome. Otherwise the usage of MKL based binaries has to be avoided in such situations, even if you don't sell something. 2014-02-02 Sturla Molden sturla.mol...@gmail.com: Carl Kleffner cmkleff...@gmail.com wrote: In the case of numpy-MKL the MKL binaries are statically linked to the pyd-files. Given the usefulness, performance and robustness of the MKL-based binaries a definite answer to this question would be desirable. Say: Can I use and re-redistribute a product with a precompiled numpy-MKL in a commercial enviroment without the need to by a Intel licence? I don't see why this is relevant. If you make commercial software chances are you can afford a commercial license for Intel's C++ or Fortran compiler. If you don't, you don't charge your customers enough. Also consider this: Can software packed and linked with MKL be sold in a store? That is also redistribution, and the store is likely not to own an MKL license. There is thus only one reasonable answer. And besides, if you make commercial software, chances are your solicitor verifies the license rights. If you don't consult a solicitor, that would be at your own risk. Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Indexing changes in 1.9
This sounds like a great and welcome work and improvements. Does it make sense to also do something about the behavior of advanced indexing when slices are interleaved between lists and integers. I know that jay borque has some preliminary work to fix this. There are a some straightforward fixes -- like doing iterative application of indexing in those cases which would be more sensical in the cases where current code gets tripped up. Travis On Feb 2, 2014 11:07 AM, Charles R Harris charlesr.har...@gmail.com wrote: Sebastian has done a lot of work to refactor/rationalize numpy indexing. The changes are extensive enough that it would be good to have more public review, so here is the release note. The NumPy indexing has seen a complete rewrite in this version. This makes most advanced integer indexing operations much faster and should have no other implications. However some subtle changes and deprecations were introduced in advanced indexing operations: * Boolean indexing into scalar arrays will always return a new 1-d array. This means that ``array(1)[array(True)]`` gives ``array([1])`` and not the original array. * Advanced indexing into one dimensional arrays used to have (undocumented) special handling regarding repeating the value array in assignments when the shape of the value array was too small or did not match. Code using this will raise an error. For compatibility you can use ``arr.flat[index] = values``, which uses the old code branch. * The iteration order over advanced indexes used to be always C-order. In NumPy 1.9. the iteration order adapts to the inputs and is not guaranteed (with the exception of a *single* advanced index which is never reversed for compatibility reasons). This means that the result is undefined if multiple values are assigned to the same element. An example for this is ``arr[[0, 0], [1, 1]] = [1, 2]``, which may set ``arr[0, 1]`` to either 1 or 2. * Equivalent to the iteration order, the memory layout of the advanced indexing result is adapted for faster indexing and cannot be predicted. * All indexing operations return a view or a copy. No indexing operation will return the original array object. * In the future Boolean array-likes (such as lists of python bools) will always be treated as Boolean indexes and Boolean scalars (including python `True`) will be a legal *boolean* index. At this time, this is already the case for scalar arrays to allow the general ``positive = a[a 0]`` to work when ``a`` is zero dimensional. * In NumPy 1.8 it was possible to use `array(True)` and `array(False)` equivalent to 1 and 0 if the result of the operation was a scalar. This will raise an error in NumPy 1.9 and, as noted above, treated as a boolean index in the future. * All non-integer array-likes are deprecated, object arrays of custom integer like objects may have to be cast explicitly. * The error reporting for advanced indexing is more informative, however the error type has changed in some cases. (Broadcasting errors of indexing arrays are reported as `IndexError`) * Indexing with more then one ellipsis (`...`) is deprecated. Thoughts? Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Fast decrementation of indices
Hi, I have run into a potential 'for loop' bottleneck. Let me outline: The following array describes bonds (connections) in a benzene molecule b = [[0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4, 5, 5, 5, 6, 7, 8, 9, 10, 11], [5, 6, 1, 0, 2, 7, 3, 8, 1, 4, 9, 2, 10, 5, 3, 4, 11, 0, 0, 1, 2, 3, 4, 5]] ie. bond 0 connects atoms 0 and 5, bond 1 connects atom 0 and 6, etc. In practical examples, the list can be much larger (N 100.000 connections. Suppose atoms with indices a = [1,2,3,7,8] are deleted, then all bonds connecting those atoms must be deleted. I achieve this doing i_0 = numpy.in1d(b[0], a) i_1 = numpy.in1d(b[1], a) b_i = numpy.where(i_0 | i_1)[0] b = b[:,~(i_0 | i_1)] If you find this approach lacking, feel free to comment. This results in the following updated bond list b = [[0, 0, 4, 4, 5, 5, 5, 6, 10, 11] [5, 6, 10, 5, 4, 11, 0, 0, 4, 5]] This list is however not correct: Since atoms [1,2,3,7,8] have been deleted, the remaining atoms with indices larger than the deleted atoms must be decremented. I do this as follows: for i in a: b = numpy.where(b i, bonds-1, bonds) (*) yielding the correct result b = [[0, 0, 1, 1, 2, 2, 2, 3, 5, 6], [2, 3, 5, 2, 1, 6, 0, 0, 1, 2]] The Python for loop in (*) may easily contain 50.000 iteration. Is there a smart way to utilize numpy functionality to avoid this? Thanks and best regards, Mads -- +-+ | Mads Ipsen | +--+--+ | Gåsebæksvej 7, 4. tv | phone: +45-29716388 | | DK-2500 Valby| email: mads.ip...@gmail.com | | Denmark | map : www.tinyurl.com/ns52fpa | +--+--+ ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Fast decrementation of indices
On Sun, Feb 2, 2014 at 2:58 PM, Mads Ipsen mads.ip...@gmail.com wrote: Since atoms [1,2,3,7,8] have been deleted, the remaining atoms with indices larger than the deleted atoms must be decremented. Let x array([[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11]]) and i = [1, 0, 2] Create a shape of x matrix with 1's at (k, i[k]) and zeros elsewhere b = zeros_like(x) b.put(i + arange(3)*4 + 1, 1) # there must be a simpler way x - b.cumsum(1) array([[ 0, 1, 1, 2], [ 4, 4, 5, 6], [ 8, 9, 10, 10]]) seems to be the result you want. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Fast decrementation of indices
Cannot test right now, but np.unique(b, return_inverse=True)[1].reshape(2, -1) should do what you are after, I think. On Feb 2, 2014 11:58 AM, Mads Ipsen mads.ip...@gmail.com wrote: Hi, I have run into a potential 'for loop' bottleneck. Let me outline: The following array describes bonds (connections) in a benzene molecule b = [[0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4, 5, 5, 5, 6, 7, 8, 9, 10, 11], [5, 6, 1, 0, 2, 7, 3, 8, 1, 4, 9, 2, 10, 5, 3, 4, 11, 0, 0, 1, 2, 3, 4, 5]] ie. bond 0 connects atoms 0 and 5, bond 1 connects atom 0 and 6, etc. In practical examples, the list can be much larger (N 100.000 connections. Suppose atoms with indices a = [1,2,3,7,8] are deleted, then all bonds connecting those atoms must be deleted. I achieve this doing i_0 = numpy.in1d(b[0], a) i_1 = numpy.in1d(b[1], a) b_i = numpy.where(i_0 | i_1)[0] b = b[:,~(i_0 | i_1)] If you find this approach lacking, feel free to comment. This results in the following updated bond list b = [[0, 0, 4, 4, 5, 5, 5, 6, 10, 11] [5, 6, 10, 5, 4, 11, 0, 0, 4, 5]] This list is however not correct: Since atoms [1,2,3,7,8] have been deleted, the remaining atoms with indices larger than the deleted atoms must be decremented. I do this as follows: for i in a: b = numpy.where(b i, bonds-1, bonds) (*) yielding the correct result b = [[0, 0, 1, 1, 2, 2, 2, 3, 5, 6], [2, 3, 5, 2, 1, 6, 0, 0, 1, 2]] The Python for loop in (*) may easily contain 50.000 iteration. Is there a smart way to utilize numpy functionality to avoid this? Thanks and best regards, Mads -- +-+ | Mads Ipsen | +--+--+ | Gåsebæksvej 7, 4. tv | phone: +45-29716388 | | DK-2500 Valby| email: mads.ip...@gmail.com | | Denmark | map : www.tinyurl.com/ns52fpa | +--+--+ ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] MKL and OpenBLAS
Carl Kleffner cmkleff...@gmail.com wrote: If you work in an academia world it can be relevant once third parties are involved in a bigger project. A situation may be reached, where you just have to prove the license situation of all of your software components. If you involve third parties outside academia, you need a commercial license. Binaries with academic license is for academic use only. Personally I pay Enthought Inc. to provide me with NumPy, and then it's their responsibility to work out the license details in their software stack. I am licensing my Python software from Enthought Inc., and I cannot go through and verify every single one of their licenses. If asked I will just refer to the license that comes with my Canopy subscription, and that will be the end of it. Numpy and scipy is 'selled' as BSD or MIT based foundation for scientific software without components with copyleft licences. For the MKL part a clear statement would be welcome. Otherwise the usage of MKL based binaries has to be avoided in such situations, even if you don't sell something. That is utter nonsence. MKL is not different from any other commercial software. With this way of backwards thinking, no commercial software could ever be used. You could e.g. never use Windows, because you might be asked to prove Microsoft's license for third-party libraries used by their operating system. That is just bullshit. I might be asked to prove my license with Microsoft, but it's Microsoft's responsibility to work out the internal license details for the software they sell. Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] MKL and OpenBLAS
Sturla Molden sturla.mol...@gmail.com wrote: Yes, it seems to be a GNU problem: http://bisqwit.iki.fi/story/howto/openmp/#OpenmpAndFork This Howto also claims Intel compilers is not affected. It seems another patch has been proposed to the libgomp team today: http://gcc.gnu.org/bugzilla/show_bug.cgi?id=60035 Unfortunaltely I don't think the libgomp team really appreciates the absurdity that a call to BLAS function like DGEMM can interfere with a subsequent call to fork. Mixing multithreading and forks is one thing, but the raison d'etre for OpenMP is to simplify vectorization of serial code. Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
