Hello community, here is the log from the commit of package python-numexpr for openSUSE:Factory checked in at 2018-08-07 09:44:18 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/python-numexpr (Old) and /work/SRC/openSUSE:Factory/.python-numexpr.new (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "python-numexpr" Tue Aug 7 09:44:18 2018 rev:8 rq:627751 version:2.6.6 Changes: -------- --- /work/SRC/openSUSE:Factory/python-numexpr/python-numexpr.changes 2017-10-05 11:59:02.282465433 +0200 +++ /work/SRC/openSUSE:Factory/.python-numexpr.new/python-numexpr.changes 2018-08-07 09:44:25.625450852 +0200 @@ -1,0 +2,13 @@ +Mon Aug 6 20:04:09 UTC 2018 - [email protected] + +- Update to 2.6.6 + * Thanks to Mark Dickinson for a fix to the thread barrier that occassionally suffered from spurious wakeups on MacOSX. +- Update to 2.6.5 + * The maximum thread count can now be set at import-time by setting the environment variable ‘NUMEXPR_MAX_THREADS’. The default number of max threads was lowered from 4096 (which was deemed excessive) to 64. + * A number of imports were removed (pkg_resources) or made lazy (cpuinfo) in order to speed load-times for downstream packages (such as pandas, sympy, and tables). Import time has dropped from about 330 ms to 90 ms. Thanks to Jason Sachs for pointing out the source of the slow-down. + * Thanks to Alvaro Lopez Ortega for updates to benchmarks to be compatible with Python 3. + * Travis and AppVeyor now fail if the test module fails or errors. + * Thanks to Mahdi Ben Jelloul for a patch that removed a bug where constants in where calls would raise a ValueError. + * Fixed a bug whereby all-constant power operations would lead to infinite recursion. + +------------------------------------------------------------------- Old: ---- numexpr-2.6.4.tar.gz New: ---- numexpr-2.6.6.tar.gz ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ python-numexpr.spec ++++++ --- /var/tmp/diff_new_pack.ehqmSU/_old 2018-08-07 09:44:26.177451812 +0200 +++ /var/tmp/diff_new_pack.ehqmSU/_new 2018-08-07 09:44:26.181451819 +0200 @@ -1,7 +1,7 @@ # # spec file for package python-numexpr # -# Copyright (c) 2017 SUSE LINUX GmbH, Nuernberg, Germany. +# Copyright (c) 2018 SUSE LINUX GmbH, Nuernberg, Germany. # # All modifications and additions to the file contributed by third parties # remain the property of their copyright owners, unless otherwise agreed @@ -18,14 +18,13 @@ %{?!python_module:%define python_module() python-%{**} python3-%{**}} Name: python-numexpr -Version: 2.6.4 +Version: 2.6.6 Release: 0 Url: https://github.com/pydata/numexpr/ Summary: Numerical expression evaluator for NumPy License: MIT Group: Development/Languages/Python Source: https://files.pythonhosted.org/packages/source/n/numexpr/numexpr-%{version}.tar.gz -BuildRoot: %{_tmppath}/%{name}-%{version}-build BuildRequires: %{python_module devel} BuildRequires: %{python_module numpy-devel >= 1.6} BuildRequires: %{python_module setuptools} @@ -33,6 +32,7 @@ BuildRequires: gcc-c++ BuildRequires: python-rpm-macros Requires: python-numpy >= 1.6 + %python_subpackages %description @@ -55,8 +55,8 @@ %python_expand %fdupes %{buildroot}%{$python_sitearch} %files %{python_files} -%defattr(-,root,root,-) -%doc ANNOUNCE.rst AUTHORS.txt LICENSE.txt README.rst RELEASE_NOTES.rst site.cfg.example +%doc ANNOUNCE.rst AUTHORS.txt README.rst RELEASE_NOTES.rst site.cfg.example +%license LICENSE.txt %{python_sitearch}/numexpr/ %{python_sitearch}/numexpr-%{version}-py*.egg-info ++++++ numexpr-2.6.4.tar.gz -> numexpr-2.6.6.tar.gz ++++++ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/numexpr-2.6.4/ANNOUNCE.rst new/numexpr-2.6.6/ANNOUNCE.rst --- old/numexpr-2.6.4/ANNOUNCE.rst 2017-09-14 08:44:35.000000000 +0200 +++ new/numexpr-2.6.6/ANNOUNCE.rst 2018-07-16 17:57:51.000000000 +0200 @@ -1,42 +1,21 @@ +========================== + Announcing Numexpr 2.6.6 +========================== + Hi everyone, -This is primarily a maintenance release that fixes a number of newly discovered -bugs. The NumPy requirement has increased from 1.6 to 1.7 due to changes with -`numpy.nditer` flags. Thanks to Caleb P. Burns `ceil` and `floor` functions are -now supported. +This is a bug-fix release. Thanks to Mark Dickinson for a fix to the thread +barrier that occassionally suffered from spurious wakeups on MacOSX. -Project documentation is now available at: +Project documentation is available at: http://numexpr.readthedocs.io/ -P.S. due to seg-faults occuring for MKL with ceil and floor we have pushed a -quick patch for 2.6.3 to 2.6.4. Thanks to Christoph Gohkle for the fixes. - -========================== - Announcing Numexpr 2.6.4 -========================== - -Changes from 2.6.3 to 2.6.4 ---------------------------- - -- Christoph Gohkle noticed a lack of coverage for the 2.6.3 - `floor` and `ceil` functions for MKL that caused seg-faults in - test, so thanks to him for that. - -Changes from 2.6.2 to 2.6.3 +Changes from 2.6.5 to 2.6.6 --------------------------- -- Documentation now available at numexpr.readthedocs.io -- Support for floor() and ceil() functions added by Caleb P. Burns. -- NumPy requirement increased from 1.6 to 1.7 due to changes in iterator - flags (#245). -- Sphinx autodocs support added for documentation on readthedocs.org. -- Fixed a bug where complex constants would return an error, fixing - problems with `sympy` when using NumExpr as a backend. -- Fix for #277 whereby arrays of shape (1,...) would be reduced as - if they were full reduction. Behavoir now matches that of NumPy. -- String literals are automatically encoded into 'ascii' bytes for - convience (see #281). +- Thanks to Mark Dickinson for a fix to the thread barrier that occassionally + suffered from spurious wakeups on MacOSX. What's Numexpr? --------------- diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/numexpr-2.6.4/PKG-INFO new/numexpr-2.6.6/PKG-INFO --- old/numexpr-2.6.4/PKG-INFO 2017-09-14 08:46:26.000000000 +0200 +++ new/numexpr-2.6.6/PKG-INFO 2018-07-16 21:37:09.000000000 +0200 @@ -1,11 +1,129 @@ Metadata-Version: 1.0 Name: numexpr -Version: 2.6.4 +Version: 2.6.6 Summary: Fast numerical expression evaluator for NumPy Home-page: https://github.com/pydata/numexpr Author: David M. Cooke, Francesc Alted and others Author-email: [email protected], [email protected] License: MIT -Description-Content-Type: UNKNOWN -Description: UNKNOWN +Description: ====================================================== + NumExpr: Fast numerical expression evaluator for NumPy + ====================================================== + + :Author: David M. Cooke, Francesc Alted and others + :Contact: [email protected] + :URL: https://github.com/pydata/numexpr + :Documentation: http://numexpr.readthedocs.io/en/latest/ + :Travis CI: |travis| + :Appveyor: |appveyor| + :PyPi: |version| + :readthedocs: |docs| + + .. |travis| image:: https://travis-ci.org/pydata/numexpr.png?branch=master + :target: https://travis-ci.org/pydata/numexpr + .. |appveyor| image:: https://ci.appveyor.com/api/projects/status/github/pydata/numexpr + :target: https://ci.appveyor.com/project/pydata/numexpr + .. |docs| image:: https://media.readthedocs.org/static/projects/badges/passing.svg + :target: http://numexpr.readthedocs.io/en/latest + .. |version| image:: https://img.shields.io/pypi/v/numexpr.png + :target: https://pypi.python.org/pypi/numexpr + + + What is NumExpr? + ---------------- + + NumExpr is a fast numerical expression evaluator for NumPy. With it, + expressions that operate on arrays (like :code:`'3*a+4*b'`) are accelerated + and use less memory than doing the same calculation in Python. + + In addition, its multi-threaded capabilities can make use of all your + cores -- which generally results in substantial performance scaling compared + to NumPy. + + Last but not least, numexpr can make use of Intel's VML (Vector Math + Library, normally integrated in its Math Kernel Library, or MKL). + This allows further acceleration of transcendent expressions. + + + How NumExpr achieves high performance + ------------------------------------- + + The main reason why NumExpr achieves better performance than NumPy is + that it avoids allocating memory for intermediate results. This + results in better cache utilization and reduces memory access in + general. Due to this, NumExpr works best with large arrays. + + NumExpr parses expressions into its own op-codes that are then used by + an integrated computing virtual machine. The array operands are split + into small chunks that easily fit in the cache of the CPU and passed + to the virtual machine. The virtual machine then applies the + operations on each chunk. It's worth noting that all temporaries and + constants in the expression are also chunked. Chunks are distributed among + the available cores of the CPU, resulting in highly parallelized code + execution. + + The result is that NumExpr can get the most of your machine computing + capabilities for array-wise computations. Common speed-ups with regard + to NumPy are usually between 0.95x (for very simple expressions like + :code:`'a + 1'`) and 4x (for relatively complex ones like :code:`'a*b-4.1*a > + 2.5*b'`), although much higher speed-ups can be achieved for some functions + and complex math operations (up to 15x in some cases). + + NumExpr performs best on matrices that are too large to fit in L1 CPU cache. + In order to get a better idea on the different speed-ups that can be achieved + on your platform, run the provided benchmarks. + + + Usage + ----- + + :: + + >>> import numpy as np + >>> import numexpr as ne + + >>> a = np.arange(1e6) # Choose large arrays for better speedups + >>> b = np.arange(1e6) + + >>> ne.evaluate("a + 1") # a simple expression + array([ 1.00000000e+00, 2.00000000e+00, 3.00000000e+00, ..., + 9.99998000e+05, 9.99999000e+05, 1.00000000e+06]) + + >>> ne.evaluate('a*b-4.1*a > 2.5*b') # a more complex one + array([False, False, False, ..., True, True, True], dtype=bool) + + >>> ne.evaluate("sin(a) + arcsinh(a/b)") # you can also use functions + array([ NaN, 1.72284457, 1.79067101, ..., 1.09567006, + 0.17523598, -0.09597844]) + + >>> s = np.array(['abba', 'abbb', 'abbcdef']) + >>> ne.evaluate("'abba' == s") # string arrays are supported too + array([ True, False, False], dtype=bool) + + + Documentation + ------------- + + Please see the official documentation at `numexpr.readthedocs.io <https://numexpr.readthedocs.io>`_. + Included is a user guide, benchmark results, and the reference API. + + + Authors + ------- + + Please see `AUTHORS.txt <https://github.com/pydata/numexpr/blob/master/AUTHORS.txt>`_. + + + License + ------- + + NumExpr is distributed under the `MIT <http://www.opensource.org/licenses/mit-license.php>`_ license. + + + .. Local Variables: + .. mode: text + .. coding: utf-8 + .. fill-column: 70 + .. End: + Platform: UNKNOWN diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/numexpr-2.6.4/RELEASE_NOTES.rst new/numexpr-2.6.6/RELEASE_NOTES.rst --- old/numexpr-2.6.4/RELEASE_NOTES.rst 2017-09-14 08:42:38.000000000 +0200 +++ new/numexpr-2.6.6/RELEASE_NOTES.rst 2018-07-12 19:35:59.000000000 +0200 @@ -2,12 +2,36 @@ Release notes for Numexpr 2.6 series ===================================== +Changes from 2.6.5 to 2.6.6 +--------------------------- + +- Thanks to Mark Dickinson for a fix to the thread barrier that occassionally + suffered from spurious wakeups on MacOSX. + +Changes from 2.6.4 to 2.6.5 +--------------------------- + +- The maximum thread count can now be set at import-time by setting the + environment variable 'NUMEXPR_MAX_THREADS'. The default number of + max threads was lowered from 4096 (which was deemed excessive) to 64. +- A number of imports were removed (pkg_resources) or made lazy (cpuinfo) in + order to speed load-times for downstream packages (such as `pandas`, `sympy`, + and `tables`). Import time has dropped from about 330 ms to 90 ms. Thanks to + Jason Sachs for pointing out the source of the slow-down. +- Thanks to Alvaro Lopez Ortega for updates to benchmarks to be compatible with + Python 3. +- Travis and AppVeyor now fail if the test module fails or errors. +- Thanks to Mahdi Ben Jelloul for a patch that removed a bug where constants + in `where` calls would raise a ValueError. +- Fixed a bug whereby all-constant power operations would lead to infinite + recursion. + Changes from 2.6.3 to 2.6.4 --------------------------- -- Christoph Gohkle noticed a lack of coverage for the 2.6.3 +- Christoph Gohlke noticed a lack of coverage for the 2.6.3 `floor` and `ceil` functions for MKL that caused seg-faults in - test, so thanks to him for that. + test, so thanks to him for that. Changes from 2.6.2 to 2.6.3 --------------------------- diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/numexpr-2.6.4/numexpr/__init__.py new/numexpr-2.6.6/numexpr/__init__.py --- old/numexpr-2.6.4/numexpr/__init__.py 2017-09-14 08:37:56.000000000 +0200 +++ new/numexpr-2.6.6/numexpr/__init__.py 2018-07-12 19:33:23.000000000 +0200 @@ -28,18 +28,17 @@ else: use_vml = False -from cpuinfo import cpu +is_cpu_amd_intel = False # DEPRECATION WARNING: WILL BE REMOVED IN FUTURE RELEASE -if cpu.is_AMD() or cpu.is_Intel(): - is_cpu_amd_intel = True -else: - is_cpu_amd_intel = False +# cpuinfo imports were moved into the test submodule function that calls them +# to improve import times. import os, os.path import platform from numexpr.expressions import E from numexpr.necompiler import NumExpr, disassemble, evaluate, re_evaluate from numexpr.tests import test, print_versions +from numexpr.interpreter import MAX_THREADS from numexpr.utils import ( get_vml_version, set_vml_accuracy_mode, set_vml_num_threads, set_num_threads, detect_number_of_cores, detect_number_of_threads) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/numexpr-2.6.4/numexpr/cpuinfo.py new/numexpr-2.6.6/numexpr/cpuinfo.py --- old/numexpr-2.6.4/numexpr/cpuinfo.py 2017-09-14 08:37:56.000000000 +0200 +++ new/numexpr-2.6.6/numexpr/cpuinfo.py 2018-07-12 19:33:23.000000000 +0200 @@ -32,6 +32,7 @@ import platform import inspect +is_cpu_amd_intel = False # DEPRECATION WARNING: WILL BE REMOVED IN FUTURE RELEASE def getoutput(cmd, successful_status=(0,), stacklevel=1): try: @@ -857,4 +858,4 @@ info.append('%s=%s' % (name[1:], r)) else: info.append(name[1:]) - print('CPU information: ' + ' '.join(info)) + print('CPU information: ' + ' '.join(info)) \ No newline at end of file diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/numexpr-2.6.4/numexpr/expressions.py new/numexpr-2.6.6/numexpr/expressions.py --- old/numexpr-2.6.4/numexpr/expressions.py 2017-09-14 08:37:56.000000000 +0200 +++ new/numexpr-2.6.6/numexpr/expressions.py 2018-07-12 19:33:23.000000000 +0200 @@ -15,8 +15,10 @@ import threading import numpy -from pkg_resources import parse_version -_np_version = parse_version(numpy.__version__) +# numpy's behavoir sometimes changes with versioning, especially in regard as +# to when ints are cast to floats. +# _np_version will be similar to ('1', '13', '1') for which we can use simple comparisons +_np_version = tuple( ver for ver in numpy.__version__.split('.') ) # Declare a double type that does not exist in Python space double = numpy.double @@ -216,8 +218,7 @@ @ophelper def where_func(a, b, c): if isinstance(a, ConstantNode): - #FIXME: This prevents where(True, a, b) - raise ValueError("too many dimensions") + return b if a.value else c if allConstantNodes([a, b, c]): return ConstantNode(numpy.where(a, b, c)) return FuncNode('where', [a, b, c]) @@ -282,7 +283,7 @@ @ophelper def pow_op(a, b): - if (_np_version >= parse_version('1.12.0b1') and + if (_np_version >= ('1', '12', '0b1') and b.astKind in ('int', 'long') and a.astKind in ('int', 'long') and numpy.any(b.value < 0)): @@ -291,7 +292,7 @@ 'Integers to negative integer powers are not allowed.') if allConstantNodes([a, b]): - return ConstantNode(a ** b) + return ConstantNode(a.value ** b.value) if isinstance(b, ConstantNode): x = b.value if get_optimization() == 'aggressive': diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/numexpr-2.6.4/numexpr/interpreter.cpp new/numexpr-2.6.6/numexpr/interpreter.cpp --- old/numexpr-2.6.4/numexpr/interpreter.cpp 2017-09-14 08:37:56.000000000 +0200 +++ new/numexpr-2.6.6/numexpr/interpreter.cpp 2018-07-12 19:49:43.000000000 +0200 @@ -772,9 +772,13 @@ pthread_mutex_lock(&gs.count_threads_mutex); if (gs.count_threads < gs.nthreads) { gs.count_threads++; - pthread_cond_wait(&gs.count_threads_cv, &gs.count_threads_mutex); + /* Beware of spurious wakeups. See issue pydata/numexpr#306. */ + do { + pthread_cond_wait(&gs.count_threads_cv, &gs.count_threads_mutex); + } while (!gs.barrier_passed); } else { + gs.barrier_passed = 1; pthread_cond_broadcast(&gs.count_threads_cv); } pthread_mutex_unlock(&gs.count_threads_mutex); @@ -783,9 +787,12 @@ pthread_mutex_lock(&gs.count_threads_mutex); if (gs.count_threads > 0) { gs.count_threads--; - pthread_cond_wait(&gs.count_threads_cv, &gs.count_threads_mutex); + do { + pthread_cond_wait(&gs.count_threads_cv, &gs.count_threads_mutex); + } while (gs.barrier_passed); } else { + gs.barrier_passed = 0; pthread_cond_broadcast(&gs.count_threads_cv); } pthread_mutex_unlock(&gs.count_threads_mutex); diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/numexpr-2.6.4/numexpr/interpreter.hpp new/numexpr-2.6.6/numexpr/interpreter.hpp --- old/numexpr-2.6.4/numexpr/interpreter.hpp 2017-09-14 08:37:56.000000000 +0200 +++ new/numexpr-2.6.6/numexpr/interpreter.hpp 2018-07-12 19:33:23.000000000 +0200 @@ -75,12 +75,17 @@ int ret_code; int *pc_error; char **errmsg; + // NOTE: memsteps, iter, and reduce_iter are arrays, they MUST be allocated + // to length `global_max_threads` before module load. // One memsteps array per thread - npy_intp *memsteps[MAX_THREADS]; + // npy_intp *memsteps[MAX_THREADS]; + npy_intp **memsteps; // One iterator per thread */ - NpyIter *iter[MAX_THREADS]; + // NpyIter *iter[MAX_THREADS]; + NpyIter **iter; // When doing nested iteration for a reduction - NpyIter *reduce_iter[MAX_THREADS]; + // NpyIter *reduce_iter[MAX_THREADS] + NpyIter **reduce_iter; // Flag indicating reduction is the outer loop instead of the inner bool reduction_outer_loop; // Flag indicating whether output buffering is needed diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/numexpr-2.6.4/numexpr/module.cpp new/numexpr-2.6.6/numexpr/module.cpp --- old/numexpr-2.6.4/numexpr/module.cpp 2017-09-14 08:37:56.000000000 +0200 +++ new/numexpr-2.6.6/numexpr/module.cpp 2018-07-12 19:49:43.000000000 +0200 @@ -21,6 +21,7 @@ // Global state. The file interpreter.hpp also has some global state // in its 'th_params' variable global_state gs; +long global_max_threads=DEFAULT_MAX_THREADS; /* Do the worker job for a certain thread */ @@ -55,9 +56,14 @@ pthread_mutex_lock(&gs.count_threads_mutex); if (gs.count_threads < gs.nthreads) { gs.count_threads++; - pthread_cond_wait(&gs.count_threads_cv, &gs.count_threads_mutex); + /* Beware of spurious wakeups. See issue pydata/numexpr#306. */ + do { + pthread_cond_wait(&gs.count_threads_cv, + &gs.count_threads_mutex); + } while (!gs.barrier_passed); } else { + gs.barrier_passed = 1; pthread_cond_broadcast(&gs.count_threads_cv); } pthread_mutex_unlock(&gs.count_threads_mutex); @@ -163,9 +169,13 @@ pthread_mutex_lock(&gs.count_threads_mutex); if (gs.count_threads > 0) { gs.count_threads--; - pthread_cond_wait(&gs.count_threads_cv, &gs.count_threads_mutex); + do { + pthread_cond_wait(&gs.count_threads_cv, + &gs.count_threads_mutex); + } while (gs.barrier_passed); } else { + gs.barrier_passed = 0; pthread_cond_broadcast(&gs.count_threads_cv); } pthread_mutex_unlock(&gs.count_threads_mutex); @@ -185,6 +195,11 @@ { int tid, rc; + if ( !(gs.nthreads > 1 && (!gs.init_threads_done || gs.pid != getpid())) ) { + /* Thread pool must always be initialized once and once only. */ + return(0); + } + /* Initialize mutex and condition variable objects */ pthread_mutex_init(&gs.count_mutex, NULL); pthread_mutex_init(&gs.parallel_mutex, NULL); @@ -193,6 +208,7 @@ pthread_mutex_init(&gs.count_threads_mutex, NULL); pthread_cond_init(&gs.count_threads_cv, NULL); gs.count_threads = 0; /* Reset threads counter */ + gs.barrier_passed = 0; /* Finally, create the threads */ for (tid = 0; tid < gs.nthreads; tid++) { @@ -220,10 +236,16 @@ int t, rc; void *status; - if (nthreads_new > MAX_THREADS) { + // if (nthreads_new > MAX_THREADS) { + // fprintf(stderr, + // "Error. nthreads cannot be larger than MAX_THREADS (%d)", + // MAX_THREADS); + // return -1; + // } + if (nthreads_new > global_max_threads) { fprintf(stderr, - "Error. nthreads cannot be larger than MAX_THREADS (%d)", - MAX_THREADS); + "Error. nthreads cannot be larger than environment variable \"NUMEXPR_MAX_THREADS\" (%ld)", + global_max_threads); return -1; } else if (nthreads_new <= 0) { @@ -240,9 +262,13 @@ pthread_mutex_lock(&gs.count_threads_mutex); if (gs.count_threads < gs.nthreads) { gs.count_threads++; - pthread_cond_wait(&gs.count_threads_cv, &gs.count_threads_mutex); + do { + pthread_cond_wait(&gs.count_threads_cv, + &gs.count_threads_mutex); + } while (!gs.barrier_passed); } else { + gs.barrier_passed = 1; pthread_cond_broadcast(&gs.count_threads_cv); } pthread_mutex_unlock(&gs.count_threads_mutex); @@ -264,9 +290,7 @@ /* Launch a new pool of threads (if necessary) */ gs.nthreads = nthreads_new; - if (gs.nthreads > 1 && (!gs.init_threads_done || gs.pid != getpid())) { - init_threads(); - } + init_threads(); return nthreads_old; } @@ -384,6 +408,22 @@ { PyObject *m, *d; + + char *max_thread_str = getenv("NUMEXPR_MAX_THREADS"); + char *end; + if (max_thread_str != NULL) { + global_max_threads = strtol(max_thread_str, &end, 10); + } + + + th_params.memsteps = (npy_intp**)calloc(sizeof(npy_intp*), global_max_threads); + th_params.iter = (NpyIter**)calloc(sizeof(NpyIter*), global_max_threads); + th_params.reduce_iter = (NpyIter**)calloc(sizeof(NpyIter*), global_max_threads); + gs.threads = (pthread_t*)calloc(sizeof(pthread_t), global_max_threads); + gs.tids = (int*)calloc(sizeof(int), global_max_threads); + // TODO: for Py3, deallocate: https://docs.python.org/3/c-api/module.html#c.PyModuleDef.m_free + // For Python 2.7, people have to exit the process to reclaim the memory. + if (PyType_Ready(&NumExprType) < 0) INITERROR; @@ -437,6 +477,8 @@ if (PyModule_AddObject(m, "allaxes", PyLong_FromLong(255)) < 0) INITERROR; if (PyModule_AddObject(m, "maxdims", PyLong_FromLong(NPY_MAXDIMS)) < 0) INITERROR; + if(PyModule_AddIntConstant(m, "MAX_THREADS", global_max_threads) < 0) INITERROR; + #if PY_MAJOR_VERSION >= 3 return m; #endif diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/numexpr-2.6.4/numexpr/module.hpp new/numexpr-2.6.6/numexpr/module.hpp --- old/numexpr-2.6.4/numexpr/module.hpp 2017-09-14 08:37:56.000000000 +0200 +++ new/numexpr-2.6.6/numexpr/module.hpp 2018-07-12 19:49:44.000000000 +0200 @@ -21,8 +21,12 @@ int nthreads; /* number of desired threads in pool */ int init_threads_done; /* pool of threads initialized? */ int end_threads; /* should exisiting threads end? */ - pthread_t threads[MAX_THREADS]; /* opaque structure for threads */ - int tids[MAX_THREADS]; /* ID per each thread */ + // pthread_t threads[MAX_THREADS]; /* opaque structure for threads */ + // int tids[MAX_THREADS]; /* ID per each thread */ + /* NOTE: threads and tids are arrays, they MUST be allocated to length + `global_max_threads` before module load. */ + pthread_t *threads; /* opaque structure for threads */ + int *tids; /* ID per each thread */ npy_intp gindex; /* global index for all threads */ int init_sentinels_done; /* sentinels initialized? */ int giveup; /* should parallel code giveup? */ @@ -32,6 +36,8 @@ /* Synchronization variables for threadpool state */ pthread_mutex_t count_mutex; int count_threads; + int barrier_passed; /* indicates if the thread pool's thread barrier + is unlocked and ready for the VM to process.*/ pthread_mutex_t count_threads_mutex; pthread_cond_t count_threads_cv; @@ -41,6 +47,7 @@ global_state() { nthreads = 1; init_threads_done = 0; + barrier_passed = 0; end_threads = 0; pid = 0; } diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/numexpr-2.6.4/numexpr/necompiler.py new/numexpr-2.6.6/numexpr/necompiler.py --- old/numexpr-2.6.4/numexpr/necompiler.py 2017-09-14 08:37:56.000000000 +0200 +++ new/numexpr-2.6.6/numexpr/necompiler.py 2018-07-13 06:47:37.000000000 +0200 @@ -13,7 +13,8 @@ import numpy import threading -from numexpr import interpreter, expressions, use_vml, is_cpu_amd_intel +is_cpu_amd_intel = False # DEPRECATION WARNING: WILL BE REMOVED IN FUTURE RELEASE +from numexpr import interpreter, expressions, use_vml from numexpr.utils import CacheDict # Declare a double type that does not exist in Python space @@ -545,8 +546,7 @@ raise ValueError("Unknown keyword argument '%s'" % d.popitem()[0]) if context['truediv'] == 'auto': caller_globals = sys._getframe(frame_depth + 1).f_globals - context['truediv'] = \ - caller_globals.get('division', None) == __future__.division + context['truediv'] = caller_globals.get('division', None) == __future__.division return context @@ -623,8 +623,7 @@ # translated to either True or False). context = getContext(kwargs, frame_depth=1) - threeAddrProgram, inputsig, tempsig, constants, input_names = \ - precompile(ex, signature, context) + threeAddrProgram, inputsig, tempsig, constants, input_names = precompile(ex, signature, context) program = compileThreeAddrForm(threeAddrProgram) return interpreter.NumExpr(inputsig.encode('ascii'), tempsig.encode('ascii'), @@ -811,8 +810,7 @@ try: compiled_ex = _numexpr_cache[numexpr_key] except KeyError: - compiled_ex = _numexpr_cache[numexpr_key] = \ - NumExpr(ex, signature, **context) + compiled_ex = _numexpr_cache[numexpr_key] = NumExpr(ex, signature, **context) kwargs = {'out': out, 'order': order, 'casting': casting, 'ex_uses_vml': ex_uses_vml} _numexpr_last = dict(ex=compiled_ex, argnames=names, kwargs=kwargs) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/numexpr-2.6.4/numexpr/numexpr_config.hpp new/numexpr-2.6.6/numexpr/numexpr_config.hpp --- old/numexpr-2.6.4/numexpr/numexpr_config.hpp 2017-09-14 08:37:56.000000000 +0200 +++ new/numexpr-2.6.6/numexpr/numexpr_config.hpp 2018-07-12 19:33:23.000000000 +0200 @@ -24,7 +24,8 @@ /* The maximum number of threads (for some static arrays). * Choose this large enough for most monsters out there. Keep in sync this with the number in __init__.py. */ -#define MAX_THREADS 4096 +// #define MAX_THREADS 4096 +#define DEFAULT_MAX_THREADS 64 #if defined(_WIN32) #include "win32/pthread.h" diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/numexpr-2.6.4/numexpr/tests/test_numexpr.py new/numexpr-2.6.6/numexpr/tests/test_numexpr.py --- old/numexpr-2.6.4/numexpr/tests/test_numexpr.py 2017-09-14 08:37:56.000000000 +0200 +++ new/numexpr-2.6.6/numexpr/tests/test_numexpr.py 2018-07-12 19:33:23.000000000 +0200 @@ -35,10 +35,11 @@ TestCase = unittest.TestCase double = np.double - +if sys.version_info[0] >= 3: + long = int # Recommended minimum versions -minimum_numpy_version = "1.6" +minimum_numpy_version = ('1', '7', '0') class test_numexpr(TestCase): @@ -295,6 +296,16 @@ b = b'a' * 40 res = evaluate('contains(a, b)') assert_equal(res, True) + + def test_where_scalar_bool(self): + a = True + b = array([1, 2]) + c = array([3, 4]) + res = evaluate('where(a, b, c)') + assert_array_equal(res, b) + a = False + res = evaluate('where(a, b, c)') + assert_array_equal(res, c) class test_numexpr2(test_numexpr): @@ -995,23 +1006,36 @@ def print_versions(): """Print the versions of software that numexpr relies on.""" - from pkg_resources import parse_version - if parse_version(np.__version__) < parse_version(minimum_numpy_version): - print("*Warning*: NumPy version is lower than recommended: %s < %s" % \ + # from pkg_resources import parse_version + from numexpr.cpuinfo import cpu + import platform + + np_version = tuple( ver for ver in np.__version__.split('.') ) + + if minimum_numpy_version < np_version: + print('*Warning*: NumPy version is lower than recommended: %s < %s' % \ (np.__version__, minimum_numpy_version)) print('-=' * 38) - print("Numexpr version: %s" % numexpr.__version__) - print("NumPy version: %s" % np.__version__) + print('Numexpr version: %s' % numexpr.__version__) + print('NumPy version: %s' % np.__version__) print('Python version: %s' % sys.version) - if os.name == 'posix': - (sysname, nodename, release, version, machine) = os.uname() - print('Platform: %s-%s' % (sys.platform, machine)) - print("AMD/Intel CPU? %s" % numexpr.is_cpu_amd_intel) - print("VML available? %s" % use_vml) + (sysname, nodename, release, os_version, machine, processor) = platform.uname() + print('Platform: %s-%s-%s' % (sys.platform, machine, os_version)) + try: + # cpuinfo doesn't work on OSX well it seems, so protect these outputs + # with a try block + cpu_info = cpu.info[0] + print('CPU vendor: %s' % cpu_info.get('VendorIdentifier', '')) + print('CPU model: %s' % cpu_info.get('ProcessorNameString', '')) + print('CPU clock speed: %s MHz' % cpu_info.get('~MHz','')) + except KeyError: + pass + print('VML available? %s' % use_vml) if use_vml: - print("VML/MKL version: %s" % numexpr.get_vml_version()) - print("Number of threads used by default: %d " - "(out of %d detected cores)" % (numexpr.nthreads, numexpr.ncores)) + print('VML/MKL version: %s' % numexpr.get_vml_version()) + print('Number of threads used by default: %d ' + '(out of %d detected cores)' % (numexpr.nthreads, numexpr.ncores)) + print('Maximum number of threads: %s' % numexpr.MAX_THREADS) print('-=' * 38) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/numexpr-2.6.4/numexpr/utils.py new/numexpr-2.6.6/numexpr/utils.py --- old/numexpr-2.6.4/numexpr/utils.py 2017-09-14 08:37:56.000000000 +0200 +++ new/numexpr-2.6.6/numexpr/utils.py 2018-07-12 19:33:23.000000000 +0200 @@ -11,7 +11,7 @@ import os import subprocess -from numexpr.interpreter import _set_num_threads +from numexpr.interpreter import _set_num_threads, MAX_THREADS from numexpr import use_vml if use_vml: @@ -128,8 +128,8 @@ if nthreads > 8: nthreads = 8 # Check that we don't surpass the MAX_THREADS in interpreter.cpp - if nthreads > 4096: - nthreads = 4096 + if nthreads > MAX_THREADS: + nthreads = MAX_THREADS return nthreads diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/numexpr-2.6.4/numexpr/version.py new/numexpr-2.6.6/numexpr/version.py --- old/numexpr-2.6.4/numexpr/version.py 2017-09-14 08:40:07.000000000 +0200 +++ new/numexpr-2.6.6/numexpr/version.py 2018-07-16 17:56:44.000000000 +0200 @@ -8,4 +8,4 @@ # rights to use. #################################################################### -version = '2.6.4' +version = '2.6.6' diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/numexpr-2.6.4/numexpr.egg-info/PKG-INFO new/numexpr-2.6.6/numexpr.egg-info/PKG-INFO --- old/numexpr-2.6.4/numexpr.egg-info/PKG-INFO 2017-09-14 08:46:26.000000000 +0200 +++ new/numexpr-2.6.6/numexpr.egg-info/PKG-INFO 2018-07-16 21:37:09.000000000 +0200 @@ -1,11 +1,129 @@ Metadata-Version: 1.0 Name: numexpr -Version: 2.6.4 +Version: 2.6.6 Summary: Fast numerical expression evaluator for NumPy Home-page: https://github.com/pydata/numexpr Author: David M. Cooke, Francesc Alted and others Author-email: [email protected], [email protected] License: MIT -Description-Content-Type: UNKNOWN -Description: UNKNOWN +Description: ====================================================== + NumExpr: Fast numerical expression evaluator for NumPy + ====================================================== + + :Author: David M. Cooke, Francesc Alted and others + :Contact: [email protected] + :URL: https://github.com/pydata/numexpr + :Documentation: http://numexpr.readthedocs.io/en/latest/ + :Travis CI: |travis| + :Appveyor: |appveyor| + :PyPi: |version| + :readthedocs: |docs| + + .. |travis| image:: https://travis-ci.org/pydata/numexpr.png?branch=master + :target: https://travis-ci.org/pydata/numexpr + .. |appveyor| image:: https://ci.appveyor.com/api/projects/status/github/pydata/numexpr + :target: https://ci.appveyor.com/project/pydata/numexpr + .. |docs| image:: https://media.readthedocs.org/static/projects/badges/passing.svg + :target: http://numexpr.readthedocs.io/en/latest + .. |version| image:: https://img.shields.io/pypi/v/numexpr.png + :target: https://pypi.python.org/pypi/numexpr + + + What is NumExpr? + ---------------- + + NumExpr is a fast numerical expression evaluator for NumPy. With it, + expressions that operate on arrays (like :code:`'3*a+4*b'`) are accelerated + and use less memory than doing the same calculation in Python. + + In addition, its multi-threaded capabilities can make use of all your + cores -- which generally results in substantial performance scaling compared + to NumPy. + + Last but not least, numexpr can make use of Intel's VML (Vector Math + Library, normally integrated in its Math Kernel Library, or MKL). + This allows further acceleration of transcendent expressions. + + + How NumExpr achieves high performance + ------------------------------------- + + The main reason why NumExpr achieves better performance than NumPy is + that it avoids allocating memory for intermediate results. This + results in better cache utilization and reduces memory access in + general. Due to this, NumExpr works best with large arrays. + + NumExpr parses expressions into its own op-codes that are then used by + an integrated computing virtual machine. The array operands are split + into small chunks that easily fit in the cache of the CPU and passed + to the virtual machine. The virtual machine then applies the + operations on each chunk. It's worth noting that all temporaries and + constants in the expression are also chunked. Chunks are distributed among + the available cores of the CPU, resulting in highly parallelized code + execution. + + The result is that NumExpr can get the most of your machine computing + capabilities for array-wise computations. Common speed-ups with regard + to NumPy are usually between 0.95x (for very simple expressions like + :code:`'a + 1'`) and 4x (for relatively complex ones like :code:`'a*b-4.1*a > + 2.5*b'`), although much higher speed-ups can be achieved for some functions + and complex math operations (up to 15x in some cases). + + NumExpr performs best on matrices that are too large to fit in L1 CPU cache. + In order to get a better idea on the different speed-ups that can be achieved + on your platform, run the provided benchmarks. + + + Usage + ----- + + :: + + >>> import numpy as np + >>> import numexpr as ne + + >>> a = np.arange(1e6) # Choose large arrays for better speedups + >>> b = np.arange(1e6) + + >>> ne.evaluate("a + 1") # a simple expression + array([ 1.00000000e+00, 2.00000000e+00, 3.00000000e+00, ..., + 9.99998000e+05, 9.99999000e+05, 1.00000000e+06]) + + >>> ne.evaluate('a*b-4.1*a > 2.5*b') # a more complex one + array([False, False, False, ..., True, True, True], dtype=bool) + + >>> ne.evaluate("sin(a) + arcsinh(a/b)") # you can also use functions + array([ NaN, 1.72284457, 1.79067101, ..., 1.09567006, + 0.17523598, -0.09597844]) + + >>> s = np.array(['abba', 'abbb', 'abbcdef']) + >>> ne.evaluate("'abba' == s") # string arrays are supported too + array([ True, False, False], dtype=bool) + + + Documentation + ------------- + + Please see the official documentation at `numexpr.readthedocs.io <https://numexpr.readthedocs.io>`_. + Included is a user guide, benchmark results, and the reference API. + + + Authors + ------- + + Please see `AUTHORS.txt <https://github.com/pydata/numexpr/blob/master/AUTHORS.txt>`_. + + + License + ------- + + NumExpr is distributed under the `MIT <http://www.opensource.org/licenses/mit-license.php>`_ license. + + + .. Local Variables: + .. mode: text + .. coding: utf-8 + .. fill-column: 70 + .. End: + Platform: UNKNOWN diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/numexpr-2.6.4/numexpr.egg-info/SOURCES.txt new/numexpr-2.6.6/numexpr.egg-info/SOURCES.txt --- old/numexpr-2.6.4/numexpr.egg-info/SOURCES.txt 2017-09-14 08:46:26.000000000 +0200 +++ new/numexpr-2.6.6/numexpr.egg-info/SOURCES.txt 2018-07-16 21:37:09.000000000 +0200 @@ -8,6 +8,7 @@ RELEASE_NOTES.rst requirements.txt setup.py +site.cfg site.cfg.example ./numexpr/tests/__init__.py ./numexpr/tests/test_numexpr.py diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/numexpr-2.6.4/setup.py new/numexpr-2.6.6/setup.py --- old/numexpr-2.6.4/setup.py 2017-09-14 08:37:56.000000000 +0200 +++ new/numexpr-2.6.6/setup.py 2018-07-16 18:11:42.000000000 +0200 @@ -10,12 +10,9 @@ #################################################################### import shutil -import os -import sys -import os.path as op +import sys, os, os.path as op, io from distutils.command.clean import clean - if sys.version_info < (2, 6): raise RuntimeError("must use python 2.6 or greater") @@ -27,6 +24,9 @@ with open('requirements.txt') as f: requirements = f.read().splitlines() +with io.open('README.rst', encoding='utf-8') as f: + LONG_DESCRIPTION = f.read() + # Fetch the version for numexpr (will be put in variable `version`) with open(os.path.join('numexpr', 'version.py')) as f: exec(f.read()) @@ -37,6 +37,7 @@ author='David M. Cooke, Francesc Alted and others', author_email='[email protected], [email protected]', url='https://github.com/pydata/numexpr', + long_description=LONG_DESCRIPTION, license='MIT', packages=['numexpr'], install_requires=requirements, diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/numexpr-2.6.4/site.cfg new/numexpr-2.6.6/site.cfg --- old/numexpr-2.6.4/site.cfg 1970-01-01 01:00:00.000000000 +0100 +++ new/numexpr-2.6.6/site.cfg 2018-07-16 18:03:13.000000000 +0200 @@ -0,0 +1,51 @@ +# if setup does not find the MKL or does not use the configuration, copy this +# file to "site.cfg" and edit the paths according to your installation of the +# Intel MKL. + +# Note: some versions of MKL need to be linked to gfortran if compiled with +# GNU C compiler. Uncomment next line if you get an error like "undefined +# symbol: _gfortran_malloc" +# +# Note2: Some Fedora users reported that they had to install a +# compatible version of the gfortran lib. See: +# http://code.google.com/p/numexpr/issues/detail?id=15 +# for more info. +[DEFAULT] +#libraries = gfortran + +#[mkl] +# Example for the MKL included in Intel C 11.0 compiler +# (you may need a recent NumPy version for being able to search libraries +# in different directories at a time) +#library_dirs = /opt/intel/Compiler/11.0/074/mkl/lib/em64t/:/opt/intel/Compiler/11.0/074/lib/intel64 +#include_dirs = /opt/intel/Compiler/11.0/074/mkl/include/ +#mkl_libs = mkl_solver_ilp64, mkl_intel_ilp64, mkl_intel_thread, mkl_core, iomp5 + +# This seems to work for MKL 11 with processors with AVX (Sandy Bridge and above) for Linux +#library_dirs = /opt/intel/composerxe/mkl/lib/intel64:/opt/intel/composer_xe_2013.3.163/compiler/lib/intel64 +#include_dirs = /opt/intel/composerxe/mkl/include/ +#mkl_libs = mkl_intel_lp64, mkl_gf_lp64, mkl_intel_thread, mkl_core, mkl_blas95_lp64, mkl_lapack95_lp64, mkl_avx, mkl_vml_avx, mkl_rt, iomp5 + +## Example for using MKL 10.0 +#library_dirs = /opt/intel/mkl/10.0.2.018/lib/em64t +#include_dirs = /opt/intel/mkl/10.0.2.018/include + +# Example for using MKL 10.2 for Windows 64-bit +#include_dirs = \Program Files\Intel\MKL\10.2.5.035\include +#library_dirs = \Program Files\Intel\MKL\10.2.5.035\em64t\lib +#mkl_libs = mkl_solver_ilp64, mkl_core, mkl_intel_thread, mkl_intel_ilp64, libiomp5md +# The next works too, but for LP64 arithmetic +#mkl_libs = mkl_core, mkl_intel_thread, mkl_intel_lp64, libiomp5md + +# Example with Intel compiler version 14.0.2 and MKL v11.1.2 on intel64 architecture +#mkl_libs = mkl_intel_lp64, mkl_gf_lp64, mkl_intel_thread, mkl_core, mkl_def, mkl_vml_avx, mkl_rt, iomp5 +# For details, see https://github.com/pydata/numexpr/issues/148 + +# Example for MKL2018 on Windows x64 +# https://software.intel.com/en-us/articles/building-numpyscipy-with-intel-mkl-and-intel-fortran-on-windows +# Please note that the valid os.pathsep is ';' on Windows and ':' on Linux +# and that numpy.distutils.system_info only accepts the first instance of library_dirs +[mkl] +library_dirs=/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/lib/intel64;/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/compiler/lib/intel64 +include_dirs=/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/include +mkl_libs = mkl_intel_lp64, mkl_intel_thread, mkl_core, libiomp5md \ No newline at end of file diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/numexpr-2.6.4/site.cfg.example new/numexpr-2.6.6/site.cfg.example --- old/numexpr-2.6.4/site.cfg.example 2017-09-14 08:38:25.000000000 +0200 +++ new/numexpr-2.6.6/site.cfg.example 2018-07-16 18:07:07.000000000 +0200 @@ -13,7 +13,7 @@ [DEFAULT] #libraries = gfortran -[mkl] +#[mkl] # Example for the MKL included in Intel C 11.0 compiler # (you may need a recent NumPy version for being able to search libraries # in different directories at a time) @@ -45,7 +45,8 @@ # https://software.intel.com/en-us/articles/building-numpyscipy-with-intel-mkl-and-intel-fortran-on-windows # Please note that the valid os.pathsep is ';' on Windows and ':' on Linux # and that numpy.distutils.system_info only accepts the first instance of library_dirs +# Last make sure you do `python setup.py install` instead of using pip. [mkl] -library_dirs=/Program Files (x86)/IntelSWTools/compilers_and_libraries_2018.0.124/windows/mkl/lib/intel64;/Program Files (x86)/IntelSWTools/compilers_and_libraries_2018.0.124/windows/compiler/lib/intel64_win -include_dirs=/Program Files (x86)/IntelSWTools/compilers_and_libraries_2018.0.124/windows/mkl/include +library_dirs=/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/lib/intel64;/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/compiler/lib/intel64 +include_dirs=/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/include mkl_libs = mkl_intel_lp64, mkl_intel_thread, mkl_core, libiomp5md \ No newline at end of file
