[Numpy-discussion] Advanced numpy tutorial for SciPy?

[Chris Barker - NOAA Federal]

In another thread, there is a discussion of a workshop on "Taking
NumPy In Stride" for PyData Barcelona.

I think it would be great to have something like that at SciPy in
Austin this year.

Jaime can't make it, and I don't think strides are going to fill a
four hour tutorial, so it would be good as part of an advanced numpy
tutorial.

I don't have the bandwidth to put together an entire tutorial, but
maybe someone would like to join forces?

Or if someone is already planning an advanced numpy tutorial, perhaps
I could contribute.

Not much time left to get a proposal in!

-Chris
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] PyData Barcelona this May

[Chris Barker]

On Mon, Mar 20, 2017 at 11:58 AM, Jaime Fernández del Río <
jaime.f...@gmail.com> wrote:

>  I have just submitted a workshop proposal with the following short
> description:
>
> Taking NumPy In Stride
> This workshop is aimed at users already familiar with NumPy. We will
> dissect
> the NumPy memory model with the help of a very powerful abstraction:
> strides.
> Participants will learn how to create different views out of the same data,
> including multidimensional ones, get a new angle on how and why
> broadcasting
> works, and explore related techniques to write faster, more efficient code.
>

I'd go!

And nice title :-)

Any thoughts on a similar one for SciPy in Austin?

-CHB





> Let's see what the organizers think of it...
>
> Jaime
>
>
> On Fri, Mar 17, 2017 at 10:59 PM, Ralf Gommers <ralf.gomm...@gmail.com>
> wrote:
>
>>
>>
>> On Sat, Mar 18, 2017 at 8:41 AM, Chris Barker <chris.bar...@noaa.gov>
>> wrote:
>>
>>> On Fri, Mar 17, 2017 at 4:37 AM, Jaime Fernández del Río <
>>> jaime.f...@gmail.com> wrote:
>>>
>>>>
>>>>- many people that use numpy in their daily work don't know what
>>>>strides are, this was a BIG surprise for me.
>>>>
>>>> I'm not surprised at all. To start with, the majority of users are
>> self-taught programmers that never used something lower level than Python
>> or Matlab. Even talking to them about memory layout presents challenges.
>>
>>
>>>
>>>>-
>>>>
>>>> Based on that experience, I was thinking that maybe a good topic for a
>>>> workshop would be NumPy's memory model: views, reshaping, strides, some
>>>> hints of buffering in the iterator...
>>>>
>>>
>> This material has been used multiple times in EuroScipy tutorials and may
>> be of use: http://www.scipy-lectures.org/advanced/advanced_numpy/index.
>> html
>>
>> Ralf
>>
>>
>>
>>> I think this is a great idea. In fact, when I do an intro to numpy, I
>>> spend a bit of time on those issues, 'cause I think it's key to "Getting"
>>> numpy, and not something that people end up learning on their own from
>>> tutorials, etc. However, in my  case, I try to jam it into a low-level
>>> intro, and I think that fails :-(
>>>
>>> So doing it on it's own with the assumption that participant already
>>> know the basics of the high level python interface is a great idea.
>>>
>>> Maybe a "advanced" numpy tutorial for SciPy 2017 in Austin also???
>>>
>>> Here is my last talk -- maybe it'll be helpful.
>>>
>>> http://uwpce-pythoncert.github.io/SystemDevelopment/scipy.html#scipy
>>>
>>> the strides stuff is covered in a notebook here:
>>>
>>> https://github.com/UWPCE-PythonCert/SystemDevelopment/blob/m
>>> aster/Examples/numpy/stride_tricks.ipynb
>>>
>>> other notebooks here:
>>>
>>> https://github.com/UWPCE-PythonCert/SystemDevelopment/tree/m
>>> aster/Examples/numpy
>>>
>>> and the source for the whole thing is here:
>>>
>>> https://github.com/UWPCE-PythonCert/SystemDevelopment/blob/m
>>> aster/slides_sources/source/scipy.rst
>>>
>>>
>>> All licensed under: Creative Commons Attribution-ShareAlike -- so please
>>> use anything you find useful.
>>>
>>> -CHB
>>>
>>>
>>>
>>> And Julian's temporary work lends itself to a very nice talk, more on
>>>> Python internals than on NumPy, but it's a very cool subject nonetheless.
>>>>
>>>> So my thinking is that I am going to propose those two, as a workshop
>>>> and a talk. Thoughts?
>>>>
>>>> Jaime
>>>>
>>>> On Thu, Mar 9, 2017 at 8:29 PM, Sebastian Berg <
>>>> sebast...@sipsolutions.net> wrote:
>>>>
>>>>> On Thu, 2017-03-09 at 15:45 +0100, Jaime Fernández del Río wrote:
>>>>> > There will be a PyData conference in Barcelona this May:
>>>>> >
>>>>> > http://pydata.org/barcelona2017/
>>>>> >
>>>>> > I am planning on attending, and was thinking of maybe proposing to
>>>>> > organize a numpy-themed workshop or tutorial.
>>>>> >
>>>>> > My personal inclination would be to look at some advanced topic that
>>>>> > I know well, like writing gufuncs in Cython, but wouldn'

Re: [Numpy-discussion] Fortran order in recarray.

[Chris Barker]

On Tue, Feb 21, 2017 at 3:05 PM, Alex Rogozhnikov <
alex.rogozhni...@yandex.ru> wrote:

> a question about numpy.recarray:
> There is a parameter order in constructor https://docs.scipy.org/doc/
> numpy-1.10.1/reference/generated/numpy.recarray.html, but it seems to
> have no effect:
> x = numpy.recarray(dtype=[('a', int), ('b', float)], shape=[1000],
> order='C')
>

you are creating a 1D array here -- there is no difference between Fortran
and C order for a 1D array. For 2D:

In [2]: x = numpy.recarray(dtype=[('a', int), ('b', float)], shape=[10,10],
order='C')


In [3]: x.strides
Out[3]: (160, 16)


In [4]: y = numpy.recarray(dtype=[('a', int), ('b', float)], shape=[10,10],
order='F')


In [5]: y.strides
Out[5]: (16, 160)

note the easier way to get the strides, too :-)

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] From Python to Numpy

[Chris Barker - NOAA Federal]

> It seems a generalized ufunc "all_equal" with signature (i),(i)->() and short 
> circuit logic once the first non equal element is encountered would be an 
> important performance improvement.

How does array_equal() perform?

-CHB
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Casting to np.byte before clearing values

[Chris Barker]

On Mon, Dec 26, 2016 at 1:34 AM, Nicolas P. Rougier <
nicolas.roug...@inria.fr> wrote:

>
> I'm trying to understand why viewing an array as bytes before clearing
> makes the whole operation faster.
> I imagine there is some kind of special treatment for byte arrays but I've
> no clue.
>

I notice that the code is simply setting a value using broadcasting -- I
don't think there is anything special about zero in that case. But your
subject refers to "clearing" an array.

So I wonder if you have a use case where the performance difference
matters, in which case _maybe_ it would be worth having a ndarray.zero()
method that efficiently zeros out an array.

Actually, there is ndarray.fill():

In [7]: %timeit Z_float[...] = 0

1000 loops, best of 3: 380 µs per loop


In [8]: %timeit Z_float.view(np.byte)[...] = 0

1000 loops, best of 3: 271 µs per loop


In [9]: %timeit Z_float.fill(0)

1000 loops, best of 3: 363 µs per loop

which seems to take an insignificantly shorter time than assignment.
Probably because it's doing exactly the same loop.

whereas a .zero() could use a memset, like it does with bytes.

can't say I have a use-case that would justify this, though.

-CHB






>
> # Native float
> Z_float = np.ones(100, float)
> Z_int   = np.ones(100, int)
>
> %timeit Z_float[...] = 0
> 1000 loops, best of 3: 361 µs per loop
>
> %timeit Z_int[...] = 0
> 1000 loops, best of 3: 366 µs per loop
>
> %timeit Z_float.view(np.byte)[...] = 0
> 1000 loops, best of 3: 267 µs per loop
>
> %timeit Z_int.view(np.byte)[...] = 0
> 1000 loops, best of 3: 266 µs per loop
>
>
> Nicolas
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] From Python to Numpy

[Chris Barker]

Nicolas,

>From a quick glance, this looks really wonderful! I intend to point my
students that are interested in numpy to it.

-CHB


On Thu, Dec 22, 2016 at 8:44 AM, Nicolas P. Rougier <
nicolas.roug...@inria.fr> wrote:

>
> Dear all,
>
> I've just put online a (kind of) book on Numpy and more specifically about
> vectorization methods. It's not yet finished, has not been reviewed and
> it's a bit rough around the edges. But I think there are some material that
> can be interesting. I'm specifically happy with the boids example that show
> a nice combination of numpy and matplotlib strengths.
>
> Book is online at: http://www.labri.fr/perso/
> nrougier/from-python-to-numpy/
> Sources are available at: https://github.com/rougier/from-python-to-numpy
>
>
> Comments/questions/fixes/ideas are of course welcome.
>
>
> Nicolas
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Ensuring one can operate on array-like argument in place

[Chris Barker]

I tend to use ndarray.copy() in python code -- no reason you couldn't do
the same in Cython.

If you want to take any array-like object that may not have a copy()
method, you could call asanyarray() first:

-CHB





On Sat, Nov 12, 2016 at 9:00 AM, Pavlyk, Oleksandr <
oleksandr.pav...@intel.com> wrote:

> Hi,
>
>
>
> In my Cython code a function processes it argument x as follows:
>
>
>
> x_arr = PyArray_CheckFromAny(
>
>   x, NULL, 0, 0,
>
>   cnp.NPY_ELEMENTSTRIDES | cnp.NPY_ENSUREARRAY |
> cnp.NPY_NOTSWAPPED, NULL)
>
>
>
> if x_arr is not x:
>
>in_place = 1  # a copy was made, so we can work in place.
>
>
>
> The logic is of the last line turns out to be incorrect, because the input
> x can be a class with an array interface:
>
>
>
> class FakeArray(object):
>
> def __init__(self, data):
>
> self._data = data
>
> self.__array_interface__ = data.__array_interface__
>
>
>
> Feeding my function FakeArray(xx),  x_arr will point into the content of
> xx, resulting in unwarranted content
> overwrite of xx.
>
>
>
> I am trying to replace that condition with
>
>
>
> if x_arr is not x and cnp.PyArray_Check(x):
>
># a copy was made, so we can work in place.
>
>in_place = 1 if cnp.PyArray_CHKFLAGS(x_arr, cnp.NPY_WRITEABLE) else
> 0
>
>
>
> I am wondering if I perhaps overlooked some case.
>
>
>
> Thank you,
>
> Sasha
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] array comprehension

[Chris Barker]

On Fri, Nov 4, 2016 at 10:36 AM, Nathaniel Smith  wrote:

> On Nov 4, 2016 10:32 AM, "Stephan Hoyer"  wrote:
> > fromiter dynamically resizes a NumPy array, like a Python list, except
> with a growth factor of 1.5
>


> Oh, right, and the dtype argument is mandatory, which is what makes this
> possible.
>
Couldn't it determine the dtype from the first element, and then barf later
if an incompatible one shows up?

And then we could adapt this code to np.array() and get nice performance
with no extra functions to think about calling...

And off the top of my head, I can't think of why it couldn't be generalized
to the nd case as well.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NumPy 1.11.2 released

[Chris Barker]

I'm pleased to announce the release of Numpy 1.11.2. This release supports
> Python 2.6 - 2.7, and 3.2 - 3.5 and fixes bugs and regressions found in
> Numpy 1.11.1.  Wheels for Linux, Windows, and OSX can be found on PyPI.
> Sources are available on both PyPI and Sourceforge
> .
>

and on conda-forge:

https://anaconda.org/conda-forge/numpy

Hmm, not Windows (darn fortran an openblas!) -- but thanks for getting that
up fast!

And of course, thanks to all in the numpy community for getting this build
out.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] automatically avoiding temporary arrays

[Chris Barker]

On Mon, Oct 3, 2016 at 3:16 AM, Julian Taylor  wrote:

> the problem with this approach is that we don't really want numpy
> hogging on to hundreds of megabytes of memory by default so it would
> need to be a user option.
>

indeed -- but one could set an LRU cache to be very small (few items, not
small memory), and then it get used within expressions, but not hold on to
much outside of expressions.

However, is the allocation the only (Or even biggest) source of the
performance hit?

If you generate a temporary as a result of an operation, rather than doing
it in-place, that temporary needs to be allocated, but it also means that
an additional array needs to be pushed through the processor -- and that
can make a big performance difference too.

I"m not entirely sure how to profile this correctly, but this seems to
indicate that the allocation is cheap compared to the operations (for a
million--element array)

* Regular old temporary creation

In [24]: def f1(arr1, arr2):
...: result = arr1 + arr2
...: return result

In [26]: %timeit f1(arr1, arr2)
1000 loops, best of 3: 1.13 ms per loop

* Completely in-place, no allocation of an extra array

In [27]: def f2(arr1, arr2):
...: arr1 += arr2
...: return arr1

In [28]: %timeit f2(arr1, arr2)
1000 loops, best of 3: 755 µs per loop

So that's about 30% faster

* allocate a temporary that isn't used -- but should catch the creation cost

In [29]: def f3(arr1, arr2):
...: result = np.empty_like(arr1)
...: arr1 += arr2
...: return arr1

In [30]: % timeit f3(arr1, arr2)

1000 loops, best of 3: 756 µs per loop

only a µs slower!

Profiling is hard, and I'm not good at it, but this seems to indicate that
the allocation is cheap.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] String & unicode arrays vs text loading in python 3

[Chris Barker]

On Tue, Sep 13, 2016 at 11:05 AM, Lluís Vilanova 
wrote:

> Great, that's the type of info I wanted to get before going forward. I
> guess
> there's code relying on the binary representation of 'S' to do mmap's or
> access
> the array's raw contents. Is that right?


yes, there is a LOT of code, most of it third party, that relies on
particular binary representations of the numpy dtypes.

There is a fundamental semantic difference between a string and a byte
> array,
> that's the core of the problem.
>

well yes. but they were mingled in py2, and the 'S' dtype is essentially a
py2 string. But in py3, it maps more closely with bytes than string --
though yes, not exactly either :-(

Here's an alternative that only handles the repr.
>


> Whenever we repr an array using 'S', we can instead show a unicode in py3.
> That
> keeps the binary representation, but will always show the expected result
> to
> users, and it's only a handful of lines added to dump_data().
>

This would probably be more confusing than helpful -- if a 'S' object
converts to a bytes object, than it's repr should show that.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] String & unicode arrays vs text loading in python 3

[Chris Barker]

We had a big long discussion about this on this list a while back (maybe 2
yrs ago???) please search the archives to find it. Though I'm pretty sure
that we never did come to a conclusion. I think it stared with wanting
better support ofr unicode in loadtxt and the like, and ended up delving
into other encodings for the 'U' dtype, and maybe a single byte string
dtype (latin-1), or maybe a variable-size unicode object like Py3's, or...

However, it is absolutely a non-starter to change the binary representation
of the 'S' type in any version of numpy. Due to the legacy of py2 (and,
indeed, most computing environments) 'S' is a single byte string
representation. And the binary representation is often really key to numpy
use.
Period, end of story.

And that maps to a py2 string and py3 bytes object.

py2 does, of course, have a Unicode object as well. If you want your code
(and doctests, and ...) to be compatible, then you should probably go to
Unicode strings everywhere. py3 now supports the u'string' no-op literal to
make this easier.

(though I guess the __repr__ won't tack on that 'u', which is going to be a
problem for docstrings).

Note also that py3 has added more an more "string-like" support to the
bytes object, so it's not too bad to go bytes-only.

-CHB


On Tue, Sep 13, 2016 at 7:21 AM, Lluís Vilanova  wrote:

> Sebastian Berg writes:
>
> > On Di, 2016-09-13 at 15:02 +0200, Lluís Vilanova wrote:
> >> Hi! I'm giving a shot to issue #3184 [1], based on the observation
> >> that the
> >> string dtype ('S') under python 3 uses byte arrays instead of unicode
> >> (the only
> >> readable string type in python 3).
> >>
> >> This brings two major problems:
> >>
> >> * numpy code has to go through loops to open and read files as binary
> >> data to
> >>   load text into a bytes array, and does not play well with users
> >> providing
> >>   string (unicode) arguments
> >>
> >> * the repr of these arrays shows strings as b'text' instead of
> >> 'text', which
> >>   breaks doctests of software built on numpy
> >>
> >> What I'm trying to do is make dtypes 'S' and 'U' equivalnt
> >> (NPY_STRING and
> >> NPY_UNICODE).
> >>
> >> Now the question. Keeping 'S' and 'U' as separate dtypes (but same
> >> internal
> >> implementation) will provide the best backwards compatibility, but is
> >> more
> >> cumbersome to implement.
>
> > I am not sure how that can be possible. Those types are fundamentally
> > different in how they store their data. String types use one byte per
> > character, unicode types will use 4 bytes per character. You can maybe
> > default to unicode in more cases in python 3, but you cannot make them
> > identical internally.
>
> BTW, by identical I mean having two externally visible types, but a common
> implementation in python 3 (that of NPY_UNICODE).
>
> The as-sane but not backwards-compatible option (I'm asking if this is
> acceptable) is to only retain 'S' (NPY_STRING), but with the NPY_UNICODE
> implementation, and making 'U' (and np.unicode_) and alias for 'S' (and
> np.string_).
>
>
> Cheers,
>   Lluis
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] State-of-the-art to use a C/C++ library from Python

[Chris Barker]

On Fri, Sep 2, 2016 at 1:16 AM, Peter Creasey  wrote:

> > I'm not quite sure which approach is state-of-the-art as of 2016. How
> would
> > you do it if you had to make a C/C++ library available in Python right
> now?
> >
> > In my case, I have a C library with some scientific functions on matrices
> > and vectors. You will typically call a few functions to configure the
> > computation, then hand over some pointers to existing buffers containing
> > vector data, then start the computation, and finally read back the data.
> > The library also can use MPI to parallelize.
>

Cython works really well for this.

ctypes is a better option if you have a "black box" shared lib you want a
call a couple functions in.

Cython works better if you want to write a little "thicker" wrapper around
youe C code -- i.e. it may do a scalar computation, and you want to apply
it to an entire numpy array, at C speed.

Either would work in this case, But I like Cyton better, as long as I don't
have compilation issues.

-Chris




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NumPy in PyPy

[Chris Barker]

>
> On Fri, Aug 5, 2016 at 3:42 AM, Papa, Florin 
>> wrote:
>>
>>>  Does anyone have knowledge of real life workloads that use NumPy and
>>> cannot be run using PyPy?
>>>
>>>
>>>
>>> We are also interested in creating a repository with relevant benchmarks
>>> for real world usage of NumPy,
>>>
>>
We have a numpy --  heavy app. bu tit, like many others, I'm sure, also
relies heavily on Cython-wrapped C++ code, as well as pure Cython
extensions.

As well as many other packages that are also wrappers around C libs, Cython
-optimized, etc.

I've never tried to run it under PyPy I've always assumed it's a
non-starter.

Is there any hope?


If you are curious:

https://github.com/NOAA-ORR-ERD/PyGnome

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Custom Dtype/Units discussion

[Chris Barker]

On Sun, Jul 10, 2016 at 8:12 PM, Nathan Goldbaum 
wrote:

>
> Maybe this can be an informal BOF session?
>

or  maybe a formal BoF? after all, how formal do they get?

Anyway, it was my understanding that we really needed to do some
significant refactoring of how numpy deals with dtypes in order to do this
kind of thing cleanly -- so where has that gone since last year?

Maybe this conversation should be about how to build a more flexible dtype
system generally, rather than specifically about unit support. (though unit
support is a great use-case to focus on)

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Is numpy.test() supposed to be multithreaded?

[Chris Barker - NOAA Federal]

> Now the user is writing back to say, "my test code is fast now, but
> numpy.test() is still about three times slower than  don't manage>".  When I watch htop as numpy.test() executes, sure enough,
> it's using one core
>

* if numpy.test() is supposed to be using multiple cores, why isn't it,
> when we've established with other test code that it's now using multiple
> cores?
>

Some numpy.linalg functions (like np.dot) will be using multiple cores, but
np.linalg.test() takes only ~1% of the time of the full test suite.
Everything else will be running single core. So your observations are not
surprising.


Though why it would run slower on one box than another comparable box is a
mystery...

-CHB
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Accelerate or OpenBLAS for numpy / scipy wheels?

[Chris Barker]

On Tue, Jun 28, 2016 at 8:15 AM, Matthew Brett 
wrote:

> > dropping the dual arch stuff is probably the way to go - the 32-bit
> builds
> > make very little sense these days.
>
> Yes, that's true, but as you know, the OSX system and Python.org
> Pythons are still dual arch, so technically a matching wheel should
> also be dual arch.


but as they say, practicality beat purity...

It's not clear yet whether 3.6 will be built dual arch at this point, but
in any case, no one is going to go back and change the builds on 2.7 or 3.4
or 3.5 

But that doesn't mean we necessarily need to support dual arch downstream.
Personally, I"d drop it and see if anyone screams.

Though it's actually a bit tricky, at least with my knowledge to build a 64
bit only extension against the dual-arch build. At least the only way I
figured out was to hack the install. ( I did this a while back when I
needed a 32bit-only build -- ironic?)


> This doesn't really matter too much imho, we have to support Accelerate
> > either way.
>

do we? -- so if we go OpenBlas, and someone want to do a simple build from
source, what happens? Do they get accelerate? or would we ship OpenBlas
source itself? or would they need to install OpenBlas some other way?

>> >> Faster to fix bugs with good support from main developer.  No
> >> >> multiprocessing crashes for Python 2.7.
>

this seems to be the compelling one.

How does the performance compare?

-CHB




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: numpy.random.random_seed

[Chris Barker]

On Sun, May 22, 2016 at 2:35 AM, Robert Kern  wrote:
>
> Well, I mean, engineers want lots of things. I suspect that most engineers
> *really* just want to call `numpy.random.seed(8675309)` at the start and
> never explicitly pass around separate streams. There's an upside to that in
> terms of code simplicity. There are also significant limitations and
> constraints. Ultimately, the upside against the alternative of passing
> around RandomState objects is usually overweighed by the limitations, so
> best practice is to pass around RandomState objects.
>

Could we do something like the logging module, and have numpy.random
"manage" a bunch of stream objects for you -- so you could get the default
single stream easily, and also get access to specific streams without
needing to pass around the objects?

That would allow folks to start off writing code the "easy" way -- then
make it easier to refactor when they realize they need multiple independent
streams.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Scipy 2016 attending

[Chris Barker]

I'll be there.

-CHB


On Wed, May 18, 2016 at 2:09 PM, Charles R Harris  wrote:

> Hi All,
>
> Out of curiosity, who all here intends to be at Scipy 2016?
>
> Chuck
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: numpy.random.random_seed

[Chris Barker]

>
> > ...anyway, the real reason I'm a bit grumpy is because there are solid
> > engineering reasons why users *want* this API,
>

Honestly, I am lost in the math -- but like any good engineer, I want to
accomplish something anyway :-) I trust you guys to get this right -- or at
least document what's "wrong" with it.

But, if I'm reading the use case that started all this correctly, it
closely matches my use-case. That is, I have a complex model with multiple
independent "random" processes. And we want to be able to re-produce
EXACTLY simulations -- our users get confused when the results are
"different" even if in a statistically insignificant way.

At the moment we are using one RNG, with one seed for everything. So we get
reproducible results, but if one thing is changed, then the entire
simulation is different -- which is OK, but it would be nicer to have each
process using its own RNG stream with it's own seed. However, it matters
not one whit if those seeds are independent -- the processes are different,
you'd never notice if they were using the same PRN stream -- because they
are used differently. So a "fairly low probability of a clash" would be
totally fine.

Granted, in a Monte Carlo simulation, it could be disastrous... :-)

I guess the point is -- do something reasonable, and document its
limitations, and we're all fine :-)

And thanks for giving your attention to this.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ndarray.T2 for 2D transpose

[Chris Barker]

On Fri, Apr 8, 2016 at 4:37 PM, Ian Henriksen <
insertinterestingnameh...@gmail.com> wrote:


> If we introduced the T2 syntax, this would be valid:
>
> a @ b.T2
>
> It makes the intent much clearer.
>

would:

a @ colvector(b)

work too? or is T2  generalized to more than one column? (though I suppose
colvector() could support more than one column also -- weird though that
might be.)

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ndarray.T2 for 2D transpose

[Chris Barker]

On Fri, Apr 8, 2016 at 9:59 AM, Charles R Harris 
wrote:

> Apropos column/row vectors, I've toyed a bit with the idea of adding a
> flag to numpy arrays to indicate that the last index is one or the other,
> and maybe neither.
>

I don't follow this. wouldn't it ony be an issue for 1D arrays, rather than
the "last index". Or maybe I'm totally missing the point.

But anyway, are (N,1) and (1, N) arrays insufficient for representing
column and row vectors for some reason? If not -- then we have a way to
express a column or row vector, we just need an easier and more obvious way
to create them.

*maybe* we could have actual column and row vector classes -- they would BE
regular arrays, with (1,N) or (N,1) dimensions, and act the same in every
way except their __repr__. and we're provide handy factor functions for
them.

These were needed to complete the old Matrix class -- which is no longer
needed now that we have @ (i.e. a 2D array IS a matrix)

Note: this is not very well thought out!

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ndarray.T2 for 2D transpose

[Chris Barker]

On Thu, Apr 7, 2016 at 11:31 AM,  wrote:

> maybe a warning?
>>
>
> AFAIR, there is a lot of code that works correctly with .T being a noop
> for 1D
> e.g. covariance matrix/inner product x.T dot y as mentioned before.
>

oh well, then no warning, either.


> write unit tests with non square 2d arrays and the exception / test error
> shows up fast.
>

Guido wrote a note to python-ideas about the conflict between the use cases
of "scripting" and "large system development" -- he urged both camps, to
respect and listen to each other.

I think this is very much a "scripters" issue -- so no unit tests, etc

For my part, I STILL have to kick myself once in a while for using square
arrays in testing/exploration!

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ndarray.T2 for 2D transpose

[Chris Barker]

On Thu, Apr 7, 2016 at 10:00 AM, Ian Henriksen <
insertinterestingnameh...@gmail.com> wrote:

> Here's another example that I've seen catch people now and again.
>
> A = np.random.rand(100, 100)
> b =  np.random.rand(10)
> A * b.T
>

typo? that was supposed to be

b =  np.random.rand(100). yes?

This is exactly what someone else referred to as the expectations of
someone that comes from MATLAB, and doesn't yet "get" that 1D arrays are 1D
arrays.

All of this is EXACTLY the motivation for the matric class -- which never
took off, and was never complete (it needed a row and column vector
implementation, if you ask me. But Ithikn the reason it didn't take off is
that it really isn't that useful, but is different enough from regular
arrays to be a greater source of confusion. And it was decided that all
people REALLY wanted was an obviou sway to get matric multiply, which we
now have with @.

So this discussion brings up that we also need an easy an obvious way to
make a column vector --

maybe:

np.col_vector(arr)

which would be a synonym for np.reshape(arr, (-1,1))

would that make anyone happy?

NOTE: having transposing a 1D array raise an exception would help remove a
lot  of the confusion, but it may be too late for that


In this case the user pretty clearly meant to be broadcasting along the
> rows of A
> rather than along the columns, but the code fails silently.
>

hence the exception idea

maybe a warning?

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ndarray.T2 for 2D transpose

[Chris Barker]

On Thu, Apr 7, 2016 at 8:13 AM, Todd  wrote:

> First you need to turn a into a 2D array.  I can think of 10 ways to do
> this off the top of my head, and there may be more:
>
> snip

Basically, my argument here is the same as the argument from pep465 for the
> inclusion of the @ operator:
>
> https://www.python.org/dev/peps/pep-0465/#transparent-syntax-is-especially-crucial-for-non-expert-programmers
>
> I think is this all a good argument for a clean and obvious way to make a
column vector, but I don't think overloading transpose is the way to do
that.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ndarray.T2 for 2D transpose

[Chris Barker]

On Wed, Apr 6, 2016 at 10:47 AM, Todd  wrote:

>
> I think that cat is already out of the bag.  As long as you can do matrix
> multiplication on arrays using the @ operator, I think they aren't really
> "pure" anymore.
>

not really -- you still need to use arrays that are the "correct" shape.
Ideally, a row vector is (1, N) and a column vector is (N,1). Though I know
there are places that a 1-D array is treated as a column vector.

>
>
>> BTW, if transposing a (N,) array gives you a (N,1) array, what does
>> transposing a (N,1) array give you?
>>
>> (1,N) or (N,) ?
>>
>
> My suggestion is that this explicitly increases the number of dimensions
> to at least 2.  The result will always have at least 2 dimensions.  So 0D
> -> 2D, 1D -> 2D, 2D -> 2D, 3D -> 3D, 4D -> 4D, etc.  So this would be
> equivalent to the existing `atleast_2d` function.
>


my point is that for 2D arrays: arr.T.T == arr, but in this case, we would
be making a one way street:

when you transpose a 1D array, you treat it as a row vector, and return a
"column vector" -- a (N,1) array.

But when you transpose a "column vector" to get a row vector, you get a
(1,N) array, not a (N) array.

So I think we need to either have proper row and column vectors (to go with
matrices) or require people to create the appropriate 2D arrays.

Perhaps there should be an easier more obvious way to spell "make this a
column vector", but I don't think .T is it.

Though arr.shape = (-1,1) has always worked fine for me.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Multidimension array access in C via Python API

[Chris Barker]

On Tue, Apr 5, 2016 at 9:48 AM, mpc  wrote:

> The idea is that I want to thin a large 2D buffer of x,y,z points to a
> given
> resolution by dividing the data into equal sized "cubes" (i.e. resolution
> is
> number of cubes along each axis) and averaging the points inside each cube
> (if any).
>

are the original x,y,z points aranged along a nice even grid? or
arbitrarily spaced?

if the former, I have Cython code that does that :-) I could dig it up,
haven't used it in a while. or scikit.image might have something.

If the latter, then Ben is right -- you NEED a spatial index --
scipy.spatial.kdtree will probably do what you want, though it would be
easier to use a sphere to average over than a cube.

Also, maybe Kernel Density Estimation could help here

https://jakevdp.github.io/blog/2013/12/01/kernel-density-estimation/

Otherwise, you could use Cython to write a non-vectorized version of your
below code -- it would be order NM where N is the number of "cubes" and M
is the number of original points. I think, but would be a lot faster than
the pure python.

-CHB

Here is where you would do the cython:
while x_idx < max_x:

> y_idx = min_y
> while y_idx < max_y:
> z_idx = min_z
> while z_idx < max_z:
> inside_block_points = np.where((x_buffer >= x_idx) &
>  (x_buffer <=
> x_idx + x_block) &
>  (y_buffer >=
> y_idx) &
>  (y_buffer <=
> y_idx + y_block) &
>  (z_buffer >=
> z_idx) &
>  (z_buffer <=
> z_idx + z_block))
>

instead of where, you could loop through all your points and find the ones
inside your extents.

though now that I think about it -- you are mapping arbitrary points to a
regular grid, so you only need to go through the points once, assigning
each one to a bin, and then compute the average in each bin.

Is this almost a histogram?

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Windows wheels, built, but should we deploy?

[Chris Barker]

+1 -- thanks for doing all this work.

There is a HUGE amount you can do with numpy that doesn't give a whit about
how fast .dot() et all are. If you really do need that to be fast as
possible, you can pug in a faster build later.

This is great.

Just as one example -- I teach a general python class every year --I do
only one session on numpy/scipy. If I can expect my students to be able to
simply pip install the core scipy stack, this will be SO much easier.

-CHB


On Thu, Mar 3, 2016 at 8:42 PM, Matthew Brett 
wrote:

> Hi,
>
> Summary:
>
> I propose that we upload Windows wheels to pypi.  The wheels are
> likely to be stable and relatively easy to maintain, but will have
> slower performance than other versions of numpy linked against faster
> BLAS / LAPACK libraries.
>
> Background:
>
> There's a long discussion going on at issue github #5479 [1], where
> the old problem of Windows wheels for numpy came up.
>
> For those of you not following this issue, the current situation for
> community-built numpy Windows binaries is dire:
>
> * We have not so far provided windows wheels on pypi, so `pip install
> numpy` on Windows will bring you a world of pain;
> * Until recently we did provide .exe "superpack" installers on
> sourceforge, but these became increasingly difficult to build and we
> gave up building them as of the latest (1.10.4) release.
>
> Despite this, popularity of Windows wheels on pypi is high.   A few
> weeks ago, Donald Stufft ran a query for the binary wheels most often
> downloaded from pypi, for any platform [2] . The top five most
> downloaded were (n_downloads, name):
>
> 6646,
> numpy-1.10.4-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
> 5445, cryptography-1.2.1-cp27-none-win_amd64.whl
> 5243, matplotlib-1.4.0-cp34-none-win32.whl
> 5241, scikit_learn-0.15.1-cp34-none-win32.whl
> 4573, pandas-0.17.1-cp27-none-win_amd64.whl
>
> So a) the OSX numpy wheel is very popular and b) despite the fact that
> we don't provide a numpy wheel for Windows, matplotlib, sckit_learn
> and pandas, that depend on numpy, are the 3rd, 4th and 5th most
> downloaded wheels as of a few weeks ago.
>
> So, there seems to be a large appetite for numpy wheels.
>
> Current proposal:
>
> I have now built numpy wheels, using the ATLAS blas / lapack library -
> the build is automatic and reproducible [3].
>
> I chose ATLAS to build against, rather than, say OpenBLAS, because
> we've had some significant worries in the past about the reliability
> of OpenBLAS, and I thought it better to err on the side of
> correctness.
>
> However, these builds are relatively slow for matrix multiply and
> other linear algebra routines compared numpy built against OpenBLAS or
> MKL (which we cannot use because of its license) [4].   In my very
> crude array test of a dot product and matrix inversion, the ATLAS
> wheels were 2-3 times slower than MKL.  Other benchmarks on Julia
> found about the same result for ATLAS vs OpenBLAS on 32-bit bit, but a
> much bigger difference on 64-bit (for an earlier version of ATLAS than
> we are currently using) [5].
>
> So, our numpy wheels likely to be stable and give correct results, but
> will be somewhat slow for linear algebra.
>
> I propose that we upload these ATLAS wheels to pypi.  The upside is
> that this gives our Windows users a much better experience with pip,
> and allows other developers to build Windows wheels that depend on
> numpy.  The downside is that these will not be optimized for
> performance on modern processors.  In order to signal that, I propose
> adding the following text to the numpy pypi front page:
>
> ```
> All numpy wheels distributed from pypi are BSD licensed.
>
> Windows wheels are linked against the ATLAS BLAS / LAPACK library,
> restricted to SSE2 instructions, so may not give optimal linear
> algebra performance for your machine. See
> http://docs.scipy.org/doc/numpy/user/install.html for alternatives.
> ```
>
> In a way this is very similar to our previous situation, in that the
> superpack installers also used ATLAS - in fact an older version of
> ATLAS.
>
> Once we are up and running with numpy wheels, we can consider whether
> we should switch to other BLAS libraries, such as OpenBLAS or BLIS
> (see [6]).
>
> I'm posting here hoping for your feedback...
>
> Cheers,
>
> Matthew
>
>
> [1] https://github.com/numpy/numpy/issues/5479
> [2] https://gist.github.com/dstufft/1dda9a9f87ee7121e0ee
> [3] https://ci.appveyor.com/project/matthew-brett/np-wheel-builder
> [4] http://mingwpy.github.io/blas_lapack.html#intel-math-kernel-library
> [5] https://github.com/numpy/numpy/issues/5479#issuecomment-185033668
> [6] https://github.com/numpy/numpy/issues/7372
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Re: [Numpy-discussion] DyND 0.7.1 Release

[Chris Barker - NOAA Federal]

> The DyND team would be happy to answer any questions people have about DyND, 
> like "what is working and what is not" or "what do we still need to do to hit 
> DyND 1.0".

OK, how about:

How does the performance. I'd DyND compare to Numpy for the core
functionality they both support?

- CHB
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] proposal: new logspace without the log in the argument

[Chris Barker]

On Thu, Feb 18, 2016 at 2:19 PM, Alan Isaac  wrote:

> Would such a new function correct the apparent mistake (?) of
> `linspace` including the endpoint by default?
> Or is the current API justified by its Matlab origins?
>

I don't think so -- we don't need no stinkin' Matlab !

But I LIKE including the endpoint in the sequence -- for the common use
cases, it's often what you want, and if it didn't include the end point but
you did want that, it would get pretty ugly to figure out how to get what
you want.

On the other hand, if I had it to do over, I would have the count specify
the number of intervals, rather than the number of items. A common cae may
be: values from zero to 10 (inclusive), and I want ten steps:

In [19]: np.linspace(0, 10, 10)
Out[19]:

array([  0.,   1.,   2.,   3.,
 4.,   5.5556,   6.6667,   7.7778,
 8.8889,  10.])

HUH? I was expecting [0,1,2,3 ] (OK, not me, this isn't my first
Rodeo), so now I need to do:
In [20]: np.linspace(0, 10, 11)
Out[20]: array([  0.,   1.,   2.,   3.,   4.,   5.,   6.,   7.,   8.,   9.,
 10.])

This gets uglier if I know what "delta" I want:

In [21]: start = 0.0; end = 9.0; delta = 1.0
In [24]: np.linspace(start, end, (end-start)/delta)
Out[24]: array([ 0.   ,  1.125,  2.25 ,  3.375,  4.5  ,  5.625,  6.75 ,
 7.875,  9.   ])

oops!

In [25]: np.linspace(start, end, (end-start)/delta + 1)
Out[25]: array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9.])

But in any case, there is no changing it now.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Suggestion: special-case np.array(range(...)) to be faster

[Chris Barker]

On Thu, Feb 18, 2016 at 10:15 AM, Antony Lee <antony@berkeley.edu>
wrote:

> Mostly so that there is no performance lost when someone passes range(...)
> instead of np.arange(...).  At least I had never realized that one is much
> faster than the other and always just passed range() as a convenience.
>

Well,  pretty much everything in numpy is faster if you use the numpy array
version rather than plain python -- this hardly seems like the extra code
would be worth it.

numpy's array() constructor can (and should) take an arbitrary iterable.

It does make some sense that you we might want to special case iterators,
as you don't want to loop through them too many times, which is what
np.fromiter() is for.

and _maybe_ it would be worth special casing python lists, as you can
access items faster, and they are really, really common (or has this
already been done?), but special casing range() is getting silly. And it
might be hard to do. At the C level I suppose you could actually know what
the parameters and state of the range object are and create an array
directly from that -- but that's what arange is for...

-CHB



> 2016-02-17 10:50 GMT-08:00 Chris Barker <chris.bar...@noaa.gov>:
>
>> On Sun, Feb 14, 2016 at 11:41 PM, Antony Lee <antony@berkeley.edu>
>> wrote:
>>
>>> So how can np.array(range(...)) even work?
>>>
>>
>> range()  (in py3) is not a generator, nor is is a iterator. it is a range
>> object, which is lazily evaluated, and satisfies both the iterator protocol
>> and the sequence protocol (at least most of it:
>>
>> In [*1*]: r = range(10)
>>
>>
>> In [*2*]: r[3]
>>
>> Out[*2*]: 3
>>
>>
>> In [*3*]: len(r)
>>
>> Out[*3*]: 10
>>
>>
>> In [*4*]: type(r)
>>
>> Out[*4*]: range
>>
>> In [*9*]: isinstance(r, collections.abc.Sequence)
>>
>> Out[*9*]: True
>>
>> In [*10*]: l = list()
>>
>> In [*11*]: isinstance(l, collections.abc.Sequence)
>>
>> Out[*11*]: True
>>
>> In [*12*]: isinstance(r, collections.abc.Iterable)
>>
>> Out[*12*]: True
>> I'm still totally confused as to why we'd need to special-case range when
>> we have arange().
>>
>> -CHB
>>
>>
>>
>> --
>>
>> Christopher Barker, Ph.D.
>> Oceanographer
>>
>> Emergency Response Division
>> NOAA/NOS/OR(206) 526-6959   voice
>> 7600 Sand Point Way NE   (206) 526-6329   fax
>> Seattle, WA  98115   (206) 526-6317   main reception
>>
>> chris.bar...@noaa.gov
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] GSoC?

[Chris Barker]

Apparetnly, NumFocus is applyign to be a GSoC Umbrella org as well:

https://github.com/numfocus/gsoc

Not sure why one might choose NumFocus vs PSF...

-Chris


On Wed, Feb 17, 2016 at 6:05 AM, Bryan Van de Ven <bry...@continuum.io>
wrote:

> [This is a complete tangent, and so I apologize in advance.]
>
> We are considering applying to GSOC for Bokeh. However, I have zero
> experience with GSOC, but non-zero questions (e.g. go it alone, vs apply
> through PSF... I think?) If anyone with experience from the mentoring
> organization side of things wouldn't mind a quick chat (or a few emails) to
> answer questions, share your experience, or offer advice, please drop me a
> line directly.
>
> Thanks,
>
> Bryan
>
>
>
> > On Feb 17, 2016, at 1:14 AM, Stephan Hoyer <sho...@gmail.com> wrote:
> >
> > On Wed, Feb 10, 2016 at 4:22 PM, Chris Barker <chris.bar...@noaa.gov>
> wrote:
> > We might consider adding "improve duck typing for numpy arrays"
> >
> > care to elaborate on that one?
> >
> > I know it come up on here that it would be good to have some code in
> numpy itself that made it easier to make array-like objects (I.e. do
> indexing the same way) Is that what you mean?
> >
> > I was thinking particularly of improving the compatibility of numpy
> functions (e.g., concatenate) with non-numpy array-like objects, but now
> that you mention it utilities to make it easier to make array-like objects
> could also be a good thing.
> >
> > In any case, I've now elaborated on my thought into a full project idea
> on the Wiki:
> >
> https://github.com/scipy/scipy/wiki/GSoC-2016-project-ideas#improved-duck-typing-support-for-n-dimensional-arrays
> >
> > Arguably, this might be too difficult for most GSoC students -- the API
> design questions here are quite contentious. But given that "Pythonic
> dtypes" is up there as a GSoC proposal still it's in good company.
> >
> > Cheers,
> > Stephan
> >
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Suggestion: special-case np.array(range(...)) to be faster

[Chris Barker]

On Sun, Feb 14, 2016 at 11:41 PM, Antony Lee 
wrote:

> So how can np.array(range(...)) even work?
>

range()  (in py3) is not a generator, nor is is a iterator. it is a range
object, which is lazily evaluated, and satisfies both the iterator protocol
and the sequence protocol (at least most of it:

In [*1*]: r = range(10)


In [*2*]: r[3]

Out[*2*]: 3


In [*3*]: len(r)

Out[*3*]: 10


In [*4*]: type(r)

Out[*4*]: range

In [*9*]: isinstance(r, collections.abc.Sequence)

Out[*9*]: True

In [*10*]: l = list()

In [*11*]: isinstance(l, collections.abc.Sequence)

Out[*11*]: True

In [*12*]: isinstance(r, collections.abc.Iterable)

Out[*12*]: True
I'm still totally confused as to why we'd need to special-case range when
we have arange().

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] GSoC?

[Chris Barker]

Thanks Ralf,

Note that we have always done a combined numpy/scipy ideas page and
> submission. For really good students numpy may be the right challenge, but
> in general scipy is easier to get started on.
>

yup -- good idea. Is there a page ready to go, or do we need to get one up?
(I don't even know where to put it...)


> Under the PSF umbrella has always worked very well, both in terms of
> communication quality and of getting the amount of slots we wanted, so yes.
>

hmm, looking here:

https://wiki.python.org/moin/SummerOfCode/2016#Sub-orgs

it seems it's time to get started. and I _think_ our ideas page can go on
that Wiki.


> Are you maybe interested in co-organizing or mentoring Chris? Updating the
> ideas page, proposal reviewing and interviewing students via video calls
> can be time-consuming, and mentoring definitely is, so the more the merrier.
>

I would love to help -- though I don't think I can commit to being a
full-on mentor.

If we get a couple people to agree to mentor, then we can get ourselves
setup up with the PSF.

-Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] GSoC?

[Chris Barker]

>
> We might consider adding "improve duck typing for numpy arrays"
>

care to elaborate on that one?

I know it come up on here that it would be good to have some code in numpy
itself that made it easier to make array-like objects (I.e. do indexing the
same way) Is that what you mean?

-CHB




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy 1.11.0b2 released

[Chris Barker]

On Sat, Feb 6, 2016 at 4:11 PM, Michael Sarahan  wrote:

> Chris,
>
> Both conda-build-all and obvious-ci are excellent projects, and we'll
> leverage them where we can (particularly conda-build-all).  Obvious CI and
> conda-smithy are in a slightly different space, as we want to use our own
> anaconda.org build service, rather than write scripts to run on other CI
> services.
>

I don't think conda-build-all or, for that matter, conda-smithy are fixed
to any particular CI server. But anyway, the anaconda.org build service
looks nice -- I'll need to give that a try. I've actually been building
everything on my own machines anyway so far.


> As I see it, the single, massive recipe repo that is conda-recipes has
> been a disadvantage for a while in terms of complexity, but now may be an
> advantage in terms of building downstream packages (how else would
> dependency get resolved?)
>

yup -- but the other issue is that conda-recipes didn't seem to be
maintained, really...


> The goal, much like ObviousCI, is to enable project maintainers to get
> their latest releases available in conda sooner, and to simplify the whole
> CI setup process.  We hope we can help each other rather than compete.
>

Great goal!

Thanks,

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] GSoC?

[Chris Barker]

ANyone interested in Google Summer of Code this year?

I think the real challenge is having folks with the time to really put into
mentoring, but if folks want to do it -- numpy could really benefit.

Maybe as a python.org sub-project?

https://wiki.python.org/moin/SummerOfCode/2016

Deadlines are approaching -- so I thought I'd ping the list and see if
folks are interested.

-Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] GSoC?

[Chris Barker]

As you can see in the timeline:

https://developers.google.com/open-source/gsoc/timeline

We are now in the stage where mentoring organizations are getting their act
together. So the question now is -- are there folks that want to mentor for
numpy projects? It can be rewarding, but it's a pretty big commitment as
well, and, I suppose depending on the project, would require some good
knowledge of the innards of numpy -- there are not a lot of those folks out
there that have that background.

So to students, I suggest you keep an eye out, and engage a little later on
in the process.

That being said, if you have a idea for a numpy improvement you'd like to
work on , by all means propose it and maybe you'll get a mentor or two
excited.

-CHB





On Mon, Feb 8, 2016 at 3:33 PM, SMRUTI RANJAN SAHOO <c99.smr...@gmail.com>
wrote:

> sir actually i am interested  very much . so can you help me about this or
> suggest some , so that i  can contribute .
>
>
>
>
> Thanks  & Regards,
> Smruti Ranjan Sahoo
>
> On Tue, Feb 9, 2016 at 1:58 AM, Chris Barker <chris.bar...@noaa.gov>
> wrote:
>
>> ANyone interested in Google Summer of Code this year?
>>
>> I think the real challenge is having folks with the time to really put
>> into mentoring, but if folks want to do it -- numpy could really benefit.
>>
>> Maybe as a python.org sub-project?
>>
>> https://wiki.python.org/moin/SummerOfCode/2016
>>
>> Deadlines are approaching -- so I thought I'd ping the list and see if
>> folks are interested.
>>
>> -Chris
>>
>>
>>
>> --
>>
>> Christopher Barker, Ph.D.
>> Oceanographer
>>
>> Emergency Response Division
>> NOAA/NOS/OR(206) 526-6959   voice
>> 7600 Sand Point Way NE   (206) 526-6329   fax
>> Seattle, WA  98115   (206) 526-6317   main reception
>>
>> chris.bar...@noaa.gov
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy 1.11.0b2 released

[Chris Barker]

On Fri, Feb 5, 2016 at 3:24 PM, Nathaniel Smith <n...@pobox.com> wrote:

> On Fri, Feb 5, 2016 at 1:16 PM, Chris Barker <chris.bar...@noaa.gov>
> wrote:
>


> >> > If we set up a numpy-testing conda channel, it could be used to cache
> >> > binary builds for all he versions of everything we want to test
> >> > against.
>   Anaconda doesn't always have the
> > latest builds of everything.


OK, this may be more or less helpful, depending on what we want to built
against. But a conda environment (maybe tied to a custom channel) really
does make  a nice contained space for testing that can be set up fast on a
CI server.

If whoever is setting up a test system/matrix thinks this would be useful,
I'd be glad to help set it up.

-Chris





-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy 1.11.0b2 released

[Chris Barker]

On Sat, Feb 6, 2016 at 3:42 PM, Michael Sarahan  wrote:

> FWIW, we (Continuum) are working on a CI system that builds conda recipes.
>

great, could be handy. I hope you've looked at the open-source systems that
do this: obvious-ci and conda-build-all. And conda-smithy to help set it
all up..

Chris, it may still be useful to use docker here (perhaps on the build
> worker, or elsewhere), also, as the distinction between build machines and
> user machines is important to make.  Docker would be great for making sure
> that all dependency requirements are met on end-user systems
>

yes -- veryhandy, I have certainly accidentally brough in other system libs
in a build

Too bad it's Linux only. Though very useful for manylinux.


-Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy 1.11.0b2 released

[Chris Barker - NOAA Federal]

> An extra ~2 hours of tests / 6-way parallelism is not that big a deal
> in the grand scheme of things (and I guess it's probably less than
> that if we can take advantage of existing binary builds)

If we set up a numpy-testing conda channel, it could be used to cache
binary builds for all he versions of everything we want to test
against.

Conda-build-all could make it manageable to maintain that channel.

-CHB
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy 1.11.0b2 released

[Chris Barker]

On Fri, Feb 5, 2016 at 9:55 AM, Nathaniel Smith  wrote:

> > If we set up a numpy-testing conda channel, it could be used to cache
> > binary builds for all he versions of everything we want to test
> > against.
> >
> > Conda-build-all could make it manageable to maintain that channel.
>
> What would be the advantage of maintaining that channel ourselves instead
> of using someone else's binary builds that already exist (e.g. Anaconda's,
> or official project wheels)?
>
other's binary wheels are only available for the versions that are
supported. Usually the latest releases, but Anaconda doesn't always have
the latest builds of everything.

Maybe we want to test against matplotlib master (or a release candidate,
or??), for instance.

And when we are testing a numpy-abi-breaking release, we'll need to have
everything tested against that release.

Usually, when you set up a conda environment, it preferentially pulls from
the default channel anyway (or any other channel you set up) , so we'd only
maintain stuff that wasn't readily available elsewhere.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Behavior of np.random.uniform

[Chris Barker]

On Tue, Jan 19, 2016 at 9:27 AM, Charles R Harris <charlesr.har...@gmail.com
> wrote:

> On Tue, Jan 19, 2016 at 9:23 AM, Chris Barker - NOAA Federal <
> chris.bar...@noaa.gov> wrote:
>
>> What does the standard lib do for rand range? I see that randint Is
>> closed on both ends, so order doesn't matter, though if it raises for b<a,
>> then that's a precedent we could follow.
>>
>
> randint is not closed on the high end. The now deprecated random_integers
> is the function that does that.
>

I was referring to the stdlib randint:

In [*7*]: [random.randint(2,5) for i in range(10)]

Out[*7*]: [5, 5, 2, 4, 5, 5, 3, 5, 2, 2]
thinking that compatibility with that would be good -- but that ship has
sailed.

randrange is open on the high end, as range is, (duh!)

 [random.randrange(2,5) for i in range(10)]

Out[*9*]: [3, 3, 4, 3, 3, 2, 4, 2, 3, 3]
but you get an exception is low >=high:

In [*10*]: random.randrange(5,2)

ValueError: empty range for randrange() (5,2, -3)
I like the exception idea best -- but backward compatibility and all that.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should I use pip install numpy in linux?

[Chris Barker - NOAA Federal]

> Last month, numpy had ~740,000 downloads from PyPI,

Hm, given that Windows and Linux wheels have not been available, then
that's mostly source installs anyway. Or failed installs  -- no
shortage of folks trying to pip install numpy on Windows and then
having questions about why it doesn't work. Unfortunately, there is no
way to know if pip downloads are successful, or if people pip install
Numpy, then find out they need some other non-pip-installable
packages, and go find another system.

> and there are
> probably hundreds of third-party projects that distribute via PyPI and
> depend on numpy.

I'm not so sure -- see above--as pip install has not been reliable for
Numpy for ages, I doubt it. Not that they aren't there, but I doubt
it's the primary distribution mechanism. There's been an explosion in
the use of conda, and there have been multiple other options for ages:
Canopy, python(x,y), Gohlke's builds, etc.

So at this point, I think the only people using pip are folks that are
set up to build -- mostly Linux. (though Mathew's efforts with the Mac
wheels may have created a different story on the Mac).



> So concretely our options as a project are:
> 1) drop support for pip/PyPI and abandon those users

There is no one to abandon -- except the Mac users -- we haven't
supported them yet.

> 2) continue to support those users fairly poorly, and at substantial
> ongoing cost

I'm curious what the cost is for this poor support -- throw the source
up on PyPi, and we're done. The cost comes in when trying to build
binaries...


> Option 1 would require overwhelming consensus of the community, which
> for better or worse is presumably not going to happen while
> substantial portions of that community are still using pip/PyPI.

Are they? Which community are we talking about? The community I'd like
to target are web developers that aren't doing what they think of as
"scientific" applications, but could use a little of the SciPy stack.
These folks are committed to pip, and are very reluctant to introduce
a difficult dependency.  Binary wheels would help these folks, but
that is not a community that exists yet ( or it's small, anyway)

All that being said, I'd be happy to see binary wheels for the core
SciPy stack on PyPi. It would be nice for people to be able to do a
bit with Numpy or pandas, it MPL, without having to jump ship to a
whole new way of doing things.

But we should be realistic about how far it can go.

> If
> folks interested in pushing things forward can convince them to switch
> to conda instead then the calculation changes, but that's just not the
> kind of thing that numpy-the-project can do or not do.

We can't convince anybody, but we can decide where to expend our efforts.

-CHB
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Behavior of np.random.uniform

[Chris Barker - NOAA Federal]

What does the standard lib do for rand range? I see that randint Is closed
on both ends, so order doesn't matter, though if it raises for b wrote:

Of the methods defined in *numpy/mtrand.pyx* (excluding helper functions
and *random_integers*, as they are all related to *randint*), *randint* is
the only other function with *low* and *high* parameters.  However, it
enforces *high* > *low*.

Greg

On Tue, Jan 19, 2016 at 1:36 PM, Benjamin Root  wrote:

> Are there other functions where this behavior may or may not be happening?
> If it isn't consistent across all np.random functions, it probably should
> be, one way or the other.
>
> Ben Root
>
> On Tue, Jan 19, 2016 at 5:10 AM, Jaime Fernández del Río <
> jaime.f...@gmail.com> wrote:
>
>> Hi all,
>>
>> There is a PR (#7026 ) that
>> documents the current behavior of np.random.uniform when the low and high
>> parameters it takes do not conform to the expected low < high. Basically:
>>
>>- if low < high, random numbers are drawn from [low, high),
>>- if low = high, all random numbers will be equal to low, and
>>- if low > high, random numbers are drawn from (high, low] (notice
>>the change in the open side of the interval.)
>>
>> My only worry is that, once we document this, we can no longer claim that
>> it is a bug.  So I would like to hear from others what do they think.  The
>> other more or less obvious options would be to:
>>
>>- Raise an error, but this would require a deprecation cycle, as
>>people may be relying on the current undocumented behavior.
>>- Check the inputs and draw numbers from [min(low, high), max(low,
>>high)), which is minimally different from current behavior.
>>
>> I will be merging the current documentation changes in the next few days,
>> so it would be good if any concerns were voiced before that.
>>
>> Thanks,
>>
>> Jaime
>>
>> --
>> (\__/)
>> ( O.o)
>> ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes
>> de dominación mundial.
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should I use pip install numpy in linux?

[Chris Barker]

hmm -- didn't mean to rev this up quite so much -- sorry!

But it's a good conversation to have, so...


On Fri, Jan 15, 2016 at 1:08 PM, Benjamin Root  wrote:

> That being said... I take exception to your assertion that anaconda is
> *the* solution to the packaging problem.
>

I think we need to keep some things straight here:

 "conda" is a binary package management system.

"Anaconda" is a python (and other stuff) distribution, built with conda.

In practice, everyone ( I know of ) uses the Anaconda distribution (or at
least the default conda channel) when using conda, but in theory, you could
maintain your an entirely distinct distribution with conda as the tool.

Also in practice, conda is so easy because continuum has done the hard work
of building a lot of the packages we all need -- there are still a lot
being maintained by the community in various ways, but frankly, we do
depend on continuum for all the hard work. But working on/with conda does
not lock you into that if you think it's not serving your needs.

And this discussion, (for me anyway) is about tools and the way forward,
not existing packages.

So onward!


> I still have a number of issues, particularly with the interactions of
> GDAL, shapely, and Basemap (they all seek out libgeos_c differently), and I
> have to use my own build of GDAL to enable many of the features that we use
> (the vanilla GDAL put out by Continuum just has the default options, and is
> quite limited).
>

Yeah, GDAL/OGR is a F%$#ing nightmare -- and I do wish that Anaconda had a
better build, but frankly, there is no system that's going to make that any
easier -- do any of the Linux distros ship a really good compatible, up to
date set of these libs -- and OS-X and Windows? yow! (Though Chris Gohlke
is a wonder!)


> If I don't set up my environment *just right* one of those packages will
> fail to import in some way due to being unable to find their particular
> version of libgeos_c. I haven't figure it out exactly why this happens, but
> it is very easy to break such an environment this way after an update.
>

Maybe conda could be improved to make this easier, I don't know (though do
checkout out the IOOS channel on anaconda.org Filipe has done some nice
work on this)



>  In a clutch, we had our IT staff manually build mod_wsgi against
> anaconda's python, but they weren't too happy about that, due to mod_wsgi
> no longer getting updated via yum.
>

I'm not sure how pip helps you out here, either. Sure for easy-to-compile
from source packages, you can just pip install, and you'll get a package
compatible with your (system) python. But binary wheels will give you the
same headaches -- so you're back to expecting your linux dstro to provide
everything, which they don't :-(

I understand that the IT folks want everything to come from their OS vendor
-- they like that -- but it simply isn't practical for scipy-based web
services. And once you've got most of your stack coming from another
source, is it really a big deal for python to come from somewhere else also
(and apache, and ???) -- conda at least is a technology that _can_ provide
an integrated system that includes all this -- I don't hink you're going to
be pip-installling apache anytime soon! (or node, or ???)


> If anaconda was the end-all, be-all solution, then it should just be a
> simple matter to do "conda install mod_wsgi". But the design of conda is
> that it is intended to be a user-space package-manager.
>

Then you can either install it as the web user (or apache user), or install
it as a system access. I haven't done this, but I don't think it all that
hard -- you're then going to need to sudo to install/upgrade anything new,
but that's expected.

So, again, I love conda for what it can do when it works well. I only take
> exception to the notion that it can address *all* problems, because there
> are some problems that it just simply isn't properly situated for.
>

But pip isn't situated for any of these either -- I'm still confused as to
the point here.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should I use pip install numpy in linux?

[Chris Barker]

On Thu, Jan 14, 2016 at 10:58 AM, Matthew Brett 
wrote:

> > but neither the culture nor the tooling support that
> > approach now, so I'm not very confident you could gather adoption.
>
> I don't think there's a very large amount of cultural work - but some
> to be sure.
>
> We already have the following on OSX:
>
> pip install numpy scipy matplotlib scikit-learn scikit-image pandas h5py
>
> where all the wheels come from pypi.  So, I don't think this is really
> outside our range, even if the problem is a little more difficult for
> Linux.
>

I'm actually less concerned about the Linux issue, I think that can be
solved reasonably with "manylinux" -- which would put us in a very similar
position to OS-X , and pretty similar to Windows -- i.e. a common platform
with the basic libs (libc, etc), but not a whole lot else.

I'm concerned about all the other libs various packages depend on. It's not
too bad if you want the core scipy stack -- a decent BLAS being the real
challenge there, but there is enough coordination between numpy and scipy
that at least efforts to solve that will be shared.

But we're still stuck with delivering dependencies on libs along with each
package -- usually statically linked.


> I thought that Anaconda.org allows pypi channels as well?


I think you can host pip-compatible wheels, etc on anaconda.org -- though
that may be deprecated... but anyway, I thought the goal here was a simple
"pip install", which will point only to PyPi -- I don't think there is a
way, ala conda, to add "channels" that will then get automatically searched
by pip. But I may be wrong there.

So here is the problem I want to solve:

> pip install numpy scipy matplotlib scikit-learn scikit-image pandas h5py

last I checked, each of those is self-contained, except for python-level
dependencies, most notably on numpy. So it doesn't' help me solve my
problem. For instance, I have my own C/C++ code that I'm wrapping that
requires netcdf (https://github.com/NOAA-ORR-ERD/PyGnome), and another that
requires image libs like libpng, libjpeg, etc.(
https://github.com/NOAA-ORR-ERD/py_gd)

netcdf is not too ugly itself, but it depends on hdf5, libcurl, zlib
(others?). So I have all these libs I need. As it happens, matplotlib
probably has the image libs I need, and h5py has hdf5 (and libcurl? and
zlib?). But even then, as far as I can tell, I need to build and provide
these libs myself for my code. Which is a pain in the @%$ and then I'm
shipping (and running) multiple copies of the same libs all over the place
-- will there be compatibility issues? apparently not, but it's still
wastes the advantage of shared libs, and makes things a pain for all of us.

With conda, on the other hand, I get netcdf libs, hdf5 libs, libpng,
libjpeg, ibtiff, and I can build my stuff against those and depend on them
-- saves me a lot of pain, and my users get a better system.

Oh, and add on the GIS stuff: GDAL, etc. (seriously a pain to build), and I
get a lot of value.

And, many of these libs (GDAL, netcdf) come with nifty command line
utilities -- I get those too.

So, pip+wheel _may_ be able to support all that, but AFAICT, no one is
doing it.

And it's really not going to support shipping cmake, and perl, and who
knows what else I might need in my toolchain that's not python or "just" a
library.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should I use pip install numpy in linux?

[Chris Barker]

On Fri, Jan 15, 2016 at 11:21 AM, Nathaniel Smith  wrote:

> Sure. Someone's already packaged those for conda, and no one has packaged
> them for pypi, so it makes sense that conda is more convenient for you. If
> someone does the work of packaging them for pypi, then that difference goes
> away.
>
This is what I meant by "cultural" issues :-)

but pypi has been an option for Windows and OS-X for ages and those
platforms are the bigger problem anyway -- and no one has done it for those
platforms -- Linux is not the issue here. I really did try to get that
effort started a while back -- I got zero support, nothing , nada. Which
doesn't mean I couldn't have gone ahead and done more myself, but I only
have so much time, and I wanted to work on my actual problems, and it it
hadn't gained any support, it would a been a big waste.

MAybe the world is ready for it now...

But besides the cultural/community/critical mass issues, pip+whell isn't
very well set up to support these use cases anyway -- so we have a 50%
solutikon now, and if we did this nifty binary-wheels-of-libs thing we'd
have a 90% solution -- and still struggle eith teh other 10%, until we
re-invented conda.

Anyway -- not trying to be a drag here -- just telling my story.

As for the many linux idea -- that's a great one -- hope we can get that
working.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should I use pip install numpy in linux?

[Chris Barker]

On Fri, Jan 15, 2016 at 2:38 PM, Matthew Brett 
wrote:

>
> I think there's a distinction between 'promote numpy as wheels' and
> 'make numpy available as a wheel'.  I don't think you'll see much
> evidence of "promotion" here - it's not really the open-source way.
>

Depends on how you define "promotion" I suppose. But I think that
supporting something is indeed promoting it.

I've been a fan of getting the scipy stack available from pip for a long
time -- I think it could be really useful for lots of folks not doing heavy
scipy-style work, but folks are very wary of introducing a new, hard to
install, dependency.

But now I'm not so sure -- the trick is what you tell newbies. When I teach
Python (not scipy), I start folks off with the python.org python. Then they
can pip install ipython, which is the only part of the "scipy stack" I want
them to have if they are not doing real numerical work.

But what if they are? Now it's pretty clear to me that anyone interested in
getting into data analysis, etc with python should just stating off with
Anaconda (or Canopy) -- or maybe Gohlke's binaries for Windows users. But
what if we have wheels on all platforms for the scipy stack (and a few
others?). Now they can learn python, pip install numpy, scipy, etc, learn
some more, get excited -- delve into some domain-specific work,and WHAM --
hit the wall of installation nightmares. NOW, they need to switch to
Anaconda or Canopy...

I think it's too bad to get that far into it and then have to switch and
learn something new.  -- Again, is there really a point to a 90% solution?

So this is the point -- heading down this road takes us to a place where
people can get much farther before hitting the wall -- but hot the wall
they will, so we'll still need other solutions, so maybe it would be better
for us to put our energies into those other solutions.

By the way, I'm seeing more and more folks that are not scipy-focused
starting off with conda, even for web apps.


> I'm not quite sure what you mean about 'circling the wagons', but the
> general approach of staying on course and seeing how things shake out
> seems to me entirely sensible.
>

well, what I've observed in the PyPa community may not be circling the
wagons, but it has been a "we're not ever going to solve the scipy
problems" attitude. I'm pretty convinced that pip/wheel will never be even
a 90% solution without some modifications -- so why go there?

Indeed, it's not uncommon for folks on the distutils list to say "go use
conda" in response to issues that pip does not address well.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Should I use pip install numpy in linux?

[Chris Barker]

hmm -- didn't mean to rev this up quite so much -- sorry!

But it's a good conversation to have, so...


On Fri, Jan 15, 2016 at 1:08 PM, Benjamin Root  wrote:

> That being said... I take exception to your assertion that anaconda is
> *the* solution to the packaging problem.
>

I think we need to keep some things straight here:

 "conda" is a binary package management system.

"Anaconda" is a python (and other stuff) distribution, built with conda.

In practice, everyone ( I know of ) uses the Anaconda distribution (or at
least the default conda channel) when using conda, but in theory, you could
maintain your an entirely distinct distribution with conda as the tool.

Also in practice, conda is so easy because continuum has done the hard work
of building a lot of the packages we all need -- there are still a lot
being maintained by the community in various ways, but frankly, we do
depend on continuum for all the hard work. But working on/with conda does
not lock you into that if you think it's not serving your needs.

And this discussion, (for me anyway) is about tools and the way forward,
not existing packages.

So onward!


> I still have a number of issues, particularly with the interactions of
> GDAL, shapely, and Basemap (they all seek out libgeos_c differently), and I
> have to use my own build of GDAL to enable many of the features that we use
> (the vanilla GDAL put out by Continuum just has the default options, and is
> quite limited).
>

Yeah, GDAL/OGR is a F%$#ing nightmare -- and I do wish that Anaconda had a
better build, but frankly, there is no system that's going to make that any
easier -- do any of the Linux distros ship a really good compatible, up to
date set of these libs -- and OS-X and Windows? yow! (Though Chris Gohlke
is a wonder!)


> If I don't set up my environment *just right* one of those packages will
> fail to import in some way due to being unable to find their particular
> version of libgeos_c. I haven't figure it out exactly why this happens, but
> it is very easy to break such an environment this way after an update.
>

Maybe conda could be improved to make this easier, I don't know (though do
checkout out the IOOS channel on anaconda.org Filipe has done some nice
work on this)



>  In a clutch, we had our IT staff manually build mod_wsgi against
> anaconda's python, but they weren't too happy about that, due to mod_wsgi
> no longer getting updated via yum.
>

I'm not sure how pip helps you out here, either. sure for easy-to-compile
from source packages, sure, you can just pip install, and you'll get a
package compatible with your (system) python. But binary wheels will give
you the same headaches -- so you're back to expecting your linux dstro to
provide everything, which they don't :-(

I understand that the IT folks want everything to come from their OS vendor
-- they like that -- but it simply isn't practical for scipy-based web
services. And once you've got most of your stack coming from another
source, is it really a big deal for python to come from somewhere else also
(and apache, and ???) -- conda at least is a technology that _can_ provide
an integrated system that includes all this -- I don't think you're going
to be pip-installling apache anytime soon!



>  But the design of conda is that it is intended to be a user-space
> package-manager. mod_wsgi is installed via root/apache user, which is
> siloed off from the user. I would have to (in theory) go and install conda
> for the apache user and likely have to install a conda "apache" package and
> mod_wsgi package.
>

This seems quite reasonable to me frankly. Plus, you can install conda
centrally as well, and then the apache user can get access to it (bot not
modify it)


> I seriously doubt this would be an acceptable solution for many IT
> administrators who would rather depend upon the upstream distributions who
> are going to be very quick about getting updates out the door and are
> updated automatically through yum or apt.
>

This is true -- but nothing to do with the technology -- it's a social
problem. And the auto updates? Ha! the real problem with admins that want
to use the system package managers is that they still insist you run python
2.6 for god's sake :-)


> So, again, I love conda for what it can do when it works well. I only take
> exception to the notion that it can address *all* problems, because there
> are some problems that it just simply isn't properly situated for.
>

still true, of course, but it can address many more than pip.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should I use pip install numpy in linux?

[Chris Barker - NOAA Federal]

>> Also, you have the problem that there is one PyPi -- so where do you put
>> your nifty wheels that depend on other binary wheels? you may need to fork
>> every package you want to build :-(
>
> Is this a real problem or a theoretical one? Do you know of some
> situation where this wheel to wheel dependency will occur that won't
> just be solved in some other way?

It's real -- at least during the whole bootstrapping period. Say I
build a nifty hdf5 binary wheel -- I could probably just grab the name
"libhdf5" on PyPI. So far so good. But the goal here would be to have
netcdf and pytables and GDAL and who knows what else then link against
that wheel. But those projects are all supported be different people,
that all have their own distribution strategy. So where do I put
binary wheels of each of those projects that depend on my libhdf5
wheel? _maybe_ I would put it out there, and it would all grow
organically, but neither the culture nor the tooling support that
approach now, so I'm not very confident you could gather adoption.

Even beyond the adoption period, sometimes you need to do stuff in
more than one way -- look at the proliferation of channels on
Anaconda.org.

This is more likely to work if there is a good infrastructure for
third parties to build and distribute the binaries -- e.g.
Anaconda.org.

Or the Linux dist to model -- for the most part, the people developing
a given library are not packaging it.

-CHB
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should I use pip install numpy in linux?

[Chris Barker]

On Mon, Jan 11, 2016 at 5:29 PM, Nathaniel Smith  wrote:

> I agree that talking about such things on distutils-sig tends to elicit a
> certain amount of puzzled incomprehension, but I don't think it matters --
> wheels already have everything you need to support this.
>
well, that's what I figured -- and I started down that path a while back
and got no support whatsoever (OK, some from Matthew Brett -- thanks!). But
I know myself well enough to know I wasn't going to get the critical mass
required to make it useful by myself, so I've moved on to an ecosystem that
is doing most of the work already.

Also, you have the problem that there is one PyPi -- so where do you put
your nifty wheels that depend on other binary wheels? you may need to fork
every package you want to build :-(

But sure, let's get the core scipy stack pip-installable as much as
possible -- and maybe some folks with more energy than me can move this all
forward.

Sorry to be a downer -- keep up the good work and energy!

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should I use pip install numpy in linux?

[Chris Barker]

On Mon, Jan 11, 2016 at 11:02 AM, David Cournapeau 
wrote:

> If we get all that worked out, we still haven't made any progress toward
>> the non-standard libs that aren't python. This is the big "scipy problem"
>> -- fortran, BLAS, hdf, ad infinitum.
>>
>> I argued for years that we could build binary wheels that hold each of
>> these, and other python packages could depend on them, but pypa never
>> seemed to like that idea.
>>
>
> I don't think that's an accurate statement. There are issues to solve
> around this, but I did not encounter push back, either on the ML or face to
> face w/ various pypa members at Pycon, etc... There may be push backs for a
> particular detail, but making "pip install scipy" or "pip install
> matplotlib" a reality on every platform is something everybody agrees o
>

sure, everyone wants that. But when it gets deeper, they don't want to have
a bunc hof pip-installable binary wheels that are simply clibs re-packaged
as a dependency. And, then you have the problelm of those being "binary
wheel" dependencies, rather than "package" dependencies.

e.g.:

this particular build of pillow depends on the libpng and libjpeg wheels,
but the Pillow package, in general, does not. And you would have different
dependencies on Windows, and OS-X, and Linux.

pip/wheel simply was not designed for that, and I didn't get any warm and
fuzzy feelings from dist-utils sig that the it ever would. And again, then
you are re-designing conda.

So the only way to do all that is to statically link all the dependent libs
in with each binary wheel (or ship a dll). Somehow it always bothered me to
ship the same lib with multiple packages -- isn't that why shared libs
exist?

-  In practice, maybe it doesn't matter, memory is cheap. But I also got
sick of building the damn things! I like that Anaconda comes with libs
someone else has built, and I can use them with my packages, too. And when
it comes to ugly stuff like HDF and GDAL, I'm really happy someone else has
built them!

Anyway -- carry on -- being able to pip install the scipy stack would be
very nice.

-CHB














>
>>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should I use pip install numpy in linux?

[Chris Barker]

On Fri, Jan 8, 2016 at 7:13 PM, Nathaniel Smith  wrote:

> > that this would potentially be able to let packages like numpy serve
> their
> > linux
> > users better without risking too much junk being uploaded to PyPI.
>
> That will never fly. But like Matthew says, I think we can probably
> get them to accept a PEP saying "here's a new well-specified platform
> tag that means that this wheel works on all linux systems meet the
> following list of criteria: ...", and then allow that new platform tag
> onto PyPI.
>

The second step is a trick though -- how does pip know, when being run on a
client, that the system meets those requirements? Do we put a bunch of code
in that checks for those libs, etc???

If we get all that worked out, we still haven't made any progress toward
the non-standard libs that aren't python. This is the big "scipy problem"
-- fortran, BLAS, hdf, ad infinitum.

I argued for years that we could build binary wheels that hold each of
these, and other python packages could depend on them, but pypa never
seemed to like that idea. In the end, if you did all this right, you'd have
something like conda -- so why not just use conda?

All that being said, if you folks can get the core scipy stack setup to pip
install on OS_X, Windows, and Linux, that would be pretty nice -- so keep
at it !

-CHB









>
> -n
>
> --
> Nathaniel J. Smith -- http://vorpus.org
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should I use pip install numpy in linux?

[Chris Barker]

On Fri, Jan 8, 2016 at 1:58 PM, Robert McGibbon  wrote:

> I'm not sure if this is the right path for numpy or not,
>

probably not -- AFAICT, the PyPa folks aren't interested in solving teh
problems we have in the scipy community -- we can tweak around the edges,
but we wont get there without a commitment to really solve the issues --
and if pip did that, it would essentially be conda -- non one wants to
re-impliment conda.


>
> Perhaps this is OT for this thread, and I should ask on distutils-sig.
>

there has been a lot of discussion of this issue on the distutils-sig in
the last couple months (quiet lately). So yes, that's the place to go.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Dynamic array list implementation

[Chris Barker]

On Wed, Dec 30, 2015 at 6:34 AM, Nicolas P. Rougier <
nicolas.roug...@inria.fr> wrote:

>
> > On 28 Dec 2015, at 19:58, Chris Barker <chris.bar...@noaa.gov> wrote:
> >
> > >>> python benchmark.py
> > Python list, append 10 items: 0.01161
> > Array list, append 10 items: 0.46854
> >
> > are you pre-allocating any extra space? if not -- it's going to be
> really, really pokey when adding a little bit at a time.
>
>
> Yes, I’m preallocating but it might not be optimal at all given your
> implementation is much faster.
> I’ll try to adapt your code. Thanks.


sounds good -- I'll try to take a look at yours soon - maybe we can merge
the projects. MIne is only operational in one small place, I think.

-CHB






>
> >
> > With my Accumulator class:
> >
> >
> https://github.com/PythonCHB/NumpyExtras/blob/master/numpy_extras/accumulator.py
> >
> > I pre-allocate a larger numpy array to start, and it gets re-allocated,
> with some extra, when filled, using ndarray.resize()
> >
> > this is quite fast.
> >
> > These are settable parameters in the class:
> >
> > DEFAULT_BUFFER_SIZE = 128 # original buffer created.
> > BUFFER_EXTEND_SIZE = 1.25 # array.array uses 1+1/16 -- that seems small
> to me.
> >
> >
> > I looked at the code in array.array (and list, I think), and it does
> stuff to optimize very small arrays, which I figured wasn't the use-case
> here :-)
> >
> > But I did a bunch of experimentation, and as long as you pre-allocate
> _some_ it doesn't make much difference how much :-)
> >
> > BTW,
> >
> > I just went in an updated and tested the Accumulator class code -- it
> needed some tweaks, but it's working now.
> >
> > The cython version is in an unknown state...
> >
> > some profiling:
> >
> > In [11]: run profile_accumulator.py
> >
> >
> > In [12]: timeit accum1(1)
> >
> > 100 loops, best of 3: 3.91 ms per loop
> >
> > In [13]: timeit list1(1)
> >
> > 1000 loops, best of 3: 1.15 ms per loop
> >
> > These are simply appending 10,000 integers in a loop -- with teh list,
> the list is turned into a numpy array at the end. So it's still faster to
> accumulate in a list, then make an array, but only a about a factor of 3 --
> I think this is because you are staring with a python integer -- with the
> accumulator function, you need to be checking type and pulling a native
> integer out with each append. but a list can append a python object with no
> type checking or anything.
> >
> > Then the conversion from list to array is all in C.
> >
> > Note that the accumulator version is still more memory efficient...
> >
> > In [14]: timeit accum2(1)
> >
> > 100 loops, best of 3: 3.84 ms per loop
> >
> > this version pre-allocated the whole internal buffer -- not much faster
> the buffer re-allocation isn't a big deal (thanks to ndarray.resize using
> realloc(), and not creating a new numpy array)
> >
> > In [24]: timeit list_extend1(10)
> >
> > 100 loops, best of 3: 4.15 ms per loop
> >
> > In [25]: timeit accum_extend1(10)
> >
> > 1000 loops, best of 3: 1.37 ms per loop
> >
> > This time, the stuff is added in chunks 100 elements at a time -- the
> chunks being created ahead of time -- a list with range() the first time,
> and an array with arange() the second. much faster to extend with arrays...
> >
> > -CHB
> >
> >
> >
> > --
> >
> > Christopher Barker, Ph.D.
> > Oceanographer
> >
> > Emergency Response Division
> > NOAA/NOS/OR(206) 526-6959   voice
> > 7600 Sand Point Way NE   (206) 526-6329   fax
> > Seattle, WA  98115   (206) 526-6317   main reception
> >
> > chris.bar...@noaa.gov
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Dynamic array list implementation

[Chris Barker]

On Wed, Dec 23, 2015 at 4:01 AM, Nicolas P. Rougier <
nicolas.roug...@inria.fr> wrote:

> But my implementation is quite slow, especially when you add one item at a
> time:
>
> >>> python benchmark.py
> Python list, append 10 items: 0.01161
> Array list, append 10 items: 0.46854
>

are you pre-allocating any extra space? if not -- it's going to be really,
really pokey when adding a little bit at a time.

With my Accumulator class:

https://github.com/PythonCHB/NumpyExtras/blob/master/numpy_extras/accumulator.py

I pre-allocate a larger numpy array to start, and it gets re-allocated,
with some extra, when filled, using ndarray.resize()

this is quite fast.

These are settable parameters in the class:

DEFAULT_BUFFER_SIZE = 128 # original buffer created.
BUFFER_EXTEND_SIZE = 1.25 # array.array uses 1+1/16 -- that seems small to
me.


I looked at the code in array.array (and list, I think), and it does stuff
to optimize very small arrays, which I figured wasn't the use-case here :-)

But I did a bunch of experimentation, and as long as you pre-allocate
_some_ it doesn't make much difference how much :-)

BTW,

I just went in an updated and tested the Accumulator class code -- it
needed some tweaks, but it's working now.

The cython version is in an unknown state...

some profiling:

In [11]: run profile_accumulator.py


In [12]: timeit accum1(1)

100 loops, best of 3: 3.91 ms per loop

In [13]: timeit list1(1)

1000 loops, best of 3: 1.15 ms per loop

These are simply appending 10,000 integers in a loop -- with teh list, the
list is turned into a numpy array at the end. So it's still faster to
accumulate in a list, then make an array, but only a about a factor of 3 --
I think this is because you are staring with a python integer -- with the
accumulator function, you need to be checking type and pulling a native
integer out with each append. but a list can append a python object with no
type checking or anything.

Then the conversion from list to array is all in C.

Note that the accumulator version is still more memory efficient...

In [14]: timeit accum2(1)

100 loops, best of 3: 3.84 ms per loop

this version pre-allocated the whole internal buffer -- not much faster the
buffer re-allocation isn't a big deal (thanks to ndarray.resize using
realloc(), and not creating a new numpy array)

In [24]: timeit list_extend1(10)

100 loops, best of 3: 4.15 ms per loop

In [25]: timeit accum_extend1(10)

1000 loops, best of 3: 1.37 ms per loop

This time, the stuff is added in chunks 100 elements at a time -- the
chunks being created ahead of time -- a list with range() the first time,
and an array with arange() the second. much faster to extend with arrays...

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Dynamic array list implementation

[Chris Barker]

On Wed, Dec 23, 2015 at 4:31 AM, Sebastian Berg <sebast...@sipsolutions.net>
wrote:

>
> Probably is a bit orthogonal since I guess you want/need cython, but
> pythons builtin array.array should get you there pretty much as well.


I don't think it's orthogonal to cython -- you can access an array.array
directly from within cython -- it's actually about the easiest way to get a
array-like object in Cython/C (which you can then access via a memoryview,
etc).

Though I don't know there is a python object (i.e. pointer) option there.
(nor text).

-CHB








> Of
> course it requires the C typecode (though that should not be hard to
> get) and does not support strings.
>
> - Sebastian
>
> >
> > In my experience, it's several times faster than using a builtin list
> > from Cython, which makes sense given that it needs to copy about 1/3
> > the data (no type or reference count for individual elements).
> > Obviously, it uses 1/3 the space to store the data, too. We currently
> > don't expose this object externally, but it could be an interesting
> > project to adapt this code into a standalone project that could be
> > more broadly useful.
> >
> >
> > Cheers,
> > Stephan
> >
> >
> >
> >
> > On Tue, Dec 22, 2015 at 8:20 PM, Chris Barker <chris.bar...@noaa.gov>
> > wrote:
> >
> >
> > sorry for being so lazy as to not go look at the project
> > pages, but
> >
> >
> > This sounds like it could be really useful, and maybe
> > supercise a coupl eof half-baked projects of mine. But -- what
> > does "dynamic" mean?
> >
> >
> > - can you append to these arrays?
> > - can it support "ragged arrrays" -- it looks like it does.
> >
> >
> > >>> L = ArrayList( [[0], [1,2], [3,4,5], [6,7,8,9]] )
> > >>> print(L)
> > [[0], [1 2], [3 4 5], [6 7 8 9]]
> > so this looks like a ragged array -- but what do you get when
> > you do:
> >
> >
> > for row in L:
> > print row
> >
> >
> >
> >
> > >>> print(L.data)
> > [0 1 2 3 4 5 6 7 8
> > is .data a regular old 1-d numpy array?
> >
> >
> > >>> L = ArrayList( np.arange(10), [3,3,4])
> > >>> print(L)
> > [[0 1 2], [3 4 5], [6 7 8 9]]
> > >>> print(L.data)
> > [0 1 2 3 4 5 6 7 8 9]
> >
> > does an ArrayList act like a numpy array in other ways:
> >
> >
> > L * 5
> >
> >
> > L* some_array
> >
> >
> > in which case, how does it do broadcasting???
> >
> >
> > Thanks,
> >
> >
> > -CHB
> >
> >
> > >>> L = ArrayList(["Hello", "world", "!"])
> > >>> print(L[0])
> > 'Hello'
> > >>> L[1] = "brave new world"
> > >>> print(L)
> > ['Hello', 'brave new world', '!']
> >
> >
> >
> >
> > Nicolas
> >
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
> >
> >
> >
> > --
> >
> > Christopher Barker, Ph.D.
> > Oceanographer
> >
> > Emergency Response Division
> > NOAA/NOS/OR(206) 526-6959   voice
> > 7600 Sand Point Way NE   (206) 526-6329   fax
> > Seattle, WA  98115   (206) 526-6317   main reception
> >
> > chris.bar...@noaa.gov
> >
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Dynamic array list implementation

[Chris Barker]

On Thu, Dec 24, 2015 at 10:19 AM, Chris Barker <chris.bar...@noaa.gov>
wrote:

> I'll try to get the code up on gitHub.
>

Hey look -- it's already there:

https://github.com/PythonCHB/NumpyExtras

too many gitHub accounts.

Here is the list/growable array/ accumulator:

https://github.com/PythonCHB/NumpyExtras/blob/master/numpy_extras/accumulator.py

And here is the ragged array:

https://github.com/PythonCHB/NumpyExtras/blob/master/numpy_extras/ragged_array.py

I haven't touched either of these for a while -- not really sure what state
they are in.

-CHB





> It would be nice to combine efforts.
>
> -CHB
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>> In my case I need to ensure a contiguous storage to allow easy upload
>> onto the GPU.
>> But my implementation is quite slow, especially when you add one item at
>> a time:
>>
>> >>> python benchmark.py
>> Python list, append 10 items: 0.01161
>> Array list, append 10 items: 0.46854
>> Array list, append 10 items at once: 0.05801
>> Python list, prepend 10 items: 1.96168
>> Array list, prepend 10 items: 12.83371
>> Array list, append 10 items at once: 0.06002
>>
>>
>>
>> I realize I did not answer all Chris' questions:
>>
>> >>> L = ArrayList( [[0], [1,2], [3,4,5], [6,7,8,9]] )
>> >>> for item in L: print(item)
>> [0]
>> [1 2]
>> [3 4 5]
>> [6 7 8 9]
>>
>> >>> print (type(L.data))
>> 
>> >>> print(L.data.dtype)
>> int64
>> >>> print(L.data.shape)
>> (10,)
>>
>>
>> I did not implement operations yet, but it would be a matter for
>> transferring call to the underlying numpy data array.
>> >>> L._data *= 2
>> >>> print(L)
>> [[0], [4 8], [12 16 20], [24 28 32 36]]
>>
>>
>>
>> > On 23 Dec 2015, at 09:34, Stephan Hoyer <sho...@gmail.com> wrote:
>> >
>> > We have a type similar to this (a typed list) internally in pandas,
>> although it is restricted to a single dimension and far from feature
>> complete -- it only has .append and a .to_array() method for converting to
>> a 1d numpy array. Our version is written in Cython, and we use it for
>> performance reasons when we would otherwise need to create a list of
>> unknown length:
>> > https://github.com/pydata/pandas/blob/v0.17.1/pandas/hashtable.pyx#L99
>> >
>> > In my experience, it's several times faster than using a builtin list
>> from Cython, which makes sense given that it needs to copy about 1/3 the
>> data (no type or reference count for individual elements). Obviously, it
>> uses 1/3 the space to store the data, too. We currently don't expose this
>> object externally, but it could be an interesting project to adapt this
>> code into a standalone project that could be more broadly useful.
>> >
>> > Cheers,
>> > Stephan
>> >
>> >
>> >
>> > On Tue, Dec 22, 2015 at 8:20 PM, Chris Barker <chris.bar...@noaa.gov>
>> wrote:
>> >
>> > sorry for being so lazy as to not go look at the project pages, but
>> >
>> > This sounds like it could be really useful, and maybe supercise a coupl
>> eof half-baked projects of mine. But -- what does "dynamic" mean?
>> >
>> > - can you append to these arrays?
>> > - can it support "ragged arrrays" -- it looks like it does.
>> >
>> > >>> L = ArrayList( [[0], [1,2], [3,4,5], [6,7,8,9]] )
>> > >>> print(L)
>> > [[0], [1 2], [3 4 5], [6 7 8 9]]
>> >
>> > so this looks like a ragged array -- but what do you get when you do:
>> >
>> > for row in L:
>> > print row
>> >
>> >
>> > >>> print(L.data)
>> > [0 1 2 3 4 5 6 7 8
>> >
>> > is .data a regular old 1-d numpy array?
>> >
>> > >>> L = ArrayList( np.arange(10), [3,3,4])
>> > >>> print(L)
>> > [[0 1 2], [3 4 5], [6 7 8 9]]
>> > >>> print(L.data)
>> > [0 1 2 3 4 5 6 7 8 9]
>> >
>> >
>> > does an ArrayList act like a numpy array in other ways:
>> >
>> > L * 5
>> >
>> > L* some_array
>> >
>> > in which case, how does it do broadcasting???
>> >
>> > Thanks,
>> >
>> > -CHB
>> >
>> > >>> L = ArrayList(["Hello", "world", "!"])
>> > >>

Re: [Numpy-discussion] Dynamic array list implementation

[Chris Barker]

On Wed, Dec 23, 2015 at 4:01 AM, Nicolas P. Rougier <
nicolas.roug...@inria.fr> wrote:
>
> Typed list in numpy would be a nice addition indeed and your cython
> implementation is nice (and small).
>

It seems we have a lot of duplicated effort here. Pernonally, I have two
needs:

1) ragged arrays
2) "growable" arrays.

I have semi-complete version of both of these, which are completely
independent -- not sure if it makes sense to combine them, I suppose not.

But we've talked a bit about  "typed list", and I'm not sure what that
means -- is it something that is entirely like a python list, except that
all the elements have the same type?

Anyway: I've been thinking about this fromt eh opposite direction: I want a
numpy array that you can append/extend. This comes from the fact that it's
not uncommon to need to build up an array where you don't know how large it
will be when you start. The common recommendation for doing that now is to
built it up in a list, and then, when you are done, turn it into an ndarray.

But that means you are limited to python types (or putting numpy scalars in
a list...), and it's not very memory efficient.

My version used a ndarray internally, and over allocates it a bit, using
ndarray.resize() to resize. this means that you can get the data pointer if
you want for Cython, etc... but also that it's getting re-allocated, so
that pointer is fragile, and you don't want other arrays to have views on
it.

Interestingly, if you are adding one float, for example, at a time to the
array, it's actually a bit faster to build it up in a list, and then make
an array out of it.

But it is more memory efficient and faster if you are using numpy dtypes
and especially if you are extend()ing it with chunks from other arrays.

I also have a not-quite finished version in Cython that statically handles
the core C data types -- that should be faster, but I haven't really
profiled it.

I'll try to get the code up on gitHub.

It would be nice to combine efforts.

-CHB
















> In my case I need to ensure a contiguous storage to allow easy upload onto
> the GPU.
> But my implementation is quite slow, especially when you add one item at a
> time:
>
> >>> python benchmark.py
> Python list, append 10 items: 0.01161
> Array list, append 10 items: 0.46854
> Array list, append 10 items at once: 0.05801
> Python list, prepend 10 items: 1.96168
> Array list, prepend 10 items: 12.83371
> Array list, append 10 items at once: 0.06002
>
>
>
> I realize I did not answer all Chris' questions:
>
> >>> L = ArrayList( [[0], [1,2], [3,4,5], [6,7,8,9]] )
> >>> for item in L: print(item)
> [0]
> [1 2]
> [3 4 5]
> [6 7 8 9]
>
> >>> print (type(L.data))
> 
> >>> print(L.data.dtype)
> int64
> >>> print(L.data.shape)
> (10,)
>
>
> I did not implement operations yet, but it would be a matter for
> transferring call to the underlying numpy data array.
> >>> L._data *= 2
> >>> print(L)
> [[0], [4 8], [12 16 20], [24 28 32 36]]
>
>
>
> > On 23 Dec 2015, at 09:34, Stephan Hoyer <sho...@gmail.com> wrote:
> >
> > We have a type similar to this (a typed list) internally in pandas,
> although it is restricted to a single dimension and far from feature
> complete -- it only has .append and a .to_array() method for converting to
> a 1d numpy array. Our version is written in Cython, and we use it for
> performance reasons when we would otherwise need to create a list of
> unknown length:
> > https://github.com/pydata/pandas/blob/v0.17.1/pandas/hashtable.pyx#L99
> >
> > In my experience, it's several times faster than using a builtin list
> from Cython, which makes sense given that it needs to copy about 1/3 the
> data (no type or reference count for individual elements). Obviously, it
> uses 1/3 the space to store the data, too. We currently don't expose this
> object externally, but it could be an interesting project to adapt this
> code into a standalone project that could be more broadly useful.
> >
> > Cheers,
> > Stephan
> >
> >
> >
> > On Tue, Dec 22, 2015 at 8:20 PM, Chris Barker <chris.bar...@noaa.gov>
> wrote:
> >
> > sorry for being so lazy as to not go look at the project pages, but
> >
> > This sounds like it could be really useful, and maybe supercise a coupl
> eof half-baked projects of mine. But -- what does "dynamic" mean?
> >
> > - can you append to these arrays?
> > - can it support "ragged arrrays" -- it looks like it does.
> >
> > >>> L = ArrayList( [[0], [1,2], [3,4,5], [6,7,8,9]] )
> > >>> print(L)
> > [[0], [1 2], [3 4 5], [

Re: [Numpy-discussion] Proposal: stop providing official win32 downloads (for now)

[Chris Barker]

On Mon, Dec 21, 2015 at 10:05 PM, Ralf Gommers 
wrote:

>
>>> There's a good chance that many downloads are from unsuspecting users
> with a 64-bit Python, and they then just get an unhelpful "cannot find
> Python" error from the installer.
>

could be -- hard to know.


> At least until we have binary wheels on PyPi.
>>
>> What's up with that, by the way?
>>
>
> I expect those to appear in 2016, built with MinGW-w64 and OpenBLAS.
>

nice. Anyway, I do think it's important to have a "official" easy way to
get numpy for pyton.org pythons.

numpy does/can/should see a lot of use outside the "scientific computing"
community. And people are wary of dependencies. people should be able to
use numpy in their projects, without requiring that their users start all
over with Anaconda or ???

The ideal is for "pip install" to "just work" -- sound sike we're getting
there.

BTW, we've been wary of putting a 32 bit wheel up 'cause of the whole "what
processor features to require" issue, but if we think it's OK to drop the
binary installer altogether, I can't see the harm in putting a 32 bit SSE2
wheel up.

Any way to know how many people are running 32 bit Python on Windows these
days??

-CHB













>
> Ralf
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Dynamic array list implementation

[Chris Barker]

sorry for being so lazy as to not go look at the project pages, but

This sounds like it could be really useful, and maybe supercise a coupl eof
half-baked projects of mine. But -- what does "dynamic" mean?

- can you append to these arrays?
- can it support "ragged arrrays" -- it looks like it does.

>
> >>> L = ArrayList( [[0], [1,2], [3,4,5], [6,7,8,9]] )
> >>> print(L)
> [[0], [1 2], [3 4 5], [6 7 8 9]]
>
> so this looks like a ragged array -- but what do you get when you do:

for row in L:
print row



> >>> print(L.data)
> [0 1 2 3 4 5 6 7 8
>
> is .data a regular old 1-d numpy array?

>>> L = ArrayList( np.arange(10), [3,3,4])
> >>> print(L)
> [[0 1 2], [3 4 5], [6 7 8 9]]
> >>> print(L.data)
> [0 1 2 3 4 5 6 7 8 9]
>
> does an ArrayList act like a numpy array in other ways:

L * 5

L* some_array

in which case, how does it do broadcasting???

Thanks,

-CHB

>>> L = ArrayList(["Hello", "world", "!"])
> >>> print(L[0])
> 'Hello'
> >>> L[1] = "brave new world"
> >>> print(L)
> ['Hello', 'brave new world', '!']
>
>
>
> Nicolas
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop providing official win32 downloads (for now)

[Chris Barker]

On Tue, Dec 22, 2015 at 11:18 AM, David Cournapeau 
wrote:

> Any way to know how many people are running 32 bit Python on Windows these
>> days??
>>
>
> I don't claim we are representative of the whole community, but as far as
> canopy is concerned, it is still a significant platform. That's the only 32
> bit platform we still support (both linux and osx 32 bits were < 1 % of our
> downloads)
>


thanks -- I think that's a good data point -- presumably, people can grab
64 bit Canopy just as easily as 64 bit -- and get all the extra packages.
And they still want, or think they do :-), 32 bit.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop providing official win32 downloads (for now)

[Chris Barker]

On Tue, Dec 22, 2015 at 11:43 AM, Matthew Brett 
wrote:

> On a virtual Windows machine I just span up, the Python.org site gave
> me default buttons to download Python 3.5 or 2.7, and the linked
> installers look like they are 32-bit.


It's probably time for python.org to change that -- but this does mean that
there will be people using 32 bit pytohn on windows purely by happenstance.

so I think it's important to continue to support those folks. Again, 32 bit
binary wheels on PyPi is probably the way to do it these days.

-CHB





>   I can also go to the full list
> where there is no preference for one over the other,
>
> Cheers,
>
> Matthew
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop providing official win32 downloads (for now)

[Chris Barker]

On Tue, Dec 22, 2015 at 11:32 AM, Matthew Brett 
wrote:

> The take-home is that about 12 percent of gamers have 32 bit Windows.
> It's easy to believe business users will use 32-bit more.
>

yup -- I tend to think gamers are on the cutting edge

Though I work on gov't which is very slow moving, and we're on 64 bit
finally...

Also, I believe that the default Windows Python.org installers are
> 32-bit, at least, that's what the filenames suggest when I try it now.
>

what is "default" -- when I go to pyton.org to downloads, I get:

https://www.python.org/downloads/release/python-2711/

where they are both there with equal footing.

though I'm running a Mac -- so it starts out suggesting a Mac download --
maybe if I was running Windows, I'd get a default.

-CHB



> Cheers,
>
> Matthew
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop providing official win32 downloads (for now)

[Chris Barker]

On Fri, Dec 18, 2015 at 1:51 PM, Ralf Gommers
>
> +1 from me. Despite the number of downloads still being high, I don't
> think there's too much value in these binaries anymore.
>

If there are a lot of downloads, then there is value. At least until we
have binary wheels on PyPi.

What's up with that, by the way?

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] array_equal too strict?

[Chris Barker - NOAA Federal]

> If you have some spare cycles, maybe you can open a pull request to add
> np.isclose to the "See Also" section?

That would be great.

Remember that equality for flits is bit-for but equality ( baring NaN
and inf...).

But you hardly ever actually want to do that with floats.

But probably np.allclose is most appropriate here.

CHB

>
> - Sebastian
>
>
>> ```
>> print(numpy.array_equal(a1, a2))
>>
>> # output: true
>> ```
>> That's expected because the difference is only in the 18th overall
>> digit, and the mantissa length of float64 is 52 bits [1], i.e., approx
>> 15.6 decimal digits. Moving the difference to the 17th overall digit
>> should also be fine, however:
>> ```
>> a1 = numpy.array(
>>[3.1415926535897930],
>>dtype=numpy.float64
>>)
>> a2 = numpy.array(
>>[3.1415926535897939],
>>dtype=numpy.float64
>>)
>>
>>
>> print(numpy.array_equal(a1, a2))
>> # output: false
>> ```
>> It gets even more visible with float32 and its 23 mantissa bits (i.e.,
>> 6.9 decimal digits):
>> ```
>> a1 = numpy.array(
>>[3.14159260],
>>dtype=numpy.float32
>>)
>> a2 = numpy.array(
>>[3.14159269],
>>dtype=numpy.float32
>>)
>>
>>
>> print(numpy.array_equal(a1, a2))
>> # output: false
>> ```
>> The difference is only in the 9th decimal digit, still `array_equal_
>> detects the difference.
>>
>>
>> I'm not sure where I'm going wrong here. Any hints?
>>
>>
>> Cheers,
>> Nico
>>
>>
>>
>>
>> [1] https://docs.scipy.org/doc/numpy-1.10.1/user/basics.types.html
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Fast vectorized arithmetic with ~32 significant digits under Numpy

[Chris Barker - NOAA Federal]

> There has also been some talk of adding a user type for ieee 128 bit doubles. 
> I've looked once for relevant code for the latter and, IIRC, the available 
> packages were GPL :(.


This looks like it's BSD-Ish:

http://www.jhauser.us/arithmetic/SoftFloat.html

Don't know if it's any good

CHB


>
> Chuck
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] When to stop supporting Python 2.6?

[Chris Barker]

drop 2.6

I still don't understand why folks insist that they need to run a (very))
old python on an old OS, but need the latest and greatest numpy.

Chuck's list was pretty long and compelling.

-CHB



On Mon, Dec 7, 2015 at 1:38 AM, Sturla Molden 
wrote:

> Charles R Harris  wrote:
>
> > As a strawman proposal, how about dropping moving to 2.7 and 3.4 minimum
> > supported version next fall, say around numpy 1.12 or 1.13 depending on
> how
> > the releases go.
> >
> > I would like to here from the scipy folks first.
>
> Personally I would be in favor of this, because 2.7 and 3.4 are the minimum
> versions anyone should consider to use. However, for SciPy which heavily
> depends on Python code, the real improvement will be when we can bump the
> minimum Python version to 3.5 and write x @ y instead of dot(x,y). I am not
> sure of bumping the minimum version to 3.4 before that is worth it or not.
> But certainly dropping 2.6 might be a good thing already now, so we can
> start to use bytes, bytearray, memoryview, etc.
>
> Sturla
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Question about structure arrays

[Chris Barker]

On Sat, Nov 7, 2015 at 1:18 PM, aerojockey 
wrote:

>  Inside a
> low-level loop, I create a structure array, populate it Python, then turn
> it
> over to some handwritten C code for processing.


can you do that inside bit of the low-level loop in C (or cython?) you
often want to put the guts of your loop in C anyway...

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] [Distutils] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

[Chris Barker - NOAA Federal]

>> I'm not talking about in place installs, I'm talking about e.g. building a
>> wheel and then tweaking one file and rebuilding -- traditionally build
>> systems go to some effort to keep track of intermediate artifacts and reuse
>> them across builds when possible, but if you always copy the source tree
>> into a temporary directory before building then there's not much the build
>> system can do.

This strikes me as an optimization -- is it an important one?

If I'm doing a lot of tweaking and re-running, I'm usually in develop mode.

I can see that when you build a wheel, you may build it, test it,
discover an wheel-specific error, and then need to repeat the cycle --
but is that a major use-case?

That being said, I have been pretty frustrated debugging conda-build
scripts -- there is a lot of overhead setting up the build environment
each time you do a build...

But with wheel building there is much less overhead, and far fewer
complications requiring the edit-build cycle.

And couldn't make-style this-has-already-been-done checking happen
with a copy anyway?

CHB

> Ah yes. So I don't think pip should do what it does. It a violation of
> the abstractions we all want to see within it. However its not me you
> need to convince ;).
>
> -Rob
>
> --
> Robert Collins 
> Distinguished Technologist
> HP Converged Cloud
> ___
> Distutils-SIG maillist  -  distutils-...@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] deprecate fromstring() for text reading?

[Chris Barker - NOAA Federal]

I was more aiming to point out a situation where the NumPy's text file
reader was significantly better than the Pandas version, so we would want
to make sure that we properly benchmark any significant changes to NumPy's
text reading code. Who knows where else NumPy beats Pandas?
Indeed. For this example, I think a fixed-with reader really is a different
animal, and it's probably a good idea to have a high performance one in
Numpy. Among other things, you wouldn't want it to try to auto-determine
data types or anything like that.

I think what's on the table now is to bring in a new delimited reader --
I.e. CSV in its various flavors.

CHB


Ben

On Mon, Nov 2, 2015 at 6:44 PM, Chris Barker <chris.bar...@noaa.gov> wrote:

> On Tue, Oct 27, 2015 at 7:30 AM, Benjamin Root <ben.v.r...@gmail.com>
> wrote:
>
>> FWIW, when I needed a fast Fixed Width reader
>>
>
> was there potentially no whitespace between fields in that case? In which
> case, it really isn a different use-case than delimited text -- if it's at
> all common, a version written in C would be nice and fast. and nat hard to
> do.
>
> But fromstring never would have helped you with that anyway :-)
>
> -CHB
>
>
>
>> for a very large dataset last year, I found that np.genfromtext() was
>> faster than pandas' read_fwf(). IIRC, pandas' text reading code fell back
>> to pure python for fixed width scenarios.
>>
>> On Fri, Oct 23, 2015 at 8:22 PM, Chris Barker - NOAA Federal <
>> chris.bar...@noaa.gov> wrote:
>>
>>> Grabbing the pandas csv reader would be great, and I hope it happens
>>> sooner than later, though alas, I haven't the spare cycles for it either.
>>>
>>> In the meantime though, can we put a deprecation Warning in when using
>>> fromstring() on text files? It's really pretty broken.
>>>
>>> -Chris
>>>
>>> On Oct 23, 2015, at 4:02 PM, Jeff Reback <jeffreb...@gmail.com> wrote:
>>>
>>>
>>>
>>> On Oct 23, 2015, at 6:49 PM, Nathaniel Smith <n...@pobox.com> wrote:
>>>
>>> On Oct 23, 2015 3:30 PM, "Jeff Reback" <jeffreb...@gmail.com> wrote:
>>> >
>>> > On Oct 23, 2015, at 6:13 PM, Charles R Harris <
>>> charlesr.har...@gmail.com> wrote:
>>> >
>>> >>
>>> >>
>>> >> On Thu, Oct 22, 2015 at 5:47 PM, Chris Barker - NOAA Federal <
>>> chris.bar...@noaa.gov> wrote:
>>> >>>
>>> >>>
>>> >>>> I think it would be good to keep the usage to read binary data at
>>> least.
>>> >>>
>>> >>>
>>> >>> Agreed -- it's only the text file reading I'm proposing to
>>> deprecate. It was kind of weird to cram it in there in the first place.
>>> >>>
>>> >>> Oh, fromfile() has the same issues.
>>> >>>
>>> >>> Chris
>>> >>>
>>> >>>
>>> >>>> Or is there a good alternative to `np.fromstring(,
>>> dtype=...)`?  -- Marten
>>> >>>>
>>> >>>> On Thu, Oct 22, 2015 at 1:03 PM, Chris Barker <
>>> chris.bar...@noaa.gov> wrote:
>>> >>>>>
>>> >>>>> There was just a question about a bug/issue with scipy.fromstring
>>> (which is numpy.fromstring) when used to read integers from a text file.
>>> >>>>>
>>> >>>>>
>>> https://mail.scipy.org/pipermail/scipy-user/2015-October/036746.html
>>> >>>>>
>>> >>>>> fromstring() is bugging and inflexible for reading text files --
>>> and it is a very, very ugly mess of code. I dug into it a while back, and
>>> gave up -- just to much of a mess!
>>> >>>>>
>>> >>>>> So we really should completely re-implement it, or deprecate it. I
>>> doubt anyone is going to do a big refactor, so that means deprecating it.
>>> >>>>>
>>> >>>>> Also -- if we do want a fast read numbers from text files function
>>> (which would be nice, actually), it really should get a new name anyway.
>>> >>>>>
>>> >>>>> (and the hopefully coming new dtype system would make it easier to
>>> write cleanly)
>>> >>>>>
>>> >>>>> I'm not sure what deprecating something means, though -- have it
>>> raise a deprecation warning in the next version?
>>> >>>>>
>>> 

Re: [Numpy-discussion] [NumPy/Swig] Return NumPy array with same size as input array (no additional length argument)

[Chris Barker]

On Fri, Oct 30, 2015 at 11:15 PM, laurentes 
wrote:

> Using Swig, I don't manage to (properly) create the Python Binding for the
> following C-like function:
>
> void add_array(double* input_array1, double* input_array2, double*
> output_array, int length);
>
> where the three arrays have all the same length.
>

do you have to use SWIG? this would be really easy in Cython

cdef cdef extern from "your_header.h":
void add_array(double* input_array1,
   double* input_array2,
   double* output_array,
   int length)


def py_add_array( np.ndarray[double, ndim=1] arr1,
  np.ndarray[double, ndim=1] arr2):

cdef int length

if arr1.shape != arr2.shape:
raise ValueError("Arrays must be the same size")

length = arr1.shape[0]

cdef np.ndarray[double, ndim=1] out_arr = np.empty((length),
dtype=np.float64)

add_array([0],
  [0],
  _arr[0],
  length)

return out_arr


Untested and from memory -- but you get the idea.

-CHB


> >
> > This is similar to  this thread
> > <
> http://numpy-discussion.10968.n7.nabble.com/Numpy-SWIG-td25709.html#a25710
> >
> > , which has never been fully addressed online.
> >
> > From Python, I would like to be able to call:
> >
> add_array(input_array1, input_array2)
>
> which would return me a newly allocated NumPy array (output_array) with the
> result.
>
> In my Swig file, I've first used the wrapper function trick described  here
> <
> http://web.mit.edu/6.863/spring2011/packages/numpy_src/doc/swig/doc/numpy_swig.html#a-common-example
> >
> , that is:
>
> %apply (double* IN_ARRAY1, int DIM1) {(double* input_array1, int length1),
> (double* input_array2, int length2)};
> %apply (double* ARGOUT_ARRAY1, int DIM1) {(double* output_array, int
> length3)};
>
> %rename (add_array) my_add_array;
> %exception my_add_array {
> $action
> if (PyErr_Occurred()) SWIG_fail;
> }
> %inline %{
> void my_add_array(double* input_array1, int length1, double* input_array2,
> int length2, double* output_array, int length3) {
>   if (length1 != length2 || length1 != length3) {
>   PyErr_Format(PyExc_ValueError,
>"Arrays of lengths (%d,%d,%d) given",
>length1, length2, length3);
>   }
>   else {
> add_array(input_array1, input_array2, output_array, length1);
>   }
> }
> %}
>
> This allows me to call the function from Python using
> add_array(input_array1, input_array2, length). But the third argument of
> this function is useless and this function does not look 'Pythonic'.
>
> Could someone help me to modify my Swig file, such that only the first two
> arguments are required for the Python API?
>
> Thanks a lot,
> Laurent
>
>
>
> --
> View this message in context:
> http://numpy-discussion.10968.n7.nabble.com/NumPy-Swig-Return-NumPy-array-with-same-size-as-input-array-no-additional-length-argument-tp41601.html
> Sent from the Numpy-discussion mailing list archive at Nabble.com.
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

[Chris Barker]

On Tue, Oct 27, 2015 at 8:25 AM, Nathan Goldbaum 
wrote:

> Interestingly, conda actually does "setup.py install" in the recipe for
> numpy:
>

indeed -- many, many conda packages do setup.py install, whihc doesn't mean
it's a good idea --personally, I'm trying hard to switch them all to:

pip install ./

:-)

Which reminds me, the conda skelaton command craes a pip install build.sh
file -- I really need to submit a PR for that ...

There are two cases where a 'pip install' run might go off and start
>> downloading packages without asking you:
>>
>
for my part, regular old setup.py isntall oftem goes off and istalls sutff
too - using easy_install, which really sucks...

This is making me want a setuptools-lite again  -- see the distutils SIG if
you're curious.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] deprecate fromstring() for text reading?

[Chris Barker]

On Tue, Oct 27, 2015 at 7:30 AM, Benjamin Root <ben.v.r...@gmail.com> wrote:

> FWIW, when I needed a fast Fixed Width reader
>

was there potentially no whitespace between fields in that case? In which
case, it really isn a different use-case than delimited text -- if it's at
all common, a version written in C would be nice and fast. and nat hard to
do.

But fromstring never would have helped you with that anyway :-)

-CHB



> for a very large dataset last year, I found that np.genfromtext() was
> faster than pandas' read_fwf(). IIRC, pandas' text reading code fell back
> to pure python for fixed width scenarios.
>
> On Fri, Oct 23, 2015 at 8:22 PM, Chris Barker - NOAA Federal <
> chris.bar...@noaa.gov> wrote:
>
>> Grabbing the pandas csv reader would be great, and I hope it happens
>> sooner than later, though alas, I haven't the spare cycles for it either.
>>
>> In the meantime though, can we put a deprecation Warning in when using
>> fromstring() on text files? It's really pretty broken.
>>
>> -Chris
>>
>> On Oct 23, 2015, at 4:02 PM, Jeff Reback <jeffreb...@gmail.com> wrote:
>>
>>
>>
>> On Oct 23, 2015, at 6:49 PM, Nathaniel Smith <n...@pobox.com> wrote:
>>
>> On Oct 23, 2015 3:30 PM, "Jeff Reback" <jeffreb...@gmail.com> wrote:
>> >
>> > On Oct 23, 2015, at 6:13 PM, Charles R Harris <
>> charlesr.har...@gmail.com> wrote:
>> >
>> >>
>> >>
>> >> On Thu, Oct 22, 2015 at 5:47 PM, Chris Barker - NOAA Federal <
>> chris.bar...@noaa.gov> wrote:
>> >>>
>> >>>
>> >>>> I think it would be good to keep the usage to read binary data at
>> least.
>> >>>
>> >>>
>> >>> Agreed -- it's only the text file reading I'm proposing to deprecate.
>> It was kind of weird to cram it in there in the first place.
>> >>>
>> >>> Oh, fromfile() has the same issues.
>> >>>
>> >>> Chris
>> >>>
>> >>>
>> >>>> Or is there a good alternative to `np.fromstring(,
>> dtype=...)`?  -- Marten
>> >>>>
>> >>>> On Thu, Oct 22, 2015 at 1:03 PM, Chris Barker <chris.bar...@noaa.gov>
>> wrote:
>> >>>>>
>> >>>>> There was just a question about a bug/issue with scipy.fromstring
>> (which is numpy.fromstring) when used to read integers from a text file.
>> >>>>>
>> >>>>>
>> https://mail.scipy.org/pipermail/scipy-user/2015-October/036746.html
>> >>>>>
>> >>>>> fromstring() is bugging and inflexible for reading text files --
>> and it is a very, very ugly mess of code. I dug into it a while back, and
>> gave up -- just to much of a mess!
>> >>>>>
>> >>>>> So we really should completely re-implement it, or deprecate it. I
>> doubt anyone is going to do a big refactor, so that means deprecating it.
>> >>>>>
>> >>>>> Also -- if we do want a fast read numbers from text files function
>> (which would be nice, actually), it really should get a new name anyway.
>> >>>>>
>> >>>>> (and the hopefully coming new dtype system would make it easier to
>> write cleanly)
>> >>>>>
>> >>>>> I'm not sure what deprecating something means, though -- have it
>> raise a deprecation warning in the next version?
>> >>>>>
>> >>
>> >> There was discussion at SciPy 2015 of separating out the text reading
>> abilities of Pandas so that numpy could include it. We should contact Jeff
>> Rebeck and see about moving that forward.
>> >
>> >
>> > IIRC Thomas Caswell was interested in doing this :)
>>
>> When he was in Berkeley a few weeks ago he assured me that every night
>> since SciPy he has dutifully been feeling guilty about not having done it
>> yet. I think this week his paltry excuse is that he's "on his honeymoon" or
>> something.
>>
>> ...which is to say that if someone has some spare cycles to take this
>> over then I think that might be a nice wedding present for him :-).
>>
>> (The basic idea is to take the text reading backend behind
>> pandas.read_csv and extract it into a standalone package that pandas could
>> depend on, and that could also be used by other packages like numpy (among
>> others -- I thing dato's SFrame package has a fork of this code as well?))
&

Re: [Numpy-discussion] deprecate fromstring() for text reading?

[Chris Barker - NOAA Federal]

Grabbing the pandas csv reader would be great, and I hope it happens sooner
than later, though alas, I haven't the spare cycles for it either.

In the meantime though, can we put a deprecation Warning in when using
fromstring() on text files? It's really pretty broken.

-Chris

On Oct 23, 2015, at 4:02 PM, Jeff Reback <jeffreb...@gmail.com> wrote:



On Oct 23, 2015, at 6:49 PM, Nathaniel Smith <n...@pobox.com> wrote:

On Oct 23, 2015 3:30 PM, "Jeff Reback" <jeffreb...@gmail.com> wrote:
>
> On Oct 23, 2015, at 6:13 PM, Charles R Harris <charlesr.har...@gmail.com>
wrote:
>
>>
>>
>> On Thu, Oct 22, 2015 at 5:47 PM, Chris Barker - NOAA Federal <
chris.bar...@noaa.gov> wrote:
>>>
>>>
>>>> I think it would be good to keep the usage to read binary data at
least.
>>>
>>>
>>> Agreed -- it's only the text file reading I'm proposing to deprecate.
It was kind of weird to cram it in there in the first place.
>>>
>>> Oh, fromfile() has the same issues.
>>>
>>> Chris
>>>
>>>
>>>> Or is there a good alternative to `np.fromstring(,
dtype=...)`?  -- Marten
>>>>
>>>> On Thu, Oct 22, 2015 at 1:03 PM, Chris Barker <chris.bar...@noaa.gov>
wrote:
>>>>>
>>>>> There was just a question about a bug/issue with scipy.fromstring
(which is numpy.fromstring) when used to read integers from a text file.
>>>>>
>>>>> https://mail.scipy.org/pipermail/scipy-user/2015-October/036746.html
>>>>>
>>>>> fromstring() is bugging and inflexible for reading text files -- and
it is a very, very ugly mess of code. I dug into it a while back, and gave
up -- just to much of a mess!
>>>>>
>>>>> So we really should completely re-implement it, or deprecate it. I
doubt anyone is going to do a big refactor, so that means deprecating it.
>>>>>
>>>>> Also -- if we do want a fast read numbers from text files function
(which would be nice, actually), it really should get a new name anyway.
>>>>>
>>>>> (and the hopefully coming new dtype system would make it easier to
write cleanly)
>>>>>
>>>>> I'm not sure what deprecating something means, though -- have it
raise a deprecation warning in the next version?
>>>>>
>>
>> There was discussion at SciPy 2015 of separating out the text reading
abilities of Pandas so that numpy could include it. We should contact Jeff
Rebeck and see about moving that forward.
>
>
> IIRC Thomas Caswell was interested in doing this :)

When he was in Berkeley a few weeks ago he assured me that every night
since SciPy he has dutifully been feeling guilty about not having done it
yet. I think this week his paltry excuse is that he's "on his honeymoon" or
something.

...which is to say that if someone has some spare cycles to take this over
then I think that might be a nice wedding present for him :-).

(The basic idea is to take the text reading backend behind pandas.read_csv
and extract it into a standalone package that pandas could depend on, and
that could also be used by other packages like numpy (among others -- I
thing dato's SFrame package has a fork of this code as well?))

-n

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion


I can certainly provide guidance on how/what to extract but don't have
spare cycles myself for this :(

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] deprecate fromstring() for text reading?

[Chris Barker - NOAA Federal]

I think it would be good to keep the usage to read binary data at least.


Agreed -- it's only the text file reading I'm proposing to deprecate. It
was kind of weird to cram it in there in the first place.

Oh, fromfile() has the same issues.

Chris


Or is there a good alternative to `np.fromstring(, dtype=...)`?  --
Marten

On Thu, Oct 22, 2015 at 1:03 PM, Chris Barker <chris.bar...@noaa.gov> wrote:

> There was just a question about a bug/issue with scipy.fromstring (which
> is numpy.fromstring) when used to read integers from a text file.
>
> https://mail.scipy.org/pipermail/scipy-user/2015-October/036746.html
>
> fromstring() is bugging and inflexible for reading text files -- and it is
> a very, very ugly mess of code. I dug into it a while back, and gave up --
> just to much of a mess!
>
> So we really should completely re-implement it, or deprecate it. I doubt
> anyone is going to do a big refactor, so that means deprecating it.
>
> Also -- if we do want a fast read numbers from text files function (which
> would be nice, actually), it really should get a new name anyway.
>
> (and the hopefully coming new dtype system would make it easier to write
> cleanly)
>
> I'm not sure what deprecating something means, though -- have it raise a
> deprecation warning in the next version?
>
> -CHB
>
>
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] deprecate fromstring() for text reading?

[Chris Barker]

There was just a question about a bug/issue with scipy.fromstring (which is
numpy.fromstring) when used to read integers from a text file.

https://mail.scipy.org/pipermail/scipy-user/2015-October/036746.html

fromstring() is bugging and inflexible for reading text files -- and it is
a very, very ugly mess of code. I dug into it a while back, and gave up --
just to much of a mess!

So we really should completely re-implement it, or deprecate it. I doubt
anyone is going to do a big refactor, so that means deprecating it.

Also -- if we do want a fast read numbers from text files function (which
would be nice, actually), it really should get a new name anyway.

(and the hopefully coming new dtype system would make it easier to write
cleanly)

I'm not sure what deprecating something means, though -- have it raise a
deprecation warning in the next version?

-CHB





-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] [Feature Suggestion]More comparison functions for floating point numbers

[Chris Barker]

On Mon, Oct 19, 2015 at 3:06 AM, cy18  wrote:

> I think these would be useful and easy to implement.
>
> greater_close(a, b) = greater_equal(a, b) | isclose(a, b)
> less_close(a, b) = less_equal(a, b) | isclose(a, b)
> greater_no_close = greater(a, b) & ~isclose(a, b)
> less_no_close = less(a, b) & ~isclose(a, b)
>

What's the use-case here? we need is_close because we want to test
equality, but precision errors are such that two floats may be as close to
equal as they can be given the computations done. And the assumption is
that you don't care about the precision to the point you specify.

But for a greater_than (or equiv) comparison, if you the precision is not
important beyond a certain level, then it's generally not important whether
you get greater than or less than when it's that close

And this would great a wierd property that some values would be greater
than, less than, and equal to a target value -- pretty weird!

note that you can get the same effect by subtracting a bit from your
comparison value for a greater than check...

But maybe there is a common use-case that I'm not thinking of..

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Making datetime64 timezone naive

[Chris Barker]

On Sun, Oct 18, 2015 at 12:20 PM, Alexander Belopolsky <ndar...@mac.com>
wrote:

>
> On Sat, Oct 17, 2015 at 6:59 PM, Chris Barker <chris.bar...@noaa.gov>
> wrote:
>
>> Off the top of my head, I think allowing a 60th second makes more sense
>> -- jsut like we do leap years.
>
>
> Yet we don't implement DST by allowing the 24th hour.  Even the countries
> that adjust the clocks at midnight don't do that.
>

Well, isn't that about conforming to already existing standards? DST is a
civil construct -- and mst (all?) implementations use the convention of
having repeated times. -- so that's what software has to deal with.

IIUC, at least +some+standards handle leap seconds by adding a 60th (61st)
second, rather than having a repeated one. So it's at least an option to do
it that way. And it can then fit into the already existing standards for
representing datetimes, etc.

Does the "fold" flag approach for representing, well, "folds" exist in a
widely used standards? It's my impression that it doesn't since we had to
argue a lot about what to call it :-)


> In some sense leap seconds are more similar to timezone changes (DST or
> political) because they are irregular and unpredictable.
>

in that regard, yes -- you need a constantly updating database to use them.
but I don't know that that has any impact on how you represent them. They
seem a lot more like leap years to me -- some februaries have a 29th day --
some hours on some days have a 61st second.


> Furthermore, the notion of "fold" is not tied to a particular 24/60/60
> system of encoding times and thus more applicable to numpy where
> times are encoded as binary integers.
>

but there are no folds in the underlying integer representation -- that is
the "continuous" time scale -- the folds (or leap seconds, or leap years,
or any of the 24/60/60 business comes in only when you want to go to-from
the "datetime" representation.

If anyone decides to actually get around to leap seconds support in numpy
> datetime, s/he can decide ...


This attitude is the reason why we will probably never have bug free
software when it comes to civil time reckoning.

OK -- fair enough -- good to think about it sooner than later.


Similarly, current numpy.datetime64 design ties arithmetic with encoding.
> This makes arithmetic easier, but in the long run may preclude designs that
> better match the problem domain.


I don't follow here -- how can you NOT tied arithmetic to encoding? sure
you could decide that you are going to overload the arithmetic, and it's up
t the object that encodes the data to do that math -- but that's pretty
much what datetime64 is doing -- defining an encoding so that it can do
math -- numpy dtypes are very much about binary representation. No reason
one couldn't make a different numpy dtype for datetimes that encoded it a
different way, and then it would have to implement math, too.



Note how the development of PEP 495 has highlighted the fact that allowing
binary operations (subtraction, comparison etc.) between times in different
timezones was a design mistake.  It will be wise to learn from such
mistakes when redesigning numpy.datetime64.

So was not considering folds -- frankly, and I this this may be your point,
I don't think timezones were well thought out at all when datetime
was first introduced -- and however well thought out it was, if you don't
provide an implementation, you are not going to find the limitations. And
despite Tim's articulate defense of the original impp;imentation decisions,
I think encoding the datetime in the local "calendar/clock" just invites a
mess. And I'm quite convinced that it wouldn't be a the way to go for numpy
use-cases.

If you ever plan to support civil time in some form, you should think about
it now.

well, the goal for now is naive time -- and unlike the original datetime --
we are not adding on a "you can implement your own timezone handling this
way" hook yet.

> In Python 3.6, datetime.now() will return different values in the first
and the second repeated hour in the "fall-back fold." > If you allow
datetime.datetime to numpy.datetime64 conversion, you should decide what
you do with that difference.

Indeed. Though will that only occur with timezones that have DST? I know
I'd be fine with NOT being able to create a numpy datetime64 from a
non-naive datetime object.  Which would force the user to think about and
convert to the timezone they want before passing off to numpy.

Unless you can suggest a sensible default way to handle this. At first
blush, I think naive time does not have folds, so there is no way to handle
them "properly"

Also -- I think we are at phase one of a (at least) two step process:

1) clean up datetime64 just enough that it is useful, and less error-prone
-- i.e. have it not pretend to support 

Re: [Numpy-discussion] Making datetime64 timezone naive

[Chris Barker - NOAA Federal]

> This is fine.  Just be aware that *naive* datetimes will also have the PEP 
> 495 "fold" attribute in Python 3.6.  You are free to ignore it, but you will 
> loose the ability to round-trip between naive stdlib datetimes and 
> numpy.datetime64.

Sigh. I can see why it's there ( primarily to support now(), I
suppose). But a naive datetime doesn't have a timezone, so how could
you know what  time one actually corresponds to if fold is True? And
what could you do with it if you did know?

I've always figured that if you are using naive time for times in a
timezone that has DST, than you'd better know wether you were in DST
or not.

(Which fold tells you, I guess) but the fold isn't guaranteed to be an
hour is it? So without more info, what can you do? And if the fold bit
is False, then you still have no idea if you are in DST or not.

And then what if you attach a timezone to it? Then the fold bit could
be wrong...

I take it back, I can't see why the fold bit could be anything but
confusing for a naive datetime. :-)

Anyway, all I can see to do here is for the datetime64 docs to say
that fold is ignored if it's there.

But what should datetime64 do when provided with a datetime with a timezone?

- Raise an exception?
- ignore the timezone?
- Convert to UTC?

If the time zone is ignored, then you could get DST and non DST times
in the same array - that could be ugly.

Is there any way to query a timezone object to ask if it's a constant-offset?

And yes, I did mean "most". There is no way I'm ever going to
introduce a three letter "timezone" abbreviation in one of these
threads!

-CHB
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Making datetime64 timezone naive

[Chris Barker]

On Fri, Oct 16, 2015 at 10:19 AM, Alexander Belopolsky 
wrote:

> Since Guido hates leap seconds, PEP 495 is silent on this issue, but
> strictly speaking UTC leap seconds are "folds."   AFAICT, a strictly POSIX
> system must repeat the same value of time_t when a leap second is
> inserted.  While datetime will never extend the second field to
>
 allow second=60, with PEP 495, it is now possible to represent 23:59:60 as
 23:59:59/fold=1.

Thanks -- If anyhone decides to actually get arond to leap seconds suport
in numpy datetime, se can decide whether to do folds or allow second: 60.

Off the top of my head, I think allowing a 60th second makes more sense --
jsut like we do leap years. Granted, external systems often don't
understand/allow a 60th second, but they generally don't understand a fold
bit, either

-CHB






> Apart from leap seconds, there is no need to use "fold" on datetimes that
> represent time in UTC or any timezone at a fixed offset from utc.
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] how is toolchain selected for compiling (OS X with python.org build)?

[Chris Barker]

you might try the python-mac list:

https://mail.python.org/mailman/listinfo/pythonmac-sig

not very active, but folks there know what they are doing :-)

-CHB


On Thu, Oct 15, 2015 at 8:49 AM, Andrew Jaffe  wrote:

> This isn't, strictly speaking, a numpy question, but I suspect it's
> something that numpy devs and users have some insight into.
>
> I am trying to compile an extension that requires a fairly advanced c++
> compiler. Using the built-in apple python, it defaults to the latest clang
> from apple, and it works just fine.
>
> Using the python.org framework build, it still selects clang, which is in
> principle a new enough compiler, but for some reason it seems to end up
> pointing to /usr/include/c++/4.2.1/ which of course is too old, and the
> build fails.
>
> So the questions I have are:
>
>  - *why* is it using such an old toolchain (I am pretty sure that the
> answer is backward compatibility, and specifically because that is the way
> the framework build python is itself compiled).
>
>  - *how* is it selecting those tools, and in particular, that include
> directory? It doesn't seem to explicitly show up in the logs, until there's
> an error. If I just use the same clang invocation as seems to be used by
> the build, it is able to compile full C++-11 code...
>
>  - Is there any way to still use the apple clang, but in full c++-11 mode
> to build extensions?
>
> The solution/workaround is to install and then explicitly select a more
> advanced compiler, e.g., from homebrew, using environment variables, but it
> would be nice if it could work out of the box, and ideally with the same
> behaviour as with apple's python build.
>
> -Andrew
>
> p.s. for the aficionados, this is for [healpy][1], and we're looking at it
> with [this issue][2].
>
> [1]: https://github.com/healpy
> [2]: https://github.com/healpy/healpy/issues/284#issuecomment-148354405
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy 1.10.1 released.

[Chris Barker]

On Mon, Oct 12, 2015 at 9:27 AM, Charles R Harris  wrote:

> * Compiling with msvc9 or msvc10 for 32 bit Windows now requires SSE2.
>   This was the easiest fix for what looked to be some miscompiled code when
>   SSE2 was not used.
>

Note that there is discusion right now on pyton-dev about requireing SSE2
for teh python.org build of python3.5 -- it does now, so it's fine for
third party pacakges to also require it. But there is some talk of removing
that requirement -- still a lot of old machines around, I guess --
particular at schools and the like.

Ideally, any binary wheels on PyPi should be compatible with the python.org
builds -- so not require SSE2, if the python.org builds don't.

Though we had this discussion a while back -- and numpy could, and maybe
should require more -- did we ever figure out a way to get a meaningful
message to the user if they try to run an SSE2 build on a machine without
SSE2?

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Making datetime64 timezone naive

[Chris Barker]

On Tue, Oct 13, 2015 at 3:58 PM, Nathaniel Smith  wrote:

> > However, numpy datetime is optimized for compact storage and fast
> computation of absolute deltas (actual hours, minutes, seconds... not
> calendar units like "the next day" ).
>
> Except that ironically it actually can't compute absolute deltas
> accurately with one second resolution, because it does the POSIX time thing
> of pretending that all UTC days have the same number of seconds, even
> though this is not true (leap seconds).
>
Note that I said "fast", not "accurate" -- but the leap second thing may be
one more reason not to call datetime64 "UTC" -- who's to say that "naive"
time should include leap seconds :-)

Also, we could certainly add a leap seconds implementation to the current
infrastructure -- the real technical problem with that is how to keep the
leap-seconds table up to date -- we have no way to know when there will be
leap-seconds in the future...

Also -- this may be one more reason to have a selectable epoch -- then
you'd likely overlap fewer leap-seconds in a given us case.

> This isn't really relevant to anything else in this thread, except as a
> reminder of how freaky date/time handling is.
>

yup -- it sure is.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Making datetime64 timezone naive

[Chris Barker]

On Tue, Oct 13, 2015 at 5:08 PM, Marten van Kerkwijk <
m.h.vankerkw...@gmail.com> wrote:

> Maybe not directly relevant, but also very clearly why one should ideally
>> not use these a
>>
> all!
>>
> I wouldn't say not at all -- I'd say "not in some circumstances"

> Perhaps even less relevant, but if you do need absolute times (and thus
>> work with UTC or TAI or GPS), have a look at astropy's `Time` class. It
>> does use two doubles,
>>
> interesting -- I wonder why not two integers?

> but with that maintains "sub-nanosecond precision over times spanning the
>> age of the universe" [1].
>>
> well, we do all need that!

Seriously, though -- if we are opening all this up, maybe it's worth
considering other options, rather than kludging datetime64 -- particularly
if there is something someone has already implemented and tested...

But for now, Stephan's patches to make datetime64 far more useful and easy
are very welcome!

-CHB

[1] http://docs.astropy.org/en/latest/time/index.html
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy 1.10.1 released.

[Chris Barker]

On Wed, Oct 14, 2015 at 9:38 AM, Nathaniel Smith  wrote:

> The change in 1.10.1 only affects msvc, which is not what most people are
> using (IIUC Enthought Canopy uses msvc, but the pypi, gohlke, and Anaconda
> builds don't).
>
Anaconda uses MSVC for the most part -- they _may_ compile numpy itself
some other way, no one but continuum knows for sure  :-)

> I'm actually not sure if anyone even uses the 32 bit builds at all :-)
>
There's a lot of 32 bit python use out there still, including numpy.

We ever figure out a way to get a meaningful message to the user if they
> try to run an SSE2 build on a machine without SSE2?
>
> It's not that difficult in principle, just someone has to do it :-).
>
yeah, there's always that 

-CHB



> -n
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy 1.10.1 released.

[Chris Barker]

On Wed, Oct 14, 2015 at 12:38 PM, Ralf Gommers 
wrote:

> I'm actually not sure if anyone even uses the 32 bit builds at all :-)
>>>
>> There's a lot of 32 bit python use out there still, including numpy.
>>
>
> If you want a quick impression, there are download stats for our binaries:
> http://sourceforge.net/projects/numpy/files/NumPy/
>
> The total number of 32-bit .exe installer downloads for the last week is
> ~5000.
>

That may be somewhat skewed by the fact that we don't provide 64 bit
intstallers a t all (or did I miss something?) But nevertheless, plenty of
users...

-CHB




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Making datetime64 timezone naive

[Chris Barker]

On Wed, Oct 14, 2015 at 10:34 AM, Phil Hodge <ho...@stsci.edu> wrote:

> On 10/14/2015 11:59 AM, Chris Barker wrote:
>
>> we have no way to know when there will be leap-seconds in the future
>>
>
> Leap seconds are announced about six months in advance.


exactly -- so more than six month, we have no idea.

and even within six months, then you'd need to update some sort of database
of leapseconds to get it.

So depending on what version of the DB someone was using, they'd get
different answers.

That could all get ugly :-(

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Make all comparisons with NaT false?

[Chris Barker]

On Sun, Oct 11, 2015 at 8:38 PM, Stephan Hoyer  wrote:

> Currently, NaT (not a time) does not have any special treatment when used
> in comparison with datetime64/timedelta64 objects.
>
> To me, this seems a little crazy for a value meant to denote a
> missing/invalid time -- NaT should really have the same comparison behavior
> as NaN.
>

Yes, indeed.


> Whether you call this an API change or a bug fix is somewhat of a judgment
> call, but I believe this change is certainly consistent with the goals of
> datetime64. It's also consistent with how NaT is used in pandas, which uses
> its own wrappers around datetime64 precisely to fix these sorts of issues.
>

Getting closer to Pandas is a Good Thing too...


> So I'm raising this here to get some opinions on the right path forward:
> 1. Is this a bug fix that we can backport to 1.10.x?
> 2. Is this an API change that should wait until 1.11?
> 3. Is this something where we need to start issuing warnings and deprecate
> the existing behavior?
>
> My vote would be for option 2.
>

I agree.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Making datetime64 timezone naive

[Chris Barker]

On Mon, Oct 12, 2015 at 11:48 AM, Alexander Belopolsky 
wrote:

> If you are going to make datetime64 more like datetime.datetime, please
> consider adding the "fold" bit.  See PEP 495. [1]
>
> [1]: https://www.python.org/dev/peps/pep-0495/
>

well, adding any timezone support is not (yet) in the table.

(no need for "fold" with purely naive time, yes?)

But yes, when we get there, absolutely.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Making datetime64 timezone naive

[Chris Barker]

On Mon, Oct 12, 2015 at 12:38 AM, Nathaniel Smith  wrote:

> > As a temporary measure, we still will parse datetimes that include a
> > timezone specification by converting them to UTC, but will issue a
> > DeprecationWarning. This is important for a smooth transition, because at
> > the very least I suspect the "Z" modifier for UTC is widely used. Another
> > option would be to preserve this conversion indefinitely, without any
> > deprecation warning.
>
> I'm dubious about supporting conversions in the long run -- even "Z"
> -- because UTC datetimes and naive datetimes are really not the same
> thing.


no -- but almost!


> OTOH maybe if we dropped this it would break everyone's code
> and they would hate us --


I think it probably would. In the current implementation, an ISO string
without an offset specifier is converted using the system's locale
timezone. So to get naive time (or UTC), we need to tack a Z (or 00:00) on
there.

So killing that would likely break a lot of code!

And excepting a Z or 00:00 and then treating it as naive, while being
perhaps misleading, would not actually change any results. So I say we keep
it.

Depreciating it eventually would be good in the long run -- but maybe when
we have actual time zone support.

I actually have no idea what people are
> doing with datetime64 outside of pandas.


What do we need to do with this not to break Panda? I'm guessing more
people use datetime64 wrapped by Pandas than any other way...

(not me, though)

> There's one (slightly) contentious API decision to make: What should we do
> > with the numpy.datetime_to_string function? As far as I can tell, it was
> > never documented as part of the NumPy API and has not been used very
> much


Well, I'm not using it :-) though I can see that it might be pretty useful.
Though once we get rid of datetime64 adjusting for the locale timezone,
maybe not anymore.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Deprecating unitless timedelta64 and "safe" casting of integers to timedelta64

[Chris Barker]

On Tue, Oct 13, 2015 at 10:48 AM, Stephan Hoyer  wrote:

> This led to my discovery that NumPy currently supports unit-less
> timedeltas (e.g., "np.timedelta64(5)"), which indicate some sort of generic
> time difference. The current behavior is to take the time units from the
> other argument when these are used in a binary operation.
>

this really is odd :-)


> Even worse, we currently support "safe" casting of integers to
> timedelta64, which means that integer + datetime64 and integer +
> timedelta64 arithmetic works:
>

which makes the above even odder -- underlying datetime64 is, "just" a 64
bit int -- so I can see how someone _may_ want to work directly with that
-- but if you can use regular integerts, why have a unitless timedelta?

Based on the principle that NumPy's datetime support should mirror the
> standard library as much as possible, both of these behaviors seem like a
> bad idea. We have datetime types precisely to disambiguate these sort of
> situations.
>
> I'd like to propose deprecating such casting in NumPy 1.11, with the
> intent of removing it entirely as soon as practical.
>

Agreed -- I can imagine use-cases, but users can cadt to/from integers if
that's what they want to do e.g. with .astype()

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Making datetime64 timezone naive

[Chris Barker]

On Tue, Oct 13, 2015 at 3:21 PM, Nathaniel Smith  wrote:


> > If you are going to make datetime64 more like datetime.datetime, please
> consider adding the "fold" bit.  See PEP 495. [1]
>
The challenge here is that we literally do not have a bit too use :-)
>

hmm -- I was first thinking that this could all be in the timezone stuff
(when we get there), but while I imagine we'll want an entire array to be
in a single timezone, each individual value would need its own "fold" flag.

But in any case, we don't need it 'till we do timezones, and my
understanding is that we aren't' going to do timezones until we have the
mythical new-and-improved-dtype-system.

So a future datetime dtype could be 64 bits + a byte of extra info, or be
63 bits plus the fold flag, or...

> Unless we make it datetime65 + 63 bits of padding, stealing a bit to use
> for fold would halve the range of representable times, and I'm guessing
> this would not be acceptable?
>
well, not now, with eh fixed epoch, but if the epoch could be adjusted,
maybe a small range would be fine -- who need nanosecond accuracy, AND
centuries of range?

Thinking a bit more here:


For those that didn't follow the massive discussion on this on Python-dev
and the new datetime list:

the fold flag is required to round-trip properly for timezones with
discontiguous time -- i.e. Daylight savings. So if you have:

2015-11-01T01:30

Do you mean the first 1:30 am or the seconds one, after the DST transition?
(i.e. in the fold, or not?)

So it is key, for Python's Datetime, to make sure to keep that information
around.

However: Python's datetime was designed to be optimized for:
  - converting between datetime and other representations in Database, etc.
  - fast math for "naive time" -- i.e. basic manipulations within the same
timezone, like "one day later"
  - Fast math for "absolute relative deltas" is of secondary concern.

The result of this is that datetime stores: year, month, day, hour minute
second, microsecond

It does NOT store some time_unit_since_an_epch, like unix time or numpy
datetime64.

Also, IIUC, when you associate a datetime with a timezone, it stores the
year, month, day, hour, second,... in the specified timezone -- NOT in UTC,
or anything else. This makes manipulations within that timezone easy -- the
next day simply  required adding a day to teh day field (then normalizing
to the month).

Given all that -- the "fold" bit is needed, as a particular datetime in a
particular timezone may have more than one meaning.

Note that to compute a proper time span between two "aware" datetimes, it
is necessary to convert to UTC, do the math, then convert back to the
timezone you want.

However, numpy datetime is optimized for compact storage and fast
computation of absolute deltas (actual hours, minutes, seconds... not
calendar units like "the next day" ).

Because of this, and because it's what we already have, datetime64 stores
times as "some number of time units since an epoch -- a simple integer.

And because we probably want fast absolute delta computation, when we add
timezones, we'll probably want to store the datetime in UTC, and apply the
timezone on I/O.

Alexander: Am I right that we don't need the "fold" bit in this case? You'd
still need it when specifying a time in a timezone with folds.. -- but
again, only on I/O

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] msvc9 comipiler problems

[Chris Barker]

>
>
> NVM. Looks like Python 2.7 also uses msvc9.
>

yup, according to Wikipedia:

*Visual C++ 2008* (known also as Visual C++ 9.0)

so py2.7

Are you testing with the "MS Visual C++ compiler for Python 2.7" here:

http://www.microsoft.com/en-us/download/details.aspx?id=44266

I think the only difference is how.where it is installed, but you never
know...

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NumFOCUS fiscal sponsorship agreement

[Chris Barker - NOAA Federal]

Looks good to me.

This pretty exciting, actually :-)

-CHB

Sent from my iPhone

> On Oct 7, 2015, at 10:57 PM, Nathaniel Smith  wrote:
>
> Hi all,
>
> Now that the governance document is in place, we need to get our legal
> ducks in a row by signing a fiscal sponsorship agreement with
> NumFOCUS.
>
> The basic idea here is that there are times when you really need some
> kind of corporation to represent the project -- the legal system for
> better or worse does not understand "a bunch of folks on a mailing
> list" as a legal entity capable of accepting donations, or holding
> funds or other assets like domain names. The obvious solution is to
> incorporate a company to represent the project -- but incorporating a
> company involves lots of super-annoying paperwork. (Like, *super*
> annoying.) So a standard trick is that a single non-profit corporation
> acts as an umbrella organization providing these services to multiple
> projects at once, and this is called "fiscal sponsorship". You can
> read more about it here:
>https://en.wikipedia.org/wiki/Fiscal_sponsorship
>
> NumFOCUS's standard comprehensive FSA agreement can be seen here:
>
>
> https://docs.google.com/document/d/11YqMX9UrgfCSgiQEUzmOFyg6Ku-vED6gMxhO6J9lCgg/edit?usp=sharing
>
> and we have the option of negotiating changes if there's anything we don't 
> like.
>
> They also have a FAQ:
>
> https://docs.google.com/document/d/1zdXp07dLvkbqBrDsw96P6mkqxnWzKJuM-1f4408I6Qs/edit?usp=sharing
>
> I've read through the document and didn't see anything that bothered
> me, except that I'm not quite sure how to make the split between the
> steering council and numfocus subcommittee that we have in our
> governance model sync up with their language about the "leadership
> body", and in particular the language in section 10 about simple
> majority votes. So I've queried them about that already.
>
> In the mean time, I'd encourage anyone with an interest to look it
> over and speak up if you see anything that you think should be changed
> before we sign.
>
> Cheers,
> -n
>
> --
> Nathaniel J. Smith -- http://vorpus.org
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Let's move forward with the current governance document.

[Chris Barker]

On Mon, Oct 5, 2015 at 2:11 PM, Sturla Molden 
wrote:

> I just envisioned a Roman patron shouting veto or a US senator
> filibustering. Expulsion would be the appropriate recation, yes :-)


Oh if only the US Senate could expulse people!

-sigh


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NumPy Governance Document.

[Chris Barker]

>
>
> The  NumPy Governance Document 
> has been merged and is now in effect.
>

whoo hoo!

And a special thanks to Nathaniel,
>

Indeed -- and everyone else that put a lot of their time into the
discussion.

Looking forward to discussing technical issues again :-)

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] composition of the steering council (was Re: Governance model request)

[Chris Barker]

On Tue, Sep 29, 2015 at 7:32 AM, Travis Oliphant 
wrote:

> I'm in a situation now where at least for 6 months or so I can help with
> NumPy more than I have been able to for 7 years.
>

great news!


> 1) 1 year of inactivity to be removed from the council is too little for a
> long-running project like NumPy
>

I always read this as "activity in council matters", not "activity in
commits to the code". If that's not what is meant, then we should re-write
that. So no, Travis is not ineligible for the council, as there has been no
council to be active on.

And in that case, my vague memory is that Travis has popped his head into
architecture discussions on this list at least once a year essentially
forever, so there is every indication that he has been and will be active
in that sense.

--- somewhere between 2 and 4 years would be more appropriate.   I suppose
> 1 year of inactivity is fine if that is defined only as "failure to vote on
> matters before the council"
>

exactly. -- I wld expand "activity" to mean more than voting, but that's
the idea. More like active participation in discussions / issues, and/or
attending meetings, if there are any.


> 2) The seed council should not just be recent contributors but should
> include as many people as are willing to help who have a long history with
> the project.
>

again, I don't think we need so much emphasis in the "seed council" that is
only needed to have SOME way to start. Though I guess as long as we're
having the discussion, we should have an outline of the expectations for
the council in general, which should indeed include a couple members with
long history.


> 3) I think people who contribute significantly generally should be able to
> re-join the steering council more easily than "going through the 1-year
> vetting process" again --- they would have to be approved by the current
> steering council but it should not be automatically disallowed (thus
> requiring the equivalent of an amendment to change it).
>

Sure -- give the Council Flexibility to keep the council current. However,
the idea here is that we want some continuity -- not have people popping on
and off the council on short time spans as their schedules allow.

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Sign of NaN

[Chris Barker]

On Tue, Sep 29, 2015 at 6:35 PM, Charles R Harris  wrote:

> For this, and other use-cases, special casing Numpy arrays stored in
>>> object arrays does make sense:
>>>
>>> "If this is s a Numpy array, pass the operation through."
>>>
>>
>> Because we now (development) use rich compare, the result looks like
>>
>> Oops, not what was intended. In fact it raises an error
>
> In [7]: b
> Out[7]: array([array([ 1.,  1.,  1.]), array([-1., -1., -1.])],
> dtype=object)
>
> In [8]: sign(b)
> ---
> ValueErrorTraceback (most recent call last)
>  in ()
> > 1 sign(b)
>
> ValueError: The truth value of an array with more than one element is
> ambiguous. Use a.any() or a.all()
>

exactly -- it seems to me that a special case for numpy arrays as objects
in object arrays makes sense, so you'd get:

In [6]: oa
Out[6]:
array([[1.0, 1.0, 1.0],
   [-1.0, -1.0, -1.0]], dtype=object)

In [7]: np.sign(oa)
Out[7]:
array([[1, 1, 1],
   [-1, -1, -1]], dtype=object)

(which you do now in the version I'm running).

Though rather than the special case, maybe we really need dtype=ndarray
arrays?

 oa = np.array([a1, a2], dtype=np.ndarray)

Then we could count on everything in the array being an array.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion