[Numpy-discussion] Advanced numpy tutorial for SciPy?

[Chris Barker - NOAA Federal]

In another thread, there is a discussion of a workshop on "Taking
NumPy In Stride" for PyData Barcelona.

I think it would be great to have something like that at SciPy in
Austin this year.

Jaime can't make it, and I don't think strides are going to fill a
four hour tutorial, so it would be good as part of an advanced numpy
tutorial.

I don't have the bandwidth to put together an entire tutorial, but
maybe someone would like to join forces?

Or if someone is already planning an advanced numpy tutorial, perhaps
I could contribute.

Not much time left to get a proposal in!

-Chris
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] From Python to Numpy

[Chris Barker - NOAA Federal]

> It seems a generalized ufunc "all_equal" with signature (i),(i)->() and short 
> circuit logic once the first non equal element is encountered would be an 
> important performance improvement.

How does array_equal() perform?

-CHB
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Is numpy.test() supposed to be multithreaded?

[Chris Barker - NOAA Federal]

> Now the user is writing back to say, "my test code is fast now, but
> numpy.test() is still about three times slower than  don't manage>".  When I watch htop as numpy.test() executes, sure enough,
> it's using one core
>

* if numpy.test() is supposed to be using multiple cores, why isn't it,
> when we've established with other test code that it's now using multiple
> cores?
>

Some numpy.linalg functions (like np.dot) will be using multiple cores, but
np.linalg.test() takes only ~1% of the time of the full test suite.
Everything else will be running single core. So your observations are not
surprising.


Though why it would run slower on one box than another comparable box is a
mystery...

-CHB
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] DyND 0.7.1 Release

[Chris Barker - NOAA Federal]

> The DyND team would be happy to answer any questions people have about DyND, 
> like "what is working and what is not" or "what do we still need to do to hit 
> DyND 1.0".

OK, how about:

How does the performance. I'd DyND compare to Numpy for the core
functionality they both support?

- CHB
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy 1.11.0b2 released

[Chris Barker - NOAA Federal]

> An extra ~2 hours of tests / 6-way parallelism is not that big a deal
> in the grand scheme of things (and I guess it's probably less than
> that if we can take advantage of existing binary builds)

If we set up a numpy-testing conda channel, it could be used to cache
binary builds for all he versions of everything we want to test
against.

Conda-build-all could make it manageable to maintain that channel.

-CHB
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should I use pip install numpy in linux?

[Chris Barker - NOAA Federal]

> Last month, numpy had ~740,000 downloads from PyPI,

Hm, given that Windows and Linux wheels have not been available, then
that's mostly source installs anyway. Or failed installs  -- no
shortage of folks trying to pip install numpy on Windows and then
having questions about why it doesn't work. Unfortunately, there is no
way to know if pip downloads are successful, or if people pip install
Numpy, then find out they need some other non-pip-installable
packages, and go find another system.

> and there are
> probably hundreds of third-party projects that distribute via PyPI and
> depend on numpy.

I'm not so sure -- see above--as pip install has not been reliable for
Numpy for ages, I doubt it. Not that they aren't there, but I doubt
it's the primary distribution mechanism. There's been an explosion in
the use of conda, and there have been multiple other options for ages:
Canopy, python(x,y), Gohlke's builds, etc.

So at this point, I think the only people using pip are folks that are
set up to build -- mostly Linux. (though Mathew's efforts with the Mac
wheels may have created a different story on the Mac).



> So concretely our options as a project are:
> 1) drop support for pip/PyPI and abandon those users

There is no one to abandon -- except the Mac users -- we haven't
supported them yet.

> 2) continue to support those users fairly poorly, and at substantial
> ongoing cost

I'm curious what the cost is for this poor support -- throw the source
up on PyPi, and we're done. The cost comes in when trying to build
binaries...


> Option 1 would require overwhelming consensus of the community, which
> for better or worse is presumably not going to happen while
> substantial portions of that community are still using pip/PyPI.

Are they? Which community are we talking about? The community I'd like
to target are web developers that aren't doing what they think of as
"scientific" applications, but could use a little of the SciPy stack.
These folks are committed to pip, and are very reluctant to introduce
a difficult dependency.  Binary wheels would help these folks, but
that is not a community that exists yet ( or it's small, anyway)

All that being said, I'd be happy to see binary wheels for the core
SciPy stack on PyPi. It would be nice for people to be able to do a
bit with Numpy or pandas, it MPL, without having to jump ship to a
whole new way of doing things.

But we should be realistic about how far it can go.

> If
> folks interested in pushing things forward can convince them to switch
> to conda instead then the calculation changes, but that's just not the
> kind of thing that numpy-the-project can do or not do.

We can't convince anybody, but we can decide where to expend our efforts.

-CHB
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Behavior of np.random.uniform

[Chris Barker - NOAA Federal]

What does the standard lib do for rand range? I see that randint Is closed
on both ends, so order doesn't matter, though if it raises for b wrote:

Of the methods defined in *numpy/mtrand.pyx* (excluding helper functions
and *random_integers*, as they are all related to *randint*), *randint* is
the only other function with *low* and *high* parameters.  However, it
enforces *high* > *low*.

Greg

On Tue, Jan 19, 2016 at 1:36 PM, Benjamin Root  wrote:

> Are there other functions where this behavior may or may not be happening?
> If it isn't consistent across all np.random functions, it probably should
> be, one way or the other.
>
> Ben Root
>
> On Tue, Jan 19, 2016 at 5:10 AM, Jaime Fernández del Río <
> jaime.f...@gmail.com> wrote:
>
>> Hi all,
>>
>> There is a PR (#7026 ) that
>> documents the current behavior of np.random.uniform when the low and high
>> parameters it takes do not conform to the expected low < high. Basically:
>>
>>- if low < high, random numbers are drawn from [low, high),
>>- if low = high, all random numbers will be equal to low, and
>>- if low > high, random numbers are drawn from (high, low] (notice
>>the change in the open side of the interval.)
>>
>> My only worry is that, once we document this, we can no longer claim that
>> it is a bug.  So I would like to hear from others what do they think.  The
>> other more or less obvious options would be to:
>>
>>- Raise an error, but this would require a deprecation cycle, as
>>people may be relying on the current undocumented behavior.
>>- Check the inputs and draw numbers from [min(low, high), max(low,
>>high)), which is minimally different from current behavior.
>>
>> I will be merging the current documentation changes in the next few days,
>> so it would be good if any concerns were voiced before that.
>>
>> Thanks,
>>
>> Jaime
>>
>> --
>> (\__/)
>> ( O.o)
>> ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes
>> de dominación mundial.
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should I use pip install numpy in linux?

[Chris Barker - NOAA Federal]

>> Also, you have the problem that there is one PyPi -- so where do you put
>> your nifty wheels that depend on other binary wheels? you may need to fork
>> every package you want to build :-(
>
> Is this a real problem or a theoretical one? Do you know of some
> situation where this wheel to wheel dependency will occur that won't
> just be solved in some other way?

It's real -- at least during the whole bootstrapping period. Say I
build a nifty hdf5 binary wheel -- I could probably just grab the name
"libhdf5" on PyPI. So far so good. But the goal here would be to have
netcdf and pytables and GDAL and who knows what else then link against
that wheel. But those projects are all supported be different people,
that all have their own distribution strategy. So where do I put
binary wheels of each of those projects that depend on my libhdf5
wheel? _maybe_ I would put it out there, and it would all grow
organically, but neither the culture nor the tooling support that
approach now, so I'm not very confident you could gather adoption.

Even beyond the adoption period, sometimes you need to do stuff in
more than one way -- look at the proliferation of channels on
Anaconda.org.

This is more likely to work if there is a good infrastructure for
third parties to build and distribute the binaries -- e.g.
Anaconda.org.

Or the Linux dist to model -- for the most part, the people developing
a given library are not packaging it.

-CHB
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] array_equal too strict?

[Chris Barker - NOAA Federal]

> If you have some spare cycles, maybe you can open a pull request to add
> np.isclose to the "See Also" section?

That would be great.

Remember that equality for flits is bit-for but equality ( baring NaN
and inf...).

But you hardly ever actually want to do that with floats.

But probably np.allclose is most appropriate here.

CHB

>
> - Sebastian
>
>
>> ```
>> print(numpy.array_equal(a1, a2))
>>
>> # output: true
>> ```
>> That's expected because the difference is only in the 18th overall
>> digit, and the mantissa length of float64 is 52 bits [1], i.e., approx
>> 15.6 decimal digits. Moving the difference to the 17th overall digit
>> should also be fine, however:
>> ```
>> a1 = numpy.array(
>>[3.1415926535897930],
>>dtype=numpy.float64
>>)
>> a2 = numpy.array(
>>[3.1415926535897939],
>>dtype=numpy.float64
>>)
>>
>>
>> print(numpy.array_equal(a1, a2))
>> # output: false
>> ```
>> It gets even more visible with float32 and its 23 mantissa bits (i.e.,
>> 6.9 decimal digits):
>> ```
>> a1 = numpy.array(
>>[3.14159260],
>>dtype=numpy.float32
>>)
>> a2 = numpy.array(
>>[3.14159269],
>>dtype=numpy.float32
>>)
>>
>>
>> print(numpy.array_equal(a1, a2))
>> # output: false
>> ```
>> The difference is only in the 9th decimal digit, still `array_equal_
>> detects the difference.
>>
>>
>> I'm not sure where I'm going wrong here. Any hints?
>>
>>
>> Cheers,
>> Nico
>>
>>
>>
>>
>> [1] https://docs.scipy.org/doc/numpy-1.10.1/user/basics.types.html
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Fast vectorized arithmetic with ~32 significant digits under Numpy

[Chris Barker - NOAA Federal]

> There has also been some talk of adding a user type for ieee 128 bit doubles. 
> I've looked once for relevant code for the latter and, IIRC, the available 
> packages were GPL :(.


This looks like it's BSD-Ish:

http://www.jhauser.us/arithmetic/SoftFloat.html

Don't know if it's any good

CHB


>
> Chuck
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] [Distutils] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

[Chris Barker - NOAA Federal]

>> I'm not talking about in place installs, I'm talking about e.g. building a
>> wheel and then tweaking one file and rebuilding -- traditionally build
>> systems go to some effort to keep track of intermediate artifacts and reuse
>> them across builds when possible, but if you always copy the source tree
>> into a temporary directory before building then there's not much the build
>> system can do.

This strikes me as an optimization -- is it an important one?

If I'm doing a lot of tweaking and re-running, I'm usually in develop mode.

I can see that when you build a wheel, you may build it, test it,
discover an wheel-specific error, and then need to repeat the cycle --
but is that a major use-case?

That being said, I have been pretty frustrated debugging conda-build
scripts -- there is a lot of overhead setting up the build environment
each time you do a build...

But with wheel building there is much less overhead, and far fewer
complications requiring the edit-build cycle.

And couldn't make-style this-has-already-been-done checking happen
with a copy anyway?

CHB

> Ah yes. So I don't think pip should do what it does. It a violation of
> the abstractions we all want to see within it. However its not me you
> need to convince ;).
>
> -Rob
>
> --
> Robert Collins 
> Distinguished Technologist
> HP Converged Cloud
> ___
> Distutils-SIG maillist  -  distutils-...@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] deprecate fromstring() for text reading?

[Chris Barker - NOAA Federal]

I was more aiming to point out a situation where the NumPy's text file
reader was significantly better than the Pandas version, so we would want
to make sure that we properly benchmark any significant changes to NumPy's
text reading code. Who knows where else NumPy beats Pandas?
Indeed. For this example, I think a fixed-with reader really is a different
animal, and it's probably a good idea to have a high performance one in
Numpy. Among other things, you wouldn't want it to try to auto-determine
data types or anything like that.

I think what's on the table now is to bring in a new delimited reader --
I.e. CSV in its various flavors.

CHB


Ben

On Mon, Nov 2, 2015 at 6:44 PM, Chris Barker <chris.bar...@noaa.gov> wrote:

> On Tue, Oct 27, 2015 at 7:30 AM, Benjamin Root <ben.v.r...@gmail.com>
> wrote:
>
>> FWIW, when I needed a fast Fixed Width reader
>>
>
> was there potentially no whitespace between fields in that case? In which
> case, it really isn a different use-case than delimited text -- if it's at
> all common, a version written in C would be nice and fast. and nat hard to
> do.
>
> But fromstring never would have helped you with that anyway :-)
>
> -CHB
>
>
>
>> for a very large dataset last year, I found that np.genfromtext() was
>> faster than pandas' read_fwf(). IIRC, pandas' text reading code fell back
>> to pure python for fixed width scenarios.
>>
>> On Fri, Oct 23, 2015 at 8:22 PM, Chris Barker - NOAA Federal <
>> chris.bar...@noaa.gov> wrote:
>>
>>> Grabbing the pandas csv reader would be great, and I hope it happens
>>> sooner than later, though alas, I haven't the spare cycles for it either.
>>>
>>> In the meantime though, can we put a deprecation Warning in when using
>>> fromstring() on text files? It's really pretty broken.
>>>
>>> -Chris
>>>
>>> On Oct 23, 2015, at 4:02 PM, Jeff Reback <jeffreb...@gmail.com> wrote:
>>>
>>>
>>>
>>> On Oct 23, 2015, at 6:49 PM, Nathaniel Smith <n...@pobox.com> wrote:
>>>
>>> On Oct 23, 2015 3:30 PM, "Jeff Reback" <jeffreb...@gmail.com> wrote:
>>> >
>>> > On Oct 23, 2015, at 6:13 PM, Charles R Harris <
>>> charlesr.har...@gmail.com> wrote:
>>> >
>>> >>
>>> >>
>>> >> On Thu, Oct 22, 2015 at 5:47 PM, Chris Barker - NOAA Federal <
>>> chris.bar...@noaa.gov> wrote:
>>> >>>
>>> >>>
>>> >>>> I think it would be good to keep the usage to read binary data at
>>> least.
>>> >>>
>>> >>>
>>> >>> Agreed -- it's only the text file reading I'm proposing to
>>> deprecate. It was kind of weird to cram it in there in the first place.
>>> >>>
>>> >>> Oh, fromfile() has the same issues.
>>> >>>
>>> >>> Chris
>>> >>>
>>> >>>
>>> >>>> Or is there a good alternative to `np.fromstring(,
>>> dtype=...)`?  -- Marten
>>> >>>>
>>> >>>> On Thu, Oct 22, 2015 at 1:03 PM, Chris Barker <
>>> chris.bar...@noaa.gov> wrote:
>>> >>>>>
>>> >>>>> There was just a question about a bug/issue with scipy.fromstring
>>> (which is numpy.fromstring) when used to read integers from a text file.
>>> >>>>>
>>> >>>>>
>>> https://mail.scipy.org/pipermail/scipy-user/2015-October/036746.html
>>> >>>>>
>>> >>>>> fromstring() is bugging and inflexible for reading text files --
>>> and it is a very, very ugly mess of code. I dug into it a while back, and
>>> gave up -- just to much of a mess!
>>> >>>>>
>>> >>>>> So we really should completely re-implement it, or deprecate it. I
>>> doubt anyone is going to do a big refactor, so that means deprecating it.
>>> >>>>>
>>> >>>>> Also -- if we do want a fast read numbers from text files function
>>> (which would be nice, actually), it really should get a new name anyway.
>>> >>>>>
>>> >>>>> (and the hopefully coming new dtype system would make it easier to
>>> write cleanly)
>>> >>>>>
>>> >>>>> I'm not sure what deprecating something means, though -- have it
>>> raise a deprecation warning in the next version?
>>> >>>>>
>>> 

Re: [Numpy-discussion] deprecate fromstring() for text reading?

[Chris Barker - NOAA Federal]

Grabbing the pandas csv reader would be great, and I hope it happens sooner
than later, though alas, I haven't the spare cycles for it either.

In the meantime though, can we put a deprecation Warning in when using
fromstring() on text files? It's really pretty broken.

-Chris

On Oct 23, 2015, at 4:02 PM, Jeff Reback <jeffreb...@gmail.com> wrote:



On Oct 23, 2015, at 6:49 PM, Nathaniel Smith <n...@pobox.com> wrote:

On Oct 23, 2015 3:30 PM, "Jeff Reback" <jeffreb...@gmail.com> wrote:
>
> On Oct 23, 2015, at 6:13 PM, Charles R Harris <charlesr.har...@gmail.com>
wrote:
>
>>
>>
>> On Thu, Oct 22, 2015 at 5:47 PM, Chris Barker - NOAA Federal <
chris.bar...@noaa.gov> wrote:
>>>
>>>
>>>> I think it would be good to keep the usage to read binary data at
least.
>>>
>>>
>>> Agreed -- it's only the text file reading I'm proposing to deprecate.
It was kind of weird to cram it in there in the first place.
>>>
>>> Oh, fromfile() has the same issues.
>>>
>>> Chris
>>>
>>>
>>>> Or is there a good alternative to `np.fromstring(,
dtype=...)`?  -- Marten
>>>>
>>>> On Thu, Oct 22, 2015 at 1:03 PM, Chris Barker <chris.bar...@noaa.gov>
wrote:
>>>>>
>>>>> There was just a question about a bug/issue with scipy.fromstring
(which is numpy.fromstring) when used to read integers from a text file.
>>>>>
>>>>> https://mail.scipy.org/pipermail/scipy-user/2015-October/036746.html
>>>>>
>>>>> fromstring() is bugging and inflexible for reading text files -- and
it is a very, very ugly mess of code. I dug into it a while back, and gave
up -- just to much of a mess!
>>>>>
>>>>> So we really should completely re-implement it, or deprecate it. I
doubt anyone is going to do a big refactor, so that means deprecating it.
>>>>>
>>>>> Also -- if we do want a fast read numbers from text files function
(which would be nice, actually), it really should get a new name anyway.
>>>>>
>>>>> (and the hopefully coming new dtype system would make it easier to
write cleanly)
>>>>>
>>>>> I'm not sure what deprecating something means, though -- have it
raise a deprecation warning in the next version?
>>>>>
>>
>> There was discussion at SciPy 2015 of separating out the text reading
abilities of Pandas so that numpy could include it. We should contact Jeff
Rebeck and see about moving that forward.
>
>
> IIRC Thomas Caswell was interested in doing this :)

When he was in Berkeley a few weeks ago he assured me that every night
since SciPy he has dutifully been feeling guilty about not having done it
yet. I think this week his paltry excuse is that he's "on his honeymoon" or
something.

...which is to say that if someone has some spare cycles to take this over
then I think that might be a nice wedding present for him :-).

(The basic idea is to take the text reading backend behind pandas.read_csv
and extract it into a standalone package that pandas could depend on, and
that could also be used by other packages like numpy (among others -- I
thing dato's SFrame package has a fork of this code as well?))

-n

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion


I can certainly provide guidance on how/what to extract but don't have
spare cycles myself for this :(

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] deprecate fromstring() for text reading?

[Chris Barker - NOAA Federal]

I think it would be good to keep the usage to read binary data at least.


Agreed -- it's only the text file reading I'm proposing to deprecate. It
was kind of weird to cram it in there in the first place.

Oh, fromfile() has the same issues.

Chris


Or is there a good alternative to `np.fromstring(, dtype=...)`?  --
Marten

On Thu, Oct 22, 2015 at 1:03 PM, Chris Barker  wrote:

> There was just a question about a bug/issue with scipy.fromstring (which
> is numpy.fromstring) when used to read integers from a text file.
>
> https://mail.scipy.org/pipermail/scipy-user/2015-October/036746.html
>
> fromstring() is bugging and inflexible for reading text files -- and it is
> a very, very ugly mess of code. I dug into it a while back, and gave up --
> just to much of a mess!
>
> So we really should completely re-implement it, or deprecate it. I doubt
> anyone is going to do a big refactor, so that means deprecating it.
>
> Also -- if we do want a fast read numbers from text files function (which
> would be nice, actually), it really should get a new name anyway.
>
> (and the hopefully coming new dtype system would make it easier to write
> cleanly)
>
> I'm not sure what deprecating something means, though -- have it raise a
> deprecation warning in the next version?
>
> -CHB
>
>
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Making datetime64 timezone naive

[Chris Barker - NOAA Federal]

> This is fine.  Just be aware that *naive* datetimes will also have the PEP 
> 495 "fold" attribute in Python 3.6.  You are free to ignore it, but you will 
> loose the ability to round-trip between naive stdlib datetimes and 
> numpy.datetime64.

Sigh. I can see why it's there ( primarily to support now(), I
suppose). But a naive datetime doesn't have a timezone, so how could
you know what  time one actually corresponds to if fold is True? And
what could you do with it if you did know?

I've always figured that if you are using naive time for times in a
timezone that has DST, than you'd better know wether you were in DST
or not.

(Which fold tells you, I guess) but the fold isn't guaranteed to be an
hour is it? So without more info, what can you do? And if the fold bit
is False, then you still have no idea if you are in DST or not.

And then what if you attach a timezone to it? Then the fold bit could
be wrong...

I take it back, I can't see why the fold bit could be anything but
confusing for a naive datetime. :-)

Anyway, all I can see to do here is for the datetime64 docs to say
that fold is ignored if it's there.

But what should datetime64 do when provided with a datetime with a timezone?

- Raise an exception?
- ignore the timezone?
- Convert to UTC?

If the time zone is ignored, then you could get DST and non DST times
in the same array - that could be ugly.

Is there any way to query a timezone object to ask if it's a constant-offset?

And yes, I did mean "most". There is no way I'm ever going to
introduce a three letter "timezone" abbreviation in one of these
threads!

-CHB
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NumFOCUS fiscal sponsorship agreement

[Chris Barker - NOAA Federal]

Looks good to me.

This pretty exciting, actually :-)

-CHB

Sent from my iPhone

> On Oct 7, 2015, at 10:57 PM, Nathaniel Smith  wrote:
>
> Hi all,
>
> Now that the governance document is in place, we need to get our legal
> ducks in a row by signing a fiscal sponsorship agreement with
> NumFOCUS.
>
> The basic idea here is that there are times when you really need some
> kind of corporation to represent the project -- the legal system for
> better or worse does not understand "a bunch of folks on a mailing
> list" as a legal entity capable of accepting donations, or holding
> funds or other assets like domain names. The obvious solution is to
> incorporate a company to represent the project -- but incorporating a
> company involves lots of super-annoying paperwork. (Like, *super*
> annoying.) So a standard trick is that a single non-profit corporation
> acts as an umbrella organization providing these services to multiple
> projects at once, and this is called "fiscal sponsorship". You can
> read more about it here:
>https://en.wikipedia.org/wiki/Fiscal_sponsorship
>
> NumFOCUS's standard comprehensive FSA agreement can be seen here:
>
>
> https://docs.google.com/document/d/11YqMX9UrgfCSgiQEUzmOFyg6Ku-vED6gMxhO6J9lCgg/edit?usp=sharing
>
> and we have the option of negotiating changes if there's anything we don't 
> like.
>
> They also have a FAQ:
>
> https://docs.google.com/document/d/1zdXp07dLvkbqBrDsw96P6mkqxnWzKJuM-1f4408I6Qs/edit?usp=sharing
>
> I've read through the document and didn't see anything that bothered
> me, except that I'm not quite sure how to make the split between the
> steering council and numfocus subcommittee that we have in our
> governance model sync up with their language about the "leadership
> body", and in particular the language in section 10 about simple
> majority votes. So I've queried them about that already.
>
> In the mean time, I'd encourage anyone with an interest to look it
> over and speak up if you see anything that you think should be changed
> before we sign.
>
> Cheers,
> -n
>
> --
> Nathaniel J. Smith -- http://vorpus.org
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Sign of NaN

[Chris Barker - NOAA Federal]

One of the usecases that has sneaked in during the last few numpy versions
is that object arrays contain numerical arrays where the shapes don't add
up to a rectangular array.


I think that's the wrong way to dve that problem -- we really should have a
"proper" ragged array implementation. But is is the easiest way at this
point.

For this, and other use-cases, special casing Numpy arrays stored in object
arrays does make sense:

"If this is s a Numpy array, pass the operation through."

-CHB


In those cases being able to apply numerical operations might be useful.

But I'm +0 since I don't work with object arrays.

Josef



> -n
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver

[Chris Barker - NOAA Federal]

This sounds pretty cool -- and I've had a use case. So it would be
nice to get into Numpy.

But: I doubt we'd want OpenMP dependence in Numpy itself.

But maybe a pure Cython non-MP version?

Are we putting Cuthon in Numpy now? I lost track.

-CHB

Sent from my iPhone

> On Sep 29, 2015, at 7:35 AM, Juha Jeronen  wrote:
>
> Hi all,
>
> I recently developed a Cython-based, OpenMP-accelerated quartic (and cubic, 
> quadratic) polynomial solver to address a personal research need for quickly 
> solving a very large number of independent low-degree polynomial equations 
> for both real and complex coefficients.
>
> For example, on an Intel i7-4710MQ, running with 8 threads this code solves 
> approximately 1.2e7 quartic polynomial equations per second. (With 
> OMP_NUM_THREADS=4 the solver gives approximately 8.9e6 solutions per second, 
> so it seems HyperThreading in this case helps about 30%.)
>
> The algorithms for cubics and quadratics come from Numerical Recipes (3rd 
> ed.), and the quartic problem is internally reduced to a cubic and two 
> quadratics, using well-known standard tricks.
>
>
> Since to my understanding, for solving polynomial equations NumPy only 
> provides roots(), which works one problem at a time, I'm writing to inquire 
> whether there would be interest to include this functionality to NumPy, if I 
> contribute the code (and clean it up for integration)?
>
>
> I have some previous open source contribution experience. I have authored the 
> IVTC and Phosphor deinterlacers for VLC, and a modular postprocessing filter 
> framework for the Panda3D game engine (posted at the forums on panda3d.org, 
> projected for version 1.10).
>
> Currently the polynomial solver is available in a git repository hosted by 
> our university, the relevant files being:
>
> https://yousource.it.jyu.fi/jjrandom2/miniprojects/blobs/master/misc/tworods/polysolve2.pyx
> https://yousource.it.jyu.fi/jjrandom2/miniprojects/blobs/master/misc/tworods/polysolve.py
> https://yousource.it.jyu.fi/jjrandom2/miniprojects/blobs/master/misc/tworods/polysolve2example.py
> https://yousource.it.jyu.fi/jjrandom2/miniprojects/blobs/master/misc/tworods/setup.py
> https://yousource.it.jyu.fi/jjrandom2/miniprojects/blobs/master/misc/tworods/compile.sh
>
> I'm working on a research grant, which allows me to hold the copyright; 
> license is BSD.
>
> Polysolve2 is the fast Cython+OpenMP version, while polysolve is an earlier 
> (slower) experimental version using only NumPy array operations. The example 
> module contains a usage example, and setup (in the standard manner) instructs 
> distutils and Cython as to how to build polysolve2 (including setting the 
> relevant math flags and enabling OpenMP).
>
> (The setup script includes also some stuff specific to my solver for the 
> Ziegler problem; basically, the "tworods" module can be ignored. I apologize 
> for the inconvenience.)
>
>
> Thoughts?
>
>
> Best regards,
>
> -J
>
> -
> Juha Jeronen, Ph.D.
> juha.jero...@jyu.fi
> University of Jyväskylä
> Department of Mathematical Information Technology
> -
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Governance model request

[Chris Barker - NOAA Federal]

> But discussing who is great community leader, etc. is frankly not obvious to 
> me related to numpy governance.

Thank you Sebastian.

Could we please try to get back to the governance issues, without
naming names? There are some specific questions on the table that need
to get hashed out.


Numpy does not have a BDFL. BDFLs are not selected, they are naturally
produced, and there is no one that fits that bill now. We *could*
decide to have a single individual executive leader, selected by some
sort of democratic process. But that is not the same as a BDFL. And I
think the almost-consensus  now is to not have that.

So here is what I think is on the table:

We have the steering council. Which leaves two questions:

-How big should it be?
-Who will be on the original, "seed" council?

Note that as I understand the current draft of the governance doc,
once established, the council itself decides who is on the council. So
at this point we really are ONLY deciding how it's going to start. It
has to be bootstrapped somehow.

However, that had been contentious enough that it would probably be a
good idea to hash out some guidelines about the council membership.

Personally, I have no idea how big the council should be. Too big, and
there is no point, consensus is harder to reach the larger the group,
and the main (only?) role of the council is to resolve issues where
consensus has not been reached in the larger community. But what is
too big?

As for make-up of the council, I think we need to expand beyond people
who have recently contributed core code.

Yes, the council does need to have expertise to make technical
decisions, but if you think about the likely contentious issues like
ABI breakage, a core-code focused view is incomplete. So there should
be representation by:

Someone(s) with a long history of working with the code -- that
institutional memory of why decisions were made the way they were
could be key.

Someone(s) that may not have worked on the core code, but is a major
player in the community, perhaps as the leader of a Numpy-dependent
project. This would provide representation for the broad community.

I do want to note that the governance document as I understand it is
consistent with these suggestions.

As for conflict of interest issues, etc:

Chill out people!

Numpy is an open source project, if it gets hijacked, it can be forked.

And the council is also democratic -- no one person can drive the
project. If a council member is not acting in the interests of the
community, s/he can be removed.

NOTE: while x.org, and egcs, Xemacs, and ... may be examples of
failures of governance, they are also examples of successes of open
source.

-Chris
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: Deprecate np.int, np.float, etc.?

[Chris Barker - NOAA Federal]

 Turns out I was passing in numpy arrays that I had typed as np.int.
 It worked OK two years ago when I was testing only on 32 bit pythons,
 but today I got a bunch of failed tests on 64 bit OS-X -- a np.int is
 now a C long!

 It has always been C long. It is the C long that varies between platforms.

Of course, it's that a c long was a c int on the platform I wrote the
code on the first time.

Which is part of the problem with C -- if two types happen to be the
same, the compiler is perfectly happy. But that was an error in the
first place, it never should have passed.

But that's just me.

;-)

Anyway, as far as concrete proposals go. I say we deprecate the Python
types in the numpy namespace (i.e int and float)

Other than that, I'm not sure there's any problem.

-Chris


 Sturla

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: Deprecate np.int, np.float, etc.?

[Chris Barker - NOAA Federal]

So one more bit of anecdotal evidence:

I just today revived some Cython code I wrote a couple years ago and
haven't tested since.

It wraps a C library that uses a lot of int typed values.

Turns out I was passing in numpy arrays that I had typed as np.int.
It worked OK two years ago when I was testing only on 32 bit pythons,
but today I got a bunch of failed tests on 64 bit OS-X -- a np.int is
now a C long!

I really thought I knew better, even a couple years ago, but I guess
it's just too easy to slip up there.

Yeah to Cython for keeping types straight (I got a run-time error).
And Yeah to me for having at least some basic tests.

But Boo to numpy for a very easy to confuse type API.

-Chris


Sent from my iPhone

 On Jul 31, 2015, at 10:45 AM, Sturla Molden sturla.mol...@gmail.com wrote:

 Chris Barker chris.bar...@noaa.gov wrote:

 What about Fortan -- I've been out of that loop for ages -- does
 semi-modern Fortran use well defined integer types?

 Modern Fortran is completely sane.

 INTEGER without kind number (Fortran 77) is the fastest integer on the CPU.
 On AMD64 that is 32 bit, because it is designed to use a 64 bit pointer
 with a 32 bit offset. (That is also why Microsoft decided to use a 32 bit
 long, because it by definition is the fastest integer of at least 32 bits.
 One can actually claim that the C standard is violated with a 64 bit long
 on AMD64.) Because of this we use a 32 bit interger in BLAS and LAPACK
 linked to NumPy and SciPy.

 The function KIND (Fortran 90) allows us to query the kind number of a
 given variable, e.g. to find out the size of INTEGER and REAL.

 The function SELECTED_INT_KIND (Fortran 90) returns the kind number of
 smallest integer with a specified range.

 The function SELECTED_REAL_KIND (Fortran 90) returns the kind number of
 smallest float with a given range and precision. THe returned kind number
 can be used for REAL and COMPLEX.

 KIND, SELECTED_INT_KIND and SELECTED_REAL_KIND will all return compile-time
 constants, and can be used to declare other variables if the return value
 is stored in a variable with the attribute PARAMETER. This allows te
 programmer to get the REAL, COMPLEX or INTEGER the algorithm needs
 numerically, without thinking about how big they need to be in bits.

 ISO_C_BINDING is a Fortran 2003 module which contains kind numbers
 corresponding to all C types, including size_t and void*, C structs, an
 attribute for using pass-by-value semantics, controlling the C name to
 avoid name mangling, as well as functions for converting between C and
 Fortran pointers. It allows portable interop between C and Fortran (either
 calling C from Fortran or calling Fortran from C).

 ISO_FORTRAN_ENV is a Fortran 2003 and 2008 module. In F2003 it contain kind
 numbers for integers with specified size: INT8, INT16, INT32, and INT64. In
 F2008 it also contains kind numbers for IEEE floating point types: REAL32,
 REAL64, and REAL128. The kind numbers for floating point types can also be
 used to declare complex numbers.

 So with modern Fortran we have a completely portable and unambiguous type
 system.

 C11/C++11 is sane as well, but not quite as sane as that of modern Fortran.


 Sturla

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] floats for indexing, reshape - too strict ?

[Chris Barker - NOAA Federal]

Sent from my iPhone


 The disadvantage I see is, that some weirder calculations would possible
 work most of the times, but not always,


  not sure if you can define a tolerance
 reasonable here unless it is exact.

You could use a relative tolerance, but you'd still have to set that.
Better to put that decision squarely in the user's hands.

 Though I guess you are right that
 `//` will also just round silently already.

Yes, but if it's in the user's code, it should be obvious -- and then
the user can choose to round, or floor, or ceiling

-CHB


 - Sebastian


 for example


 5.0 == 5
 True


 np.ones(10 / 2)
 array([ 1.,  1.,  1.,  1.,  1.])
 10 / 2 == 5
 True


 or the python 2 version


 np.ones(10. / 2)
 array([ 1.,  1.,  1.,  1.,  1.])
 10. / 2 == 5
 True


 I'm using now 10 // 2, or int(10./2 + 1)   but this is unconditional
 and doesn't raise if the numbers are not close or equal to an integer
 (which would be a bug)




 Josef




 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Video meeting this week

[Chris Barker - NOAA Federal]

I _may_ be able to join -- but don't go setting up an alternative
conferencing system just for me.

But I'm planning on ring in Austin Tues in any case.

-Chris


Sent from my iPhone

 On Jun 29, 2015, at 9:59 PM, Nathaniel Smith n...@pobox.com wrote:

 On Fri, Jun 26, 2015 at 2:32 AM, Nathaniel Smith n...@pobox.com wrote:
 Hi all,

 In a week and a half, this is happening:

https://github.com/numpy/numpy/wiki/SciPy-2015-developer-meeting

 It's somewhat short notice (my bad :-/), but I think it would be good
 to have a short video meeting sometime this week as a kind of
 pre-meeting -- to at least briefly go over the main issues we see
 facing the project to prime the pump, get a better idea about what we
 want to accomplish at the meeting itself, and gather some early
 feedback from anyone who won't be able to make it to SciPy (we'll miss
 you).

 The obligatory doodle:
http://doodle.com/6b4s6thqt9xt4vnh

 Okay, let's aim for:

   Thursday July 2 at 20:00 UTC.

 I believe that's 1pm California / 4 pm New York / 9pm London / 10pm
 western Europe

 And so far it looks like we'll be under the 10 person Google Hangouts
 limit, which I'm assuming is simpler for everybody, so let's assume
 we're doing that unless otherwise specified. (This does mean that I'd
 appreciate a quick email if you're planning on dialling in but haven't
 otherwise responded to the poll, though!)

 -n

 --
 Nathaniel J. Smith -- http://vorpus.org
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] binary wheels for numpy?

[Chris Barker - NOAA Federal]

Thanks for the update Matthew, it's great to see so much activity on this issue.

Looks like we are headed in the right direction --and getting close.

Thanks to all that are putting time into this.

-Chris

 On May 15, 2015, at 1:37 PM, Matthew Brett matthew.br...@gmail.com wrote:

 Hi,

 On Fri, May 15, 2015 at 1:07 PM, Chris Barker chris.bar...@noaa.gov wrote:
 Hi folks.,

 I did a little intro to scipy session as part of a larger Python class the
 other day, and was dismayed to find that pip install numpy still dosn't
 work on Windows.

 Thanks mostly to Matthew Brett's work, the whole scipy stack is
 pip-installable on OS-X, it would be really nice if we had that for Windows.

 And no, saying you should go get Python(x,y) or Anaconda, or Canopy, or...)
 is really not a good solution. That is indeed the way to go if someone is
 primarily focusing on computational programming, but if you have a web
 developer, or someone new to Python for general use, they really should be
 able to just grab numpy and play around with it a bit without having to
 start all over again.


 My solution was to point folks to Chris Gohlke's site -- which is a Fabulous
 resource --

 THANK YOU CHRISTOPH!

 But I still think that we should have the basic scipy stack on PyPi as
 Windows Wheels...

 IIRC, the last run through on this discussion got stuck on the what
 hardware should it support -- wheels do not allow a selection at installc
 time, so we'd have to decide what instruction set to support, and just stick
 with that. Which would mean that:

 some folks would get a numpy/scipy that would run a bit slower than it might
 and
 some folks would get one that wouldn't run at all on their machine.

 But I don't see any reason that we can't find a compromise here -- do a
 build that supports most machines, and be done with it. Even now, people
 have to go get (one way or another) a MKL-based build to get optimum
 performance anyway -- so if we pick an instruction set support by, say (an
 arbitrary, and impossible to determine) 95% of machines out there -- we're
 good to go.

 I take it there are licensing issues that prevent us from putting Chris'
 Binaries up on PyPi?

 Yes, unfortunately we can't put MKL binaries on pypi because of the
 MKL license - see
 https://github.com/numpy/numpy/wiki/Numerical-software-on-Windows#blas--lapack-libraries.
 Also see discussion in the containing thread of
 http://mail.scipy.org/pipermail/numpy-discussion/2014-March/069701.html
 .

 But are there technical issues I'm forgetting here, or do we just need to
 come to a consensus as to hardware version to support and do it?

 There has been some progress on this - see

 https://github.com/scipy/scipy/issues/4829

 I think there's a move afoot to have a Google hangout or similar on
 this exact topic :
 https://github.com/scipy/scipy/issues/2829#issuecomment-101303078 -
 maybe we could hammer out a policy there?  Once we have got numpy and
 scipy built in a reasonable way, I think we will be most of the way
 there...

 Cheers,

 Matthew
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] [SciPy-User] Congratulations to Chris Barker...

[Chris Barker - NOAA Federal]

 Are there plans to write a vectorized version for NumPy? :)

 np.isclose isn't identical, but IIRC the only difference is the defaults.

There are subtle differences in the algorithm as well. But not enough
that it makes sense to change the numpy one.

The results will be similar in most cases, and identical fir a
relative tolerance less than 1e-8 (for float64).

-Chris


 -n

 --
 Nathaniel J. Smith -- http://vorpus.org
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] String type again.

[Chris Barker - NOAA Federal]

 But HDF5
 additionally has a fixed-storage-width UTF8 type, so we could map to a
 NumPy fixed-storage-width type trivially.

Sure -- this is why *nix uses utf-8 for filenames -- it can just be a
char*. But that just punts the problem to client code.

I think a UTF-8 string type does not match the numpy model well, and I
don't think we should support it just because it would be easier for
the HDF 5 wrappers.

( to be fair, there are probably other similar systems numpy wants to
interface with that cod use this...)

It seems if you want a 1:1 binary mapping between HDF and numpy for
utf strings, then a bytes type in numpy makes more sense. Numpy
could/should have encode and decode methods for converting byte arrays
to/from Unicode arrays (does it already? ).

 Custom in this context means a user-created HDF5 data-conversion
 filter, which is necessary since all data conversion is handled inside
 the HDF5 library.

 As far as generic Unicode goes, we currently don't support the NumPy
 U dtype in h5py for similar reasons; there's no destination type in
 HDF5 which (1) would preserve the dtype for round-trip write/read
 operations and (2) doesn't risk truncation.

It sounds to like HDF5 simply doesn't support Unicode. Calling an
array of bytes utf-8 simple pushes the problem on to client libs. As
that's where the problem lies, then the PyHDF may be the place to
address it.

If we put utf-8 in numpy, we have the truncation problem there instead
-- which is exactly what I think we should avoid.

 A Latin-1 based 'a' type
 would have similar problems.

Maybe not -- latin1 is fixed width.

 Does HDF enforce ascii-only? what does it do with the  127 values?

 Unfortunately/fortunately the charset is not enforced for either ASCII

So you can dump Latin-1 into and out of the HDF 'ASCII' type -- it's
essentially the old char* / py2 string. An ugly situation, but why not
use it?

 or UTF-8,

So ASCII and utf-8 are really the same thing, with different meta-data...

 although the HDF Group has been thinking about it.

I wonder if they would consider going Latin-1 instead of ASCII --
similarly to utf-8 it's backward compatible with ASCII, but gives you
a little more.

I don't know that there is another 1byte encoding worth using -- it
maybe be my English bias, but it seems Latin-1 gives us ASCII+some
extra stuff handy for science ( I use the degree symbol a lot, for
instance) with nothing lost.

 Ideally, NumPy would support variable-length
 strings, in which case all these headaches would go away.

Would they? That would push the problem back to PyHDF -- which I'm
arguing is where it belongs, but I didn't think you were ;-)

 But I
 imagine that's also somewhat complicated. :)

That's a whole other kettle of fish, yes.


-Chris
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Short-hand array creation in `numpy.mat` style

[Chris Barker - NOAA Federal]

If you are going to introduce this functionality, please don't call it
np.arr.


I agree, but..,

I would suggest calling it something like np.array_simple or
np.array_from_string, but the best choice IMO, would be
np.ndarray.from_string (a static constructor method).


Except the entire point of his is that it's easy to type...

-1 on the whole idea -- this isn't Matlab, I'd saving a little typing worth
it?

CHB


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Short-hand array creation in `numpy.mat` style

[Chris Barker - NOAA Federal]

On Jul 7, 2014, at 7:28 AM, Sebastian Berg sebast...@sipsolutions.net wrote:
 not sure that many use np.r_ or np.c_

I actually really like those ;-)

-Chris
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Teaching Scipy BoF at SciPy

[Chris Barker - NOAA Federal]

On Jul 4, 2014, at 7:02 AM, Phil Elson pelson@gmail.com wrote:

Nice idea. Just a repository of courses would be a great first step.


Yup -- or really even a curated page of links and refrrences.

Maybe we can get first draft of such a thing put together during the BoF.
Feel free to add this idea to the Wiki :-)

I hope you can come to the BoF, I know there are a number of others at the
same time.

-CHB


For example, I know Jake Vanderplas's course at
https://github.com/jakevdp/2013_fall_ASTR599 is useful, and I have a few
introduction (3hr) courses at https://github.com/SciTools/courses.


On 3 July 2014 16:59, Chris Barker chris.bar...@noaa.gov wrote:

 HI  Folks,

 I will be hosting a Teaching the SciPy Stack BoF at SciPy this year:

 https://conference.scipy.org/scipy2014/schedule/presentation/1762/

 (Actually, I proposed it for the conference, but would be more than happy
 to have other folks join me in facilitating, hosting, etc.)

 I've put up a Wiki page to collect ideas for topics. Please take a look
 and add your $0.02:

 https://github.com/numpy/numpy/wiki/TeachingSciPy-BoF-at-Scipy-2014

 See you there,

   -Chris

 --

 Christopher Barker, Ph.D.
 Oceanographer

 Emergency Response Division
 NOAA/NOS/ORR(206) 526-6959   voice
 7600 Sand Point Way NE   (206) 526-6329   fax
 Seattle, WA  98115   (206) 526-6317   main reception

 chris.bar...@noaa.gov

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] SciPy 2014 BoF NumPy Participation

[Chris Barker - NOAA Federal]

On Jun 27, 2014, at 8:44 PM, Charles R Harris charlesr.har...@gmail.com wrote:

 Hi Kyle,

 On Tue, Jun 17, 2014 at 2:40 I'd like to propose adopting something like the 
 Blaze standard for datetime,

+1 for some focused discussion of datetime. This has been lingering
far too long.

-Chris
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

[Chris Barker - NOAA Federal]

On Apr 23, 2014, at 8:23 PM, Sankarshan Mudkavi smudk...@uwaterloo.ca
wrote:

I've been quite busy for the past few weeks but I should be much freer
after next week and can pick up on this (fixing the code and actually
implement things).


wonderful! Thanks.

Chris

Cheers,
Sankarshan

On Apr 23, 2014, at 5:58 PM, Chris Barker chris.bar...@noaa.gov wrote:

On Wed, Mar 19, 2014 at 7:07 PM, Sankarshan Mudkavi
smudk...@uwaterloo.cawrote:


 I've written a rather rudimentary NEP, (lacking in technical details which
 I will hopefully add after some further discussion and receiving
 clarification/help on this thread).

 Please let me know how to proceed and what you think should be added to
 the current proposal (attached to this mail).

 Here is a rendered version of the same:

 https://github.com/Sankarshan-Mudkavi/numpy/blob/Enhance-datetime64/doc/neps/datetime-improvement-proposal.rst


I've done a bit of copy-editing, and added some more from this discussion.
See the pull request on gitHub.

There are a fair number of rough edges, but I think we have a consensus
among the small group of folks that participated in this discussion anyway,
so now all we need is someone to actually fix the code.

If someone steps up, then we should also go in and add a bunch of unit
tests, as discussed in this thread.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


-- 
Sankarshan Mudkavi
Undergraduate in Physics, University of Waterloo
www.smudkavi.com






___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: NumPy 1.8.1 release

[Chris Barker - NOAA Federal]

On Apr 1, 2014, at 4:36 PM, Nathaniel Smith n...@pobox.com wrote:

 We could just ship all numpy's extension modules in the same directory
 if we wanted. It would be pretty easy to stick some code at the top of
 numpy/__init__.py to load them from numpy/all_dlls/ and then slot them
 into the appropriate places in the package namespace.

I like this, and it could play well with wheels and virtualenv.

(which reminds me -- looking at what Anaconda does may be instructive)

 Of course scipy and numpy will still both have to ship BLAS etc., and
 so I guess it will get loaded at least twice in *any* binary install

Since scipy has a strict dependency on numpy, and the same people are
maintaining the builds, couldn't scipy use the libs provided by numpy?
( maybe adding more)

-Chris
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] python array

[Chris Barker - NOAA Federal]

On Mar 13, 2014, at 9:39 AM, Nicolas Rougier nicolas.roug...@inria.fr wrote:


 Seems to be related to the masked values:

Good hint -- a masked array keeps the junk values in the main array.

What abs are you using -- it may not be mask-aware. ( you want a
numpy abs anyway)

Also -- I'm not sure I know what happens with Boolean operators on
masked arrays when you use them to index. I'd investigate that.
(sorry, not at a machine I can play with now)

Chris


 print r2010[:3,:3]
 [[-- -- --]
 [-- -- --]
 [-- -- --]]

 print abs(r2010)[:3,:3]
 [[-- -- --]
 [-- -- --]
 [-- -- --]]


 print r2010[ r2010[:3,:3] 0 ]
 [-- -- -- -- -- -- -- -- --]

 print r2010[ abs(r2010)[:3,:3]  0]
 []

 Nicolas



 On 13 Mar 2014, at 16:52, Sudheer Joseph sudheer.jos...@yahoo.com wrote:

 Dear experts,
 I am encountering a strange behaviour of python data 
 array as below. I have been trying to use the data from a netcdf 
 file(attached herewith) to do certain calculation using below code. If I 
 take absolute value of the same array and look for values .5  I get a 
 different value than the original array. But the fact is that this 
 particular case do not have any negative values in the array( but there are 
 other files where it can have negative values so the condition is put). I do 
 not see any reason for getting different numbers for values .5 in case of 
 bt and expected it to be same as that of r2010. If any one has a guess on 
 what is behind this behaviour please help.


 In [14]: from netCDF4 import Dataset as nc

 In [15]: nf=nc('r2010.nc')
 In [16]: r2010=nf.variables['R2010'][:]
 In [17]: bt=abs(r2010)
 In [18]: bt[bt=.5].shape
 Out[18]: (2872,)
 In [19]: r2010[r2010.5].shape
 Out[19]: (36738,)


 bt.min()
 Out[20]: 0.0027588337040836768

 In [21]: bt.max()
 Out[21]: 3.5078965479057089
 In [22]: r2010.max()
 Out[22]: 3.5078965479057089
 In [23]: r2010.min()
 Out[23]: 0.0027588337040836768



 ***
 Sudheer Joseph
 Indian National Centre for Ocean Information Services
 Ministry of Earth Sciences, Govt. of India
 POST BOX NO: 21, IDA Jeedeemetla P.O.
 Via Pragathi Nagar,Kukatpally, Hyderabad; Pin:5000 55
 Tel:+91-40-23886047(O),Fax:+91-40-23895011(O),
 Tel:+91-40-23044600(R),Tel:+91-40-9440832534(Mobile)
 E-mail:sjo.in...@gmail.com;sudheer.jos...@yahoo.com
 Web- http://oppamthadathil.tripod.com
 ***r2010.nc___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ndarray is not a sequence

[Chris Barker - NOAA Federal]

On Feb 28, 2014, at 1:04 AM, Sebastian Berg sebast...@sipsolutions.net wrote:


 because the sequence check like that seems standard in python 3.

Whatever happened to duck typing?

Sigh.

-Chris
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Catching out-of-memory error before it happens

[Chris Barker - NOAA Federal]

c = a + b: 3N
c = a + 2*b: 4N

Does python garbage collect mid-expression? I.e. :

C = (a + 2*b) + b

4 or 5 N?

Also note that when memory gets tight, fragmentation can be a problem. I.e.
if two size-n arrays where just freed, you still may not be able to
allocate a size-2n array. This seems to be worse on windows, not sure why.

a += b: 2N
np.add(a, b, out=a): 2N
b *= 2; a += b: 2N

Note that simply loading a and b requires 2N memory, so the latter code
samples are near-optimal.

And will run quite a bit faster for large arrays--pushing that memory
around takes time.

-Chris
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] using loadtxt to load a text file in to a numpy array

[Chris Barker - NOAA Federal]

On Jan 22, 2014, at 1:13 PM, Oscar Benjamin oscar.j.benja...@gmail.com wrote:


 It's not safe to stop removing the null bytes. This is how numpy determines
 the length of the strings in a dtype='S' array. The strings are not
 fixed-width but rather have a maximum width.

Exactly--but folks have told us on this list that they want (and are)
using the 'S' style for arbitrary bytes, NOT for text. In which case
you wouldn't want to remove null bytes. This is more evidence that 'S'
was designed to handle c-style one-byte-per-char strings, and NOT
arbitrary bytes, and thus not to map directly to the py2 string type
(you can store null bytes in a py2 string

Which brings me back to my original proposal: properly map the 'S'
type to the py3 data model, and maybe add some kind of fixed width
bytes style of there is a use case for that. I still have no idea what
the use case might be.

 If the trailing nulls are not removed then you would get:

 a[0]
 b'a\x00\x00\x00\x00\x00\x00\x00\x00\x00'
 len(a[0])
 9

 And I'm sure that someone would get upset about that.

Only if they are using it for text-which you should not do with py3.

 Having the null bytes removed and a str (on Py2) object returned is precisely
 the use case that distinguishes it from np.uint8.

But that was because it was designed to be used with text . And if you
want text, then you should use py3 strings, not bytes. And if you
really want bytes, then you wouldn't want null bytes removed.

 The other differences are the
 removal of arithmetic operations.

And 'S' is treated as an atomic element, I'm not sure how you can do
that cleanly with uint8.

 Some more oddities:

 a[0] = 1
 a
 array([b'1', b'string', b'of', b'different', b'length', b'words'],
  dtype='|S9')
 a[0] = None
 a
 array([b'None', b'string', b'of', b'different', b'length', b'words'],
  dtype='|S9')

More evidence that this is a text type.

-Chris
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] A one-byte string dtype?

[Chris Barker - NOAA Federal]

On Jan 21, 2014, at 4:58 PM, David Goldsmith d.l.goldsm...@gmail.com wrote:


 OK, well that's definitely beyond my level of expertise.

Well, it's in github--now's as good a time as any to learn github
collaboration...

-Fork the numpy source.

-Create a new file in:
numpy/doc/neps

Point folks to it here so they can comment, etc.

At some point, issue a pull request, and it can get merged into the
main source for final polishing...

-Chris







 DG
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] [Pythonmac-SIG] Building extension modules with Xcode 5

[Chris Barker - NOAA Federal]

Oops!

Wrong list--darn auto complete!

Sorry about that,

Chris

On Oct 25, 2013, at 5:08 PM, Chris Barker chris.bar...@noaa.gov wrote:

Ned,

I think this fell off the list (I really don't like when reply is not set
to the list...)

On Fri, Oct 25, 2013 at 4:33 PM, Ned Deily n...@acm.org wrote:


 On Oct 25, 2013, at 15:17 , Chris Barker chris.bar...@noaa.gov wrote:

  On Thu, Oct 24, 2013 at 7:29 PM, Ned Deily n...@acm.org wrote:
  I'm really confused here.   Using the current 2.7.5 64-bit installer
 (from
  May) on 10.8.5 with Xcode 5.0.1 works just fine for me.
 
  I'm not on 10.8 or XCode5, but does it make a difference whether you
 installed XCode 5 after Xcode 4? i.e. someone may have gcc4.2 around form
 XCode 4, but not if they installed XCode 5 on a raw machine.

 Not that I know of.  gcc-4.2 is a red herring as far as I can tell.
  Distutils has known how to deal with a missing compiler for a couple of
 releases now.


Good to know -- I know I really need to move forward -- I wonder if there
is an XCode5 for 10.6? pProbably not -- but for some screwy reason Apple
does not provide a free XCode4 for 10.6, either (though you can hack the
installer for 10.7 ...


 However, I did just realize that I had forgotten about a problem when
 using the current 32-bit-only installers with Xcode 5.  I've just posted
 about that; it's described now in http://bugs.python.org/issue19400

 Good to fix that, but does anyone need it anymore?

Though I use it when I need to force 32 bit, even on a 64 bit system (I'm
working with some not-64-bit safe C++ code...)

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] official binaries on web page.

[Chris Barker - NOAA Federal]

 Ralf Gommers ralf.gomm...@gmail.com wrote:
 but the layout of that page is on
 purpose. scipy.org is split into two parts: (a) a SciPy Stack part, and
 (b)
 a numpy  scipy library part. You're looking at the stack part, and the
 preferred method to install that stack is a Python distribution.

OK, I'm not sure that's a great idea, but if we take that as a given:

That page could use some clarification about what the heck the stack
is, and what its relationship to the scipy and numpy packages is.

And I still think it wouldn't hurt to more obviously point people to
how to get either numpy or scipy themselves.

So maybe my section about the official binaries, but lower on the
page. I don't like custom as a title, as that makes it sound like
advanced numpy-fu, not where a newbie looking for just numpy is going
to look.

But it sounds like the real problem is with the surrounding
pages--that's the page you find when you try to figure out how to get
numpy--if that page is about the stack, it should not be linked to
directly from the numpy.org page without explanation.

We do have a branding problem: scipy is a package, a stack and a
ecosystem/community. It should be clear which one is being referred to
when.

-Chris
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Funny business with 'is' operator?

[Chris Barker - NOAA Federal]

On Fri, Sep 6, 2013 at 9:19 AM, James Bergstra bergs...@iro.umontreal.cawrote:

 def test_is(self):
 a = np.empty(1)
 b = np.empty(1)
 if a.data is not b.data:
 assert id(a.data) != id(b.data) # -- fail


I'm not familiar with the internals, but:

In [27]: a = np.empty(1)

In [28]: a.data
Out[28]: read-write buffer for 0x122df30, size 8, offset 0 at 0x3501860

so .data is a buffer object, wrapped around a particular pointer.

In [29]: id(a.data)
Out[29]: 55581376

In [30]: id(a.data)
Out[30]: 55581728

but it looks like the buffer object itself is created on the fly -- so you
are getting a different pyton object, even though it wraps the same data
pointer. So you need to compare the value of the pointer compared, not the
identity of the buffer object.

Not sure how to get that value though, other that parsing __repr__

In [44]: a.data.__repr__().split()[-1][:-1]
Out[44]: '0x3512100'

ugly hack -- there must be a better way!

But:

In [38]: a = np.zeros((100,))

In [39]: b = a[50:]

In [40]: a.data
Out[40]: read-write buffer for 0x1443990, size 800, offset 0 at 0x3501da0

In [41]: b.data
Out[41]: read-write buffer for 0x1617130, size 400, offset 0 at 0x3501c20

a and b share memory, but don't have the same data pointer as b's is offset.

HTH,
   -Chris









 I'm trying to write an alternate may_share_memory function.

 Thanks,
  - James


 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] fast_any_all , a trivial but fast/useful helper function for numpy

[Chris Barker - NOAA Federal]

This is good stuff, but I can't help thinking that if I needed to do an
any/all test on a number of arrays with common and/or combos --
I'd probably write a Cython function to do it.

It could be a bit tricky to make it really general, but not bad for a
couple specific dtypes / use cases.

-just a thought...

Also -- how does this work with numexpr? It would be nice if it could
handle these kinds of cases.

-Chris





On Thu, Sep 5, 2013 at 1:54 AM, Graeme B. Bell g...@skogoglandskap.nowrote:



 Hi Julian,

 Thanks for the post.  It's great to hear that the main numpy function is
 improving in 1.8, though I think there is still plenty of value here for
 performance junkies :-)

 I don't have 1.8beta installed (and I can't conveniently install it on my
 machines just now). If you have time, and have the beta installed, could
 you try this and mail me the output from the benchmark?  I'm curious to
 know.

 # git clone https://github.com/gbb/numpy-fast-any-all.git
 # cd numpy-fast-any-all
 # python test-fast-any-all.py

 Graeme


 On Sep 4, 2013, at 7:38 PM, Julian Taylor jtaylor.deb...@googlemail.com
 wrote:

 
  The result is 14 to 17x faster than np.any() for this use case.*
 
  any/all and boolean operations have been significantly speed up by
  vectorization in numpy 1.8 [0].
  They are now around 10 times faster than before, especially if the
  boolean array fits into one of the cpu caching layers.

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: Numpy 1.8.0 beta 1 release

[Chris Barker - NOAA Federal]

FWIW,

You all may know this already, but a long is 64 bit on most 64 bit
platforms, but 32 bit on Windows.

Can we start using stdint.h and int32_t and friends?

-CHB

On Sep 3, 2013, at 5:18 PM, Charles R Harris charlesr.har...@gmail.com
wrote:




On Tue, Sep 3, 2013 at 6:09 PM, Christoph Gohlke cgoh...@uci.edu wrote:

 On 9/3/2013 4:45 PM, Charles R Harris wrote:
 
 
 
  On Tue, Sep 3, 2013 at 5:40 PM, Christoph Gohlke cgoh...@uci.edu
  mailto:cgoh...@uci.edu wrote:
 
  On 9/3/2013 2:51 PM, Charles R Harris wrote:
  
  
  
   On Tue, Sep 3, 2013 at 3:23 PM, Christoph Gohlke 
  cgoh...@uci.edumailto:
 cgoh...@uci.edu
   mailto:cgoh...@uci.edu mailto:cgoh...@uci.edu wrote:
  
   On 9/1/2013 9:54 AM, Charles R Harris wrote:
  
   Hi all,
  
   I'm happy to announce the first beta release of Numpy
 1.8.0.
   Please try
   this beta and report any issues on the numpy-dev mailing
 list.
  
   Source tarballs and release notes can be found at
  https://sourceforge.net/__projects/numpy/files/NumPy/1.__8.0b1/
   
 https://sourceforge.net/projects/numpy/files/NumPy/1.8.0b1/.
   The Windows
   and OS X installers will follow when the infrastructure
 issues
   are dealt
   with.
  
   Chuck
  
  
   Hello,
  
   I tried numpy-1.8.0.dev-86a6e6c with msvc9 and MKL 11.1 on
   win-amd64-py2.7. It builds OK but there are 23 test errors and
 6
   failures (attached).
  
   Some 3rd party packages (e.g. scipy, numexpr, pytables,
 bottleneck,
   pandas and matplotlib) that were built against numpy-MKL 1.7
 fail
   tests when used with numpy-MKL 1.8. Other packages test OK
 (e.g.
   skimage, sklearn, statsmodels, mahotas, pygame). See
   
 http://www.lfd.uci.edu/~__gohlke/pythonlibs/tests/__20130902-win-amd64-py2.7-__numpy-1.8.0.dev-86a6e6c/
  
 http://www.lfd.uci.edu/%7E__gohlke/pythonlibs/tests/__20130902-win-amd64-py2.7-__numpy-1.8.0.dev-86a6e6c/
 
   
 http://www.lfd.uci.edu/%7Egohlke/pythonlibs/tests/20130902-win-amd64-py2.7-numpy-1.8.0.dev-86a6e6c/
 
   compared to
   
 http://www.lfd.uci.edu/~__gohlke/pythonlibs/tests/__20130902-win-amd64-py2.7/
  
 http://www.lfd.uci.edu/%7E__gohlke/pythonlibs/tests/__20130902-win-amd64-py2.7/
 
   
 http://www.lfd.uci.edu/%7Egohlke/pythonlibs/tests/20130902-win-amd64-py2.7/
 .
  
  
   I have not looked in more detail or at other Python versions
 yet.
  
  
   Thanks Christoph,
  
   Looks like some work to do. I wonder what is different between
 windows
   and linux here?
  
   Chuck
  
 
  Looks like the fundamental PyArray_PyIntAsIntp function is broken on
 64
  bit Windows. 64 bit PyLong values are intermediately stored in a 32
 bit
  C long variable. But maybe I am missing something...
  
 https://github.com/numpy/numpy/blob/maintenance/1.8.x/numpy/core/src/multiarray/conversion_utils.c#L729
 
  
 https://github.com/numpy/numpy/blob/maintenance/1.8.x/numpy/core/src/multiarray/conversion_utils.c#L767
 
 
  My, that does look suspicious. That function is new in 1.8 I believe.
  Looks like it needs fixing whatever else it fixes.
 
  Chuck
 

 In fact, using a npy_longlong instead of npy_long fixes all numpy test
 errors and failures. But it probably foils the recent optimizations.


Great! I think the function is not used for numeric things so I'm not sure
what optimizations could be affected. I'll put up a PR and backport it.

Chuck

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Stick intersection path algorithm

[Chris Barker - NOAA Federal]

On Sun, Sep 1, 2013 at 3:55 PM, Josè Luis Mietta 
joseluismie...@yahoo.com.ar wrote:

 Given two arbitrary sticks, i need a simple and effective algorithm that
 determinate if that two sticks are conected by a 'intersected-sticks' path.


do you mean a test to see if two line segments intersect?

This looks reasonable:

http://wiki.processing.org/w/Line-Line_intersection

It probably makes sense to translate to Cython (or use the C and call with
cython).

Ive also got similar code in a little package of mine:

https://github.com/ChrisBarker-NOAA/py_geometry

Already Cython, and includes code to check a whole bunch at once, stored in
numpy arrays:

https://github.com/ChrisBarker-NOAA/py_geometry/blob/master/py_geometry/line_crossings.pyx

I hope it's useful to you.

-Chris


-Chris








-Chris







 Any idea for that?

 Thanks a lot!


 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] What is a numpy.long type?

[Chris Barker - NOAA Federal]

On Aug 22, 2013, at 11:57 PM, David Cournapeau courn...@gmail.com wrote:

npy_long is indeed just an alias to C long,


Which means it's likely broken on 32 bit platforms and 64 bit MSVC.

np.long is an alias to python's long:

But python's long is an unlimited type--it can't be mapped to a c type at
all.

arch -32 python -c import numpy as np; print np.dtype(np.int); print
np.dtype(np.long)
int32
int64


So this is giving us a 64 bit int--not a bad compromise, but not a python
long--I've got to wonder why the alias is there at all.

arch -64 python -c import numpy as np; print np.dtype(np.int); print
np.dtype(np.long)
int64
int64

Same thing on 64 bit.

So while np.long is an alias to python long--it apparently is translated
internally as 64 bit -- everywhere?

So apparently there is no way to get a platform long. ( or, for that
matter, a platform anything else, it's just that there is more consistancy
among common platforms for the others)

-Chris


All this is on python 2.7, I am not sure how/if that changes on python 3
(that consolidated python int/long).

David


 -Chris




 --

 Christopher Barker, Ph.D.
 Oceanographer

 Emergency Response Division
 NOAA/NOS/ORR(206) 526-6959   voice
 7600 Sand Point Way NE   (206) 526-6329   fax
 Seattle, WA  98115   (206) 526-6317   main reception

 chris.bar...@noaa.gov
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] What is a numpy.long type?

[Chris Barker - NOAA Federal]

On Fri, Aug 23, 2013 at 8:11 AM, Sebastian Berg
sebast...@sipsolutions.net wrote:
 So this is giving us a 64 bit int--not a bad compromise, but not a
 python long--I've got to wonder why the alias is there at all.

 It is there because you can't remove it :).

sure we could -- not that we'd want to

 So while np.long is an alias to python long--it apparently is
 translated internally as 64 bit -- everywhere?

 Not sure how a python long is translated...

The big mystery!

 An np.int_ is a platform long, since the python ints are C longs. It is
 a bit weird naming, but it is transparent. Check
 http://docs.scipy.org/doc/numpy-dev/reference/arrays.scalars.html#built-in-scalar-types

cool, thanks.

 you got everything platform dependend there really. `intc` is an int,
 `int_` is a long, and you got `longlong`, as well as `intp` which is an
 ssize_t, etc.

great, thanks. That's helpful.

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] What is a numpy.long type?

[Chris Barker - NOAA Federal]

On Fri, Aug 23, 2013 at 8:15 AM, Charles R Harris
charlesr.har...@gmail.com wrote:
 I use 'bBhHiIlLqQ' for the C types. Long varies between 32  64 bit,
 depending on the platform and 64 bit convention chosen. The C int is always
 32 bits as far as I know.

Well, not in the spec, but in practice, probably. Maybe not on some
embedded platfroms?

not my issue...

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] OS X binaries for releases

[Chris Barker - NOAA Federal]

On Thu, Aug 22, 2013 at 12:14 PM, Russell E. Owen ro...@uw.edu wrote:
 I'll be interested to learn how you make binary installers for python
 3.x because the standard version of bdist_mpkg will not do it. I have
 heard of two other projects (forks or variants of bdist_mpkg) that will,
 but I have no idea of either is supported.

Ideally -- we go to binary wheels -- but don't know if the world is
ready for that.

 I have been able to building packages on 10.8 using
 MACOSX_DEPLOYMENT_TARGET=10.6 that will run on 10.6, so it will probably
 work. However I have run into several odd problems over the years
 building a binary installer on a newer system only to find it won't work
 on older systems for various reasons. Thus my personal recommendation is
 that you build on 10.6 if you want an installer that reliably works for
 10.6 and later.

That is the easiest and most robust, for sure. And you can also keep
that system clean -- i.e. no libfreetypes around to acidentally link
to...

-Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] What is a numpy.long type?

[Chris Barker - NOAA Federal]

Hi folks,

I had thought that maybe a numpy.long dtype was a system
(compiler)-native C long.

But on both 32 and 64 bit python on OS-X, it seems to be 64 bit. I'm
pretty sure that on OS-X 32 bit, a C long is 32 bits. (gdd, of course)

I don't have other machines to test on , but as the MS compilers and
gcc do different things with a C long on 64 bit platforms, Im curious
how numpy defines it.

In general, I prefer the explicit is better than implicit approach
and use, e.g. int32 and int64. However, in this case, we are using
Cython, and calling C code that used long -- so what I want is
whatever the compiler thinks is a long -- is there  a way to do that
without my own kludgy platform-dependent code?

I note that the Cython numpy.pxd assigns:

ctypedef signed long  npy_long

Is that a bug? (or maybe npy_long is not supposed to match np.long

-Chris




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] OS X binaries for releases

[Chris Barker - NOAA Federal]

Ralf,

Thanks for doing all this!

 Building binaries for releases is currently quite complex and
 time-consuming.

It sure would be nice to clean that up.

 For OS X we need two different machines, because we still
 provide binaries for OS X 10.5 and PPC machines. I propose to not do this
 anymore. It doesn't mean we completely drop support for 10.5 and PPC, just
 that we don't produce binaries. PPC was phased out in 2006 and OS X 10.6
 came out in 2009, so there can't be a lot of demand for it (and the download
 stats at http://sourceforge.net/projects/numpy/files/NumPy/1.7.1/ confirm
 this).

 Furthermore I propose to not provide 2.6 binaries anymore. Downloads of 2.6
 OS X binaries were 5% of the 2.7 ones. We did the same with 2.4 for a long
 time - support it but no binaries.

Adding 2.6 is not a huge deal, but I agree -- what's the point? And it
really is time to drop PPC an 10.5, though I'd be interested to see if
anyone IS still using it.

 So what we'd have left at the moment is only the 64-bit/32-bit universal
 binary for 10.6 and up. What we finally need to add is 3.x OS X binaries. We
 can make an attempt to build these on 10.8 - since we have access to a
 hosted 10.8 Mac Mini it would allow all devs to easily do a release (leaving
 aside the Windows issue). If anyone has tried the 10.6 SDK on 10.8 and knows
 if it actually works, that would be helpful.

haven't tried it yet, but it should be doable -- if a pain and a laudable goal.

 Any concerns, objections?

nope -- good plan.

Note that I am trying to unify some of this binary building -- the
same techniques used for matplotlib and useful for other packages, too
(numpy/scipy for sure), but also PIL, netCDF, who knows what?

I'll post here when I get a little further and there is something for
others to look at and/or contribute to.

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] PEP 450 (stats module for standard library)

[Chris Barker - NOAA Federal]

On Fri, Aug 16, 2013 at 8:20 AM, Alan G Isaac alan.is...@gmail.com wrote:
 http://www.python.org/dev/peps/pep-0450/
 https://groups.google.com/forum/#!topic/comp.lang.python/IV-3mobU7L0

as numpy is the right way to do this sort of stuff, I think this is
a better argument for a numpy-lite in the stdlib than anything else.

But I doubt that will happen. Oh well.

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Upcoming 1.8 release.

[Chris Barker - NOAA Federal]

On Thu, Aug 15, 2013 at 12:31 PM, Matthew Brett matthew.br...@gmail.com wrote:

 I'm afraid I don't understand the discussion on timezones in
 datetime64, but I have the impression that those who do think it needs
 an urgent decision and some action for the short term.  Is that right,
 datetimers?

 If that's so, and there are worthwhile changes that are practical in
 the next few weeks, it seems reasonable to wait.

Well, it's only urgent in the sense that there are indeed a couple
small changes that would really help, and if we don't use a release to
motivate us, when will we it ever get done?

But it'll still take someone to do it -- I'm afraid it's out of my
depth to do so.

There is a chance that Mark W. or Travis O. could do it, but it does
seem unlikely that they'll find the time in the next week or two, so I
guess we'll put it off, and keep the experimental label on there.

 My impression is that we will have something for 1.9. If it comes in for
 1.8, fine.

I sure hope we can at least get the rip out the ugly default I/O TZ
behavior fix in time for 1.9. Whether something more ambitious can be
done, we'll have to see.

 OK - then I guess you are saying it is up you, our datetimer friends,
 to make a proposal and timetable and implementation, if y'all think it
 can be done in the next few weeks,

Yup -- if anyone wants to pipe up and offer to do it, speak now or
forever hold your piece.

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Upcoming 1.8 release.

[Chris Barker - NOAA Federal]

On Thu, Aug 15, 2013 at 3:16 PM, Matthew Brett matthew.br...@gmail.com wrote:
 Chris B - are you the point man on this one?  What do you think?

Only the point man in the sense that I'm poking at people to try to
get what I want ;-)

But see my other note.

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] About ready to start 1.8 release process.

[Chris Barker - NOAA Federal]

On Tue, Aug 13, 2013 at 5:54 PM, Charles R Harris
charlesr.har...@gmail.com wrote:
 I wish it were. It seems unreasonably difficult to get constructive
 feedback. Chris is pretty much the only one making the attempt and that
 discussion petered out.

well, Nathaniel Smith chimed in, and Mark Weibe commented a bit (as he
wrote the code in the first place). Wes McKinney also chimed in,
essentially saying that Pandas mostly has to do its own thing, and I'm
pretty sure would prefer that the current TZ converting was ripped out
as I suggest.

In the thread on the mialing list I started:

http://numpy-discussion.10968.n7.nabble.com/timezones-and-datetime64-tt33407.html#none

A bunch of core people chimed in. I just re-read that thread and
provide a summary below:

 - No one said they liked it as it is, and there were a number of +1
(and even a +10) comment to cleaning up the current conversion issues.

Some folks would like real improvements:
  - proper, full time zone handling
  - proper (or maybe just better?) handling of leap-seconds, and all that.

 - Marc Weibe had a suggestion or two that were not a trivial
addition -- so off the table for this release, and would need some
real discussion, a NEP, and developer time...

- Marc also pointed out that handling the Locale time zone is helpful
for the interactive use case, which is a major numpy use-case.
However, doing it half-way isn't really helpful anyway.

- Marc also commented that making datetime64 time-zone naive would be
the easy way

- There was also discussion about having a settable epoch -- I think
that would be really useful, but not for this release, and would
require a NEP, etc...

- From Travis O.:It seems to me that the biggest issue is just the
automatic conversion that is occurring on string or date-time input.
We should stop using the local time-zone (explicit is better than
implicit strikes again) and not use any time-zone unless time-zone
information is provided in the string. I am definitely +1 on that.

My conclusions:

The current code is not really usable for ANY use without careful
checking of inputs and/or work-arounds.

No one involved in the numpy list would mind removing the time zone
translation on I/O. A number of people would find it a real
improvement. (granted most users don't get involved with these
discussion on the list)

There are some great ideas for how to improve it that would require
NEPs, discussion and real developer time -- that's not happening any
time soon.

Ripping out the translation on I/O would be fairly easy (at least for
someone familiar with the code)

So I say let's do it!

But who? Travis offered up himself and Mark as possibilities, but who
knows what their schedules allow at this point -- are either of you
following this thread?

It's probably not too big a deal for anyone familiar with the numpy
code base -- if no one else steps up, I can give it a try, though I'm
sure I wouldn't have even found what code to edit in the time it would
take Mark to fix it.

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] About ready to start 1.8 release process.

[Chris Barker - NOAA Federal]

On Wed, Aug 14, 2013 at 10:35 AM, Mark Wiebe mwwi...@gmail.com wrote:

Hi Mark, great to have you thinking (and coding) about this!

 - Marc also commented that making datetime64 time-zone naive would be
 the easy way

 I've experimented a little bit with one possibility in this direction within
 dynd. Basically, adding a time zone to the metadata which initially only can
 be UTC or abstract. (I'm calling it abstract because I don't think naive
 communicates the right idea about it.)

I like that term better, too -- I only borrowed naive from the
stdlib datetime docs.

And having these two is good -- saves us the argument of which to do


  I'm -1 on simply removing a
 distinction between UTC and abstract ISO 8601 strings, I think equivocating
 the two would be as bad as the current situation.

well, I wouldn't equate them (or did you mean equate, thought that
would be equivocating...), I'd say one or the other if you couldnt' do
both. though Im still confused about the difference if you don't
provide any other time zone capability.

 Adding a limited timezone
 with just these two options might be a way to improve the situation without
 too much development effort, while maintaining a clear path to proper
 timezone support.

yup -- though my concern is that calling a datetime64 UTC may imply
that there is some other time zone functionality in there -- which
there isn't (at least not yet) i.e. it's really jsut a hint ot the
user as to how it should be thought of (and makes it clear what to do
when you parse an ISO sting with an offset...)

 In this formulation, strings without a timezone are parsed as abstract
 datetimes, and strings with a timezone as UTC datetimes, without allowing
 implicit conversions:

 from dynd import nd, ndt
 nd.array('2011-03-22T12:00', dtype='datetime(min)')
 nd.array(2011-03-22T12:00, datetimeunit=minute,tz=abstract)
 nd.array('2011-03-22T12:00Z', dtype='datetime(min, UTC)')
 nd.array(2011-03-22T12:00Z, datetimeunit=minute,tz=utc)
 nd.array('2011-03-22T12:00Z', dtype='datetime(min)')
 Traceback (most recent call last):
   File stdin, line 1, in module
   File _pydynd.pyx, line 892, in _pydynd.w_array.__cinit__
 (_pydynd.cxx:5774)
 RuntimeError: cannot parse 2011-03-22T12:00Z as an abstract datetime using
 rule strict, because a timezone was present in the string
 nd.array('2011-03-22T12:00', dtype='datetime(min, UTC)')
 Traceback (most recent call last):
   File stdin, line 1, in module
   File _pydynd.pyx, line 892, in _pydynd.w_array.__cinit__
 (_pydynd.cxx:5774)
 RuntimeError: cannot parse 2011-03-22T12:00 as a datetime with timezone
 using
 rule strict, because no timezone was present in the string

This is looking good to me.

 I've also implemented one way of getting a more relaxed parsing mode, with
 the parameter that controls casting strictness. Note that in this case it is
 throwing away the timezone information, so the -0400 doesn't get
 incorporated in the third value.

 nd.array(['2011-03-22T12:00', '2012-04-23T01:00Z',
 '2012-05-01T13:00-0400'])
 nd.array([2011-03-22T12:00, 2012-04-23T01:00, 2012-05-01T13:00],
 strided_dimconvertto=datetimeunit=minute,tz=abstract, from=string,
 errmode=none)

hmm -- it is explicite, but still seems fragile to me -- one coudl
test with Z and 0, and then  later get a different offset and things
would go tot heck -- but buyer beware with a non-default setting, I
guess.

 This stuff is all experimental/work in progress, but if you want to try
 these examples you can install Anaconda and use its conda package manager to
 update the package dynd-python.
 (https://github.com/ContinuumIO/dynd-python/blob/master/README.md#trying-out-dynd).

Great, thanks!

Any prospects of something similar in numpy relatively soon?

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Do I Still Have to Edit intelcompiler.py For ICC?

[Chris Barker - NOAA Federal]

On Wed, Aug 14, 2013 at 2:12 PM, Ralf Gommers ralf.gomm...@gmail.com wrote:

 with the Intel compilers, I have to supply --compiler and --fcompiler. Is
 there any way to just do this in site config?

  Maybe pip has a way to supply that info but I've never
 bothered to look for it - python setup.py  works just as well.

It look slike you can pass distuitls commands through pip:


--install-option options

Extra arguments to be supplied to the setup.py install command (use
like –install-option=”–install-scripts=/usr/local/bin”). Use multiple
–install-option options to pass multiple options to setup.py install.
If you are using an option with a directory path, be sure to use
absolute path.


There is also a way to specify default arguments for pip install, so
that it woudl get used every time.

http://www.pip-installer.org/en/latest/usage.html#pip-install

-HTH, Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Reading footer lines

[Chris Barker - NOAA Federal]

On Tue, Aug 13, 2013 at 6:01 AM, Daπid davidmen...@gmail.com wrote:

 Alternatively, you could also use seek to put the pointer a certain
 distance from the end of the file and start from there,

That's what I'd do if the file(s) may be too large to simply dump into memory.

 but this could cause problems in Windows.

what problems? this is pretty straightforward stuff!

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] About ready to start 1.8 release process.

[Chris Barker - NOAA Federal]

On Mon, Aug 12, 2013 at 2:41 PM, Charles R Harris
charlesr.har...@gmail.com wrote:

 Datetime64 will not be modified in this release.

I now there is neither the time nor the will for all that it needs,
but please, please, please, can we yank out the broken timezone
handling at least?

https://github.com/numpy/numpy/issues/3290

-Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] About ready to start 1.8 release process.

[Chris Barker - NOAA Federal]

On Mon, Aug 12, 2013 at 3:14 PM, Charles R Harris
charlesr.har...@gmail.com wrote:

  Datetime64 will not be modified in this release.

 I now there is neither the time nor the will for all that it needs,
 but please, please, please, can we yank out the broken timezone
 handling at least?

 https://github.com/numpy/numpy/issues/3290

 You need to be a bit more specific, what parts should be yanked out?

It's pretty well discussed in issue 3290 -- but what I'm suggesting is
essentially to ignore time zones completely -- i.e. make the
datetime64 naive with respect to time zones.

In fact, it already is -- the only timezone handling it has is that if
it parses an ISO string and no timezone is specified, it applies the
Locale time zone -- this is pretty broken behavior. At the moment, I
can't recall what it does with a datetime.datetime instance, but it's
not quite consitent with what it does parsing string.

I _think_ the only point of contention in that issue 3290 discussion
is how datetime64 should parse and ISO string that provides an offset:

1) ignore it -- probably a bad idea
2) raise an error -- you can't do it.
3) apply the offset so that the resulting datetime64 is assumed to be UTC.

Personally, I think (2) is the way to go, with the possible addition
of allowing zero offset, 'cause, why not?

 I'm
 also worried about breaking peoples' work arounds this late in the game.

well, that is an issue -- though I think most work-arounds will not be
broken, and the discussion about this didn't reveal a single user that
actually used the use the Locale time zone feature -- granted,
people contributing to the discussion is a pretty small subset of
users.

But in 1.7 datetime64 is officially experimental, and keeping a broken
implementation around longer will only make for more broken code
later.

-Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] add .H attribute?

[Chris Barker - NOAA Federal]

On Jul 23, 2013, at 4:57 PM, Alan G Isaac alan.is...@gmail.com wrote:

 Finally, I think (?) everyone (proponents and opponents)
 would be happy if .H could provide access to an iterative
 view of the conjugate transpose.

Except those of us that don't think numpy needs it at all.

But I'll call it a -0

-Chris
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] add .H attribute?

[Chris Barker - NOAA Federal]

On Jul 23, 2013, at 11:54 PM, Stéfan van der Walt ste...@sun.ac.za wrote:

 The .H property has been implemented in Numpy matrices and Scipy's
 sparse matrices for many years.

 Then we're done. Numpy  is an array package, NOT a matrix package, and
 while you can implement matrix math with arrays (and we do), having
 quick and easy mnemonics for common matrix math operations (but
 uncommon general purpose array operations) is not eh job of numpy.
 That's what the matrix object is for.

 I would argue that the ship sailed when we added .T already.  Most
 users see no difference between the addition of .T and .H.

I don't know who can speak for most users, but I see them quite
differently. Transposing is a common operation outside of linear
algebra--I, for one, use it to work with image arrays, which are often
stored in a way by image libraries that is the transpose of the
natural numpy way.

But anyway, just because we have one domain-specific convenience
attribute, doesn't mean we should add them all.

 The matrix class should probably be deprecated and removed from NumPy
 in the long run--being a second class citizen not used by the
 developers themselves is not sustainable.

I agree, but the fact that no one has stepped up to maintain and
improve it tells me that there is not a very large community that
wants a clean linear algebra interface, not that we should try to
build such an interface directly into numpy.

Is there really a point to a clean interface to the Hermetian
transpose, but not plain old matrix multiply?

  And, now that we have dot
 as a method,

Agh, but dot is a method--so we still don't have a clean
relationship with the math in text books:

AB = A.dot(B)

Anyway, adding .H is clearly not a big deal, I just don't think it's
going to satisfy anyone anyway.

-Chris
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] add .H attribute?

[Chris Barker - NOAA Federal]

  plan
 would be to remove the Matrix class from numpy over two or three
 releases, and publish it as a separate package on PyPi.

Anyone willing to take ownership of it? Maybe we should still do it of
not-- at least it will make it clear that it is orphaned.

Though one plus to having matrix in numpy is that it was a testbed for
ndarray subclassing...

-Chris

 Please do! There are some sticky issues to work through (e.g. how to
 deprecate the matrix entry in the numpy namespace, what to do with
 scipy.sparse), and I don't know whether we'll decide to go through
 with it in the end, but the way to figure that out is to, you know,
 work through them :-).

 -n
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] add .H attribute?

[Chris Barker - NOAA Federal]

On Tue, Jul 23, 2013 at 6:09 AM, Pauli Virtanen p...@iki.fi wrote:

 The .H property has been implemented in Numpy matrices and Scipy's
 sparse matrices for many years.

Then we're done. Numpy  is an array package, NOT a matrix package, and
while you can implement matrix math with arrays (and we do), having
quick and easy mnemonics for common matrix math operations (but
uncommon general purpose array operations) is not eh job of numpy.
That's what the matrix object is for.

Yes, I know the matrix object isn't really what it should be, and
doesn't get much  use, but if you want something that is natural for
doing matrix math, and particularly natural for teaching it -- that's
what it's for -- work to make it what it could be, rather than
polluting numpy with this stuff.

One of the things I've loved about numpy after moving from MATLAB is
that matrixes are second-class citizens, not the other way around.

(OK, I'll go away now)

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] PIL and NumPy

[Chris Barker - NOAA Federal]

On Jul 12, 2013, at 8:51 PM, Brady McCary brady.mcc...@gmail.com wrote:


 something to do with an alpha channel being present.

I'd check and see how PIL is storing the alpha channel. If it's RGBA,
then I'd expect it to work.

But I'd PIL is storing the alpha channel as a separate band, then I'm
not surprised you have an issue.

Can you either drop the alpha or convert to RGBA?

There is also a package called something line imageArray that loads
and saves image formats directly to/from numpy arrays-maybe that would
be helpful.

CHB


 When I remove the
 alpha channel, things appear to work as I expect. Any discussion on
 the matter?

 Brady

 On Fri, Jul 12, 2013 at 10:00 PM, Brady McCary brady.mcc...@gmail.com wrote:
 NumPy Folks,

 I want to load images with PIL and then operate on them with NumPy.
 According to the PIL and NumPy documentation, I would expect the
 following to work, but it is not.



 Python 2.7.4 (default, Apr 19 2013, 18:28:01)
 [GCC 4.7.3] on linux2
 Type help, copyright, credits or license for more information.
 import numpy
 numpy.version.version

 import Image
 Image.VERSION
 '1.1.7'

 im = Image.open('big-0.png')
 im.size
 (2550, 3300)

 ar = numpy.asarray(im)
 ar.size
 1
 ar.shape
 ()
 ar
 array(PIL.PngImagePlugin.PngImageFile image mode=LA size=2550x3300 at
 0x1E5BA70, dtype=object)



 By not working I mean that I would have expected the data to be
 loaded/available in ar. PIL and NumPy/SciPy seem to be working fine
 independently of each other. Any guidance?

 Brady
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Avoiding messy code in Python

[Chris Barker - NOAA Federal]

 I have some code more or less 500 lines, but very messy code. All codes
 containing several functions are in one module, besides, without
 documentation and testing.

 Could anyone give me some advice to organize my messy code in an accurate
 style including test function as well?

This is a really broad, non-numpy specific question. You'd best read
up on good programming practices in general. But you've answered a
good portion of your questions already:

All codes containing several functions are in one module

Nothing wrong with that -- functions that are likely to be used
together belong in one module -- and while Python supports namespaces
(one honking great idea), pythonic style supports keeping those
namespaces flat (flat is better than nested)

So do you need to do any different stucture? I'd look for two things:

1) the DRY principle (Don't Repeat Yourself) -- if you have
essentially the same code in multiple places, refactor that out into
utility functions, or classes, or...

2) Do classes make sense? Look for patterns like:
function_a(data_structure_1, some, other, arguments)
function_b(data_structure_1, some, other, arguments)
function_c(data_structure_1, some, other, arguments)

that is the C procedural style -- nothing wring with that, but it may
be better suited to a class -- the class holds your data structure and
the functions that operate on it in one place.

But:
if a class has only two methods, and one of them is __init__ -- you
don't need a class:

http://pyvideo.org/video/880/stop-writing-classes

also see:

http://pyvideo.org/video/879/the-art-of-subclassing

For why you would want to use classes...

 without documentation and testing.

This one is easy: add documentation and testing!

put docstings in your functions and classes -- if the package is big
enough to warrant it, use Sphinx to document it -- it can extract the
docstrings, and give you a nice framework for writing up more
extensive docs.

Testing:
This sounds like a collection of functions -- you want unit tests --
read up on unit testing (and test-driven development) and the write
them.

I suggest nose or pytest (I prefer pytest, it's got some nice
features, but either is fine). You can also use the stdlib unittest,
but I found it overly wordy and klunky.

HTH,
  -Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy.filled, again

[Chris Barker - NOAA Federal]

On Wed, Jun 12, 2013 at 5:10 AM, Nathaniel Smith n...@pobox.com wrote:

 Personally I think that overloading np.empty is horribly ugly, will
 continue confusing newbies and everyone else indefinitely, and I'm
 100% convinced that we'll regret implementing such a warty interface
 for something that should be so idiomatic.
...
 deprecate np.ma.filled in favor
 of masked_array.filled (which does exactly the same thing) and
 eventually switch np.ma.filled to be consistent with the new
 np.filled.

+1

 I also don't really see why an np.empty() constructor exists, it seems
 to do the same thing that np.ndarray() does.

I had always assumed that np.ndarray() was a low-level interce that
you really don't want to use in regular code (maybe for subclassing
array...), as the docs say:


Arrays should be constructed using `array`, `zeros` or `empty` (refer
to the See Also section below).  The parameters given here refer to
a low-level method (`ndarray(...)`) for instantiating an array.


Am I wrong? is there any reason )other than history to have np.empty()

But in any case, I like np.filled(), as being analogous to ones(),
zeros() and empty()...

-Chris





-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy.filled, again

[Chris Barker - NOAA Federal]

On Wed, Jun 12, 2013 at 11:49 AM, Eric Firing efir...@hawaii.edu wrote:
 On 2013/06/12 4:18 AM, Nathaniel Smith wrote:
 Now imagine a different new version of this page, if we overload
 'empty' to add a fill= option. I don't even know how we document that
 on this page. The list will remain:
empty
ones
zeros

values

kind of fits with ones and zeros...

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy.filled, again

[Chris Barker - NOAA Federal]

On Wed, Jun 12, 2013 at 12:00 PM, Phil Hodge ho...@stsci.edu wrote:
 On 06/12/2013 02:55 PM, Eric Firing wrote:

 I would interpret np.filled as a test, asking whether the array is
 filled.  If the function is supposed to do something related to
 assigning values, the name should be a verb.

or a noun: ones, zeros -- but I agree, better not an adjective.

-Chris




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Dropping support for, Accelerate/veclib?

[Chris Barker - NOAA Federal]

On Tue, Jun 11, 2013 at 10:22 AM, Nils Becker n.bec...@amolf.nl wrote:
 fwiw, homebrew is not macports. it's a more recent replacement that
 seems to be taking over gradually.

And then there is (or was) fink

Anyway, it would be really nice if numpy could work well out-of-the
box with the pyton.org python (and or the Apple-supplied one) without
any need fro macports, homebrew, etc. It's  actually pretty cool that
Apple provides veclib, and and I liked that it got used by default
with a simple setup.py build.

It would be nice to keep it that way.

If a user (or distributor of binaries) wants to build the whole scipy
stack, they'll need to figure out all this *^% anyway, changing a
config in numpy to build differently would be the easiest part.

And yes, there really are folks that use numpy a lot without scipy.

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] stdint.h

[Chris Barker - NOAA Federal]

On Thu, May 23, 2013 at 1:44 PM, Charles R Harris
charlesr.har...@gmail.com wrote:

 Just seeking some info here. The file stdint.h was part of the C99 standard
 and has types for integers of specified width and thus could be used to
 simplify some of the numpy configuration. I'm curious as to which compilers
 might be a problem and what folks think of that possibility.

MSVC 2008, which is the (still!) the standard compiler for Python =
2.7, does not include that header. Newer MSVC versions do.

However, I'm a big fan of stdint.h, so I think it should be used. I'll
bet we can find a header file for MSVC 2008 that could be included
with numpy.

(OK -- not bet much, but worth looking for!)

-Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Possible conversion bug with record array

[Chris Barker - NOAA Federal]

On Wed, May 22, 2013 at 10:07 AM, Nicolas Rougier
 U = np.zeros(1, dtype=[('x', np.float32, (4,4))])

 U[0] = np.eye(4)
 print U[0]
 # output:  ([[0.0, 1.875, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 
 1.875], [0.0, 0.0, 0.0, 0.0]],)

I get the same thing. Note:

In [86]: U[0].shape
Out[86]: ()

it's a rank-zero array, not a 4X4 array -- some odd tings can happen there...

This works:

In [94]: U = np.zeros(1, dtype=[('x', np.float32, (4,4))])

In [95]: U['x'][0] = np.eye(4)

In [96]: U
Out[96]:
array([ ([[1.0, 0.0, 0.0, 0.0], [0.0, 1.0, 0.0, 0.0], [0.0, 0.0, 1.0,
0.0], [0.0, 0.0, 0.0, 1.0]],)],
  dtype=[('x', 'f4', (4, 4))])

Not that the first version isn't a bug!


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Possible conversion bug with record array

[Chris Barker - NOAA Federal]

On Wed, May 22, 2013 at 11:15 AM, eat e.antero.ta...@gmail.com wrote:

 FWIW, apparently bug related to dtype of np.eye(.)

sort of -- the issue shows up when assigning a float64 array (default
for eye()) to a rank-0 array with a custom dtype that has a single
object filed that is an arraynumpy should do the right thing
there. If you pull the arra yout of the custom dtype object, it does
do the right thing. See the example I posted earlier.

-Chris



 On Wed, May 22, 2013 at 8:07 PM, Nicolas Rougier nicolas.roug...@inria.fr
 wrote:



 Hi all,

 I got a weird output from the following script:

 import numpy as np

 U = np.zeros(1, dtype=[('x', np.float32, (4,4))])

 U[0] = np.eye(4)
 print U[0]
 # output:  ([[0.0, 1.875, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0,
 1.875], [0.0, 0.0, 0.0, 0.0]],)

 U[0] = np.eye(4, dtype=np.float32)
 print U[0]
 # output:  ([[1.0, 0.0, 0.0, 0.0], [0.0, 1.0, 0.0, 0.0], [0.0, 0.0, 1.0,
 0.0], [0.0, 0.0, 0.0, 1.0]],)


 The first output is obviously wrong. Can anyone confirm ?
 (using numpy 1.7.1 on osx 10.8.3)

 In []: sys.version
 Out[]: '2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)]'
 In []: np.__version__
 Out[]: '1.6.0'
 In []: U= np.zeros(1, dtype= [('x', np.float32, (4, 4))])

 In []: U[0]= np.eye(4)
 In []: U
 Out[]:
 array([ ([[0.0, 1.875, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0,
 1.875], [0.0, 0.0, 0.0, 0.0]],)],
   dtype=[('x', 'f4', (4, 4))])

 In []: U[0]= np.eye(4, dtype= np.float32)
 In []: U
 Out[]:
 array([ ([[1.0, 0.0, 0.0, 0.0], [0.0, 1.0, 0.0, 0.0], [0.0, 0.0, 1.0, 0.0],
 [0.0, 0.0, 0.0, 1.0]],)],
   dtype=[('x', 'f4', (4, 4))])

 and
 In []: sys.version
 Out[]: '3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64
 bit (AMD64)]'
 In []: np.__version__
 Out[]: '1.7.0rc1'
 In []: U= np.zeros(1, dtype= [('x', np.float32, (4, 4))])

 In []: U[0]= np.eye(4)
 In []: U
 Out[17]:
 array([ ([[0.0, 1.875, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0,
 1.875], [0.0, 0.0, 0.0, 0.0]],)],
   dtype=[('x', 'f4', (4, 4))])

 U[0]= np.eye(4, dtype= np.float32)

 In []: U
 Out[]:
 array([ ([[1.0, 0.0, 0.0, 0.0], [0.0, 1.0, 0.0, 0.0], [0.0, 0.0, 1.0, 0.0],
 [0.0, 0.0, 0.0, 1.0]],)],
   dtype=[('x', 'f4', (4, 4))])


 My 2 cents,
 -eat



 Nicolas
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion



 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Equvalent function for Ceil() and Floor()

[Chris Barker - NOAA Federal]

On Mon, May 20, 2013 at 8:54 AM, Bakhtiyor Zokhidov
bakhtiyor_zokhi...@mail.ru wrote:

 what about the following example:
new_ceil(-0.24, 0.25)
 -0.0

ceil rounds toward +inf (and floor towards -inf) -- this is exactly
what you want if you're doing what I think you are...(note that
round() rounds towards and away from zero -- so different for negative
numbers...)

the minus sign means minus 0.0, which is a seemingly odd floating
point thing -- but FP numbers have a sign bit, which, in this case, is
set to negative, even though, mathematically speaking, 0.0 doesn't
have a sign. But it will compare to 0.0 as you'd expect:

In [8]: -0.0 == 0.0
Out[8]: True

In [9]: -0.0   0.0
Out[9]: False

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Newbie trying to install NumPy

[Chris Barker - NOAA Federal]

On May 18, 2013, at 1:53 AM, Daπid davidmen...@gmail.com wrote:

 On 18 May 2013 07:11, Joe
 You probably have a different version installed. Grab Python 2.7 from
 python.org and install it;

Make sure you match 32/64 bit. The message is a bit out of date,
you'll get the same error if you try to install a 32 bit numpy and
have a 64 bit python.

-Chris


 then, run the numpy installer. It should
 just work out of the box.
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] printing array in tabular form

[Chris Barker - NOAA Federal]

On Thu, May 9, 2013 at 1:06 AM, Sudheer Joseph sudheer.jos...@yahoo.comwrote:

 Thank you Gomersall,
   However writing a formatted out put looks to be bit tricky with
 python relative to other programing languages.


this is just plain wrong -- working with text in python is as easy, or
easier, than most languages.

Take a little time to learn a bit of pyton, and a bit of numpy, then come
back if you can't figure it out.

But as a rule -- if it's seems hard and/or awkward, you are probably doing
it wrong!

 -Chris
-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] nanmean(), nanstd() and other missing functions for 1.8

[Chris Barker - NOAA Federal]

On Wed, May 1, 2013 at 6:52 AM, Benjamin Root ben.r...@ou.edu wrote:

 How about a tuple: (min, max)?



 I am not familiar enough with numpy internals to know which is the better
 approach to implement.  I kind of feel that the 2xN array approach would be
 more flexible in case a user wants all of this information in a single
 array, while still allowing for unpacking as if it was a tuple.


hmm, my thinking is that the min and max values really are two different
results, so getting two separate arrays makes sense to me.

however, you are right, Python's nifty generic sequence unpacking lets you
use a (2X...) array similarly to a tuple of two arrays, so why not?

Food for thought on one reason:

min, max = np.minmax(arr)

would result in two arrays, but they would be views on the same array, Not
sure if that matters, though.

-Chris



-Chris




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] bug in deepcopy() of rank-zero arrays?

[Chris Barker - NOAA Federal]

hmm -- I suppose one of us should post an issue on github -- then ask for
it ti be fixed before 1.8  ;-)

I'll try to get to the issue if no one beats me to it -- got to run now...

-Chris



On Tue, Apr 30, 2013 at 5:35 AM, Richard Hattersley
rhatters...@gmail.comwrote:

 +1 for getting rid of this inconsistency

 We've hit this with Iris (a met/ocean analysis package - see github), and
 have had to add several workarounds.


 On 19 April 2013 16:55, Chris Barker - NOAA Federal chris.bar...@noaa.gov
  wrote:

  Hi folks,

 In [264]: np.__version__
 Out[264]: '1.7.0'

 I just noticed that deep copying a rank-zero array yields a scalar --
 probably not what we want.

 In [242]: a1 = np.array(3)

 In [243]: type(a1), a1
 Out[243]: (numpy.ndarray, array(3))

 In [244]: a2 = copy.deepcopy(a1)

 In [245]: type(a2), a2
 Out[245]: (numpy.int32, 3)

 regular copy.copy() seems to work fine:

 In [246]: a3 = copy.copy(a1)

 In [247]: type(a3), a3
 Out[247]: (numpy.ndarray, array(3))

 Higher-rank arrays seem to work fine:

 In [253]: a1 = np.array((3,4))

 In [254]: type(a1), a1
 Out[254]: (numpy.ndarray, array([3, 4]))

 In [255]: a2 = copy.deepcopy(a1)

 In [256]: type(a2), a2
 Out[256]: (numpy.ndarray, array([3, 4]))

 Array scalars seem to work fine as well:

 In [257]: s1 = np.float32(3)

 In [258]: s2 = copy.deepcopy(s1)

 In [261]: type(s1), s1
 Out[261]: (numpy.float32, 3.0)

 In [262]: type(s2), s2
 Out[262]: (numpy.float32, 3.0)

 There are other ways to copy arrays, but in this case, I had a dict
 with a bunch of arrays in it, and needed a deepcopy of the dict. I was
 surprised to find that my rank-0 array got turned into a scalar.

 -Chris

 --

 Christopher Barker, Ph.D.
 Oceanographer

 Emergency Response Division
 NOAA/NOS/ORR(206) 526-6959   voice
 7600 Sand Point Way NE   (206) 526-6329   fax
 Seattle, WA  98115   (206) 526-6317   main reception

 chris.bar...@noaa.gov
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion



 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] nanmean(), nanstd() and other missing functions for 1.8

[Chris Barker - NOAA Federal]

On Apr 30, 2013, at 6:37 PM, Benjamin Root ben.r...@ou.edu wrote:
 I can not think of any reason not to include these functions in v1.8.

+1


 Of course, the documentation for discussed before: np.minmax().  My thinking 
 is that it would return a 2xN array

How about a tuple: (min, max)?

-Chris
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] 1.8 release

[Chris Barker - NOAA Federal]

On Thu, Apr 25, 2013 at 8:19 AM, Dave Hirschfeld
dave.hirschf...@gmail.comwrote:

  Hi All,I think it is time to start the runup to the 1.8 release. I don't
 know of any outstanding blockers but if anyone has a PR/issue that they
 feel
 needs to be in the next Numpy release now is the time to make it
 known.Chuck
 

 It would be good to get the utc-everywhere fix for datetime64 in there if
 someone has time to look into it.

 +1

I've been on vacation, so haven't written up the various notes and comments
as a NEP  yet -- I'll try to do that soon. There are some larger proposals
in the mix, which I doubt could be done by 1.8, but we really should fix
the utc-everywhere issue ASAP. I think it will be pretty easy to do, but
someone still needs to do it...

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] 1.8 release

[Chris Barker - NOAA Federal]

On Mon, Apr 29, 2013 at 12:07 PM, Charles R Harris 
charlesr.har...@gmail.com wrote:


 It would be good to get the utc-everywhere fix for datetime64 in there if
 someone has time to look into it.

 +1

 I've been on vacation, so haven't written up the various notes and
 comments as a NEP  yet -- I'll try to do that soon. There are some larger
 proposals in the mix, which I doubt could be done by 1.8, but we really
 should fix the utc-everywhere issue ASAP. I think it will be pretty easy
 to do, but someone still needs to do it...


 Is there a issue for this?


no -- I don't think so -- just a bunch of discussion on the list. I was
starting down the path of a proper NEP, etc, but I think that:

a) Datetime64 is still officially experimental, so we can change things
more rapidly than we might otherwise.

b) It's really pretty broken as it is.

I'll see if I can open an issue for the easy fix.

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] 1.8 release

[Chris Barker - NOAA Federal]

On Mon, Apr 29, 2013 at 12:12 PM, Chris Barker - NOAA Federal 
chris.bar...@noaa.gov wrote:


  It would be good to get the utc-everywhere fix for datetime64 in there if
 someone has time to look into it.

 I'll see if I can open an issue for the easy fix.



DONE:  Issue #3290


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy scalars and savez -- bug?

[Chris Barker - NOAA Federal]

On Apr 18, 2013, at 11:33 PM, Nathaniel Smith n...@pobox.com wrote:

On 18 Apr 2013 01:29, Chris Barker - NOAA Federal chris.bar...@noaa.gov
wrote:
 This has been annoying, particular as rank-zero scalars are kind of a
pain.

BTW, while we're on the topic, can you elaborate on this? I tend to think
scalars (as opposed to 0d ndarrays) are kind of a pain, so I'm curious if
you have specific issues you've run into with 0d ndarrays.

Well, I suppose what's really a pain is that we have both, and they are not
the same, and neither can be used in all cases one may want.

In the case at hand, I really wanted a datetime64 scalar. By saving and
re-loading in an npz, it got converted to a rank-zero array, which had
different behavior. In this case, the frustrating bit was how to extract a
scalar again ( which I really wanted to turn into a datetime object).

After the fact, I discovered .item(), which seems to do what I want.

On a phone now, so sorry about the lack of examples.

Note: I've lost track of why we need both scalers and rank-zero arrays. I
can't help thinking that there could be an object that acts like a scalar
in most contexts, but also has the array methods that make sense.

But I know it's far from simple.

-Chris
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] type conversion question

[Chris Barker - NOAA Federal]

On Thu, Apr 18, 2013 at 10:04 PM, K.-Michael Aye kmichael@gmail.com wrote:
 On 2013-04-19 01:02:59 +, Benjamin Root said:
 So why is there an error in the 2nd case, but no error in the first
 case? Is there a logic to it?

 When you change a dtype like that in the first one, you aren't really
 upcasting anything.  You are changing how numpy interprets the
 underlying bits.  Because you went from a 32-bit element size to a
 64-bit element size, you are actually seeing the double-precision
 representation of 2 of your original data points together.

I was wondering what would happen if there were not the right number
of points available...

In [225]: a = np.array((2.0, 3.0), dtype=np.float32)

In [226]: a.dtype=np.float64

In [227]: a
Out[227]: array([ 32.0763])

OK , but:

In [228]: a = np.array((2.0,), dtype=np.float32)

In [229]: a.dtype=np.float64
---
ValueErrorTraceback (most recent call last)
ipython-input-229-0d494747aee1 in module()
 1 a.dtype=np.float64

ValueError: new type not compatible with array.

so numpy is smart enough to not let you do it .. good thing.

Final note -- changing the dtype in place like that is a very powerful
and useful tool, but not likely to be used often -- it's  really for
things like working with odd binary data and the like.

-Chris





-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy scalars and savez -- bug?

[Chris Barker - NOAA Federal]

Robert,

As I think you wrote the code, you may have a quick answer:

Given that numpy scalars do exist, and have their uses -- I found this
wiki page to remind me:

http://projects.scipy.org/numpy/wiki/ZeroRankArray

It would be nice if the .npy format could support them. Would that be
a major change? I'm trying to decide if this bugs me enough to work on
that.

-Chris


On Fri, Apr 19, 2013 at 8:03 AM, Chris Barker - NOAA Federal
chris.bar...@noaa.gov wrote:
 On Apr 18, 2013, at 11:33 PM, Nathaniel Smith n...@pobox.com wrote:

 On 18 Apr 2013 01:29, Chris Barker - NOAA Federal chris.bar...@noaa.gov
 wrote:
 This has been annoying, particular as rank-zero scalars are kind of a
 pain.

 BTW, while we're on the topic, can you elaborate on this? I tend to think
 scalars (as opposed to 0d ndarrays) are kind of a pain, so I'm curious if
 you have specific issues you've run into with 0d ndarrays.

 Well, I suppose what's really a pain is that we have both, and they are not
 the same, and neither can be used in all cases one may want.

 In the case at hand, I really wanted a datetime64 scalar. By saving and
 re-loading in an npz, it got converted to a rank-zero array, which had
 different behavior. In this case, the frustrating bit was how to extract a
 scalar again ( which I really wanted to turn into a datetime object).

 After the fact, I discovered .item(), which seems to do what I want.

 On a phone now, so sorry about the lack of examples.

 Note: I've lost track of why we need both scalers and rank-zero arrays. I
 can't help thinking that there could be an object that acts like a scalar in
 most contexts, but also has the array methods that make sense.

 But I know it's far from simple.

 -Chris





-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: NumPy 1.7.1 release

[Chris Barker - NOAA Federal]

On Fri, Apr 19, 2013 at 8:12 AM, Ondřej Čertík ondrej.cer...@gmail.com wrote:

 I'm pleased to announce the availability of the final NumPy 1.7.1 release.

Nice work -- but darn! I was hoping a change/fix to teh datetime64
timezone handlien could get into the next release -- oh well.

When do we expect the next one may be? I've lost track, are you trying
to keep to a schedule now, or just waiting until there is something
compelling to release?

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] bug in deepcopy() of rank-zero arrays?

[Chris Barker - NOAA Federal]

Hi folks,

In [264]: np.__version__
Out[264]: '1.7.0'

I just noticed that deep copying a rank-zero array yields a scalar --
probably not what we want.

In [242]: a1 = np.array(3)

In [243]: type(a1), a1
Out[243]: (numpy.ndarray, array(3))

In [244]: a2 = copy.deepcopy(a1)

In [245]: type(a2), a2
Out[245]: (numpy.int32, 3)

regular copy.copy() seems to work fine:

In [246]: a3 = copy.copy(a1)

In [247]: type(a3), a3
Out[247]: (numpy.ndarray, array(3))

Higher-rank arrays seem to work fine:

In [253]: a1 = np.array((3,4))

In [254]: type(a1), a1
Out[254]: (numpy.ndarray, array([3, 4]))

In [255]: a2 = copy.deepcopy(a1)

In [256]: type(a2), a2
Out[256]: (numpy.ndarray, array([3, 4]))

Array scalars seem to work fine as well:

In [257]: s1 = np.float32(3)

In [258]: s2 = copy.deepcopy(s1)

In [261]: type(s1), s1
Out[261]: (numpy.float32, 3.0)

In [262]: type(s2), s2
Out[262]: (numpy.float32, 3.0)

There are other ways to copy arrays, but in this case, I had a dict
with a bunch of arrays in it, and needed a deepcopy of the dict. I was
surprised to find that my rank-0 array got turned into a scalar.

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: NumPy 1.7.1 release

[Chris Barker - NOAA Federal]

On Fri, Apr 19, 2013 at 8:46 AM, Nathaniel Smith n...@pobox.com wrote:

 Nice work -- but darn! I was hoping a change/fix to teh datetime64
 timezone handlien could get into the next release -- oh well.

 That's probably too big a behavioural chance to go into a point
 release in any case...

well, datetime64 is marked as experimental, and clearly quite broken
(in this regard), so I think a point release change would be fine --
the fewer people use it this way the better.

 I think the current schedule for 1.8 is any time now, as soon as we
 manage to get around to it.

then the point-release question is a non-issue anyway.

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy scalars and savez -- bug?

[Chris Barker - NOAA Federal]

On Fri, Apr 19, 2013 at 10:21 AM, Robert Kern robert.k...@gmail.com wrote:
 On Fri, Apr 19, 2013 at 8:45 PM, Chris Barker - NOAA Federal
 chris.bar...@noaa.gov wrote:

 Given that numpy scalars do exist, and have their uses -- I found this
 wiki page to remind me:

 http://projects.scipy.org/numpy/wiki/ZeroRankArray

 It would be nice if the .npy format could support them. Would that be
 a major change? I'm trying to decide if this bugs me enough to work on
 that.

 I think that is significant scope creep for the .npy format, and I
 would like to avoid it.

hmm -- maybe it's more work that we want, but it seems to me that
numpy scalars are part and parcel of numpy -- so it makes sense for
.npy to save them.

 A case might be made for letting np.savez()
 simply pickle non-arrays that it is given. I have a vague recollection
 that that was discussed when savez() was designed and rejected as a
 moral hazard, but I could be wrong.

That could be a nice solution -- I'm not _so_ worried about moral hazards!

 The .npy and .npz formats are
 intentionally limited by design. As soon as you feel constrained by
 those limitations, you should start using more full-fledged and
 standard file formats.

well, maybe -- in this case, I'm using it to cache a bunch  of data on
disk. The data are all in a dict of numpy arrays, so it was really
natural and easy (and I presume fast) to use npz. All I want to is
dump it to disk, and get back the same way. It worked great.

Then I needed a datetime stored with it all -- so I figured a
datetime64 scalar would be perfect. It's not a huge deal to use a
rank-zero array instead, but it would have been nicer to be able to
store a scalar (I suppose one trick may be that there are numpy
scalars, and there are regular old pyton scalars...)

Anyway -- going to HDF, or netcdf, or role-your-own really seems like
overkill for this. I just need something fast and simple and it
doesn't need to interchange with anything else.

-Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy scalars and savez -- bug?

[Chris Barker - NOAA Federal]

On Fri, Apr 19, 2013 at 11:31 AM, Nathaniel Smith n...@pobox.com wrote:
 On 19 Apr 2013 19:22, Chris Barker - NOAA Federal chris.bar...@noaa.gov
 wrote:
 Anyway -- going to HDF, or netcdf, or role-your-own really seems like
 overkill for this. I just need something fast and simple and it
 doesn't need to interchange with anything else.

 Just use pickle...?

hmm -- for some reason, I always have thought as pickle as unreliable
and ill-suited to numpy arrays -- we developed savez for a reason...
but maybe I just need to give it a shot and see how it works.

Thanks,
  -Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy scalars and savez -- bug?

[Chris Barker - NOAA Federal]

On Thu, Apr 18, 2013 at 4:04 AM, Robert Kern robert.k...@gmail.com wrote:

 np.save() and company (and the NPY format itself) are for arrays, not
 for scalars. np.save() uses an np.asanyarray() to coerce its input
 which is why your scalar gets converted to a rank-zero array.

Fair enough -- so a missing feature, not bug -- I'll need to look at
the docs and see if that can be clarified - I note that it never
dawned on me to pass anything other than an array in (like a list),
but I guess if I did, it would likely work, but return an array when
re-loaded. I'm ambivalent about whether I like this feature -- in this
case, it resulted in confusion. If I'd gotten an exception in the
first place, it would have been simple enough to fix, as it was it
took some poking around.

As for numpy scalars -- would it be a major lift to support them directly?

-Chris










 --
 Robert Kern
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] datetime64 1970 issue

[Chris Barker - NOAA Federal]

On Wed, Apr 17, 2013 at 6:05 PM, Benjamin Root ben.r...@ou.edu wrote:

 Aren't we on standard time at Jan 1st?  So, at that date, you would have
 been -8.

yes, of course, pardon me for being an idiot.

-Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] datetime64 1970 issue

[Chris Barker - NOAA Federal]

On Wed, Apr 17, 2013 at 11:27 PM, Joris Van den Bossche
jorisvandenboss...@gmail.com wrote:
 Anyone tested this on Windows?

 On Windows 7, numpy 1.7.0 (Anaconda 1.4.0 64 bit), I don't even get a wrong
 answer, but an error:

 In [3]: np.datetime64('1969-12-31 00')
 Out[3]: numpy.datetime64('1969-12-31T00:00Z','h')

 In [4]: np.datetime64('1970-01-01 00')
 ---
 OSError   Traceback (most recent call last)
 ipython-input-4-ebf323268a4e in module()
  1 np.datetime64('1970-01-01 00')

 OSError: Failed to use 'mktime' to convert local time to UTC

OK -- so that confirms the issue is from using the system libs to do this.

though I'm surprised it fails on that date -- I would have expected it
to fail on the 1960 date -- unless there is code that bypasses the
system libs (Or modifies it some way) for pre-1970 dates.

 We REALLY need to fix this!

There is a lot that we _could_ do to handle all tis well, but one
thing is clear to me -- poor timezone support is MUCH worse than no
timezone support!

I'll get that NEP up, but  I hope someone can get on this -- at least
before any next release.

-Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy scalars and savez -- bug?

[Chris Barker - NOAA Federal]

On Thu, Apr 18, 2013 at 8:31 AM, Chris Barker - NOAA Federal
chris.bar...@noaa.gov wrote:

 Fair enough -- so a missing feature, not bug -- I'll need to look at
 the docs and see if that can be clarified -

All I've found is the docstring docs (which also show up in the Sphinx
docs). I suggest some slight modification:

def save(file, arr):

Save an array to a binary file in NumPy ``.npy`` format.

Parameters
--
file : file or str
File or filename to which the data is saved.  If file is a file-object,
then the filename is unchanged.  If file is a string, a ``.npy``
extension will be appended to the file name if it does not already
have one.
arr : array_like
Array data to be saved. Any object that is not an array will be
converted to an array with asanyarray(). When reloaded, the array
version of the object will be returned.

See Also

savez : Save several arrays into a ``.npz`` archive
savetxt, load

Notes
-
For a description of the ``.npy`` format, see `format`.

Examples

 from tempfile import TemporaryFile
 outfile = TemporaryFile()

 x = np.arange(10)
 np.save(outfile, x)

 outfile.seek(0) # Only needed here to simulate closing  reopening file
 np.load(outfile)
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])



I also see:

For a description of the ``.npy`` format, see `format`.

but no idea where to find 'format' -- it looks like it should be a
link in the Sphinx docs, but it's not.

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] datetime64 1970 issue

[Chris Barker - NOAA Federal]

On Tue, Apr 16, 2013 at 9:32 PM, Charles R Harris
charlesr.har...@gmail.com wrote:

 Dude, it was the 60's, no one remembers.

I can't say I remember much from then -- but probably because I was 4
years old, not because of too much partying

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] datetime64 1970 issue

[Chris Barker - NOAA Federal]

On Tue, Apr 16, 2013 at 8:23 PM, Zachary Ploskey zplos...@gmail.com wrote:
 The problem does not appear to exist on Linux with numpy version 1.6.2.

datetime64 was re-vampded a fair bit between 1.6 and 1.7

something is up here for sure with 1.7

We can be more dramatic about it:

In [5]: np.datetime64('1970-01-01T00') - np.datetime64('1969-12-31T23:59 ')
Out[5]: numpy.timedelta64(481,'m')

In [6]: np.datetime64('1970-01-01T00:00') - np.datetime64('1969-12-31T23:59 ')
Out[6]: numpy.timedelta64(481,'m')

1970-01-01T00 is the epoch, of course, so no surprise that if there is
something wrong, it would be there.

Interesting that no one noticed this before -- I guess I haven't
happened to use pre-1970 dates -- and or didn't check for accuracy
down to a few hours...

I'd say we need some more unit-tests!

We've been discussion some changes to datetime64 anyway, so a good
time to open it up.

-CHris


 In [1]: import numpy as np

 In [2]: np.datetime64('1970-01-01') - np.datetime64('1969-12-31')
 Out[2]: 1 day, 0:00:00

 In [3]: np.datetime64('1970-01-01 00') - np.datetime64('1969-12-31 00')
 Out[3]: 1 day, 0:00:00

 In [4]: np.__version__
 Out[4]: '1.6.2'

 Zach


 On Tue, Apr 16, 2013 at 4:45 PM, Ondřej Čertík ondrej.cer...@gmail.com
 wrote:

 On Tue, Apr 16, 2013 at 4:55 PM, Bob Nnamtrop bob.nnamt...@gmail.com
 wrote:
  I am curious if others have noticed an issue with datetime64 at the
  beginning of 1970. First:
 
  In [144]: (np.datetime64('1970-01-01') - np.datetime64('1969-12-31'))
  Out[144]: numpy.timedelta64(1,'D')
 
  OK this look fine, they are one day apart. But look at this:
 
  In [145]: (np.datetime64('1970-01-01 00') - np.datetime64('1969-12-31
  00'))
  Out[145]: numpy.timedelta64(31,'h')
 
  Hmmm, seems like there are 7 extra hours? Am I missing something? I
  don't
  see this at any other year. This discontinuity makes it hard to use the
  datetime64 object without special adjustment in ones code. I assume this
  a
  bug?

 Indeed, this looks like a bug, I can reproduce it on linux as well:

 In [1]: import numpy as np

 In [2]: np.datetime64('1970-01-01') - np.datetime64('1969-12-31')
 Out[2]: numpy.timedelta64(1,'D')

 In [3]: np.datetime64('1970-01-01 00') - np.datetime64('1969-12-31 00')
 Out[3]: numpy.timedelta64(31,'h')

 In [4]: np.__version__
 Out[4]: '1.7.1'


 We need to look into the sources to see what is going on.

 Ondrej
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion



 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] datetime64 1970 issue

[Chris Barker - NOAA Federal]

On Wed, Apr 17, 2013 at 9:04 AM, Chris Barker - NOAA Federal
chris.bar...@noaa.gov wrote:
 On Tue, Apr 16, 2013 at 8:23 PM, Zachary Ploskey zplos...@gmail.com wrote:
 I'd say we need some more unit-tests!

speaking of which, where are the tests? I just did a quick poke at
github, and found:

https://github.com/numpy/numpy/tree/master/numpy/testing
and
https://github.com/numpy/numpy/tree/master/numpy/test

but there's very little in there.

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] datetime64 1970 issue

[Chris Barker - NOAA Federal]

On Tue, Apr 16, 2013 at 3:55 PM, Bob Nnamtrop bob.nnamt...@gmail.com wrote:

 pss It would be most handy if datetime64 had a constructor of the form
 np.datetime64(year,month,day,hour,min,sec) where these inputs were numpy
 arrays and the output would have the same shape as the input arrays (but be
 of type datetime64). The hour,min,sec would be optional. Scalar inputs would
 be broadcast to the size of the array inputs, etc. Maybe this is a topic for
 another post.

Indeed -- there are a couple other threads about DateTime64 right now,
and we need to get a new NEP draft going (I've started discussion, but
haven't had a chance to put it in a NEP yet...)

It would be good to add this.

How/Where do I start a NEP, anyway? It  looks like they go here:

https://github.com/numpy/numpy/tree/master/doc/neps

But do I get it in pretty good shape first? Do I ask someone to commit
it for me?

-Chris



 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] datetime64 1970 issue

[Chris Barker - NOAA Federal]

On Wed, Apr 17, 2013 at 1:09 PM, Bob Nnamtrop bob.nnamt...@gmail.com wrote:
 It would seem that before 1970 the dates do not include the time zone
 adjustment while after 1970 they do. This is the source of the extra 7
 hours.

 In [21]: np.datetime64('1970-01-01 00')
 Out[21]: numpy.datetime64('1970-01-01T00:00-0700','h')

 In [22]: np.datetime64('1969-12-31 00')
 Out[22]: numpy.datetime64('1969-12-31T00:00Z','h')

wow! that is so wrong, and confusing -- I thought I had an idea what
was going on here:

datetime64 currently does a timezone adjustment at two places:

1) when constructing a datetime64 from an ISO string
2) when constructing an ISO string from a datetime64

This:
In [110]: np.datetime64('1969-12-31 00').view(np.int64)
Out[110]: -24

In [111]: np.datetime64('1970-01-01 00').view(np.int64)
Out[111]: 8

indicates that it is doing the input transition differently, as the
underlying value is wrong for one.
(another weird note -- I;m in pacific time, which is -7 now, with
DSTso why the 8?)

That explains the timedelta error.

But the output is odd, too:

In [117]: np.datetime64(datetime.datetime(1969, 12, 31, 0))
Out[117]: numpy.datetime64('1969-12-31T00:00:00.00Z')

In [118]: np.datetime64(datetime.datetime(1970, 1, 1, 0))
Out[118]: numpy.datetime64('1969-12-31T16:00:00.00-0800')

(when converting datetime.datetime objects, no timezone adjustment is applied)

I suspect that it's trying to use the system time functions (which wil
apply the locale), but that they don't work before 1970...at least on
*nix machines.

ANyone tested this on Windows?

We REALLY need to fix this!

-Chris









 I saw the other thread about the time zone issues and I think getting rid of
 timezones (perhaps unless they are explicitly requested) is the right thing
 to do.

 Bob


 On Tue, Apr 16, 2013 at 4:55 PM, Bob Nnamtrop bob.nnamt...@gmail.com
 wrote:

 I am curious if others have noticed an issue with datetime64 at the
 beginning of 1970. First:

 In [144]: (np.datetime64('1970-01-01') - np.datetime64('1969-12-31'))
 Out[144]: numpy.timedelta64(1,'D')

 OK this look fine, they are one day apart. But look at this:

 In [145]: (np.datetime64('1970-01-01 00') - np.datetime64('1969-12-31
 00'))
 Out[145]: numpy.timedelta64(31,'h')

 Hmmm, seems like there are 7 extra hours? Am I missing something? I don't
 see this at any other year. This discontinuity makes it hard to use the
 datetime64 object without special adjustment in ones code. I assume this a
 bug?

 Thanks,
 Bob

 ps I'm using the most recent anaconda release on mac os x 10.6.8 which
 includes numpy 1.7.0.

 pss It would be most handy if datetime64 had a constructor of the form
 np.datetime64(year,month,day,hour,min,sec) where these inputs were numpy
 arrays and the output would have the same shape as the input arrays (but be
 of type datetime64). The hour,min,sec would be optional. Scalar inputs would
 be broadcast to the size of the array inputs, etc. Maybe this is a topic for
 another post.



 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] numpy scalars and savez -- bug?

[Chris Barker - NOAA Federal]

Folks,

I've discovered somethign intertesting (bug?) with numpy scalars ans
savz. If I save a numpy scalar, then reload it, ot comes back as
rank-0 array -- similar, but not the same thing:

In [144]: single_value, type(single_value)
Out[144]: (2.0, numpy.float32)

In [145]: np.savez('test.npz', single_value=single_value)

In [146]: single_value2 = np.load('test.npz')['single_value']

In [147]: single_value, type(single_value)
Out[147]: (2.0, numpy.float32)

In [148]: single_value2, type(single_value2)
Out[148]: (array(2.0, dtype=float32), numpy.ndarray)

straight np.save has the same issue (which makes sense, I'm sure savez
uses the save code under the hood):

In [149]: single_value, type(single_value)
Out[149]: (2.0, numpy.float32)

In [150]: np.save('test.npy', single_value)

In [151]: single_value2 = np.load('test.npy')

In [152]: single_value2, type(single_value2)
Out[152]: (array(2.0, dtype=float32), numpy.ndarray)


This has been annoying, particular as rank-zero scalars are kind of a pain.

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Time Zones and datetime64

[Chris Barker - NOAA Federal]

On Fri, Apr 12, 2013 at 1:36 PM, Anthony Scopatz scop...@gmail.com wrote:
 Option (2), what datetime does, is the wrong model.  This is more
 complicated in both the implementation and API, and leads to lots of broken
 code, weird errors, and no clear right way of doing thing.

could you elaborate a bit more? I've never tried to use timezone
support with datetime, so I have no idea what goes wrong -- but it
looks reasonable to me. though it really punts on the hard stuff, so
maybe no point.

-Chris






 Be Well
 Anthony



 On Fri, Apr 12, 2013 at 2:57 PM, Chris Barker - NOAA Federal
 chris.bar...@noaa.gov wrote:

 On Fri, Apr 12, 2013 at 9:52 AM, Riccardo De Maria
 riccardodema...@gmail.com wrote:
  Not related to leap seconds and physically accurate time deltas, I have
  just
  noticed that SQLite has a nice API:
 
  http://www.sqlite.org/lang_datefunc.html
 
  that one can be inspired from. The source contains a date.c which looks
  reasonably clear.

 well, I don't see any timezone support in there at all. It appears the
 use UTC, though Im not entierly sure from the docs what now() would
 return.

 So I think it's pretty much like my use UTC proposal.


 -Chris




 --

 Christopher Barker, Ph.D.
 Oceanographer

 Emergency Response Division
 NOAA/NOS/ORR(206) 526-6959   voice
 7600 Sand Point Way NE   (206) 526-6329   fax
 Seattle, WA  98115   (206) 526-6317   main reception

 chris.bar...@noaa.gov
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion



 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Time Zones and datetime64

[Chris Barker - NOAA Federal]

On Fri, Apr 12, 2013 at 9:52 AM, Riccardo De Maria
riccardodema...@gmail.com wrote:
 Not related to leap seconds and physically accurate time deltas, I have just
 noticed that SQLite has a nice API:

 http://www.sqlite.org/lang_datefunc.html

 that one can be inspired from. The source contains a date.c which looks
 reasonably clear.

well, I don't see any timezone support in there at all. It appears the
use UTC, though Im not entierly sure from the docs what now() would
return.

So I think it's pretty much like my use UTC proposal.


-Chris




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion