Re: [Numpy-discussion] migration of all scipy.org mailing lists

[Benjamin Root]

nabble is the other site that probably should be notified of the mailing
list move.

On Wed, Mar 22, 2017 at 10:11 AM, Nathan Goldbaum 
wrote:

> Are you sure about astropy? They recently moved to google groups.
>
>
> On Wednesday, March 22, 2017, Ralf Gommers  wrote:
>
>> Hi all,
>>
>> The server for the scipy.org mailing list is in very bad shape, so we
>> (led by Didrik Pinte) are planning to complete the migration of active
>> mailing lists to the python.org infrastructure and to decommission the
>> lists than seem dormant/obsolete.
>>
>> The scipy-user mailing list was already moved to python.org a month or
>> two ago, and that migration went smoothly.
>>
>> These are the lists we plan to migrate:
>>
>> astropy
>> ipython-dev
>> ipython-user
>> numpy-discussion
>> numpy-svn
>> scipy-dev
>> scipy-organizers
>> scipy-svn
>>
>> And these we plan to retire:
>>
>> Announce
>> APUG
>> Ipython-tickets
>> Mailman
>> numpy-refactor
>> numpy-refactor-git
>> numpy-tickets
>> Pyxg
>> scipy-tickets
>> NiPy-devel
>>
>> There will be a short period (<24 hours) where messages to the list may
>> be refused, with an informative message as to why. The mailing list address
>> will change from listn...@scipy.org to listn...@python.org
>>
>> This will happen asap, likely within a day or two. So two requests:
>> 1.  If you see any issue with this plan, please reply and keep Didrik and
>> myself on Cc (we are not subscribed to all lists).
>> 2. If you see this message on a numpy/scipy list, but not on another list
>> (could be due to a moderation queue) then please forward this message again
>> to that other list.
>>
>> Thanks,
>> Ralf
>>
>>
>>
>>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Move scipy.org docs to Github?

[Benjamin Root]

How is it that scipy.org's bandwidth usage is that much greater than
matplotlib.org's? We are quite image-heavy, but we haven't hit any
bandwidth limits that I am aware of.

Ben Root

On Wed, Mar 15, 2017 at 9:16 AM, Bryan Van de ven 
wrote:

> NumPy is a NumFocus fiscally sponsored project, perhaps they can help with
> the costs of different/better hosting.
>
> Bryan
>
>
> > On Mar 15, 2017, at 07:24, Nathaniel Smith  wrote:
> >
> > On Mar 15, 2017 02:47, "Ralf Gommers"  wrote:
> >
> >
> > On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett 
> wrote:
> > Hi,
> >
> > The scipy.org site is down at the moment, and has been for more than 36
> hours:
> >
> > https://github.com/numpy/numpy/issues/8779#issue-213781439
> >
> > This has happened before:
> >
> > https://github.com/scipy/scipy.org/issues/187#issue-186426408
> >
> > I think it was down for about 24 hours that time.
> >
> > From the number of people opening issues or commenting on the
> > scipy.org website this time, it seems to be causing quite a bit of
> > disruption.
> >
> > It seems to me that we would have a much better chances of avoiding
> > significant down-time, if we switched to hosting the docs on github
> > pages.
> >
> > What do y'all think?
> >
> > Once the site is back up we should look at migrating to a better
> (hosted) infrastructure. I suspect that Github Pages won't work, we'll
> exceed or be close to exceeding both the 1 GB site size limit and the 100
> GB/month bandwidth limit [1].
> >
> > Rough bandwidth estimate (using page size from http://scipy.github.io/
> devdocs/ and Alexa stats): 2 million visits per month, 2.5 page views per
> visit, 5 kb/page = 25 GB/month (html). Add to that pdf docs, which are ~20
> MB in size: if only a small fraction of visitors download those, we'll be
> at >100 GB.
> >
> > No matter where we go, we can likely reduce the endpoint bandwidth
> requirements substantially by putting something like cloudflare's free tier
> in front. That doesn't help for the actual disk size though of course...
> >
> > -n
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy Overhead

[Benjamin Root]

You are going to need to provide much more context than that. Overhead
compared to what? And where (io, cpu, etc.)? What are the size of your
arrays, and what sort of operations are you doing? Finally, how much
overhead are you seeing?

There can be all sorts of reasons for overhead, and some can easily be
mitigated, and others not so much.

Cheers!
Ben Root


On Tue, Feb 28, 2017 at 4:47 PM, Sebastian K  wrote:

> Hello everyone,
>
> I'm interested in the numpy project and tried a lot with the numpy array.
> I'm wondering what is actually done that there is so much overhead when I
> call a function in Numpy. What is the reason?
> Thanks in advance.
>
> Regards
>
> Sebastian Kaster
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] automatically avoiding temporary arrays

[Benjamin Root]

Ah, that wheel would be a huge help. Most of the packages I have as
dependencies for this project were compiled against v1.10, so I am hoping
that there won't be too big of a problem. Are the mostlinux wheels still
compatible with CentOS5, or has the base image been bumped to CentOS6?

Cheers!
Ben Root

On Mon, Feb 27, 2017 at 2:31 PM, Matthew Brett <matthew.br...@gmail.com>
wrote:

> Hi,
>
> On Mon, Feb 27, 2017 at 11:27 AM, Charles R Harris
> <charlesr.har...@gmail.com> wrote:
> >
> >
> > On Mon, Feb 27, 2017 at 11:43 AM, Benjamin Root <ben.v.r...@gmail.com>
> > wrote:
> >>
> >> What's the timeline for the next release? I have the perfect usecase for
> >> this (Haversine calculation on large arrays that takes up ~33% of one
> of my
> >> processing scripts). However, to test it out, I have a huge dependency
> mess
> >> to wade through first, and there are no resources devoted to that
> project
> >> for at least a few weeks. I want to make sure I get feedback to y'all.
> >
> >
> > I'd like to branch 1.13.x at the end of March. The planned features that
> > still need to go in are the `__array_ufunc__` work and the `lapack_lite`
> > update. The first RC should not take much longer. I believe Matthew is
> > building wheels for testing on the fly but I don't know where you can
> find
> > them.
>
> Latest wheels at :
> https://7933911d6844c6c53a7d-47bd50c35cd79bd838daf386af554a
> 83.ssl.cf2.rackcdn.com/
>
> PRE_URL=https://7933911d6844c6c53a7d-47bd50c35cd79bd838daf386af554a
> 83.ssl.cf2.rackcdn.com
> pip install -f $PRE_URL --pre numpy
>
> Cheers,
>
> Matthew
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Deprecating matrices.

[Benjamin Root]

That's not a bad idea. Matplotlib is currently considering something
similar for its mlab module. It has been there since the beginning, but it
is very outdated and very out-of-scope for matplotlib. However, there are
still lots of code out there that depends on it. So, we are looking to
split it off as its own package. The details still need to be worked out
(should we initially depend on the package and simply alias its import with
a DeprecationWarning, or should we go cold turkey and have a good message
explaining the change).

Ben Root


On Tue, Jan 3, 2017 at 2:31 PM, Todd  wrote:

> On Mon, Jan 2, 2017 at 8:36 PM, Charles R Harris <
> charlesr.har...@gmail.com> wrote:
>
>> Hi All,
>>
>> Just throwing this click bait out for discussion. Now that the `@`
>> operator is available and things seem to be moving towards Python 3,
>> especially in the classroom, we should consider the real possibility of
>> deprecating the matrix type and later removing it. No doubt there are old
>> scripts that require them, but older versions of numpy are available for
>> those who need to run old scripts.
>>
>> Thoughts?
>>
>> Chuck
>>
>>
> What if the matrix class was split out into its own project, perhaps as a
> scikit.  That way those who really need it can still use it.  If there is
> sufficient desire for it, those who need it can maintain it.  If not, it
> will hopefully it will take long enough for it to bitrot that everyone has
> transitioned.
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Casting to np.byte before clearing values

[Benjamin Root]

Might be os-specific, too. Some virtual memory management systems might
special case the zeroing out of memory. Try doing the same thing with a
different value than zero.

On Dec 26, 2016 6:15 AM, "Nicolas P. Rougier" 
wrote:


Thanks for the explanation Sebastian, makes sense.

Nicolas


> On 26 Dec 2016, at 11:48, Sebastian Berg 
wrote:
>
> On Mo, 2016-12-26 at 10:34 +0100, Nicolas P. Rougier wrote:
>> Hi all,
>>
>>
>> I'm trying to understand why viewing an array as bytes before
>> clearing makes the whole operation faster.
>> I imagine there is some kind of special treatment for byte arrays but
>> I've no clue.
>>
>
> Sure, if its a 1-byte width type, the code will end up calling
> `memset`. If it is not, it will end up calling a loop with:
>
> while (N > 0) {
> *dst = output;
> *dst += 8;  /* or whatever element size/stride is */
> --N;
> }
>
> now why this gives such a difference, I don't really know, but I guess
> it is not too surprising and may depend on other things as well.
>
> - Sebastian
>
>
>>
>> # Native float
>> Z_float = np.ones(100, float)
>> Z_int   = np.ones(100, int)
>>
>> %timeit Z_float[...] = 0
>> 1000 loops, best of 3: 361 µs per loop
>>
>> %timeit Z_int[...] = 0
>> 1000 loops, best of 3: 366 µs per loop
>>
>> %timeit Z_float.view(np.byte)[...] = 0
>> 1000 loops, best of 3: 267 µs per loop
>>
>> %timeit Z_int.view(np.byte)[...] = 0
>> 1000 loops, best of 3: 266 µs per loop
>>
>>
>> Nicolas
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] How to use user input as equation directly

[Benjamin Root]

"only be used by engineers/scientists for research"

Famous last words. I know plenty of scientists who would love to "do
research" with an exposed eval(). Full disclosure, I personally added a
security hole into matplotlib thinking I covered all my bases in protecting
an eval() statement.

Ben Root

On Thu, Oct 27, 2016 at 4:21 PM, djxvillain  wrote:

> This will not be a public product and will only be used by other
> engineers/scientists for research.  I don't think security should be a huge
> issue, but I appreciate your input and concern for the quality of my code.
>
>
>
> --
> View this message in context: http://numpy-discussion.10968.
> n7.nabble.com/How-to-use-user-input-as-equation-directly-
> tp43665p43670.html
> Sent from the Numpy-discussion mailing list archive at Nabble.com.
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] How to use user input as equation directly

[Benjamin Root]

Perhaps the numexpr package might be safer? Not exactly meant for this
situation (meant for optimizations), but the evaluator is pretty darn safe.

Ben Root

On Thu, Oct 27, 2016 at 5:33 PM, John Ladasky  wrote:

> This isn't just a Numpy issue.  You are interested in Python's eval().
>
> Keep in mind that any programming language that blurs the line between
> code and data (many do not) has a potential security vulnerability.  What
> if your user doesn't type
>
> "x = 2*np.sin(2*np.pi*44100*t+np.pi/2)"
>
> but instead types this:
>
> "import os ; os.remove('/home')"
>
> I do NOT recommend that you eval() the second statement.
>
> You can try to write code which traps unwanted input before you eval()
> it.  It's apparently quite hard to stop everything bad from getting through.
>
>
> On Thu, Oct 27, 2016 at 12:58 PM, djxvillain  wrote:
>
>> Hello all,
>>
>> I am an electrical engineer and new to numpy.  I need the ability to take
>> in
>> user input, and use that input as a variable.  For example:
>>
>> t = input('enter t: ')
>> x = input('enter x: ')
>>
>> I need the user to be able to enter something like x =
>> 2*np.sin(2*np.pi*44100*t+np.pi/2) and it be the same as if they just
>> typed
>> it in the .py file.  There's no clean way to cast or evaluate it that I've
>> found.
>>
>> I could make a function to parse this string character by character, but I
>> figured this is probably a common problem and someone else has probably
>> figured it out and created an object for it.  I can't find a library that
>> does it though.
>>
>> If I can provide any more information please let me know.  Thank you in
>> advance for your help.
>>
>>
>>
>> --
>> View this message in context: http://numpy-discussion.10968.
>> n7.nabble.com/How-to-use-user-input-as-equation-directly-tp43665.html
>> Sent from the Numpy-discussion mailing list archive at Nabble.com.
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
>
> --
> *John J. Ladasky Jr., Ph.D.*
> *Research Scientist*
> *International Technological University*
> *2711 N. First St, San Jose, CA 95134 USA*
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] assert_allclose equal_nan default value.

[Benjamin Root]

+1. I was almost always setting it to True anyway.

On Thu, Oct 20, 2016 at 1:18 PM, Nathan Goldbaum 
wrote:

> Agreed, especially given the prevalence of using this function in
> downstream test suites:
>
> https://github.com/search?utf8=%E2%9C%93=numpy+assert_
> allclose=Code=searchresults
>
> On Thu, Oct 20, 2016 at 12:16 PM, Charles R Harris <
> charlesr.har...@gmail.com> wrote:
>
>> Hi All,
>>
>> Just a heads up that there is a PR changing the default value of
>> `equal_nan` to `True` in the `assert_allclose` test function. The
>> `equal_nan` argument was previously ineffective due to a bug that has
>> recently been fixed. The current default value of `False` is not backward
>> compatible and causes test failures in  scipy. See the extended argument at
>> https://github.com/numpy/numpy/pull/8184. I think this change is the
>> right thing to do but want to make sure everyone is aware of it.
>>
>> Chuck
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] how to name "contagious" keyword in np.ma.convolve

[Benjamin Root]

Why not "propagated"?

On Fri, Oct 14, 2016 at 1:08 PM, Sebastian Berg 
wrote:

> On Fr, 2016-10-14 at 13:00 -0400, Allan Haldane wrote:
> > Hi all,
> >
> > Eric Wieser has a PR which defines new functions np.ma.correlate and
> > np.ma.convolve:
> >
> > https://github.com/numpy/numpy/pull/7922
> >
> > We're deciding how to name the keyword arg which determines whether
> > masked elements are "propagated" in the convolution sums. Currently
> > we
> > are leaning towards calling it "contagious", with default of True:
> >
> > def convolve(a, v, mode='full', contagious=True):
> >
> > Any thoughts?
> >
>
> Sounds a bit overly odd to me to be honest. Just brain storming, you
> could think/name it the other way around maybe? Should the masked
> values be considered as zero/ignored?
>
> - Sebastian
>
>
> > Cheers,
> > Allan
> >
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] automatically avoiding temporary arrays

[Benjamin Root]

With regards to arguments about holding onto large arrays, I would like to
emphasize that my original suggestion mentioned weakref'ed numpy arrays.
Essentially, the idea is to claw back only the raw memory blocks during
that limbo period between discarding the numpy array python object and when
python garbage-collects it.

Ben Root

On Mon, Oct 3, 2016 at 2:43 PM, Julian Taylor  wrote:

> On 03.10.2016 20:23, Chris Barker wrote:
> >
> >
> > On Mon, Oct 3, 2016 at 3:16 AM, Julian Taylor
> > >
> > wrote:
> >
> > the problem with this approach is that we don't really want numpy
> > hogging on to hundreds of megabytes of memory by default so it would
> > need to be a user option.
> >
> >
> > indeed -- but one could set an LRU cache to be very small (few items,
> > not small memory), and then it get used within expressions, but not hold
> > on to much outside of expressions.
>
> numpy doesn't see the whole expression so we can't really do much.
> (technically we could in 3.5 by using pep 523, but that would be a
> larger undertaking)
>
> >
> > However, is the allocation the only (Or even biggest) source of the
> > performance hit?
> >
>
> on large arrays the allocation is insignificant. What does cost some
> time is faulting the memory into the process which implies writing zeros
> into the pages (a page at a time as it is being used).
> By storing memory blocks in numpy we would save this portion. This is
> really the job of the libc, but these are usually tuned for general
> purpose workloads and thus tend to give back memory back to the system
> much earlier than numerical workloads would like.
>
> Note that numpy already has a small memory block cache but its only used
> for very small arrays where the allocation cost itself is significant,
> it is limited to a couple megabytes at most.
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Vendorize tempita

[Benjamin Root]

This is the first I am hearing of tempita (looks to be a templating
language). How is it a dependency of numpy? Do I now need tempita in order
to use numpy, or is it a build-time-only dependency?

Ben

On Fri, Sep 30, 2016 at 11:13 AM, Stephan Hoyer  wrote:

> One way to do this is to move to vendorized dependencies into an submodule
> of numpy itself (e.g., sklearn.externals.joblib, though maybe even a little
> more indirection than that would be valuable to make it clear that it isn't
> part of NumPy public API). This would avoid further enlarging the set of
> namespaces we use.
>
> In any case, I'm perfectly OK with using something like npy_tempita
> internally, too, as long as we can be sure that we're using NumPy's
> vendorized version, not whatever version is installed locally. We're not
> planning to actually install "npy_tempita" when installing numpy (even for
> dev installs), right?
>
>
> On Fri, Sep 30, 2016 at 7:30 AM, Charles R Harris <
> charlesr.har...@gmail.com> wrote:
>
>> Hi All,
>>
>> There is a PR  to vendorize
>> tempita. This removes tempita as a dependency and simplifies some things.
>> Feedback on this step is welcome. One question is whether the package
>> should be renamed to something like `npy_tempita`, as otherwise installed
>> tempita, if any has priority.
>>
>> Thoughts?
>>
>> Chuck
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Which NumPy/Numpy/numpy spelling?

[Benjamin Root]

There was similar discussion almost two years ago with respect to
capitalization of matplotlib in prose. Most of the time, it was lower-case
in our documentation, but then the question was if it should be upper-case
at the beginning of the sentence... or should it always be upper-cased like
a proper noun. I don't think a clear consensus was reached, but I ended up
treating it as a proper noun in my book.

I am also pretty sure I used "NumPy" in most places, even for "NumPy
arrays", which still looks weird to me when I go back over my book.

Ben Root


On Mon, Aug 29, 2016 at 12:56 PM,  wrote:

> https://mail.scipy.org/pipermail/scipy-user/2010-June/025756.html
>
> "NumPy and SciPy to refer to the projects. numpy and scipy to refer to
> the packages, specifically. When in doubt, use the former."
>
> I thought there was also another discussion about capital letters, but
> I don't find it.
>
> Josef
>
>
> On Mon, Aug 29, 2016 at 9:22 AM, Bartosz Telenczuk 
> wrote:
> > Hi,
> >
> > I would not mind any choice as long as it's consistent.
> >
> > I agree that using all-lowercase spelling may avoid some common errors.
> However,
> > PEP8 requires all module/package names to be lower case [1].  If we
> force the
> > name of the library and the corresponding package to be the same, all
> Python
> > libraries would be named in lowercase. This would not be the best choice
> for
> > libraries, which have multi-component names (like NumPy = Numerical
> Python).
> >
> > Note also that both the Wikipedia page [2] and the official NumPy logo
> [3] use
> > "NumPy" spelling.
> >
> > Some other popular [4] libraries use similar dichotomies:
> >
> > - Django - import django
> > - Cython - import cython
> > - PyYAML - import yaml
> > - scikit-learn - import sklearn
> >
> > On the other hand all standard Python libraries are lower-case named.
> >
> > Cheers,
> >
> > Bartosz
> >
> > [1] https://www.python.org/dev/peps/pep-0008/#package-and-module-names
> > [2] https://en.wikipedia.org/wiki/NumPy
> > [3] http://www.numpy.org/_static/numpy_logo.png
> > [4] http://pypi-ranking.info/alltime
> >
> >> On 08/29/2016 07:43 AM, m...@telenczuk.pl wrote:
> >> > What is the official spelling of NumPy/Numpy/numpy?
> >>
> >> IMHO it should be written numpy, because ...
> >>
> >>  >>> import NumPy
> >> Traceback (most recent call last):
> >>File "", line 1, in 
> >> ImportError: No module named NumPy
> >>  >>> import Numpy
> >> Traceback (most recent call last):
> >>File "", line 1, in 
> >> ImportError: No module named Numpy
> >>  >>> import numpy
> >>  >>>
> >>
> >> Phil
> >> ___
> >> NumPy-Discussion mailing list
> >> NumPy-Discussion@scipy.org
> >> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NumPy in PyPy

[Benjamin Root]

Don't know if it is what you are looking for, but NumPy has a built-in
suite of benchmarks:
http://docs.scipy.org/doc/numpy/reference/generated/numpy.testing.Tester.bench.html

Also, some projects have taken to utilizing the "airspeed velocity" utility
to track benchmarking stats for their projects. I know astropy utilizes it.
So, maybe their benchmarks might be a good starting point since they
utilize numpy heavily?

Cheers!
Ben Root


On Fri, Aug 5, 2016 at 3:42 AM, Papa, Florin  wrote:

> Hi,
>
>
>
> This is Florin Papa from the Dynamic Scripting Languages Optimizations
> team in Intel Corporation.
>
>
>
> Our team is working on optimizing the PyPy interpreter and part of this
> work is to find and fix incompatibilities between NumPy and PyPy. Does
> anyone have knowledge of real life workloads that use NumPy and cannot be
> run using PyPy?
>
>
>
> We are also interested in creating a repository with relevant benchmarks
> for real world usage of NumPy, like GUPB for CPython, but we have not found
> such workloads for NumPy.
>
>
>
> Thank you,
>
> Florin
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Is there any official position on PEP484/mypy?

[Benjamin Root]

One thing that I have always wished for from a project like mypy is the
ability to annotate what the expected shape should be. Too often, I get a
piece of code from a coworker and it comes with no docstring explaining the
expected dimensions of the input arrays and what the output array is going
to be. What would be really awesome is the ability to do something like
annotate that "a" is MxN, and "b" is NxP, and that "c" is Px3. Even if the
linter can't really check to make sure that the shapes would be respected,
it would still be nice to have a common way of expressing the expected
shapes in this annotation format.

As for matplotlib, we would need to express much more complicated
annotations, because our API is so flexible. It would be useful to keep an
eye out to those needs as well.

Cheers!
Ben Root


On Fri, Jul 29, 2016 at 5:33 AM, Daniel Moisset 
wrote:

> Hi Sebastian, thanks for your reply
>
> I'm glad to hear that you see value in having type annotations. Just to
> clarify, my question was aimed at surveying if there was interest in
> accepting the work we're already doing if we contribute it and if it has
> value for the numpy project. I'm aware there's effort involved; some
> colleagues and me are already involved doing that at
> https://github.com/machinalis/mypy-data because it's valuable for
> ourselves, so the volunteers are already here. You of course are invited to
> comment on the existing code and try it :) (or joining the effort, goes
> without saying)
>
> Running the checker on the test suite is probably the best way to validate
> the annotations (the normal way would be checking the annotations against
> the code, but that doesn't work with C extensions like numpy). That's
> something we haven't been doing yet but it's an obvious next step now that
> some simple examples are working.
> WRT "I wonder if all or most of numpy can be easily put into it.", we've
> covered ndarray (and matrix soon) which are the core types, things built
> upon that shouldn't be too hard. We found some snags along the way [1] [2],
> but no of it is a showstopper and I'm quite sure we'll fix those in time.
> But of course, if someone wants to try it out it will be a better
> validation than my optimism to see if this makes sense :)
>
> Thanks again and I'd be happy to hear more opinions from other numpy devs!
>
> Best,
>D.
>
> [1] http://www.machinalis.com/blog/writing-type-stubs-for-numpy/
> [2] https://github.com/machinalis/mypy-data/issues
>
>
> On 29 Jul 2016 08:31, "Sebastian Berg"  wrote:
>
>> On Mi, 2016-07-27 at 20:07 +0100, Daniel Moisset wrote:
>> >
>> > Hi,
>> >
>> > I work at Machinalis were we use a lot of numpy (and the pydata stack
>> > in general). Recently we've also been getting involved with mypy,
>> > which is a tool to type check (not on runtime, think of it as a
>> > linter) annotated python code (the way of annotating python types has
>> > been recently standarized in PEP 484).
>> >
>> > As part of that involvement we've started creating type annotations
>> > for the Python libraries we use most, which include numpy. Mypy
>> > provides a way to specify types with annotations in separate files in
>> > case you don't have control over a library, so we have created an
>> > initial proof of concept at [1], and we are actively improving it.
>> > You can find some additional information about it and some problems
>> > we've found on the way at this blogpost [2].
>> >
>> > What I wanted to ask is if the people involved on the numpy project
>> > are aware of PEP484 and if you have some interest in starting using
>> > them. The main benefit is that annotations serve as clear (and
>> > automatically testable) documentation for users, and secondary
>> > benefits is that users discovers bugs more quickly and that some IDEs
>> > (like pycharm) are starting to use this information for smart editor
>> > features (autocompletion, online checking, refactoring tools);
>> > eventually tools like jupyter could take advantage of these
>> > annotations in the future. And the cost of writing and including
>> > these are relatively low.
>> >
>>
>> There is currently no plan to do it as far as I know, but with these
>> things it is often more of a problem that someone volunteers to
>> maintain it then to convince everyone that it is a good idea.
>> If there is enough interest we could talk about hosting it on the numpy
>> github group as a separate project to make it a bit more
>> visible/obvious that such a project exists.
>>
>> For inclusion in numpy, it seems to me that currently this would
>> probably be better of improved separately? In the long run, would it be
>> possible to do something like run all numpy tests and then check
>> whether the definitions cover (almost) everything, or test against the
>> documentation or so? Otherwise it might get tricky to keep things quite
>> up to date, at least until these type checks are very widely used. 

Re: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d

[Benjamin Root]

I don't see how one could define a spec that would take an arbitrary array
of indices at which to place new dimensions. By definition, you don't know
how many dimensions are going to be added. If you knew, then you wouldn't
be calling this function. I can only imagine simple rules such as 'left' or
'right' or maybe something akin to what at_least3d() implements.

On Wed, Jul 6, 2016 at 3:20 PM, Joseph Fox-Rabinovitz <
jfoxrabinov...@gmail.com> wrote:

> On Wed, Jul 6, 2016 at 2:57 PM, Eric Firing <efir...@hawaii.edu> wrote:
> > On 2016/07/06 8:25 AM, Benjamin Root wrote:
> >>
> >> I wouldn't have the keyword be "where", as that collides with the notion
> >> of "where" elsewhere in numpy.
> >
> >
> > Agreed.  Maybe "side"?
>
> I have tentatively changed it to "pos". The reason that I don't like
> "side" is that it implies only a subset of the possible ways that that
> the position of the new dimensions can be specified. The current
> implementation only puts things on one side or the other, but I have
> considered also allowing an array of indices at which to place new
> dimensions, and/or a dictionary keyed by the starting ndims. I do not
> think "side" would be appropriate for these extended cases, even if
> they are very unlikely to ever materialize.
>
> -Joe
>
> > (I find atleast_1d and atleast_2d to be very helpful for handling
> inputs, as
> > Ben noted; I'm skeptical as to the value of atleast_3d and atleast_nd.)
> >
> > Eric
> >
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d

[Benjamin Root]

I wouldn't have the keyword be "where", as that collides with the notion of
"where" elsewhere in numpy.

On Wed, Jul 6, 2016 at 2:21 PM, Joseph Fox-Rabinovitz <
jfoxrabinov...@gmail.com> wrote:

> I still think this function is useful. I have made a change so that it
> only accepts one array, as Marten suggested, making the API much
> cleaner than that of its siblings. The side on which the new
> dimensions will be added is configurable via the `where` parameter,
> which currently accepts 'before' and 'after', but can be changed to
> accept sequences or even dicts. The change also resulted in finding a
> bug in the masked array versions of the atleast functions, which the
> PR now fixes and adds regression tests for. If the devs do decide to
> discard this PR, I will of course submit the bug fix separately.
>
> -Joe
>
> On Wed, Jul 6, 2016 at 1:43 PM, Stephan Hoyer  wrote:
> > On Tue, Jul 5, 2016 at 10:06 PM, Nathaniel Smith  wrote:
> >>
> >> I don't know how typical I am in this. But it does make me wonder if the
> >> atleast_* functions act as an attractive nuisance, where new users take
> >> their presence as an implicit recommendation that they are actually a
> useful
> >> thing to reach for, even though they... aren't that. And maybe we
> should be
> >> recommending folk move away from them rather than trying to extend them
> >> further?
> >
> > Agreed. I would avoid adding atleast_nd. We could discourage using
> > atleast_3d (certainly the behavior is indeed surprising), but I'm not
> sure
> > it's worth the trouble.
> >
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d

[Benjamin Root]

While atleast_1d/2d/3d predates my involvement in numpy, I am probably
partly to blame for popularizing them as I helped to fix them up a fair
amount. I wouldn't call its use "guessing". Rather, I would treat them as
useful input sanitizers. If your function is going to be doing 2d indexing
on an input, then it is very convenient to have atleast_2d() at the top of
your function, not only to sanitize the input, but to make it clear that
your code expects at least two dimensions.

One place where it is used is in np.loadtxt(..., ndmin=N) to protect
against the situation of a single row of data becoming a 1-D array rather
than a 2-D array (or an empty text file returning something completely
useless).

I have previously pointed out the oddity with atleast_3d(). I can't
remember the explanation I got though. Maybe someone can find the old
thread that has the explanation, if any?

I think the keyword argument approach for controlling the behavior might be
a good approach, provided that a suitable design could be devised. 1 & 2
dimensions is fairly trivial to control, but 3+ dimensions has too many
degrees of freedom for me to consider.

Cheers!
Ben Root


On Wed, Jul 6, 2016 at 9:12 AM, Joseph Fox-Rabinovitz <
jfoxrabinov...@gmail.com> wrote:

> I can add a keyword-only argument that lets you put the new dims
> before or after the existing ones. I am not sure how to specify
> arbitrary patterns for the new dimensions, but that should take care
> of most use cases.
>
> The use case that motivated this function in the first place is that I
> am doing some processing on 4D arrays and I need to reduce them but
> return a result with the original dimensionality (but not shape).
> atleast_nd seemed like a better solution than atleast_4d.
>
> -Joe
>
>
> On Wed, Jul 6, 2016 at 3:41 AM,   wrote:
> >
> >
> > On Wed, Jul 6, 2016 at 3:29 AM,  wrote:
> >>
> >>
> >>
> >> On Wed, Jul 6, 2016 at 2:21 AM, Ralf Gommers 
> >> wrote:
> >>>
> >>>
> >>>
> >>> On Wed, Jul 6, 2016 at 7:06 AM, Nathaniel Smith  wrote:
> >>>
>  On Jul 5, 2016 9:09 PM, "Joseph Fox-Rabinovitz"
>   wrote:
>  >
>  > Hi,
>  >
>  > I have generalized np.atleast_1d, np.atleast_2d, np.atleast_3d with
> a
>  > function np.atleast_nd in PR#7804
>  > (https://github.com/numpy/numpy/pull/7804).
>  >
>  > As a result of this PR, I have a couple of questions about
>  > `np.atleast_3d`. `np.atleast_3d` appears to do something weird with
>  > the dimensions: If the input is 1D, it prepends and appends a size-1
>  > dimension. If the input is 2D, it appends a size-1 dimension. This
> is
>  > inconsistent with `np.atleast_2d`, which always prepends (as does
>  > `np.atleast_nd`).
>  >
>  >   - Is there any reason for this behavior?
>  >   - Can it be cleaned up (e.g., by reimplementing `np.atleast_3d` in
>  > terms of `np.atleast_nd`, which is actually much simpler)? This
> would
>  > be a slight API change since the output would not be exactly the
> same.
> 
>  Changing atleast_3d seems likely to break a bunch of stuff...
> 
>  Beyond that, I find it hard to have an opinion about the best design
> for
>  these functions, because I don't think I've ever encountered a
> situation
>  where they were actually what I wanted. I'm not a big fan of coercing
>  dimensions in the first place, for the usual "refuse to guess"
> reasons. And
>  then generally if I do want to coerce an array to another dimension,
> then I
>  have some opinion about where the new dimensions should go, and/or I
> have
>  some opinion about the minimum acceptable starting dimension, and/or
> I have
>  a maximum dimension in mind. (E.g. "coerce 1d inputs into a column
> matrix;
>  0d or 3d inputs are an error" -- atleast_2d is zero-for-three on that
>  requirements list.)
> 
>  I don't know how typical I am in this. But it does make me wonder if
> the
>  atleast_* functions act as an attractive nuisance, where new users
> take
>  their presence as an implicit recommendation that they are actually a
> useful
>  thing to reach for, even though they... aren't that. And maybe we
> should be
>  recommending folk move away from them rather than trying to extend
> them
>  further?
> 
>  Or maybe they're totally useful and I'm just missing it. What's your
> use
>  case that motivates atleast_nd?
> >>>
> >>> I think you're just missing it:) atleast_1d/2d are used quite a bit in
> >>> Scipy and Statsmodels (those are the only ones I checked), and in the
> large
> >>> majority of cases it's the best thing to use there. There's a bunch of
> >>> atleast_2d calls with a transpose appended because the input needs to
> be
> >>> treated as columns instead of rows, but that's still efficient and
> readable
> >>> enough.
> >>
> >>
> >>
> >> 

Re: [Numpy-discussion] Axis argument to np.unique

[Benjamin Root]

That seems like the only reasonable behavior, but I will admit that my
initial desire is essentially a vectorized "unique" such that it returns
the unique values of the stated axis. But that isn't possible because there
can be different number of unique values in the given axis, resulting in a
ragged array, which numpy does not support.

Ben Root

On Wed, Jun 15, 2016 at 9:16 AM, Martino Sorbaro 
wrote:

> Hi all,
> I've opened a new pull request
> (https://github.com/numpy/numpy/pull/7742) trying to revive a previous
> one that was left abandoned (#3584, by another contributor), regarding
> the possibility of adding an 'axis=' argument to numpy.unique.
>
> There had been a debate
> (
> http://numpy-discussion.10968.n7.nabble.com/Adding-an-axis-argument-to-numpy-unique-td34841.html
> )
> about what the axis argument should mean. The current behaviour in the
> code I propose (written by the previous contributor) looks for unique
> rows if "axis=0" and unique columns if "axis=1", in other words:
>
> [In]  a = array([[0, 0, 0],
>  [1, 1, 1],
>  [0, 0, 0]])
>
> [In]  unique(a, axis=0)
> [Out] array([[0, 0, 0],
>  [1, 1, 1]])
>
>
> So, I would just like to ask whether a conclusion can be reached about
> that discussion.
> Thanks!
> Martino
>
> --
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] A numpy based Entity-Component-System

[Benjamin Root]

As a bit of a real-life example where things can go wrong with naming. The
"pylab" name was accidentally hijacked a couple years ago on pypi, and
caused several bug reports to be filed against matplotlib for failing
scripts.  Some people thought that one should do "pip install pylab" to do
"from pylab import *" -- crazy, I know, right? ;-)

That was at least two years ago, and we are just now getting the person who
uploaded that errant package to fix the problem. We also have not yet been
able to find out the owner of the long defunct matplotlib twitter handle
(we started the process to reclaim it, though). There are a couple other
situations that have caused confusion among users, but these have been
mostly an issue of trademarks (which I believe NumFOCUS holds for
matplotlib?).

What you really have to watch out for is when someone creates a package
that uses some open-sourced library, and then claims that it is "supported"
by it. This can cause an undue burden on the original maintainers in
fielding bug reports and other issues. It also creates a false sense of
association/coordination -- which gets me to the issue at hand. I highly
suggest coming up with a unique name. It benefits you as it becomes
something distinct from numpy (so search results are more relevant), and it
benefits the numpy developers because support requests go exactly where
they are supposed to go.


On Fri, May 20, 2016 at 9:40 PM, Nathaniel Smith  wrote:

> Like naming your child "Human-legsarms"?



You owe me a new monitor!

Cheers!
Ben Root
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy arrays shareable among related processes (PR #7533)

[Benjamin Root]

Oftentimes, if one needs to share numpy arrays for multiprocessing, I would
imagine that it is because the array is huge, right? So, the pickling
approach would copy that array for each process, which defeats the purpose,
right?

Ben Root

On Wed, May 11, 2016 at 2:01 PM, Allan Haldane 
wrote:

> On 05/11/2016 04:29 AM, Sturla Molden wrote:
> > 4. The reason IPC appears expensive with NumPy is because multiprocessing
> > pickles the arrays. It is pickle that is slow, not the IPC. Some would
> say
> > that the pickle overhead is an integral part of the IPC ovearhead, but i
> > will argue that it is not. The slowness of pickle is a separate problem
> > alltogether.
>
> That's interesting. I've also used multiprocessing with numpy and didn't
> realize that. Is this true in python3 too?
>
> In python2 it appears that multiprocessing uses pickle protocol 0 which
> must cause a big slowdown (a factor of 100) relative to protocol 2, and
> uses pickle instead of cPickle.
>
> a = np.arange(40*40)
>
> %timeit pickle.dumps(a)
> 1000 loops, best of 3: 1.63 ms per loop
>
> %timeit cPickle.dumps(a)
> 1000 loops, best of 3: 1.56 ms per loop
>
> %timeit cPickle.dumps(a, protocol=2)
> 10 loops, best of 3: 18.9 µs per loop
>
> Python 3 uses protocol 3 by default:
>
> %timeit pickle.dumps(a)
> 1 loops, best of 3: 20 µs per loop
>
>
> > 5. Share memory does not improve on the pickle overhead because also
> NumPy
> > arrays with shared memory must be pickled. Multiprocessing can bypass
> > pickling the RawArray object, but the rest of the NumPy array is pickled.
> > Using shared memory arrays have no speed advantage over normal NumPy
> arrays
> > when we use multiprocessing.
> >
> > 6. It is much easier to write concurrent code that uses queues for
> message
> > passing than anything else. That is why using a Queue object has been the
> > popular Pythonic approach to both multitreading and multiprocessing. I
> > would like this to continue.
> >
> > I am therefore focusing my effort on the multiprocessing.Queue object. If
> > you understand the six points I listed you will see where this is going:
> > What we really need is a specialized queue that has knowledge about NumPy
> > arrays and can bypass pickle. I am therefore focusing my efforts on
> > creating a NumPy aware queue object.
> >
> > We are not doing the users a favor by encouraging the use of shared
> memory
> > arrays. They help with nothing.
> >
> >
> > Sturla Molden
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Help with bit arrays

[Benjamin Root]

What behavior is unexpected?  For the (256, 256) images, matplotlib applies
its default colormap to the grayscale (v1.5 and previous, that is jet,
+v2.0, that will be viridis). The numpy array as loaded from PIL will never
carry any additional information that came from the TIFF.

As for PIL, it will return an RGB[A] array if there is colormap data in the
TIFF. If there is no colormap specified in the TIFF, it'll give you a
simple 2D array. Now, maybe you'd like it to always return an RGB[A] array,
but without a colormap in the TIFF, it makes sense to return the data
as-is. This makes sense for people treating the TIFF as a data format
rather than a visualization data format.

Ben Root


On Fri, Apr 29, 2016 at 12:47 PM, Henrique Almeida <hdante.l...@gmail.com>
wrote:

>  I think in any case, the result is unexpected, PIL is loading garbage
> from memory when loading black and white images because it sends the
> wrong buffer size, and matplotlib correctly loads the black and white
> image, but stores it in a 3D array.
>
> 2016-04-29 13:43 GMT-03:00 Henrique Almeida <hdante.l...@gmail.com>:
> >  For 1 bit images, the resulting array has shape (256, 256, 4). For
> > grayscale images, the shape is (256, 256). So the image seems to have
> > been loaded as a color image.
> >
> > 2016-04-29 13:38 GMT-03:00 Benjamin Root <ben.v.r...@gmail.com>:
> >> What kind of array is "img"? What is its dtype and shape?
> >>
> >> plt.imshow() will use the default colormap for matplotlib if the given
> array
> >> is just 2D. But if it is 3D (a 2D array of RGB[A] channels), then it
> will
> >> forego the colormap and utilize that for the colors. It knows nothing
> of the
> >> colormap contained in the TIFF.
> >>
> >> Ben Root
> >>
> >>
> >> On Fri, Apr 29, 2016 at 12:31 PM, Henrique Almeida <
> hdante.l...@gmail.com>
> >> wrote:
> >>>
> >>>  Paul, yes, imread() worked for reading the black and white TIFF. The
> >>> situation improved, but now, there seems to be some problem with the
> >>> color map. Example code:
> >>>
> >>> #!/usr/bin/env python3
> >>> import numpy
> >>> from matplotlib import pyplot, cm
> >>>
> >>> img = pyplot.imread('oi-00.tiff')
> >>> pyplot.imshow(img)
> >>> pyplot.colorbar()
> >>> pyplot.show()
> >>>
> >>>  The code can open both 1-bit and 8-bit images, but only with 8 bits
> >>> the image is shown with the colormap colors. The 1 bit image is shown
> >>> as black and white.
> >>>
> >>>  The questions:
> >>>  1) Should Image.open() behave like pyplot.imread() ? Is this a bug in
> PIL
> >>> ?
> >>>  2) Why isn't the colormap working with black and white images ?
> >>>
> >>> 2016-04-29 13:06 GMT-03:00 Paul Hobson <pmhob...@gmail.com>:
> >>> > Does using pyplot.imgread work?
> >>> >
> >>> > On Fri, Apr 29, 2016 at 8:27 AM, Henrique Almeida
> >>> > <hdante.l...@gmail.com>
> >>> > wrote:
> >>> >>
> >>> >>  Any help with this problem ?
> >>> >>
> >>> >> 2016-04-27 11:35 GMT-03:00 Henrique Almeida <hdante.l...@gmail.com
> >:
> >>> >> >  Hello, what's the current status on numpy for loading bit-arrays
> ?
> >>> >> >
> >>> >> > I'm currently unable to correctly load black and white (1-bit)
> TIFF
> >>> >> > images. Code example follows:
> >>> >> >
> >>> >> > from PIL import Image
> >>> >> > import numpy
> >>> >> > from matplotlib import pyplot
> >>> >> >
> >>> >> > img = Image.open('oi-00.tiff')
> >>> >> > a = numpy.array(img)
> >>> >> >
> >>> >> > ^ does not work for 1-bit TIFF images
> >>> >> >
> >>> >> > PIL source shows that it incorrectly uses typestr == '|b1'. I
> tried
> >>> >> > to
> >>> >> > change this to '|t1', but I get :
> >>> >> >
> >>> >> > TypeError: data type "|t1" not understood
> >>> >> >
> >>> >> > My goal is to make the above code to work for black and white TIFF
> >>> >> > images the same way it works for grayscale images. Any help ?
> >>> >> ___
> >>> >> NumPy-Discussion mailing list
> >>> >> NumPy-Discussion@scipy.org
> >>> >> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> >>> >
> >>> >
> >>> >
> >>> > ___
> >>> > NumPy-Discussion mailing list
> >>> > NumPy-Discussion@scipy.org
> >>> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> >>> >
> >>> ___
> >>> NumPy-Discussion mailing list
> >>> NumPy-Discussion@scipy.org
> >>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> >>
> >>
> >>
> >> ___
> >> NumPy-Discussion mailing list
> >> NumPy-Discussion@scipy.org
> >> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> >>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Help with bit arrays

[Benjamin Root]

What kind of array is "img"? What is its dtype and shape?

plt.imshow() will use the default colormap for matplotlib if the given
array is just 2D. But if it is 3D (a 2D array of RGB[A] channels), then it
will forego the colormap and utilize that for the colors. It knows nothing
of the colormap contained in the TIFF.

Ben Root


On Fri, Apr 29, 2016 at 12:31 PM, Henrique Almeida 
wrote:

>  Paul, yes, imread() worked for reading the black and white TIFF. The
> situation improved, but now, there seems to be some problem with the
> color map. Example code:
>
> #!/usr/bin/env python3
> import numpy
> from matplotlib import pyplot, cm
>
> img = pyplot.imread('oi-00.tiff')
> pyplot.imshow(img)
> pyplot.colorbar()
> pyplot.show()
>
>  The code can open both 1-bit and 8-bit images, but only with 8 bits
> the image is shown with the colormap colors. The 1 bit image is shown
> as black and white.
>
>  The questions:
>  1) Should Image.open() behave like pyplot.imread() ? Is this a bug in PIL
> ?
>  2) Why isn't the colormap working with black and white images ?
>
> 2016-04-29 13:06 GMT-03:00 Paul Hobson :
> > Does using pyplot.imgread work?
> >
> > On Fri, Apr 29, 2016 at 8:27 AM, Henrique Almeida  >
> > wrote:
> >>
> >>  Any help with this problem ?
> >>
> >> 2016-04-27 11:35 GMT-03:00 Henrique Almeida :
> >> >  Hello, what's the current status on numpy for loading bit-arrays ?
> >> >
> >> > I'm currently unable to correctly load black and white (1-bit) TIFF
> >> > images. Code example follows:
> >> >
> >> > from PIL import Image
> >> > import numpy
> >> > from matplotlib import pyplot
> >> >
> >> > img = Image.open('oi-00.tiff')
> >> > a = numpy.array(img)
> >> >
> >> > ^ does not work for 1-bit TIFF images
> >> >
> >> > PIL source shows that it incorrectly uses typestr == '|b1'. I tried to
> >> > change this to '|t1', but I get :
> >> >
> >> > TypeError: data type "|t1" not understood
> >> >
> >> > My goal is to make the above code to work for black and white TIFF
> >> > images the same way it works for grayscale images. Any help ?
> >> ___
> >> NumPy-Discussion mailing list
> >> NumPy-Discussion@scipy.org
> >> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
> >
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Possible negative impact of future change to None comparison

[Benjamin Root]

Working on closing out some bug reports at work, and I ran into one about
comparisons to 'None' will result in elementwise object comparison in the
future. Now, I totally get the idea behind the change, and I am not here to
argue that decision. However, I have come across a situation where the
change might have an unexpected negative consequence.

Consider the following:

p = Pool(min(len(tiles), maxprocs))
res = p.map(_wrap_tilesum, izip(repeat(args.varname),
tiles,
repeat(request_start),
repeat(args.timeLen),
repeat(args.srcdir),
repeat(args.tarredInputs),
repeat(args.dataset)))
p.close()
p.join()

(tiles, tile_start_dates, tile_end_dates,
tile_lons, tile_lats) = zip(*res)

if None in tiles:
logging.critical("At least one tile was invalid. Aborting")
raise Exception("Invalid data retrieved!")

Essentailly, in the nominal case, "tiles" would be a list of numpy arrays.
However, my error handling is such that if one of my subprocesses errors
out, then it returns a None instead of a numpy array. So, all I am doing is
testing to see if any of the items in the "tiles" list is None. I have zero
desire to actually compare None with the elements in the arrays that
happens to be in the list.

Of course, I can rewrite this if statement as `any(tile is None for tile in
tiles)`, but that isn't my point. Is `if None in tiles:` an unreasonable
idiom?

Cheers!
Ben Root
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] linux wheels coming soon

[Benjamin Root]

Yeah! That's the bug I encountered! So, that would explain why this seems
to work fine now (I tried it out a bit on Friday on a CentOS6 system, but
didn't run the test suite).

Cheers!
Ben Root

On Sun, Apr 17, 2016 at 1:46 PM, Olivier Grisel 
wrote:

> Thanks for the clarification, I read your original report too quickly.
>
> I wonder why the travis maintainers built Python 2.7 with a
> non-standard unicode option.
>
> Edit (after googling): this is a known issue. The image with Python
> 2.7.11 will be fixed:
>
> https://github.com/travis-ci/travis-ci/issues/5107
>
> --
> Olivier
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] linux wheels coming soon

[Benjamin Root]

I am honestly surprised that these worked (I haven't gotten around to
testing for myself). I could have sworn there was a difference in how
Continuum compiled python such that any binaries built against a stock
python would not work in a conda environment. I ran into issues a couple
years ago where a modwsgi package provided through yum wouldn't work with
miniconda because of link-time differences.

I cannot for the life of me remember the error message, though.

Ben Root


On Thu, Apr 14, 2016 at 3:59 PM, Paul Hobson <pmhob...@gmail.com> wrote:

>
>
> On Thu, Apr 14, 2016 at 12:07 PM, Nathaniel Smith <n...@pobox.com> wrote:
>
>> On Apr 14, 2016 11:11 AM, "Benjamin Root" <ben.v.r...@gmail.com> wrote:
>> >
>> > Are we going to have to have documentation somewhere making it clear
>> that the numpy wheel shouldn't be used in a conda environment? Not that I
>> would expect this issue to come up all that often, but I could imagine a
>> scenario where a non-scientist is simply using a base conda distribution
>> because that is what IT put on their system. Then they do "pip install
>> ipython" that indirectly brings in numpy (through the matplotlib
>> dependency), and end up with an incompatible numpy because they would have
>> been linked against different pythons?
>> >
>> > Or is this not an issue?
>>
>> There are always issues when you have two different package managers
>> maintaining separate and out-of-sync metadata about what they think is
>> installed, but that's true for any mixed use of conda and pip.
>>
>> But:
>> - pip won't install a numpy that is incompatible with your python, unless
>> Anaconda is actively breaking cpython's standard abi (they aren't) or
>> there's a bug in pip (possible, but no reports yet).
>> - conda packages for python packages like numpy do generally include the
>> .egg-info / .dist-info directories that pip uses to store its installation
>> metadata, so pip can "see" packages installed by conda (but not
>> vice-versa). So "pip install matplotlib" won't drag in a pypi numpy if
>> there's already a conda numpy installed.
>>
> Minor clarification:. I believe conda can see pip-installed packages.
>
> If I execute "conda list" in an environment, I can see packaged installed
> by both pip, conda, and locally (i.e., "pip install . -e").
>
> -paul
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] linux wheels coming soon

[Benjamin Root]

Are we going to have to have documentation somewhere making it clear that
the numpy wheel shouldn't be used in a conda environment? Not that I would
expect this issue to come up all that often, but I could imagine a scenario
where a non-scientist is simply using a base conda distribution because
that is what IT put on their system. Then they do "pip install ipython"
that indirectly brings in numpy (through the matplotlib dependency), and
end up with an incompatible numpy because they would have been linked
against different pythons?

Or is this not an issue?

Ben Root


On Thu, Apr 14, 2016 at 2:04 PM, Matthew Brett 
wrote:

> Hi,
>
> On Thu, Apr 14, 2016 at 8:02 AM, Jens Nielsen 
> wrote:
> > I have tried testing the wheels in a project that runs tests on Travis's
> > Trusty infrastructure which. The wheels work great for python 3.5 and
> saves
> > us several minuts of runtime.
> >
> > However, I am having trouble using the wheels on python 2.7 on the same
> > Trusty machines. It seems to be because the wheels are tagged as
> cp27-cp27mu
> > (numpy-1.11.0-cp27-cp27mu-manylinux1_x86_64.whl) where as
> > pip.pep425tags.get_abi_tag() returns cp27m on this particular python
> > version. (Stock python 2.7 installed on Travis 14.04 VMs) Any chance of a
> > cp27m compatible wheel build?
>
> Ouch - do you know where travis-ci's Python 2.7 comes from?I see
> that the standard apt-get install -y python is a wide (mu) build...
>
> Cheers,
>
> Matthew
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Multidimension array access in C via Python API

[Benjamin Root]

You might do better using scipy.spatial. It has very useful data structures
for handling spatial coordinates. I am not exactly sure how to use them for
this specific problem (not a domain expert), but I would imagine that the
QHull wrappers there might give you some useful tools.

Ben Root

On Tue, Apr 5, 2016 at 12:48 PM, mpc  wrote:

> The idea is that I want to thin a large 2D buffer of x,y,z points to a
> given
> resolution by dividing the data into equal sized "cubes" (i.e. resolution
> is
> number of cubes along each axis) and averaging the points inside each cube
> (if any).
>
>
> *# Fill up buffer data for demonstration purposes with initial buffer
> of
> size 10,000,000 to reduce to 1,000,000
> size = 1000
> buffer = np.ndarray(shape=(size,3), dtype=np.float)
> # fill it up
> buffer[:, 0] = np.random.ranf(size)
> buffer[:, 1] = np.random.ranf(size)
> buffer[:, 2] = np.random.ranf(size)
>
> # Create result buffer to size of cubed resolution (i.e. 100 ^ 3 =
> 1,000,000)
> resolution = 100
> thinned_buffer = np.ndarray(shape=(resolution ** 3,3), dtype=np.float)
>
> # Trying to convert the following into C to speed it up
> x_buffer = buffer[:, 0]
> y_buffer = buffer[:, 1]
> z_buffer = buffer[:, 2]
> min_x = x_buffer.min()
> max_x = x_buffer.max()
> min_y = y_buffer.min()
> max_y = y_buffer.max()
> min_z = z_buffer.min()
> max_z = z_buffer.max()
> z_block = (max_z - min_z) / resolution
> x_block = (max_x - min_x) / resolution
> y_block = (max_y - min_y) / resolution
>
> current_idx = 0
> x_idx = min_x
> while x_idx < max_x:
> y_idx = min_y
> while y_idx < max_y:
> z_idx = min_z
> while z_idx < max_z:
> inside_block_points = np.where((x_buffer >= x_idx) &
>  (x_buffer <=
> x_idx + x_block) &
>  (y_buffer >=
> y_idx) &
>  (y_buffer <=
> y_idx + y_block) &
>  (z_buffer >=
> z_idx) &
>  (z_buffer <=
> z_idx + z_block))
> if inside_block_points[0].size > 0:
> mean_point =
> buffer[inside_block_points[0]].mean(axis=0)
> thinned_buffer[current_idx] = mean_point
> current_idx += 1
> z_idx += z_block
> y_idx += y_block
> x_idx += x_block
> return thin_buffer
> *
>
>
>
> --
> View this message in context:
> http://numpy-discussion.10968.n7.nabble.com/Multidimension-array-access-in-C-via-Python-API-tp42710p42726.html
> Sent from the Numpy-discussion mailing list archive at Nabble.com.
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Reflect array?

[Benjamin Root]

Along those lines, yes, but you have to be careful of even/odd dimension
lengths. Would be nice if it was some sort of stride trick so that I don't
have to allocate a new array twice as we do in the concatenation steps.

Cheers!

Ben Root

On Tue, Mar 29, 2016 at 1:58 PM, Joseph Fox-Rabinovitz <
jfoxrabinov...@gmail.com> wrote:

> On Tue, Mar 29, 2016 at 1:46 PM, Benjamin Root <ben.v.r...@gmail.com>
> wrote:
> > Is there a quick-n-easy way to reflect a NxM array that represents a
> > quadrant into a 2Nx2M array? Essentially, I am trying to reduce the size
> of
> > an expensive calculation by taking advantage of the fact that the first
> part
> > of the calculation is just computing gaussian weights, which is radially
> > symmetric.
> >
> > It doesn't seem like np.tile() could support this (yet?). Maybe we could
> > allow negative repetitions to mean "reflected"? But I was hoping there
> was
> > some existing function or stride trick that could accomplish what I am
> > trying.
> >
> > x = np.linspace(-5, 5, 20)
> > y = np.linspace(-5, 5, 24)
> > z = np.hypot(x[None, :], y[:, None])
> > zz = np.hypot(x[None, :int(len(x)//2)], y[:int(len(y)//2), None])
> > zz = some_mirroring_trick(zz)
>
> Are you looking for something like this:
>
> zz = np.hypot.outer(y[:len(y)//2], x[:len(x)//2])
> zz = np.concatenate((zz[:, ::-1], zz), axis=1)
> zz = np.concatenate((zz, zz[::-1, :]))
>
> > assert np.all(z == zz)
> >
> > What can be my "some_mirroring_trick()"? I am hoping for something a
> little
> > better than using hstack()/vstack().
> >
> > Thanks,
> > Ben Root
> >
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Reflect array?

[Benjamin Root]

Is there a quick-n-easy way to reflect a NxM array that represents a
quadrant into a 2Nx2M array? Essentially, I am trying to reduce the size of
an expensive calculation by taking advantage of the fact that the first
part of the calculation is just computing gaussian weights, which is
radially symmetric.

It doesn't seem like np.tile() could support this (yet?). Maybe we could
allow negative repetitions to mean "reflected"? But I was hoping there was
some existing function or stride trick that could accomplish what I am
trying.

x = np.linspace(-5, 5, 20)
y = np.linspace(-5, 5, 24)
z = np.hypot(x[None, :], y[:, None])
zz = np.hypot(x[None, :int(len(x)//2)], y[:int(len(y)//2), None])
zz = some_mirroring_trick(zz)
assert np.all(z == zz)

What can be my "some_mirroring_trick()"? I am hoping for something a little
better than using hstack()/vstack().

Thanks,
Ben Root
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] reshaping empty array bug?

[Benjamin Root]

On Tue, Feb 23, 2016 at 3:30 PM, Nathaniel Smith  wrote:

> What should this do?
>
> np.zeros((12, 0)).reshape((10, -1, 2))
>


It should error out, I already covered that. 12 != 20.

Ben Root
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] reshaping empty array bug?

[Benjamin Root]

On Tue, Feb 23, 2016 at 3:14 PM, Nathaniel Smith  wrote:

> Sure, it's totally ambiguous. These are all legal:



I would argue that except for the first reshape, all of those should be an
error, and that the current algorithm is buggy.

This isn't a heuristic. It isn't guessing. It is making the semantics
consistent. The fact that I can do:
a.shape = (-1, 5, 64)
or
a.shape = (0, 5, 64)

but not
a.shape = (0, 5, -1)

is totally inconsistent.

Ben Root
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] reshaping empty array bug?

[Benjamin Root]

I'd be more than happy to write up the patch. I don't think it would be
quite like make zeros be ones, but it would be along those lines. One case
I need to wrap my head around is to make sure that a 0 would happen if the
following was true:

>>> a = np.ones((0, 5*64))
>>> a.shape = (-1, 5, 64)

EDIT: Just tried the above, and it works as expected (zero in the first
dim)!

Just tried out a couple of other combos:
>>> a.shape = (-1,)
>>> a.shape
(0,)
>>> a.shape = (-1, 5, 64)
>>> a.shape
(0, 5, 64)


This is looking more and more like a bug to me.

Ben Root


On Tue, Feb 23, 2016 at 1:58 PM, Sebastian Berg <sebast...@sipsolutions.net>
wrote:

> On Di, 2016-02-23 at 11:45 -0500, Benjamin Root wrote:
> > but, it isn't really ambiguous, is it? The -1 can only refer to a
> > single dimension, and if you ignore the zeros in the original and new
> > shape, the -1 is easily solvable, right?
>
> I think if there is a simple logic (like using 1 for all zeros in both
> input and output shape for the -1 calculation), maybe we could do it. I
> would like someone to think about it carefully that it would not also
> allow some unexpected generalizations. And at least I am getting a
> BrainOutOfResourcesError right now trying to figure that out :).
>
> - Sebastian
>
>
> > Ben Root
> >
> > On Tue, Feb 23, 2016 at 11:41 AM, Warren Weckesser <
> > warren.weckes...@gmail.com> wrote:
> > >
> > >
> > > On Tue, Feb 23, 2016 at 11:32 AM, Benjamin Root <
> > > ben.v.r...@gmail.com> wrote:
> > > > Not exactly sure if this should be a bug or not. This came up in
> > > > a fairly general function of mine to process satellite data.
> > > > Unexpectedly, one of the satellite files had no scans in it,
> > > > triggering an exception when I tried to reshape the data from it.
> > > >
> > > > >>> import numpy as np
> > > > >>> a = np.zeros((0, 5*64))
> > > > >>> a.shape
> > > > (0, 320)
> > > > >>> a.shape = (0, 5, 64)
> > > > >>> a.shape
> > > > (0, 5, 64)
> > > > >>> a.shape = (0, 5*64)
> > > > >>> a.shape = (0, 5, -1)
> > > > Traceback (most recent call last):
> > > >   File "", line 1, in 
> > > > ValueError: total size of new array must be unchanged
> > > >
> > > > So, if I know all of the dimensions, I can reshape just fine. But
> > > > if I wanted to use the nifty -1 semantic, it completely falls
> > > > apart. I can see arguments going either way for whether this is a
> > > > bug or not.
> > > >
> > >
> > > When you try `a.shape = (0, 5, -1)`, the size of the third
> > > dimension is ambiguous.  From the Zen of Python:  "In the face of
> > > ambiguity, refuse the temptation to guess."
> > >
> > > Warren
> > >
> > >
> > >
> > > >
> > > > Thoughts?
> > > >
> > > > Ben Root
> > > >
> > > > ___
> > > > NumPy-Discussion mailing list
> > > > NumPy-Discussion@scipy.org
> > > > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> > > >
> > >
> > > ___
> > > NumPy-Discussion mailing list
> > > NumPy-Discussion@scipy.org
> > > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> > >
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] reshaping empty array bug?

[Benjamin Root]

but, it isn't really ambiguous, is it? The -1 can only refer to a single
dimension, and if you ignore the zeros in the original and new shape, the
-1 is easily solvable, right?

Ben Root

On Tue, Feb 23, 2016 at 11:41 AM, Warren Weckesser <
warren.weckes...@gmail.com> wrote:

>
>
> On Tue, Feb 23, 2016 at 11:32 AM, Benjamin Root <ben.v.r...@gmail.com>
> wrote:
>
>> Not exactly sure if this should be a bug or not. This came up in a fairly
>> general function of mine to process satellite data. Unexpectedly, one of
>> the satellite files had no scans in it, triggering an exception when I
>> tried to reshape the data from it.
>>
>> >>> import numpy as np
>> >>> a = np.zeros((0, 5*64))
>> >>> a.shape
>> (0, 320)
>> >>> a.shape = (0, 5, 64)
>> >>> a.shape
>> (0, 5, 64)
>> >>> a.shape = (0, 5*64)
>> >>> a.shape = (0, 5, -1)
>> Traceback (most recent call last):
>>   File "", line 1, in 
>> ValueError: total size of new array must be unchanged
>>
>> So, if I know all of the dimensions, I can reshape just fine. But if I
>> wanted to use the nifty -1 semantic, it completely falls apart. I can see
>> arguments going either way for whether this is a bug or not.
>>
>
>
> When you try `a.shape = (0, 5, -1)`, the size of the third dimension is
> ambiguous.  From the Zen of Python:  "In the face of ambiguity, refuse the
> temptation to guess."
>
> Warren
>
>
>
>
>> Thoughts?
>>
>> Ben Root
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] reshaping empty array bug?

[Benjamin Root]

Not exactly sure if this should be a bug or not. This came up in a fairly
general function of mine to process satellite data. Unexpectedly, one of
the satellite files had no scans in it, triggering an exception when I
tried to reshape the data from it.

>>> import numpy as np
>>> a = np.zeros((0, 5*64))
>>> a.shape
(0, 320)
>>> a.shape = (0, 5, 64)
>>> a.shape
(0, 5, 64)
>>> a.shape = (0, 5*64)
>>> a.shape = (0, 5, -1)
Traceback (most recent call last):
  File "", line 1, in 
ValueError: total size of new array must be unchanged

So, if I know all of the dimensions, I can reshape just fine. But if I
wanted to use the nifty -1 semantic, it completely falls apart. I can see
arguments going either way for whether this is a bug or not.

Thoughts?

Ben Root
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] [Suggestion] Labelled Array

[Benjamin Root]

matplotlib would be more than happy if numpy could take those functions off
our hands! They don't get nearly the correct visibility in matplotlib
because no one is expecting them to be in a plotting library, and they
don't have any useful unit-tests. None of us made them, so we are very
hesitant to update them because of that.

Cheers!
Ben Root

On Fri, Feb 19, 2016 at 1:39 PM, <josef.p...@gmail.com> wrote:

>
>
> On Fri, Feb 19, 2016 at 12:08 PM, Allan Haldane <allanhald...@gmail.com>
> wrote:
>
>> I also want to add a historical note here, that 'groupby' has been
>> discussed a couple times before.
>>
>> Travis Oliphant even made an NEP for it, and Wes McKinney lightly hinted
>> at adding it to numpy.
>>
>>
>> http://thread.gmane.org/gmane.comp.python.numeric.general/37480/focus=37480
>>
>> http://thread.gmane.org/gmane.comp.python.numeric.general/38272/focus=38299
>> http://docs.scipy.org/doc/numpy-1.10.1/neps/groupby_additions.html
>>
>> Travis's idea for a ufunc method 'reduceby' is more along the lines of
>> what I was originally thinking. Just musing about it, it might cover few
>> small cases pandas groupby might not: It could work on arbitrary ufuncs,
>> and over particular axes of multidimensional data. Eg, to sum over
>> pixels from NxNx3 image data. But maybe pandas can cover the
>> multidimensional case through additional index columns or with Panel.
>>
>
> xarray is now covering that area.
>
> There are also recfunctions in numpy.lib that never got a lot of attention
> and expansion.
> There were plans to cover more of the matplotlib versions in numpy, but I
> have no idea and didn't check what happened to it..
>
> Josef
>
>
>
>>
>> Cheers,
>> Allan
>>
>> On 02/15/2016 05:31 PM, Paul Hobson wrote:
>> > Just for posterity -- any future readers to this thread who need to do
>> > pandas-like on record arrays should look at matplotlib's mlab submodule.
>> >
>> > I've been in situations (::cough:: Esri production ::cough::) where I've
>> > had one hand tied behind my back and unable to install pandas. mlab was
>> > a big help there.
>> >
>> > https://goo.gl/M7Mi8B
>> >
>> > -paul
>> >
>> >
>> >
>> > On Mon, Feb 15, 2016 at 1:28 PM, Lluís Vilanova <vilan...@ac.upc.edu
>> > <mailto:vilan...@ac.upc.edu>> wrote:
>> >
>> > Benjamin Root writes:
>> >
>> > > Seems like you are talking about xarray:
>> https://github.com/pydata/xarray
>> >
>> > Oh, I wasn't aware of xarray, but there's also this:
>> >
>> >
>> >
>> https://people.gso.ac.upc.edu/vilanova/doc/sciexp2/user_guide/data.html#basic-indexing
>> >
>> >
>> https://people.gso.ac.upc.edu/vilanova/doc/sciexp2/user_guide/data.html#dimension-oblivious-indexing
>> >
>> >
>> > Cheers,
>> >   Lluis
>> >
>> >
>> >
>> > > Cheers!
>> > > Ben Root
>> >
>> > > On Fri, Feb 12, 2016 at 9:40 AM, Sérgio <filab...@gmail.com
>> > <mailto:filab...@gmail.com>> wrote:
>> >
>> > > Hello,
>> >
>> >
>> > > This is my first e-mail, I will try to make the idea simple.
>> >
>> >
>> > > Similar to masked array it would be interesting to use a label
>> > array to
>> > > guide operations.
>> >
>> >
>> > > Ex.:
>> > >>>> x
>> > > labelled_array(data =
>> >
>> > > [[0 1 2]
>> > > [3 4 5]
>> > > [6 7 8]],
>> > > label =
>> > > [[0 1 2]
>> > > [0 1 2]
>> > > [0 1 2]])
>> >
>> >
>> > >>>> sum(x)
>> > > array([9, 12, 15])
>> >
>> >
>> > > The operations would create a new axis for label indexing.
>> >
>> >
>> > > You could think of it as a collection of masks, one for each
>> > label.
>> >
>> >
>> > > I don't know a way to make something like this efficiently
>> > without a loop.
>> > > Just wondering...
>> >
>> >
>> > > Sérgio.
>> >
>> > > ___

Re: [Numpy-discussion] [Suggestion] Labelled Array

[Benjamin Root]

Seems like you are talking about xarray: https://github.com/pydata/xarray

Cheers!
Ben Root

On Fri, Feb 12, 2016 at 9:40 AM, Sérgio  wrote:

> Hello,
>
> This is my first e-mail, I will try to make the idea simple.
>
> Similar to masked array it would be interesting to use a label array to
> guide operations.
>
> Ex.:
> >>> x
> labelled_array(data =
>  [[0 1 2]
>  [3 4 5]
>  [6 7 8]],
> label =
>  [[0 1 2]
>  [0 1 2]
>  [0 1 2]])
>
> >>> sum(x)
> array([9, 12, 15])
>
> The operations would create a new axis for label indexing.
>
> You could think of it as a collection of masks, one for each label.
>
> I don't know a way to make something like this efficiently without a loop.
> Just wondering...
>
> Sérgio.
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] [Suggestion] Labelled Array

[Benjamin Root]

Re-reading your post, I see you are talking about something different. Not
exactly sure what your use-case is.

Ben Root

On Fri, Feb 12, 2016 at 9:49 AM, Benjamin Root <ben.v.r...@gmail.com> wrote:

> Seems like you are talking about xarray: https://github.com/pydata/xarray
>
> Cheers!
> Ben Root
>
> On Fri, Feb 12, 2016 at 9:40 AM, Sérgio <filab...@gmail.com> wrote:
>
>> Hello,
>>
>> This is my first e-mail, I will try to make the idea simple.
>>
>> Similar to masked array it would be interesting to use a label array to
>> guide operations.
>>
>> Ex.:
>> >>> x
>> labelled_array(data =
>>  [[0 1 2]
>>  [3 4 5]
>>  [6 7 8]],
>> label =
>>  [[0 1 2]
>>  [0 1 2]
>>  [0 1 2]])
>>
>> >>> sum(x)
>> array([9, 12, 15])
>>
>> The operations would create a new axis for label indexing.
>>
>> You could think of it as a collection of masks, one for each label.
>>
>> I don't know a way to make something like this efficiently without a
>> loop. Just wondering...
>>
>> Sérgio.
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Deprecating `numpy.iterable`

[Benjamin Root]

Huh... matplotlib could use that! We have been using our own internal
function left over from the numerix days, I think.

Ben Root

On Thu, Feb 11, 2016 at 2:12 PM, Nathaniel Smith  wrote:

> Oh wow, yeah, there are tons of uses:
>
> https://github.com/search?q=%22np.iterable%22=simplesearch=Code=%E2%9C%93
>
> Meh, I dunno, maybe we're stuck with it. It's not a major maintenance
> burden at least.
>
> -n
>
> On Thu, Feb 11, 2016 at 10:25 AM, Stephan Hoyer  wrote:
> > We certainly can (and probably should) deprecate this, but we can't
> remove
> > it for a very long time.
> >
> > np.iterable is used in a lot of third party code.
> >
> > On Wed, Feb 10, 2016 at 7:09 PM, Joseph Fox-Rabinovitz
> >  wrote:
> >>
> >> I have created a PR to deprecate `np.iterable`
> >> (https://github.com/numpy/numpy/pull/7202). It is a very old function,
> >> introduced as a utility in 2005
> >>
> >> (
> https://github.com/numpy/numpy/commit/052a7b2e3276a303be1083022fc24d43084d2e14
> ),
> >> and there is no good reason for it to be part of the public API. It is
> >> used internally 10 times within numpy. I have repaced those usages
> >> with a private function `np.lib.function_base._iterable` and added a
> >> `DeprecationWarning` to the public function.
> >>
> >> Is there anyone that objects to deprecating this function?
> >>
> >> Regards,
> >>
> >> -Joseph
> >> ___
> >> NumPy-Discussion mailing list
> >> NumPy-Discussion@scipy.org
> >> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
> >
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
>
>
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Bump warning stacklevel

[Benjamin Root]

I like the idea of bumping the stacklevel in principle, but I am not sure
it is all that practical. For example, if a warning came up when doing "x /
y", I am assuming that it is emitted from within the ufunc np.divide(). So,
you would need two stacklevels based on whether the entry point was the
operator or a direct call to np.divide()? Also, I would imagine it might
get weird for numpy functions called within other numpy functions. Or
perhaps I am not totally understanding how this would be done?

Ben Root


On Wed, Jan 27, 2016 at 5:07 PM, Ralf Gommers 
wrote:

>
>
> On Wed, Jan 27, 2016 at 11:02 PM, sebastian 
> wrote:
>
>> On 2016-01-27 21:01, Ralf Gommers wrote:
>>
>>>
>>> One issue will be how to keep this consistent. `stacklevel` is used so
>>> rarely that new PRs will always omit it for new warnings. Will we just
>>> rely on code review, or would a private wrapper around `warn` to use
>>> inside numpy plus a test that checks that the wrapper is used
>>> everywhere be helpful here?
>>>
>>>
>> Yeah, I mean you could add tests for the individual functions in
>> principle.
>> I am not sure if adding an alias helps much, how are we going to test that
>> warnings.warn is not being used? Seems like quite a bit of voodoo
>> necessary
>> for that.
>>
>
> I was thinking something along these lines, but with a regexp checking for
> warnings.warn:
> https://github.com/scipy/scipy/blob/master/scipy/fftpack/tests/test_import.py
>
> Probably more trouble than it's worth though.
>
> Ralf
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Behavior of np.random.uniform

[Benjamin Root]

Are there other functions where this behavior may or may not be happening?
If it isn't consistent across all np.random functions, it probably should
be, one way or the other.

Ben Root

On Tue, Jan 19, 2016 at 5:10 AM, Jaime Fernández del Río <
jaime.f...@gmail.com> wrote:

> Hi all,
>
> There is a PR (#7026 ) that
> documents the current behavior of np.random.uniform when the low and high
> parameters it takes do not conform to the expected low < high. Basically:
>
>- if low < high, random numbers are drawn from [low, high),
>- if low = high, all random numbers will be equal to low, and
>- if low > high, random numbers are drawn from (high, low] (notice the
>change in the open side of the interval.)
>
> My only worry is that, once we document this, we can no longer claim that
> it is a bug.  So I would like to hear from others what do they think.  The
> other more or less obvious options would be to:
>
>- Raise an error, but this would require a deprecation cycle, as
>people may be relying on the current undocumented behavior.
>- Check the inputs and draw numbers from [min(low, high), max(low,
>high)), which is minimally different from current behavior.
>
> I will be merging the current documentation changes in the next few days,
> so it would be good if any concerns were voiced before that.
>
> Thanks,
>
> Jaime
>
> --
> (\__/)
> ( O.o)
> ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes
> de dominación mundial.
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should I use pip install numpy in linux?

[Benjamin Root]

Travis -

I will preface the following by pointing out how valuable miniconda and
anaconda has been for our workplace because we were running into issues
with ensuring that everyone in our mixed platform office had access to all
the same tools, particularly GDAL, NetCDF4 and such. For the longest time,
we were all stuck on an ancient "Corporate Python" that our IT staff
managed to put together, but never had the time to update. So, I do
absolutely love conda for the problems that it solved for us.

That being said... I take exception to your assertion that anaconda is
*the* solution to the packaging problem. I still have a number of issues,
particularly with the interactions of GDAL, shapely, and Basemap (they all
seek out libgeos_c differently), and I have to use my own build of GDAL to
enable many of the features that we use (the vanilla GDAL put out by
Continuum just has the default options, and is quite limited). If I don't
set up my environment *just right* one of those packages will fail to
import in some way due to being unable to find their particular version of
libgeos_c. I haven't figure it out exactly why this happens, but it is very
easy to break such an environment this way after an update.

But that problem is a solvable problem within the framework of conda, so I
am not too concerned about that. The bigger problem is Apache. In
particular, mod_wsgi. About a year ago, one of our developers was happily
developing a webtool that utilized numpy, NetCDF4 and libxml via conda
environments. All of the testing was done in flask and everything was
peachy. We figured that final deployment out to the Apache server would be
a cinch, right? Wrong. Because our mod_wsgi was built against the system's
python because it came in through RPMs, it was completely incompatible with
the compiled numpy because of the differences in the python compile
options. In a clutch, we had our IT staff manually build mod_wsgi against
anaconda's python, but they weren't too happy about that, due to mod_wsgi
no longer getting updated via yum.

If anaconda was the end-all, be-all solution, then it should just be a
simple matter to do "conda install mod_wsgi". But the design of conda is
that it is intended to be a user-space package-manager. mod_wsgi is
installed via root/apache user, which is siloed off from the user. I would
have to (in theory) go and install conda for the apache user and likely
have to install a conda "apache" package and mod_wsgi package. I seriously
doubt this would be an acceptable solution for many IT administrators who
would rather depend upon the upstream distributions who are going to be
very quick about getting updates out the door and are updated automatically
through yum or apt.

So, again, I love conda for what it can do when it works well. I only take
exception to the notion that it can address *all* problems, because there
are some problems that it just simply isn't properly situated for.

Cheers!
Ben Root


On Fri, Jan 15, 2016 at 3:20 PM, Nathaniel Smith  wrote:

> On Fri, Jan 15, 2016 at 12:09 PM, Chris Barker 
> wrote:
> > On Fri, Jan 15, 2016 at 11:21 AM, Nathaniel Smith  wrote:
> >>
> >> Sure. Someone's already packaged those for conda, and no one has
> packaged
> >> them for pypi, so it makes sense that conda is more convenient for you.
> If
> >> someone does the work of packaging them for pypi, then that difference
> goes
> >> away.
> >
> > This is what I meant by "cultural" issues :-)
> >
> > but pypi has been an option for Windows and OS-X for ages and those
> > platforms are the bigger problem anyway -- and no one has done it for
> those
> > platforms
>
> I think what's going on here is that Windows *hasn't* been an option
> for numpy/scipy due to the toolchain issues, and on OS-X we've just
> been using the platform BLAS (Accelerate), so the core packages
> haven't had any motivation to sort out the whole library dependency
> issue, and no-one else is motivated enough to do it. My prediction is
> that after the core packages sort it out in order to solve their own
> problems then we might see others picking it up.
>
> -n
>
> --
> Nathaniel J. Smith -- http://vorpus.org
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Fast Access to Container of Numpy Arrays on Disk?

[Benjamin Root]

A warning about HDF5. It is not a database format, so you have to be
extremely careful if the data is getting updated while it is open for
reading by anybody else. If it is strictly read-only, and no body else is
updating it, then have at it!

Cheers!
Ben Root

On Thu, Jan 14, 2016 at 9:16 AM, Edison Gustavo Muenz <
edisongust...@gmail.com> wrote:

> From what I know this would be the use case that Dask seems to solve.
>
> I think this blog post can help:
> https://www.continuum.io/content/xray-dask-out-core-labeled-arrays-python
>
> Notice that I haven't used any of these projects myself.
>
> On Thu, Jan 14, 2016 at 11:48 AM, Francesc Alted  wrote:
>
>> Well, maybe something like a simple class emulating a dictionary that
>> stores a key-value on disk would be more than enough.  Then you can use
>> whatever persistence layer that you want (even HDF5, but not necessarily).
>>
>> As a demonstration I did a quick and dirty implementation for such a
>> persistent key-store thing (
>> https://gist.github.com/FrancescAlted/8e87c8762a49cf5fc897).  On it, the
>> KeyStore class (less than 40 lines long) is responsible for storing the
>> value (2 arrays) into a key (a directory).  As I am quite a big fan of
>> compression, I implemented a couple of serialization flavors: one using the
>> .npz format (so no other dependencies than NumPy are needed) and the other
>> using the ctable object from the bcolz package (bcolz.blosc.org).  Here
>> are some performance numbers:
>>
>> python key-store.py -f numpy -d __test -l 0
>> ## Checking method: numpy (via .npz files) 
>> Building database.  Wait please...
>> Time (creation) --> 1.906
>> Retrieving 100 keys in arbitrary order...
>> Time (   query) --> 0.191
>> Number of elements out of getitem: 10518976
>> faltet@faltet-Latitude-E6430:~/blosc/bcolz$ du -sh __test
>>
>> 75M __test
>>
>> So, with the NPZ format we can deal with the 75 MB quite easily.  But NPZ
>> can compress data as well, so let's see how it goes:
>>
>> $ python key-store.py -f numpy -d __test -l 9
>> ## Checking method: numpy (via .npz files) 
>> Building database.  Wait please...
>> Time (creation) --> 6.636
>> Retrieving 100 keys in arbitrary order...
>> Time (   query) --> 0.384
>> Number of elements out of getitem: 10518976
>> faltet@faltet-Latitude-E6430:~/blosc/bcolz$ du -sh __test
>> 28M __test
>>
>> Ok, in this case we have got almost a 3x compression ratio, which is not
>> bad.  However, the performance has degraded a lot.  Let's use now bcolz.
>> First in non-compressed mode:
>>
>> $ python key-store.py -f bcolz -d __test -l 0
>> ## Checking method: bcolz (via ctable(clevel=0, cname='blosclz')
>> 
>> Building database.  Wait please...
>> Time (creation) --> 0.479
>> Retrieving 100 keys in arbitrary order...
>> Time (   query) --> 0.103
>> Number of elements out of getitem: 10518976
>> faltet@faltet-Latitude-E6430:~/blosc/bcolz$ du -sh __test
>> 82M __test
>>
>> Without compression, bcolz takes a bit more (~10%) space than NPZ.
>> However, bcolz is actually meant to be used with compression on by default:
>>
>> $ python key-store.py -f bcolz -d __test -l 9
>> ## Checking method: bcolz (via ctable(clevel=9, cname='blosclz')
>> 
>> Building database.  Wait please...
>> Time (creation) --> 0.487
>> Retrieving 100 keys in arbitrary order...
>> Time (   query) --> 0.98
>> Number of elements out of getitem: 10518976
>> faltet@faltet-Latitude-E6430:~/blosc/bcolz$ du -sh __test
>> 29M __test
>>
>> So, the final disk usage is quite similar to NPZ, but it can store and
>> retrieve lots faster.  Also, the data decompression speed is on par to
>> using non-compression.  This is because bcolz uses Blosc behind the scenes,
>> which is much faster than zlib (used by NPZ) --and sometimes faster than a
>> memcpy().  However, even we are doing I/O against the disk, this dataset is
>> so small that fits in the OS filesystem cache, so the benchmark is actually
>> checking I/O at memory speeds, not disk speeds.
>>
>> In order to do a more real-life comparison, let's use a dataset that is
>> much larger than the amount of memory in my laptop (8 GB):
>>
>> $ PYTHONPATH=. python key-store.py -f bcolz -m 100 -k 5000 -d
>> /media/faltet/docker/__test -l 0
>> ## Checking method: bcolz (via ctable(clevel=0, cname='blosclz')
>> 
>> Building database.  Wait please...
>> Time (creation) --> 133.650
>> Retrieving 100 keys in arbitrary order...
>> Time (   query) --> 2.881
>> Number of elements out of getitem: 91907396
>> faltet@faltet-Latitude-E6430:~/blosc/bcolz$ du -sh
>> /media/faltet/docker/__test
>>
>> 39G /media/faltet/docker/__test
>>
>> and now, with compression on:
>>
>> $ PYTHONPATH=. python key-store.py -f bcolz -m 100 -k 5000 -d
>> 

Re: [Numpy-discussion] Should I use pip install numpy in linux?

[Benjamin Root]

The other half of the fun is how to deal with weird binary issues with
libraries like libgeos_c, libhdf5 and such.  You have to get all of the
compile options right for your build of those libraries to get your build
of GDAL and pyhdf working right. You also have packages like gdal and
netcdf4 have diamond dependencies -- not only are they built and linked
against numpy binaries, some of their binary dependencies are built against
numpy binaries as well. Joys!

I don't envy anybody that tries to take on the packaging problem in any
language.

Ben Root

On Mon, Jan 11, 2016 at 1:25 PM, Chris Barker  wrote:

> On Fri, Jan 8, 2016 at 7:13 PM, Nathaniel Smith  wrote:
>
>> > that this would potentially be able to let packages like numpy serve
>> their
>> > linux
>> > users better without risking too much junk being uploaded to PyPI.
>>
>> That will never fly. But like Matthew says, I think we can probably
>> get them to accept a PEP saying "here's a new well-specified platform
>> tag that means that this wheel works on all linux systems meet the
>> following list of criteria: ...", and then allow that new platform tag
>> onto PyPI.
>>
>
> The second step is a trick though -- how does pip know, when being run on
> a client, that the system meets those requirements? Do we put a bunch of
> code in that checks for those libs, etc???
>
> If we get all that worked out, we still haven't made any progress toward
> the non-standard libs that aren't python. This is the big "scipy problem"
> -- fortran, BLAS, hdf, ad infinitum.
>
> I argued for years that we could build binary wheels that hold each of
> these, and other python packages could depend on them, but pypa never
> seemed to like that idea. In the end, if you did all this right, you'd have
> something like conda -- so why not just use conda?
>
> All that being said, if you folks can get the core scipy stack setup to
> pip install on OS_X, Windows, and Linux, that would be pretty nice -- so
> keep at it !
>
> -CHB
>
>
>
>
>
>
>
>
>
>>
>> -n
>>
>> --
>> Nathaniel J. Smith -- http://vorpus.org
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] what would you expect A[none] to do?

[Benjamin Root]

TBH, I wouldn't have expected it to work, but now that I see it, it does
make some sense. I would have thought that it would error out as being
ambiguous (prepend? append?). I have always used ellipses to make it
explicit where the new axis should go. But, thinking in terms of how
regular indexing works, I guess it isn't all that ambiguous.

Ben Root

On Thu, Dec 31, 2015 at 11:56 AM, Joe Kington 
wrote:

> Slicing with None adds a new dimension.  It's a common paradigm, though
> usually you'd use A[np.newaxis] or A[np.newaxis, ...] instead for
> readibility.   (np.newaxis is None, but it's a lot more readable)
>
> There's a good argument to be made that slicing with a single None
> shouldn't add a new axis, and only the more readable forms like A[None, :],
> A[..., None], etc should.
>
> However, that would rather seriously break backwards compatibility.
> There's a fair amount of existing code that assumes "A[None]" prepends a
> new axis.
>
> On Thu, Dec 31, 2015 at 10:36 AM, Neal Becker  wrote:
>
>> Neal Becker wrote:
>>
>> > In my case, what it does is:
>> >
>> > A.shape = (5760,)
>> > A[none] -> (1, 5760)
>> >
>> > In my case, use of none here is just a mistake.  But why would you want
>> > this to be accepted at all, and how should it be interpreted?
>>
>> Actually, in my particular case, if it just acted as a noop, returning the
>> original array, that would have been perfect.  No idea if that's a good
>> result in general.
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] How to find indices of values in an array (indirect in1d) ?

[Benjamin Root]

Maybe use searchsorted()? I will note that I have needed to do something
like this once before, and I found that the list comprehension form of
calling .index() for each item was faster than jumping through hoops to
vectorize it using searchsorted (needing to sort and then map the sorted
indices to the original indices), and was certainly clearer, but that might
depend upon the problem size.

Cheers!
Ben Root

On Wed, Dec 30, 2015 at 10:02 AM, Andy Ray Terrel 
wrote:

> Using pandas one can do:
>
> >>> A = np.array([2,0,1,4])
> >>> B = np.array([1,2,0])
> >>> s = pd.Series(range(len(B)), index=B)
> >>> s[A].values
> array([  1.,   2.,   0.,  nan])
>
>
>
> On Wed, Dec 30, 2015 at 8:45 AM, Nicolas P. Rougier <
> nicolas.roug...@inria.fr> wrote:
>
>>
>> I’m scratching my head around a small problem but I can’t find a
>> vectorized solution.
>> I have 2 arrays A and B and I would like to get the indices (relative to
>> B) of elements of A that are in B:
>>
>> >>> A = np.array([2,0,1,4])
>> >>> B = np.array([1,2,0])
>> >>> print (some_function(A,B))
>> [1,2,0]
>>
>> # A[0] == 2 is in B and 2 == B[1] -> 1
>> # A[1] == 0 is in B and 0 == B[2] -> 2
>> # A[2] == 1 is in B and 1 == B[0] -> 0
>>
>> Any idea ? I tried numpy.in1d with no luck.
>>
>>
>> Nicolas
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop providing official win32 downloads (for now)

[Benjamin Root]

I believe that a lot can be learned from matplotlib's recent foray into
appveyor. Don't hesitate to ask questions on our dev mailing list (I wasn't
personally involved, so I don't know what was learned).

Cheers!
Ben Root

On Fri, Dec 18, 2015 at 5:27 PM, Nathaniel Smith  wrote:

> On Dec 18, 2015 2:22 PM, "Ian Henriksen" <
> insertinterestingnameh...@gmail.com> wrote:
> >
> > An appveyor setup is a great idea. An appveyor build matrix with the
> > various supported MSVC versions would do a lot more to prevent
> > compatibility issues than periodically building installers with old
> versions of
> > MinGW. The effort toward a MinGW-based build is valuable, but having a
> > CI system test for MSVC compatibility will be valuable regardless of
> where
> > things go with that.
>
> Yes, definitely. Would you by chance have any interest in getting this set
> up?
>
> -n
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] A minor clarification no why count_nonzero is faster for boolean arrays

[Benjamin Root]

Would it make sense to at all to bring that optimization to np.sum()? I
know that I have np.sum() all over the place instead of count_nonzero,
partly because it is a MatLab-ism and partly because it is easier to write.
I had no clue that there was a performance difference.

Cheers!
Ben Root


On Thu, Dec 17, 2015 at 1:37 PM, CJ Carey  wrote:

> I believe this line is the reason:
>
> https://github.com/numpy/numpy/blob/c0e48cfbbdef9cca954b0c4edd0052e1ec8a30aa/numpy/core/src/multiarray/item_selection.c#L2110
>
> On Thu, Dec 17, 2015 at 11:52 AM, Raghav R V  wrote:
>
>> I was just playing with `count_nonzero` and found it to be significantly
>> faster for boolean arrays compared to integer arrays
>>
>>
>> >>> a = np.random.randint(0, 2, (100, 5))
>> >>> a_bool = a.astype(bool)
>>
>> >>> %timeit np.sum(a)
>> 10 loops, best of 3: 5.64 µs per loop
>>
>> >>> %timeit np.count_nonzero(a)
>> 100 loops, best of 3: 1.42 us per loop
>>
>> >>> %timeit np.count_nonzero(a_bool)
>> 100 loops, best of 3: 279 ns per loop (but why?)
>>
>> I tried looking into the code and dug my way through to this line
>> .
>> I am unable to dig further.
>>
>> I know this is probably a trivial question, but was wondering if anyone
>> could provide insight on why this is so?
>>
>> Thanks
>>
>> R
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] FeatureRequest: support for array construction from iterators

[Benjamin Root]

Devil's advocate here: np.array() has become the de-facto "constructor" for
numpy arrays. Right now, passing it a generator results in what, IMHO, is a
useless result:

>>> np.array((i for i in range(10)))
array( at 0x7f28b2beca00>, dtype=object)

Passing pretty much any dtype argument will cause that to fail:

>>> np.array((i for i in range(10)), dtype=np.int_)
Traceback (most recent call last):
  File "", line 1, in 
TypeError: long() argument must be a string or a number, not 'generator'

Therefore, I think it is not out of the realm of reason that passing a
generator object and a dtype could then delegate the work under the hood to
np.fromiter()? I would even go so far as to raise an error if one passes a
generator without specifying dtype to np.array(). The point is to reduce
the number of entry points for creating numpy arrays.


By the way, any reason why this works?
>>> np.array(xrange(10))
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])


Cheers!
Ben Root


On Sat, Dec 12, 2015 at 6:02 PM, Juan Nunez-Iglesias 
wrote:

> Hey Nathaniel,
>
> Fascinating! Thanks for the primer! I didn't know that it would check
> dtype of values in the whole array. In that case, I would agree that it
> would be bad to infer it magically from just the first value, and this can
> be left to the users.
>
> Thanks!
>
> Juan.
>
> On Sat, Dec 12, 2015 at 7:00 PM, Nathaniel Smith  wrote:
>
>> On Fri, Dec 11, 2015 at 11:32 PM, Juan Nunez-Iglesias
>>  wrote:
>> > Nathaniel,
>> >
>> >> IMO this is better than making np.array(iter) internally call
>> list(iter)
>> >> or equivalent
>> >
>> > Yeah but that's not the only option:
>> >
>> > from itertools import chain
>> > def fromiter_awesome_edition(iterable):
>> > elem = next(iterable)
>> > dtype = whatever_numpy_does_to_infer_dtypes_from_lists(elem)
>> > return np.fromiter(chain([elem], iterable), dtype=dtype)
>> >
>> > I think this would be a huge win for usability. Always getting tripped
>> up by
>> > the dtype requirement. I can submit a PR if people like this pattern.
>>
>> This isn't the semantics of np.array, though -- np.array will look at
>> the whole input and try to find a common dtype, so this can't be the
>> implementation for np.array(iter). E.g. try np.array([1, 1.0])
>>
>> I can see an argument for making the dtype= argument to fromiter
>> optional, with a warning in the docs that it will guess based on the
>> first element and that you should specify it if you don't want that.
>> It seems potentially a bit error prone (in the sense that it might
>> make it easier to end up with code that works great when you test it
>> but then breaks later when something unexpected happens), but maybe
>> the usability outweighs that. I don't use fromiter myself so I don't
>> have a strong opinion.
>>
>> > btw, I think np.array(['f', 'o', 'o']) would be exactly the expected
>> result
>> > for np.array('foo'), but I guess that's just me.
>>
>> In general np.array(thing_that_can_go_inside_an_array) returns a
>> zero-dimensional (scalar) array -- np.array(1), np.array(True), etc.
>> all work like this, so I'd expect np.array("foo") to do the same.
>>
>> -n
>>
>> --
>> Nathaniel J. Smith -- http://vorpus.org
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] FeatureRequest: support for array construction from iterators

[Benjamin Root]

Heh, never noticed that. Was it implemented more like a generator/iterator
in older versions of Python?

Thanks,
Ben Root

On Mon, Dec 14, 2015 at 12:38 PM, Robert Kern <robert.k...@gmail.com> wrote:

> On Mon, Dec 14, 2015 at 3:56 PM, Benjamin Root <ben.v.r...@gmail.com>
> wrote:
>
> > By the way, any reason why this works?
> > >>> np.array(xrange(10))
> > array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>
> It's not a generator. It's a true sequence that just happens to have a
> special implementation rather than being a generic container.
>
> >>> len(xrange(10))
> 10
> >>> xrange(10)[5]
> 5
>
> --
> Robert Kern
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ENH: Add the function 'expand_view'

[Benjamin Root]

How is this different from using np.newaxis and broadcasting? Or am I
misunderstanding this?

Ben Root

On Tue, Nov 24, 2015 at 9:13 PM,  wrote:

>
>
> On Tue, Nov 24, 2015 at 7:13 PM, Nathaniel Smith  wrote:
>
>> On Nov 24, 2015 11:57 AM, "John Kirkham"  wrote:
>> >
>> > Takes an array and tacks on arbitrary dimensions on either side, which
>> is returned as a view always. Here are the relevant features:
>> >
>> > * Creates a view of the array that has the dimensions before and after
>> tacked on to it.
>> > * Takes the before and after arguments independent of each other and
>> the current shape.
>> > * Allows for read and write access to the underlying array.
>>
>> Can you expand this with some discussion of why you want this function,
>> and why you chose these specific features? (E.g. as mentioned in the PR
>> comments already, the reason broadcast_to returns a read-only array is that
>> it was decided that this was less confusing for users, not because of any
>> technical issue.)
>>
>
> Why is this a stride_trick?
>
> I thought this looks similar to expand_dims and could maybe be implemented
> with some extra options there.
>
>
>
> Josef
>
>
>
>> -n
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] New functions added in pull request

[Benjamin Root]

Oooh, this will be nice to have. This would be one of the few times I would
love to see unicode versions of these function names supplied, too.

On Wed, Nov 25, 2015 at 5:31 PM, Antonio Lara  wrote:

> Hello, I have added three new functions to the file function_base.py in
> the numpy/lib folder. These are divergence, curl and laplacian (for the
> moment, laplacian of a scalar field, maybe in the future I will try
> laplacian for a vector field). The calculation method is based in the
> existing one for numpy.gradient, with central differences.
> The changes are in this pull request:
>
> https://github.com/numpy/numpy/pull/6727
>
> Thank you,
>
> Antonio
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] loadtxt and usecols

[Benjamin Root]

Just pointing out np.loadtxt(..., ndmin=2) will always return a 2D array.
Notice that without that option, the result is effectively squeezed. So if
you don't specify that option, and you load up a CSV file with only one
row, you will get a very differently shaped array than if you load up a CSV
file with two rows.

Ben Root

On Tue, Nov 10, 2015 at 10:07 AM, Irvin Probst <
irvin.pro...@ensta-bretagne.fr> wrote:

> On 10/11/2015 14:17, Sebastian Berg wrote:
>
>> Actually, it is the "sequence special case" type ;). (matlab does not
>> have this, since matlab always returns 2-D I realized).
>>
>> As I said, if usecols is like indexing, the result should mimic:
>>
>> arr = np.loadtxt(f)
>> arr = arr[usecols]
>>
>> in which case a 1-D array is returned if you put in a scalar into
>> usecols (and you could even generalize usecols to higher dimensional
>> array-likes).
>> The way you implemented it -- which is fine, but I want to stress that
>> there is a real decision being made here --, you always see it as a
>> sequence but allow a scalar for convenience (i.e. always return a 2-D
>> array). It is a `sequence of ints or int` type argument and not an
>> array-like argument in my opinion.
>>
>
> I think we have two separate problems here:
>
> The first one is whether loadtxt should always return a 2D array or should
> it match the shape of the usecol argument. From a CS guy point of view I do
> understand your concern here. Now from a teacher point of view I know many
> people expect to get a "matrix" (thank you Matlab...) and the "purity" of
> matching the dimension of the usecol variable will be seen by many people
> [1] as a nerdy useless heavyness noone cares of (no offense). So whatever
> you, seadoned numpy devs from this mailing list, decide I think it should
> be explained in the docstring with a very clear wording.
>
> My own opinion on this first problem is that loadtxt() should always
> return a 2D array, no less, no more. If I write np.loadtxt(f)[42] it means
> I want to read the whole file and then I explicitely ask for transforming
> the 2-D array loadtxt() returned into a 1-D array. Otoh if I write
> loadtxt(f, usecol=42) it means I don't want to read the other columns and I
> want only this one, but it does not mean that I want to change the returned
> array from 2-D to 1-D. I know this new behavior might break a lot of
> existing code as usecol=(42,) used to return a 1-D array, but
> usecol=42, also returns a 1-D array so the current behavior is not
> consistent imho.
>
> The second problem is about the wording in the docstring, when I see
> "sequence of int or int" I uderstand I will have to cast into a 1-D python
> list whatever wicked N-dimensional object I use to store my column indexes,
> or hope list(my_object) will do it fine. On the other hand when I read
> "array-like" the function is telling me I don't have to worry about my
> object, as long as numpy knows how to cast it into an array it will be fine.
>
> Anyway I think something like that:
>
> import numpy as np
> a=[[[2,],[],[],],[],[],[]]
> foo=np.loadtxt("CONCARNEAU_2010.txt", usecols=a)
>
> should just work and return me a 2-D (or 1-D if you like) array with the
> data I asked for and I don't think "a" here is an int or a sequence of int
> (but it's a good example of why loadtxt() should not match the shape of the
> usecol argument).
>
> To make it short, let the reading function read the data in a consistent
> and predictible way and then let the user explicitely change the data's
> shape into anything he likes.
>
> Regards.
>
> [1] read non CS people trying to switch to numpy/scipy
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] loadtxt and usecols

[Benjamin Root]

My personal rule for flexible inputs like that is that it should be
encouraged so long as it does not introduce ambiguity. Furthermore,
Allowing a scalar as an input doesn't add a congitive disconnect on the
user on how to specify multiple columns. Therefore, I'd give this a +1.

On Mon, Nov 9, 2015 at 4:15 AM, Irvin Probst  wrote:

> Hi,
> I've recently seen many students, coming from Matlab, struggling against
> the usecols argument of loadtxt. Most of them tried something like:
> loadtxt("foo.bar", usecols=2) or the ones with better documentation
> reading skills tried loadtxt("foo.bar", usecols=(2)) but none of them
> understood they had to write usecols=[2] or usecols=(2,).
>
> Is there a policy in numpy stating that this kind of arguments must be
> sequences ? I think that being able to an int or a sequence when a single
> column is needed would make this function a bit more user friendly for
> beginners. I would gladly submit a PR if noone disagrees.
>
> Regards.
>
> --
> Irvin
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] New behavior of allclose

[Benjamin Root]

allclose() needs to return a bool so that one can do "if np.allclose(foo,
bar) is True" or some such. The "good behavior" is for np.isclose() to
return a memmap, which still confuses the heck out of me, but I am not a
memmap expert.

On Thu, Nov 5, 2015 at 4:50 PM, Ralf Gommers  wrote:

>
>
> On Wed, Nov 4, 2015 at 8:28 PM, Charles R Harris <
> charlesr.har...@gmail.com> wrote:
>
>> Hi All,
>>
>> This is to open a discussion of a change of behavior of `np.allclose`.
>> That function uses `isclose` in numpy 1.10 with the result that array
>> subtypes are preserved whereas before they were not. In particular, memmaps
>> are returned when at least one of the inputs is a memmap. By and large I
>> think this is a good thing, OTOH, it is a change in behavior. It is easy to
>> fix, just run `np.array(result, copy=False)` on the current `result`, but I
>> thought I'd raise the topic on the list in case there is a good argument to
>> change things.
>>
>
> Why would it be good to return a memmap? And am I confused or does your
> just merged PR [1] revert the behavior you say here is a good thing?
>
> Ralf
>
> [1] https://github.com/numpy/numpy/pull/6628
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] New behavior of allclose

[Benjamin Root]

I am not sure I understand what you mean. Specifically that np.isclose will
return a memmap if one of the inputs is a memmap. The result is a brand new
array, right? So, what is that result memmapping from? Also, how does this
impact np.allclose()? That function returns a scalar True/False, so what is
the change in behavior there?

By the way, the docs for isclose in 1.10.1 does not mention any behavior
changes.

Ben Root

On Wed, Nov 4, 2015 at 2:28 PM, Charles R Harris 
wrote:

> Hi All,
>
> This is to open a discussion of a change of behavior of `np.allclose`.
> That function uses `isclose` in numpy 1.10 with the result that array
> subtypes are preserved whereas before they were not. In particular, memmaps
> are returned when at least one of the inputs is a memmap. By and large I
> think this is a good thing, OTOH, it is a change in behavior. It is easy to
> fix, just run `np.array(result, copy=False)` on the current `result`, but I
> thought I'd raise the topic on the list in case there is a good argument to
> change things.
>
> Chuck
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] deprecate fromstring() for text reading?

[Benjamin Root]

Correct, there were entries that would sometimes take up their entire
width. The delimited text readers could not read this particular dataset.
The dataset I am referring to is the processed ISD data:
https://www.ncdc.noaa.gov/isd

As for fromstring() not being able to help there, I didn't mean to imply
that it would. I was more aiming to point out a situation where the NumPy's
text file reader was significantly better than the Pandas version, so we
would want to make sure that we properly benchmark any significant changes
to NumPy's text reading code. Who knows where else NumPy beats Pandas?

Ben

On Mon, Nov 2, 2015 at 6:44 PM, Chris Barker <chris.bar...@noaa.gov> wrote:

> On Tue, Oct 27, 2015 at 7:30 AM, Benjamin Root <ben.v.r...@gmail.com>
> wrote:
>
>> FWIW, when I needed a fast Fixed Width reader
>>
>
> was there potentially no whitespace between fields in that case? In which
> case, it really isn a different use-case than delimited text -- if it's at
> all common, a version written in C would be nice and fast. and nat hard to
> do.
>
> But fromstring never would have helped you with that anyway :-)
>
> -CHB
>
>
>
>> for a very large dataset last year, I found that np.genfromtext() was
>> faster than pandas' read_fwf(). IIRC, pandas' text reading code fell back
>> to pure python for fixed width scenarios.
>>
>> On Fri, Oct 23, 2015 at 8:22 PM, Chris Barker - NOAA Federal <
>> chris.bar...@noaa.gov> wrote:
>>
>>> Grabbing the pandas csv reader would be great, and I hope it happens
>>> sooner than later, though alas, I haven't the spare cycles for it either.
>>>
>>> In the meantime though, can we put a deprecation Warning in when using
>>> fromstring() on text files? It's really pretty broken.
>>>
>>> -Chris
>>>
>>> On Oct 23, 2015, at 4:02 PM, Jeff Reback <jeffreb...@gmail.com> wrote:
>>>
>>>
>>>
>>> On Oct 23, 2015, at 6:49 PM, Nathaniel Smith <n...@pobox.com> wrote:
>>>
>>> On Oct 23, 2015 3:30 PM, "Jeff Reback" <jeffreb...@gmail.com> wrote:
>>> >
>>> > On Oct 23, 2015, at 6:13 PM, Charles R Harris <
>>> charlesr.har...@gmail.com> wrote:
>>> >
>>> >>
>>> >>
>>> >> On Thu, Oct 22, 2015 at 5:47 PM, Chris Barker - NOAA Federal <
>>> chris.bar...@noaa.gov> wrote:
>>> >>>
>>> >>>
>>> >>>> I think it would be good to keep the usage to read binary data at
>>> least.
>>> >>>
>>> >>>
>>> >>> Agreed -- it's only the text file reading I'm proposing to
>>> deprecate. It was kind of weird to cram it in there in the first place.
>>> >>>
>>> >>> Oh, fromfile() has the same issues.
>>> >>>
>>> >>> Chris
>>> >>>
>>> >>>
>>> >>>> Or is there a good alternative to `np.fromstring(,
>>> dtype=...)`?  -- Marten
>>> >>>>
>>> >>>> On Thu, Oct 22, 2015 at 1:03 PM, Chris Barker <
>>> chris.bar...@noaa.gov> wrote:
>>> >>>>>
>>> >>>>> There was just a question about a bug/issue with scipy.fromstring
>>> (which is numpy.fromstring) when used to read integers from a text file.
>>> >>>>>
>>> >>>>>
>>> https://mail.scipy.org/pipermail/scipy-user/2015-October/036746.html
>>> >>>>>
>>> >>>>> fromstring() is bugging and inflexible for reading text files --
>>> and it is a very, very ugly mess of code. I dug into it a while back, and
>>> gave up -- just to much of a mess!
>>> >>>>>
>>> >>>>> So we really should completely re-implement it, or deprecate it. I
>>> doubt anyone is going to do a big refactor, so that means deprecating it.
>>> >>>>>
>>> >>>>> Also -- if we do want a fast read numbers from text files function
>>> (which would be nice, actually), it really should get a new name anyway.
>>> >>>>>
>>> >>>>> (and the hopefully coming new dtype system would make it easier to
>>> write cleanly)
>>> >>>>>
>>> >>>>> I'm not sure what deprecating something means, though -- have it
>>> raise a deprecation warning in the next version?
>>> >>>>>
>>> >>
>>> >> There was disc

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

[Benjamin Root]

Conda is for binary installs and largely targeted for end-users. This topic
pertains to source installs, and is mostly relevant to developers, testers,
and those who like to live on the bleeding edge of a particular project.

On Tue, Oct 27, 2015 at 10:31 AM, Edison Gustavo Muenz <
edisongust...@gmail.com> wrote:

> I'm sorry if this is out-of-topic, but I'm curious on why nobody mentioned
> Conda yet.
>
> Is there any particular reason for not using it?
>
> On Tue, Oct 27, 2015 at 11:48 AM, James E.H. Turner 
> wrote:
>
>> Apparently it is not well known that if you have a Python project
>>> source tree (e.g., a numpy checkout), then the correct way to install
>>> it is NOT to type
>>>
>>>python setup.py install   # bad and broken!
>>>
>>> but rather to type
>>>
>>>pip install .
>>>
>>
>> Though I haven't studied it exhaustively, it always seems to me that
>> pip is bad & broken, whereas python setup.py install does what I
>> expect (even if it's a mess internally). In particular, when
>> maintaining a distribution of Python packages, you try to have some
>> well-defined, reproducible build from source tarballs and then you
>> find that pip is going off and downloading stuff under the radar
>> without being asked (etc.). Stopping that can be a pain & I always
>> groan whenever some package insists on using pip. Maybe I don't
>> understand it well enough but in this role its dependency handling
>> is an unnecessary complication with no purpose. Just a comment that
>> not every installation is someone trying to get numpy on their
>> laptop...
>>
>> Cheers,
>>
>> James.
>>
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] deprecate fromstring() for text reading?

[Benjamin Root]

FWIW, when I needed a fast Fixed Width reader for a very large dataset last
year, I found that np.genfromtext() was faster than pandas' read_fwf().
IIRC, pandas' text reading code fell back to pure python for fixed width
scenarios.

On Fri, Oct 23, 2015 at 8:22 PM, Chris Barker - NOAA Federal <
chris.bar...@noaa.gov> wrote:

> Grabbing the pandas csv reader would be great, and I hope it happens
> sooner than later, though alas, I haven't the spare cycles for it either.
>
> In the meantime though, can we put a deprecation Warning in when using
> fromstring() on text files? It's really pretty broken.
>
> -Chris
>
> On Oct 23, 2015, at 4:02 PM, Jeff Reback  wrote:
>
>
>
> On Oct 23, 2015, at 6:49 PM, Nathaniel Smith  wrote:
>
> On Oct 23, 2015 3:30 PM, "Jeff Reback"  wrote:
> >
> > On Oct 23, 2015, at 6:13 PM, Charles R Harris 
> wrote:
> >
> >>
> >>
> >> On Thu, Oct 22, 2015 at 5:47 PM, Chris Barker - NOAA Federal <
> chris.bar...@noaa.gov> wrote:
> >>>
> >>>
>  I think it would be good to keep the usage to read binary data at
> least.
> >>>
> >>>
> >>> Agreed -- it's only the text file reading I'm proposing to deprecate.
> It was kind of weird to cram it in there in the first place.
> >>>
> >>> Oh, fromfile() has the same issues.
> >>>
> >>> Chris
> >>>
> >>>
>  Or is there a good alternative to `np.fromstring(,
> dtype=...)`?  -- Marten
> 
>  On Thu, Oct 22, 2015 at 1:03 PM, Chris Barker 
> wrote:
> >
> > There was just a question about a bug/issue with scipy.fromstring
> (which is numpy.fromstring) when used to read integers from a text file.
> >
> > https://mail.scipy.org/pipermail/scipy-user/2015-October/036746.html
> >
> > fromstring() is bugging and inflexible for reading text files -- and
> it is a very, very ugly mess of code. I dug into it a while back, and gave
> up -- just to much of a mess!
> >
> > So we really should completely re-implement it, or deprecate it. I
> doubt anyone is going to do a big refactor, so that means deprecating it.
> >
> > Also -- if we do want a fast read numbers from text files function
> (which would be nice, actually), it really should get a new name anyway.
> >
> > (and the hopefully coming new dtype system would make it easier to
> write cleanly)
> >
> > I'm not sure what deprecating something means, though -- have it
> raise a deprecation warning in the next version?
> >
> >>
> >> There was discussion at SciPy 2015 of separating out the text reading
> abilities of Pandas so that numpy could include it. We should contact Jeff
> Rebeck and see about moving that forward.
> >
> >
> > IIRC Thomas Caswell was interested in doing this :)
>
> When he was in Berkeley a few weeks ago he assured me that every night
> since SciPy he has dutifully been feeling guilty about not having done it
> yet. I think this week his paltry excuse is that he's "on his honeymoon" or
> something.
>
> ...which is to say that if someone has some spare cycles to take this over
> then I think that might be a nice wedding present for him :-).
>
> (The basic idea is to take the text reading backend behind pandas.read_csv
> and extract it into a standalone package that pandas could depend on, and
> that could also be used by other packages like numpy (among others -- I
> thing dato's SFrame package has a fork of this code as well?))
>
> -n
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
> I can certainly provide guidance on how/what to extract but don't have
> spare cycles myself for this :(
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Nansum function behavior

[Benjamin Root]

The change to nansum() happened several years ago. The main thrust of it
was to make the following consistent:

np.sum([])  # zero
np.nansum([np.nan])  # zero
np.sum([1])  # one
np.nansum([np.nan, 1])  # one

If you want to propagate masks and such, use masked arrays.
Ben Root


On Fri, Oct 23, 2015 at 12:45 PM, Charles Rilhac 
wrote:

> Hello,
>
> I noticed the change regarding nan function and especially nansum
> function. I think this choice is a big mistake. I know that Matlab and R
> have made this choice but it is illogical and counterintuitive.
>
> First argument is about logic. An arithmetic operation between Nothing and
> Nothing cannot make a figure or an object. Nothing + Object can be an
> object or something else, but from nothing, it cannot ensue something else
> than nothing. I hope you see what I mean.
>
> Secondly, it's counterintuitive and not convenient. Because, if you want
> to fill the result of nanfunction you can do that easily :
>
> a = np.array([[np.nan, np.nan], [1,np.nan]])
> a = np.nansum(a, axis=1)print(a)
> array([np.nan,  1.])
> a[np.isnan(a)] = 0
>
> Whereas, if the result is already filled with zero on NaN-full rows, you
> cannot replace the result of NaN-full rows by NaN easily. In the case
> above, you cannot because you lost information about NaN-full rows.
>
> I know it is tough to come back to a previous stage but I really think
> that it is wrong to absolutely fill with zeros the result of arithmetic
> operation containing NaN.
> Thank for your work guys ;-)
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Behavior of numpy.copy with sub-classes

[Benjamin Root]

In many other parts of numpy, calling the numpy function that had an
equivalent array method would result in the method being called. I would
certainly be surprised if the copy() method behaved differently from the
np.copy() function.

Now it is time for me to do some grepping of my code-bases...

On Mon, Oct 19, 2015 at 10:40 PM, Charles R Harris <
charlesr.har...@gmail.com> wrote:

>
>
> On Mon, Oct 19, 2015 at 8:28 PM, Nathan Goldbaum 
> wrote:
>
>>
>>
>> On Mon, Oct 19, 2015 at 7:23 PM, Jonathan Helmus 
>> wrote:
>>
>>> In GitHub issue #3474, a number of us have started a conversation on how
>>> NumPy's copy function should behave when passed an instance which is a
>>> sub-class of the array class.  Specifically, the issue began by noting that
>>> when a MaskedArray is passed to np.copy, the sub-class is not passed
>>> through but rather a ndarray is returned.
>>>
>>> I suggested adding a "subok" parameter which controls how sub-classes
>>> are handled and others suggested having the function call a copy method on
>>> duck arrays.  The "subok" parameter is implemented in PR #6509 as an
>>> example. Both of these options would change the API of numpy.copy and
>>> possibly break backwards compatibility.  Do others have an opinion of how
>>> np.copy should handle sub-classes?
>>>
>>> For a concrete example of this behavior and possible changes, what type
>>> should copy_x be in the following snippet:
>>>
>>> import numpy as np
>>> x = np.ma.array([1,2,3])
>>> copy_x = np.copy(x)
>>>
>>
>> FWIW, it looks like np.copy() is never used in our code to work with the
>> ndarray subclass we maintain in yt. Instead we use the copy() method much
>> more often, and that returns the appropriate type. I guess it makes sense
>> to have the type of the return value of np.copy() agree with the type of
>> the copy() member function.
>>
>> That said, breaking backwards compatibility here before numpy 2.0 might
>> very well break real code. It might be worth it search e.g. github for all
>> instances of np.copy() to see if they're dealing with subclasses.
>>
>
> The problem with github searches is that there are a ton of numpy forks.
> ISTR once finding a method to avoid them, but can't remember what is was.
> If anyone knows how to do that, I'd appreciate learning.
>
> Chuck
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Making datetime64 timezone naive

[Benjamin Root]

I'd be totally in support of switching to timezone naive form. While it
would be ideal that everyone stores their dates in UTC, the real world is
messy and most of the time, people are just loading dates as-is and don't
even care about timezones. I work on machines with different TZs, and I
hate it when I save a bunch of data on one machine in UTC, but then go to
view it on my local machine and everything is shifted. It gets even more
confusing around DST switches because it gets all mixed up.

Ben Root

On Mon, Oct 12, 2015 at 2:48 PM, Alexander Belopolsky 
wrote:

>
> On Mon, Oct 12, 2015 at 3:10 AM, Stephan Hoyer  wrote:
>
>> The tentative consensus from last year's discussion was that we should
>> make datetime64 timezone naive, like the standard library's
>> datetime.datetime
>
>
>
> If you are going to make datetime64 more like datetime.datetime, please
> consider adding the "fold" bit.  See PEP 495. [1]
>
> [1]: https://www.python.org/dev/peps/pep-0495/
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Let's move forward with the current governance document.

[Benjamin Root]

There is the concept of consensus-driven development, which centers on veto
rights. It does assume that all actors are driven by a common goal to
improve the project. For example, the fact that we didn't have consensus
back during the whole NA brouhaha was actually a good thing because IMHO
including NA into NumPy would have hurt the community more than it would
have helped.

Ben Root

On Mon, Oct 5, 2015 at 5:11 PM, Sturla Molden 
wrote:

> Nathaniel Smith  wrote:
>
> > Are you planning to go around vetoing things
>
> I don't consider myself qualified.
>
> > for ridiculous reasons and causing havoc?
>
> That would be unpolite.
>
> > And if not, then who is it that you're worried about?
>
> I am not sure :)
>
> I just envisioned a Roman patron shouting veto or a US senator
> filibustering. Expulsion would be the appropriate recation, yes :-)
>
>
> Sturla
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Python needs goto

[Benjamin Root]

Ow! Ow! Ow! I am just a meteorologist that has an obsession with looking up
unfamiliar technology terms.

I need a Tylenol...
Ben Root

On Fri, Sep 25, 2015 at 12:27 PM, Anne Archibald 
wrote:

> goto! and comefrom! Together with exceptions, threads, lambda, super,
> generators, and coroutines, all we're lacking is
> call-with-current-continuation for the full list of impenetrable
> control-flow constructs. Oh, and lisp-style resumable exception handling.
> (Suggested syntax: drop(exception, value) to return control to where the
> exception was raised and make the raise statement return value.)
>
> On Thu, Sep 24, 2015 at 8:42 PM Charles R Harris <
> charlesr.har...@gmail.com> wrote:
>
>> On Thu, Sep 24, 2015 at 12:13 PM, Yarko Tymciurak 
>> wrote:
>>
>>>
>>>
>>> I think there are more valid uses - I've read that "goto" basically is
>>> what a state machine does.
>>> Have a read of the brief implementation notes for "goto" in golang, for
>>> example.  Goto may not be unreasonable to use, just most people would
>>> abuse.  Sort of like "everyone shouldn't write assembly, but if you
>>> understand the machine, you can make good things happen".  Without
>>> compiler/interpreter checks, more responsibility rests on the coder to keep
>>> out of trouble.
>>>
>>
>> I would agree about state machines. When implemented using the standard
>> control flow constructs they always look a bit artificial.
>>
>>
> That depends what your "standard" control flow constructs are. Has anyone
> tried implementing a state machine using coroutines? They seem like a
> rather natural setup: each state is a coroutine that loops, doing the
> appropriate actions for the state and then handing control over to the
> coroutine for the next state.
>
> Anne
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Python needs goto

[Benjamin Root]

Most of the time when I wanted to use goto in my early days, I found that
breaks and continues were better and easier to understand. I will admit
that there are occasional nested if/elif/else code that get messy without a
goto. But which smells worse? A "goto" package or a complex if/elif/else?

Ben Root

On Thu, Sep 24, 2015 at 2:41 PM, Charles R Harris  wrote:

>
>
> On Thu, Sep 24, 2015 at 12:13 PM, Yarko Tymciurak 
> wrote:
>
>>
>>
>> On Thu, Sep 24, 2015 at 12:54 PM, Alexander Eberspächer <
>> alex.eberspaec...@gmail.com> wrote:
>>
>>> On 24.09.2015 13:25, Christophe Bal wrote:
>>>
>>> > Can you give an example where GOTO is useful ?
>>>
>>> I think those pieces of code are best understood with some humour..
>>>
>>> However, basically I can think two main causes for using goto:
>>>
>>> 1. Stop whatever your code is doing and jump towards the end of the
>>> program. However, this is mainly something useful for languages without
>>> exception handling and garbage collection.
>>>
>>> 2. Get out of something deeply nested. Also, this probably isn't very
>>> useful in Python as there's exception handling.
>>>
>>
>> I think there are more valid uses - I've read that "goto" basically is
>> what a state machine does.
>> Have a read of the brief implementation notes for "goto" in golang, for
>> example.  Goto may not be unreasonable to use, just most people would
>> abuse.  Sort of like "everyone shouldn't write assembly, but if you
>> understand the machine, you can make good things happen".  Without
>> compiler/interpreter checks, more responsibility rests on the coder to keep
>> out of trouble.
>>
>
> I would agree about state machines. When implemented using the standard
> control flow constructs they always look a bit artificial.
>
> Chuck
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Governance model request

[Benjamin Root]

To expand on Ryan's point a bit about recusal... this is why we have a
general policy against self-merging and why peer review is so valuable. A
ban on self-merging is much like recusal, and I think it is a fantastic
policy.

As for a BDFL, I used to like that idea having seen it work well for Linux
and Python, but I have found it at odds with the policy of recusal and no
self-merging. That said, as I am sure Thomas Caswell can attest, a
non-self-merging policy can result in a lot of ideas getting stalled,
waiting for review that may or may not happen. I don't know what the
solution is, but I am sympathetic to those who are apprehensive about a
BDFL -- regardless of who is in that role.

Ben Root


On Tue, Sep 22, 2015 at 2:20 PM, Stefan van der Walt 
wrote:

> Hi Travis
>
> On 2015-09-22 03:44:12, Travis Oliphant  wrote:
> > I'm actually offended that so many at BIDS seem eager to crucify my
> > intentions when I've done nothing but give away my time, my energy, my
> > resources, and my sleep to NumPy for many, many years.I guess if your
> > intent is to drive me away, then you are succeeding.
>
> I guess we've gone off the rails pretty far at this point, so let me at
> least take a step back, and make sure that you know that:
>
> - I have never doubted that your intensions for NumPy are anything but
>   good (I know they are!),
> - I *want* the community to be a welcoming place for companies to
>   contribute (otherwise, I guess I'd not be such a fervent supporter of
>   the scientific eco-system using the BSD license), and
> - I love your enthusiasm for the project.  After all, that is a big part
>   of what inspired me to become involved in the first place.
>
> My goal is not to spread uncertainty, fear nor doubt—if that was the
> perception left, I apologize.
>
> I'll re-iterate that I wanted to highlight a concern about the
> interactions of a (somewhat weakly cohesive) community and strong,
> driven personalities such as yourself backed by a formidable amount of
> development power.  No matter how good your intensions are, there are
> risks involved in this kind of interaction, and if we fail to even
> *admit* that, we are in trouble.
>
> Lest the above be read in a negative light again, let me state it
> up-front: *I don't think you will hijack the project, use it for your
> own gain, or attempt to do anything you don't believe to be in the best
> interest of NumPy.* What I'm saying is that we absolutely need to move
> forward in a way that brings everyone along, and makes everyone rest
> assured that their voice will be heard.
>
> Also, please know that I have not discussed these matters with Nathaniel
> behind the scenes, other than an informal hour-long discussion about his
> original governance proposal.  There is no BIDS conspiracy or attempts
> at crucifixion.  After all, you were an invited guest speaker at an
> event I organized this weekend, since I value your opinion and insights.
>
> Either way, let me again apologize if my suggested lack of insight hurt
> people's feelings.  I can only hope that, in educating me, we all learn
> a few lessons.
>
> Stéfan
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] 1.10.0rc1

[Benjamin Root]

Ok, I tested matplotlib master against numpy master, and there were no
errors. I did get a bunch of new deprecation warnings though such as:

/nas/home/broot/centos6/lib/python2.7/site-packages/matplotlib-1.5.dev1-py2.7-linux-x86_64.egg/matplotlib/colorbar.py:539:
VisibleDeprecationWarning: boolean index did not match indexed array along
dimension 0; dimension is 5 but corresponding boolean dimension is 3
  colors = np.asarray(colors)[igood]

The message isn't exactly clear. I suspect the problem is a shape mismatch,
like colors is 5x3, and igood is just 3 for some reason. Could somebody
shine some light on this, please?

Ben Root
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] 1.10.0rc1

[Benjamin Root]

Ok, I just wanted to make sure I understood the issue before going bug
hunting. Chances are, it has been a bug on our end for a while now. Just to
make sure, is the following valid?

arr = np.zeros((5, 3))
ind = np.array([True, True, True, False, True])
arr[ind]   # gives a 4x3 result

Running that at the REPL doesn't produce a warning, so i am guessing that
it is valid.

Ben Root

On Thu, Aug 27, 2015 at 10:44 AM, Sebastian Berg sebast...@sipsolutions.net
 wrote:

 On Do, 2015-08-27 at 08:04 -0600, Charles R Harris wrote:
 
 
  On Thu, Aug 27, 2015 at 7:52 AM, Benjamin Root ben.v.r...@gmail.com
  wrote:
 
 
  Ok, I tested matplotlib master against numpy master, and there
  were no errors. I did get a bunch of new deprecation warnings
  though such as:
 
 
  
 /nas/home/broot/centos6/lib/python2.7/site-packages/matplotlib-1.5.dev1-py2.7-linux-x86_64.egg/matplotlib/colorbar.py:539:
 VisibleDeprecationWarning: boolean index did not match indexed array along
 dimension 0; dimension is 5 but corresponding boolean dimension is 3
colors = np.asarray(colors)[igood]
 
 
  The message isn't exactly clear. I suspect the problem is a
  shape mismatch, like colors is 5x3, and igood is just 3 for
  some reason. Could somebody shine some light on this, please?
 
 
 
  IIRC, Boolean indexing would fill out the dimension, i.e., len 3 would
  be expanded to len 5 in this case. That behavior is deprecated.
 

 Yes, this is exactly the case, you have something like:

 arr = np.zeros((5, 3))
 ind = np.array([True, False, False])
 arr[ind, :]

 and numpy nowadays thinks that such code is likely a bug (when the ind
 is shorter than arr it is somewhat OK, the other way around gets more
 creepy). If you have an idea of how to make the error message clearer,
 or objections to the change, I am happy to hear it!

 - Sebastian


 
  Chuck
 
 
 
  ___
  NumPy-Discussion mailing list
  NumPy-Discussion@scipy.org
  http://mail.scipy.org/mailman/listinfo/numpy-discussion


 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] 1.10.0rc1

[Benjamin Root]

The reason why we don't have that extra slice is because we may not know
ahead of time that we are dealing with a 2D array. It could be a 1D array.
I guess we could use ellipses, but I wanted to make sure that the numpy
devs consider the above to be perfectly valid semantics because it is
entrenched in our codebase.

Ben Root

On Thu, Aug 27, 2015 at 11:33 AM, Sebastian Berg sebast...@sipsolutions.net
 wrote:

 On Do, 2015-08-27 at 11:15 -0400, Benjamin Root wrote:
  Ok, I just wanted to make sure I understood the issue before going bug
  hunting. Chances are, it has been a bug on our end for a while now.
  Just to make sure, is the following valid?
 
 
  arr = np.zeros((5, 3))
 
  ind = np.array([True, True, True, False, True])
 
  arr[ind]   # gives a 4x3 result
 
 
  Running that at the REPL doesn't produce a warning, so i am guessing
  that it is valid.
 

 Sure, that is perfect (you can add the slice and write `arr[ind, :]` to
 make it a bit more clear if you like I guess).

 - Sebastian


 
  Ben Root
 
 
  On Thu, Aug 27, 2015 at 10:44 AM, Sebastian Berg
  sebast...@sipsolutions.net wrote:
  On Do, 2015-08-27 at 08:04 -0600, Charles R Harris wrote:
  
  
   On Thu, Aug 27, 2015 at 7:52 AM, Benjamin Root
  ben.v.r...@gmail.com
   wrote:
  
  
   Ok, I tested matplotlib master against numpy master,
  and there
   were no errors. I did get a bunch of new deprecation
  warnings
   though such as:
  
  
 
 /nas/home/broot/centos6/lib/python2.7/site-packages/matplotlib-1.5.dev1-py2.7-linux-x86_64.egg/matplotlib/colorbar.py:539:
 VisibleDeprecationWarning: boolean index did not match indexed array along
 dimension 0; dimension is 5 but corresponding boolean dimension is 3
 colors = np.asarray(colors)[igood]
  
  
   The message isn't exactly clear. I suspect the
  problem is a
   shape mismatch, like colors is 5x3, and igood is
  just 3 for
   some reason. Could somebody shine some light on
  this, please?
  
  
  
   IIRC, Boolean indexing would fill out the dimension, i.e.,
  len 3 would
   be expanded to len 5 in this case. That behavior is
  deprecated.
  
 
  Yes, this is exactly the case, you have something like:
 
  arr = np.zeros((5, 3))
  ind = np.array([True, False, False])
  arr[ind, :]
 
  and numpy nowadays thinks that such code is likely a bug (when
  the ind
  is shorter than arr it is somewhat OK, the other way around
  gets more
  creepy). If you have an idea of how to make the error message
  clearer,
  or objections to the change, I am happy to hear it!
 
  - Sebastian
 
 
  
   Chuck
  
  
  
   ___
   NumPy-Discussion mailing list
   NumPy-Discussion@scipy.org
   http://mail.scipy.org/mailman/listinfo/numpy-discussion
 
 
 
  ___
  NumPy-Discussion mailing list
  NumPy-Discussion@scipy.org
  http://mail.scipy.org/mailman/listinfo/numpy-discussion
 
 
 
  ___
  NumPy-Discussion mailing list
  NumPy-Discussion@scipy.org
  http://mail.scipy.org/mailman/listinfo/numpy-discussion


 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] 1.10.0rc1

[Benjamin Root]

Just a data point, I just tested 1.9.0rc1 (built from source) with
matplotlib master, and things appear to be fine there. In fact, matplotlib
was built against 1.7.x (I was hunting down a regression), and worked
against the 1.9.0 install, so the ABI appears intact.

Cheers!
Ben Root

On Wed, Aug 26, 2015 at 9:52 AM, Charles R Harris charlesr.har...@gmail.com
 wrote:



 On Wed, Aug 26, 2015 at 7:32 AM, Charles R Harris 
 charlesr.har...@gmail.com wrote:



 On Wed, Aug 26, 2015 at 7:31 AM, Charles R Harris 
 charlesr.har...@gmail.com wrote:



 On Wed, Aug 26, 2015 at 7:11 AM, Antoine Pitrou solip...@pitrou.net
 wrote:

 On Tue, 25 Aug 2015 10:26:02 -0600
 Charles R Harris charlesr.har...@gmail.com wrote:
  Hi All,
 
  The silence after the 1.10 beta has been eerie. Consequently, I'm
 thinking
  of making a first release candidate this weekend. If you haven't yet
 tested
  the beta, please do so. It would be good to discover as many problems
 as we
  can before the first release.

 Has typing of ufunc parameters become much stricter? I can't find
 anything in the release notes, but see (1.10b1):

  arr = np.linspace(0, 5, 10)
  out = np.empty_like(arr, dtype=np.intp)
  np.round(arr, out=out)
 Traceback (most recent call last):
   File stdin, line 1, in module
   File
 /home/antoine/np110/lib/python3.4/site-packages/numpy/core/fromnumeric.py,
 line 2778, in round_
 return round(decimals, out)
 TypeError: ufunc 'rint' output (typecode 'd') could not be coerced to
 provided output parameter (typecode 'l') according to the casting rule
 ''same_kind''


 It used to work (1.9):

  arr = np.linspace(0, 5, 10)
  out = np.empty_like(arr, dtype=np.intp)
  np.round(arr, out=out)
 array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5])
  out
 array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5])


 The default casting mode has been changed. I think this has been raising
 a warning since 1.7 and was mentioned as a future change in 1.10, but you
 are right, it needs to be mentioned in the 1.10  release notes.


 Make that warned of in the 1.9.0 release notes.


 Here it is in 1.9.0 with deprecation warning made visible.
 ```
 In [3]: import warnings

 In [4]: warnings.simplefilter('always')

 In [5]: arr = np.linspace(0, 5, 10)

 In [6]: out = np.empty_like(arr, dtype=np.intp)

 In [7]: np.round(arr, out=out)
 /home/charris/.local/lib/python2.7/site-packages/numpy/core/fromnumeric.py:2640:
 DeprecationWarning: Implicitly casting between incompatible kinds. In a
 future numpy release, this will raise an error. Use casting=unsafe if
 this is intentional.
   return round(decimals, out)
 Out[7]: array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5])
 ```

 Chuck

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] 1.10.0rc1

[Benjamin Root]

Aw, crap... I looked at the list of tags and saw the rc1... I'll test again
in the morning Grumble, grumble...
On Aug 26, 2015 10:53 PM, Nathaniel Smith n...@pobox.com wrote:

 On Aug 26, 2015 7:03 PM, Benjamin Root ben.v.r...@gmail.com wrote:
 
  Just a data point, I just tested 1.9.0rc1 (built from source) with
 matplotlib master, and things appear to be fine there. In fact, matplotlib
 was built against 1.7.x (I was hunting down a regression), and worked
 against the 1.9.0 install, so the ABI appears intact.

 1.9, or 1.10?

 -n

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Development workflow (not git tutorial)

[Benjamin Root]

I used to be a huge advocate for the develop mode, but not anymore. I
have run into way too many Heisenbugs that would clear up if I nuked my
source tree and re-clone.

I should also note that there is currently an open issue with pip install
-e and namespace packages. This has been reported to matplotlib with
regards to mpl_toolkits. Essentially, if you have namespace packages, it
doesn't get installed correctly in this mode, and python won't find them.

On Fri, Aug 14, 2015 at 12:12 PM, Chris Barker chris.bar...@noaa.gov
wrote:

 On Thu, Aug 13, 2015 at 11:25 AM, Stefan van der Walt 
 stef...@berkeley.edu wrote:

 (for
  example python setup.py develop, although suggested by
  setup.py itself, claims that develop is not a command).


 develop is a command provided by setuptools, not distutils itself.

 I find it absolutely invaluable -- it is THE way to go when actively
 working on any package under development.

 if numpy doesn't currently use setuptools, it probably should (though
 maybe it's gets messy with numpy's distutils extensions...)

 Nowadays, you can use

 pip install -e .


 pip injects setuptools into the mix -- so this may be develope mode with
 a different name. but yes, a fine option for a package that doesn't use
 setuptools out of the box.

 -Chris

 --

 Christopher Barker, Ph.D.
 Oceanographer

 Emergency Response Division
 NOAA/NOS/ORR(206) 526-6959   voice
 7600 Sand Point Way NE   (206) 526-6329   fax
 Seattle, WA  98115   (206) 526-6317   main reception

 chris.bar...@noaa.gov

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Multiarray API size mismatch 301 302?

[Benjamin Root]

Did you do a git clean -fxd before re-installing?

On Thu, Aug 13, 2015 at 2:34 PM, Sebastian Berg sebast...@sipsolutions.net
wrote:

 Hey,

 just for hacking/testing, I tried to add to shape.c:


 /*NUMPY_API
  *
  * Checks if memory overlap exists
  */
 NPY_NO_EXPORT int
 PyArray_ArraysShareMemory(PyArrayObject *arr1, PyArrayObject *arr2, int
 work) {
 return solve_may_share_memory(arr1, arr2, work);
 }



 and to numpy_api.py:

 # End 1.10 API
 'PyArray_ArraysShareMemory':(301,),


 But I am getting the error:

   File numpy/core/code_generators/generate_numpy_api.py, line 230, in
 do_generate_api
 (len(multiarray_api_dict), len(multiarray_api_index)))
 AssertionError: Multiarray API size mismatch 301 302

 It is puzzling me, so anyone got a quick idea?

 - Sebastian

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] np.in1d() sets, bug?

[Benjamin Root]

 Not really, it is simply because ``np.asarray(set([1, 2, 3]))``
 returns an object array

Holy crap! To be pedantic, it looks like it turns it into a numpy scalar,
but still! I wouldn't have expected np.asarray() on a set (or dictionary,
for that matter) to work because order is not guaranteed. Is this expected
behavior?

Digging into the implementation of in1d(), I can see now how passing a
set() wouldn't be useful at all (as an aside, pretty clever algorithm). I
know sets aren't array-like, but the code that used this seemed to work at
first, and this problem wasn't revealed until I created some unit tests to
exercise some possible corner cases. Silently producing possibly erroneous
results is dangerous. Don't know if better documentation or some better
sanity checking would be called for here, though.

Ben Root


On Mon, Aug 10, 2015 at 1:10 PM, Sebastian Berg sebast...@sipsolutions.net
wrote:

 On Mo, 2015-08-10 at 12:09 -0400, Benjamin Root wrote:
  Just came across this one today:
 
   np.in1d([1], set([0, 1, 2]), assume_unique=True)
  array([ False], dtype=bool)
 
   np.in1d([1], [0, 1, 2], assume_unique=True)
 
  array([ True], dtype=bool)
 
 
  I am assuming this has something to do with the fact that order is not
  guaranteed with set() objects? I was kind of hoping that setting
  assume_unique=True would be sufficient to overcome that problem.
  Should sets be rejected as an error?
 

 Not really, it is simply because ``np.asarray(set([1, 2, 3]))``
 returns an object array and 1 is not the same as ``set([1, 2, 3])``.

 I think earlier numpy versions may have had short cuts for short lists
 or something so this may have worked in some cases

 - Sebastian


 
  This was using v1.9.0
 
 
  Cheers!
 
  Ben Root
 
  ___
  NumPy-Discussion mailing list
  NumPy-Discussion@scipy.org
  http://mail.scipy.org/mailman/listinfo/numpy-discussion


 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] np.in1d() sets, bug?

[Benjamin Root]

Just came across this one today:

 np.in1d([1], set([0, 1, 2]), assume_unique=True)
array([ False], dtype=bool)
 np.in1d([1], [0, 1, 2], assume_unique=True)
array([ True], dtype=bool)

I am assuming this has something to do with the fact that order is not
guaranteed with set() objects? I was kind of hoping that setting
assume_unique=True would be sufficient to overcome that problem. Should
sets be rejected as an error?

This was using v1.9.0

Cheers!
Ben Root
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy where and dtype in 1.9

[Benjamin Root]

What a coincidence! A very related bug just got re-opened today at my
behest: https://github.com/numpy/numpy/issues/5095

Not the same, but I wouldn't be surprised if it stems from the same
sources. The short of it...

np.where(x, 0, x)

where x is a masked array, will return a masked array in 1.8.2 and earlier,
but will return a regular numpy array in 1.9 and above (drops the mask).
That bug took a long time for me to track down!

Ben Root

On Wed, Jul 29, 2015 at 5:16 PM, Nathan Jensen ndjen...@gmail.com wrote:

 Hi,

 The numpy.where() function was rewritten in numpy 1.9 to speed it up.  I
 traced it to this changeset.
 https://github.com/numpy/numpy/commit/593e3c30c24f0c61a271dc883c614724d7a57e1e

 The weird thing is the 1.9 behavior changed the resulting dtype in some
 situations when using scalar values as the second or third argument.  To
 try and illustrate, I wrote a simple test script and ran it against both
 numpy 1.7 and 1.9.  Here are the results:

 2.7.9 (default, Jul 25 2015, 03:06:43)
 [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)]
 * numpy version 1.7.2 *

 === testing numpy.where with NaNs ===
 numpy.where([True], numpy.float32(1.0), numpy.NaN).dtype
 float64
 numpy.where([True], [numpy.float32(1.0)], numpy.NaN).dtype
 float32
 numpy.where([True], numpy.float32(1.0), [numpy.NaN]).dtype
 float64
 numpy.where([True], [numpy.float32(1.0)], [numpy.NaN]).dtype
 float64


 === testing numpy.where with integers ===
 numpy.where([True], [numpy.float32(1.0)], 65535).dtype
 float32
 numpy.where([True], [numpy.float32(1.0)], 65536).dtype
 float32
 numpy.where([True], [numpy.float32(1.0)], -32768).dtype
 float32
 numpy.where([True], [numpy.float32(1.0)], -32769).dtype
 float32



 2.7.9 (default, Mar 10 2015, 09:26:44)
 [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)]
 * numpy version 1.9.2 *

 === testing numpy.where with NaNs ===
 numpy.where([True], numpy.float32(1.0), numpy.NaN).dtype
 float64
 numpy.where([True], [numpy.float32(1.0)], numpy.NaN).dtype
 float32
 numpy.where([True], numpy.float32(1.0), [numpy.NaN]).dtype
 float64
 numpy.where([True], [numpy.float32(1.0)], [numpy.NaN]).dtype
 float64


 === testing numpy.where with integers ===
 numpy.where([True], [numpy.float32(1.0)], 65535).dtype
 float32
 numpy.where([True], [numpy.float32(1.0)], 65536).dtype
 float64
 numpy.where([True], [numpy.float32(1.0)], -32768).dtype
 float32
 numpy.where([True], [numpy.float32(1.0)], -32769).dtype
 float64



 Regarding the NaNs with where, the behavior does not differ between 1.7
 and 1.9.  But it's a little odd that the one scenario returns a dtype of
 float32 where the other three scenarios return dtype of float64.  I'm not
 sure if that was intentional or a bug?

 Regarding using ints with where, in 1.7 the resulting dtype is consistent
 but then in 1.9 the resulting dtype is influenced by the value of the int.
 It appears it is somehow related to whether the value falls within the
 range of a short.  I'm not sure if this was a side effect of the
 performance improvement or was intentional?

 At the very least I think this change in where() should probably be noted
 in the release notes for 1.9.  Our project saw an increase in memory usage
 with 1.9 due to where(cond, array, scalar) returning arrays of dtype
 float64 when using scalars not within that limited range.

 I've attached my simple script if you're interested in running it.

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy V1.4.1 software license

[Benjamin Root]

I think there is a misunderstanding. What you are seeing is documentation
on how to use f90 for compiling, which then outputs some stuff to the
terminal, which is being shown in the documentation. We don't actually
include any compilers in numpy.

Ben Root

On Wed, Jul 8, 2015 at 11:53 AM, Nguyen, Theresa X 
theresa.x.ngu...@lmco.com wrote:

  Dear Sir/Madam,



 Currently, I have a chance to review the software Numpy
 V1.4.1,  I found some software that they had the Copyright but *no License*
 show as below:



 1.   File: … numpy-1.4.1\numpy\distutils\fcompiler\
 absoft.py



# on windows: f90 -V -c dummy.f

# f90: Copyright Absoft Corporation 1994-1998 mV2; Cray
 Research, Inc. 1994-1996 CF90 (2.x.x.x  f36t87) Version 2.3 Wed Apr 19,
 2006  13:05:16



# samt5735(8)$ f90 -V -c dummy.f

# f90: Copyright Absoft Corporation 1994-2002; Absoft Pro
 FORTRAN Version 8.0



 2.   File:… numpy-1.4.1\numpy\distutils\fcompiler\intel.py



 #Intel(R) Fortran Itanium(R) Compiler for Itanium(R)-based applications

 #Version 9.1Build 20060928 Package ID: l_fc_c_9.1.039

 #Copyright (C) 1985-2006 Intel Corporation.  All rights reserved.

 #30 DAY EVALUATION LICENSE





 Would you please provide a License for these files.



 Numpy V1.4.1 is released under BSD Style License.  I wonder if these
 software can be released under the same license?



 Thank you very much for your assistant.





 Best Regards,
 *Theresa X. Nguyen*





 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] DEP: Deprecate boolean array indices with non-matching shape #4353

[Benjamin Root]

On Thu, Jun 4, 2015 at 10:41 PM, Nathaniel Smith n...@pobox.com wrote:

 My comment was about the second type. Are your comments about the
 second type? The second type definitely does not produce a flattened
 array:



I was talking about the second type in that I never even knew it existed.
My understanding of boolean indexing has always been that it flattens, so
the second type is a surprise to me.

Ben Root
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] DEP: Deprecate boolean array indices with non-matching shape #4353

[Benjamin Root]

On Thu, Jun 4, 2015 at 9:04 PM, Nathaniel Smith n...@pobox.com wrote:

 On Thu, Jun 4, 2015 at 5:57 PM, Nathaniel Smith n...@pobox.com wrote:

 One place where the current behavior is particularly baffling and annoying
 is when you have multiple boolean masks in the same indexing operation. I
 think everyone would expect this to index separately on each axis (outer
 product indexing style, like slices do), and that's really the only useful
 interpretation, but that's not what it does...:


As a huge user of boolean indexes, I have never expected this to work in
any way, shape or form. I don't think it works in matlab (but someone
should probably check that), so you wouldn't have to worry about converts
missing a feature from there. I have always been told that boolean indexing
will produce a flattened array, and I wouldn't want to be dealing with
magic when the array does not match up right.

Now, what if the boolean array is broadcastable (dimension-wise, not
length-wise)? I do see some uses there. Modulo that, my vote is to
deprecate.

Ben Root
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Verify your sourceforge windows installer downloads

[Benjamin Root]

Speaking from the matplotlib project, our binaries are substantial due to
our suite of test images. Pypi worked with us on relaxing size constraints.
Also, I think the new cheese shop/warehouse server they are using scales
better, so size is not nearly the same concern as before.

Ben Root
On May 29, 2015 1:43 AM, Todd toddr...@gmail.com wrote:

 On May 28, 2015 7:06 PM, David Cournapeau courn...@gmail.com wrote:
  On Fri, May 29, 2015 at 2:00 AM, Andrew Collette 
 andrew.colle...@gmail.com wrote:
 
  In any case I've always been surprised that NumPy is distributed
  through SourceForge, which has been sketchy for years now. Could it
  simply be hosted on PyPI?
 
 
  They don't accept arbitrary binaries like SF does, and some of our
 installer formats can't be uploaded there.
 
  David

 Is that something that could be fixed? Has anyone asked the pypi
 maintainers whether they could change those rules, either in general or by
 granting exceptions on a case-by-case basis to projects that have proven
 track records and importance?

 It would seem to me that if the rules on pypi are forcing critical
 projects like numpy to host elsewhere, then the rules are flawed and are
 preventing pypi from serving is intended purpose.

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Two questions about PEP 465 dot product

[Benjamin Root]

That assumes that the said recently-confused ever get to the point of
understanding it... and I personally don't do much matrix math work, so I
don't have the proper mental context. I just know that coworkers are going
to be coming to me asking questions because I am the de facto python guy.
So, having a page I can point them to would be extremely valuable.

Ben Root

On Fri, May 22, 2015 at 4:05 PM, Nathaniel Smith n...@pobox.com wrote:

 On May 22, 2015 11:34 AM, Benjamin Root ben.r...@ou.edu wrote:
 
  At some point, someone is going to make a single documentation page
 describing all of this, right? Tables, mathtex, and such? I get woozy
 whenever I see this discussion go on.

 That does seem like a good idea, doesn't it. Following the principle that
 recently-confused users write the best docs, any interest in taking a shot
 at writing such a thing?

 -n

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Two questions about PEP 465 dot product

[Benjamin Root]

Then add in broadcasting behavior...

On Fri, May 22, 2015 at 4:58 PM, Nathaniel Smith n...@pobox.com wrote:

 On May 22, 2015 1:26 PM, Benjamin Root ben.r...@ou.edu wrote:
 
  That assumes that the said recently-confused ever get to the point of
 understanding it...

 Well, I don't think it's that complicated really. For whatever that's
 worth :-). My best attempt is here, anyway:

   https://www.python.org/dev/peps/pep-0465/#semantics

 The short version is, for 1d and 2d inputs it acts just like dot(). For
 higher dimension inputs like (i, j, n, m) it acts like any other gufunc
 (e.g., everything in np.linalg) -- it treats this as an i-by-j stack of
 n-by-m matrices and is vectorized over the i, j dimensions. And 0d inputs
 are an error.

 -n

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Two questions about PEP 465 dot product

[Benjamin Root]

At some point, someone is going to make a single documentation page
describing all of this, right? Tables, mathtex, and such? I get woozy
whenever I see this discussion go on.

Ben Root

On Fri, May 22, 2015 at 2:23 PM, Nathaniel Smith n...@pobox.com wrote:

 On May 22, 2015 11:00 AM, Alexander Belopolsky ndar...@mac.com wrote:
 
 
  On Thu, May 21, 2015 at 9:37 PM, Nathaniel Smith n...@pobox.com wrote:
  
   .. there's been some discussion of the possibility of
 
   adding specialized gufuncs for broadcasted vector-vector,
   vector-matrix, matrix-vector multiplication, which wouldn't do the
   magic vector promotion that dot and @ do.
 
 
  This would be nice.  What I would like to see is some consistency
 between multi-matrix
  support in linalg methods and dot.
 
  For example, when A is a matrix and b is a vector and
 
  a = linalg.solve(A, b)
 
  then
 
  dot(A, a) returns b, but if either or both A and b are stacks, this
 invariant does not hold.  I would like
  to see a function (say xdot) that I can use instead of dot and have
 xdot(A, a) return b whenever a = linalg.solve(A, b).

 I believe this equivalence holds if xdot(x, y) = x @ y, because solve()
 does follow the pep 465 semantics for shape handling. Or at least, it's
 intended to. Of course we will also expose pep 465 matmul semantics under
 some name that doesn't require the new syntax (probably not xdot though
 ;-)).

  Similarly, if w,v =  linalg.eig(A), then dot(A,v) returns w * v, but
 only if A is 2d.

 Again A @ v I believe does the right thing, though I'm not positive -- you
 might need a swapaxes or matvec or something. Let us know if you work it
 out :-).

 Note that it still won't be equivalent to w * v because w * v doesn't
 broadcast the way you want :-). You need w[..., np.newaxis, :] * v, I think.

 -n

 
 
  ___
  NumPy-Discussion mailing list
  NumPy-Discussion@scipy.org
  http://mail.scipy.org/mailman/listinfo/numpy-discussion
 

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Bug in np.nonzero / Should index returning functions return ndarray subclasses?

[Benjamin Root]

On Sat, May 9, 2015 at 4:03 PM, Nathaniel Smith n...@pobox.com wrote:

 Not sure what this has to do with Jaime's post about nonzero? There is
 indeed a potential question about what 3-argument where() should do with
 subclasses, but that's effectively a different operation entirely and to
 discuss it we'd need to know things like what it historically has done and
 why that was causing problems.



Because my train of thought started at np.nonzero(), which I have always
just mentally mapped to np.where(), and then... squirrel!

Indeed, np.where() has no bearing here.

Ben Root
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Bug in np.nonzero / Should index returning functions return ndarray subclasses?

[Benjamin Root]

Absolutely, it should be writable. As for subclassing, that might be messy.
Consider the following:

inds = np.where(data  5)

In that case, I'd expect a normal, bog-standard ndarray because that is
what you use for indexing (although pandas might have a good argument for
having it return one of their special indexing types if data was a pandas
array...). Next:

foobar = np.where(data  5, 1, 2)

Again, I'd expect a normal, bog-standard ndarray because the scalar
elements are very simple. This question gets very complicated when
considering array arguments. Consider:

merged_data = np.where(data  5, data, data2)

So, what should merged_data be? If both data and data2 are the same
types, then it would be reasonable to return the same type, if possible.
But what if they aren't the same? Maybe use array_priority to determine the
return type? Or, perhaps it does make sense to say sod it all and always
return an ndarray?

I don't know the answer. I do find it interesting that the result from a
multi-dimensional array is not writable. I don't know why I have never
encountered that.


Ben Root


On Sat, May 9, 2015 at 2:42 PM, Nathaniel Smith n...@pobox.com wrote:

 On May 9, 2015 10:48 AM, Jaime Fernández del Río jaime.f...@gmail.com
 wrote:
 
  There is a reported bug (issue #5837) regarding different returns from
 np.nonzero with 1-D vs higher dimensional arrays. A full summary of the
 differences can be seen from the following output:
 
   class C(np.ndarray): pass
  ...
   a = np.arange(6).view(C)
   b = np.arange(6).reshape(2, 3).view(C)
   anz = a.nonzero()
   bnz = b.nonzero()
 
   type(anz[0])
  type 'numpy.ndarray'
   anz[0].flags
C_CONTIGUOUS : True
F_CONTIGUOUS : True
OWNDATA : True
WRITEABLE : True
ALIGNED : True
UPDATEIFCOPY : False
   anz[0].base
 
   type(bnz[0])
  class '__main__.C'
   bnz[0].flags
C_CONTIGUOUS : False
F_CONTIGUOUS : False
OWNDATA : False
WRITEABLE : False
ALIGNED : True
UPDATEIFCOPY : False
   bnz[0].base
  array([[0, 1],
 [0, 2],
 [1, 0],
 [1, 1],
 [1, 2]])
 
  The original bug report was only concerned with the non-writeability of
 higher dimensional array returns, but there are more differences: 1-D
 always returns an ndarray that owns its memory and is writeable, but higher
 dimensional arrays return views, of the type of the original array, that
 are non-writeable.
 
  I have a branch that attempts to fix this by making both 1-D and n-D
 arrays:
  return a view, never the base array,

 This doesn't matter, does it? View isn't a thing, only view of is
 meaningful. And in this case, none of the returned arrays share any memory
 with any other arrays that the user has access to... so whether they were
 created as a view or not should be an implementation detail that's
 transparent to the user?

  return an ndarray, never a subclass, and
  return a writeable view.
  I guess the most controversial choice is #2, and in fact making that
 change breaks a few tests. I nevertheless think that all of the index
 returning functions (nonzero, argsort, argmin, argmax, argpartition) should
 always return a bare ndarray, not a subclass. I'd be happy to be corrected,
 but I can't think of any situation in which preserving the subclass would
 be needed for these functions.

 I also can't see any logical reason why the return type of these functions
 has anything to do with the type of the inputs. You can index me with my
 phone number but my phone number is not a person. OTOH logic and ndarray
 subclassing don't have much to do with each other; the practical effect is
 probably more important. Looking at the subclasses I know about (masked
 arrays, np.matrix, and astropy quantities), though, I also can't see much
 benefit in copying the subclass of the input, and the fact that we were
 never consistent about this suggests that people probably aren't depending
 on it too much.

 So in summary my feeling is: +1 to making then writable, no objection to
 the view thing (though I don't see how it matters), and provisional +1 to
 consistently returning ndarray (to be revised if the people who use the
 subclassing functionality disagree).

 -n

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] read not byte aligned records

[Benjamin Root]

I have been very happy with the bitarray package. I don't know if it is
faster than bitstring, but it is worth a mention. Just watch out for any
hashing operations on its objects, it doesn't seem to do them right (set(),
dict(), etc...), but comparison operations work just fine.

Ben Root
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] how to set a fixed sized dtype suitable for bitwise operations

[Benjamin Root]

I have a need to have a numpy array of 17 byte (more specifically, at least
147 bits) values that I would be doing some bit twiddling on. I have found
that doing a dtype of i17 yields a dtype of int32, which is completely
not what I intended. Doing 'u17' gets an data type not understood. I have
tried 'a17', but then bitwise_or() and left_shift() do not work (returns
NotImplemented).

How should I be going about this?

Cheers!
Ben Root
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] how to set a fixed sized dtype suitable for bitwise operations

[Benjamin Root]

Yeah, I am not seeing any way around it at the moment. I guess I will have
to use the bitarray package for now. I was hoping for some fast per-element
processing, but at the moment, I guess I will have to sacrifice that just
to have something that worked correctly.

Ben Root

On Tue, Apr 28, 2015 at 10:19 AM, Jaime Fernández del Río 
jaime.f...@gmail.com wrote:

 On Tue, Apr 28, 2015 at 7:00 AM, Benjamin Root ben.r...@ou.edu wrote:

 I have a need to have a numpy array of 17 byte (more specifically, at
 least 147 bits) values that I would be doing some bit twiddling on. I have
 found that doing a dtype of i17 yields a dtype of int32, which is
 completely not what I intended. Doing 'u17' gets an data type not
 understood. I have tried 'a17', but then bitwise_or() and left_shift() do
 not work (returns NotImplemented).


 How should I be going about this?


 The correct type to use would be a void dtype:

  dt = np.dtype('V17')
  dt.itemsize
 17

 Unfortunately, it does not support bitwise operations either, which seems
 like an oddity to me:

  a = np.empty(2, dt)
  a[0] = 'abcdef'
  a[1] = bytearray([64, 56, 78])
  a[0] | a[1]
 Traceback (most recent call last):
   File stdin, line 1, in module
 TypeError: unsupported operand type(s) for |: 'numpy.void' and 'numpy.void'

 Any fundamental reason for this?

 Jaime

 --
 (\__/)
 ( O.o)
 (  ) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes
 de dominación mundial.

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Automatic number of bins for numpy histograms

[Benjamin Root]

Then you can set about convincing matplotlib and friends to
use it by default

Just to note, this proposal was originally made over in the matplotlib
project. We sent it over here where its benefits would have wider reach.
Matplotlib's plan is not to change the defaults, but to offload as much as
possible to numpy so that it can support these new features if they are
available. We might need to do some input validation so that users running
older version of numpy can get a sensible error message.

Cheers!
Ben Root


On Tue, Apr 14, 2015 at 7:12 PM, Nathaniel Smith n...@pobox.com wrote:

 On Mon, Apr 13, 2015 at 8:02 AM, Neil Girdhar mistersh...@gmail.com
 wrote:
  Can I suggest that we instead add the P-square algorithm for the dynamic
  calculation of histograms?
  (
 http://pierrechainais.ec-lille.fr/Centrale/Option_DAD/IMPACT_files/Dynamic%20quantiles%20calcultation%20-%20P2%20Algorythm.pdf
 )
 
  This is already implemented in C++'s boost library
  (
 http://www.boost.org/doc/libs/1_44_0/boost/accumulators/statistics/extended_p_square.hpp
 )
 
  I implemented it in Boost Python as a module, which I'm happy to share.
  This is much better than fixed-width histograms in practice.  Rather than
  adjusting the number of bins, it adjusts what you really want, which is
 the
  resolution of the bins throughout the domain.

 This definitely sounds like a useful thing to have in numpy or scipy
 (though if it's possible to do without using Boost/C++ that would be
 nice). But yeah, we should leave the existing histogram alone (in this
 regard) and add a new name for this like adaptive_histogram or
 something. Then you can set about convincing matplotlib and friends to
 use it by default :-)

 -n

 --
 Nathaniel J. Smith -- http://vorpus.org
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Advanced indexing: fancy vs. orthogonal

[Benjamin Root]

The distinction that boolean indexing has over the other 2 methods of
indexing is that it can guarantee that it references a position at most
once. Slicing and scalar indexes are also this way, hence why these methods
allow for in-place assignments. I don't see boolean indexing as an
extension of orthogonal indexing because of that.

Ben Root

On Thu, Apr 2, 2015 at 2:41 PM, Stephan Hoyer sho...@gmail.com wrote:

 On Thu, Apr 2, 2015 at 11:03 AM, Eric Firing efir...@hawaii.edu wrote:

 Fancy indexing is a horrible design mistake--a case of cleverness run
 amok.  As you can read in the Numpy documentation, it is hard to
 explain, hard to understand, hard to remember.


 Well put!

 I also failed to correct predict your example.


 So I think you should turn the question around and ask, What is the
 actual real-world use case for fancy indexing?  How often does real
 code rely on it?


 I'll just note that Indexing with a boolean array with the same shape as
 the array (e.g., x[x  0] when x has greater than 1 dimension) technically
 falls outside a strict interpretation of orthogonal indexing. But there's
 not any ambiguity in adding that as an extension to orthogonal indexing
 (which otherwise does not allow ndim  1), so I think your point still
 stands.

 Stephan

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] IDE's for numpy development?

[Benjamin Root]

mixed C and python development? I would just wait for the Jupyter folks to
create IC and maybe even IC++!

On Wed, Apr 1, 2015 at 12:04 PM, Charles R Harris charlesr.har...@gmail.com
 wrote:

 Hi All,

 In a recent exchange Mark Wiebe suggested that the lack of support for
 numpy development in Visual Studio might limit the number of developers
 attracted to the project. I'm a vim/console developer myself and make no
 claim of familiarity with modern development tools, but I wonder if such
 tools might now be available for Numpy. A quick google search turns up a
 beta plugin for Visual Studio, https://pytools.codeplex.com/, and there
 is an xcode IDE for the mac that apparently offers some Python support. The
 two things that I think are required are: 1) support for mixed C, python
 developement and 2) support for building and testing numpy. I'd be
 interested in information from anyone with experience in using such an IDE
 and ideas of how Numpy might make using some of the common IDEs easier.

 Thoughts?

 Chuck

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Adding 'where' to ufunc methods?

[Benjamin Root]

Another usecase would be for MaskedArrays. ma.masked_array.min() wouldn't
have to make a copy anymore (there is a github issue about that). It could
just pass its mask into the where= argument of min() and be done with it.
Problem would be generalizing situations where where= effectively results
in nowhere.

Cheers!
Ben Root

On Wed, Apr 1, 2015 at 2:34 PM, Jaime Fernández del Río 
jaime.f...@gmail.com wrote:

 This question on StackOverflow:


 http://stackoverflow.com/questions/29394377/minimum-of-numpy-array-ignoring-diagonal

 Got me thinking that I had finally found a use for the 'where' kwarg of
 ufuncs. Unfortunately it is only provided for the ufunc itself, but not for
 any of its methods.

 Is there any fundamental reason these were not implemented back in the
 day? Any frontal opposition to having them now?

 Jaime

 --
 (\__/)
 ( O.o)
 (  ) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes
 de dominación mundial.

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Do you find this behavior surprising?

[Benjamin Root]

Ah, *that* example is surprising to me. Regardless of whether it is a C int
of the PyArrayObject struct or not, the way it is presented at the python
code level should make sense. From my perspective, a.flags is a mutable
object of some sort. Updating it should act like a mutable object, not some
other magical object that doesn't work like anything else in python.

Ben Root

On Wed, Mar 25, 2015 at 5:11 PM, Jaime Fernández del Río 
jaime.f...@gmail.com wrote:

 On Wed, Mar 25, 2015 at 1:45 PM, Benjamin Root ben.r...@ou.edu wrote:

 I fail to see the wtf.


 flags = a.flags

 So, flags at this point is just an alias to a.flags, just like any
 other variable in python

 flags.writeable = False would then be equivalent to a.flags.writeable
 = False. There is nothing numpy-specific here. a.flags is mutable object.
 This is how Python works.

 Ben Root


 Ah, yes indeed. If you think of it that way it does make all the sense in
 the world.

 But of course that is not what is actually going on, as flags is a single
 C int of the PyArrayObject struct, and a.flags is just a proxy built from
 it, and great coding contortions have to be made to have changes to the
 proxy rewritten into the owner array.

 I guess then the surprising behavior is this other one, which was the one
 I (wrongly) expected intuitively:

  a = np.arange(10)
  flags = a.flags
  a.flags.writeable = False
  flags
   C_CONTIGUOUS : True
   F_CONTIGUOUS : True
   OWNDATA : True
   WRITEABLE : True
   ALIGNED : True
   UPDATEIFCOPY : False

 This could be fixed to work properly, although it is probably not worth
 worrying much.

 Properties of properties are weird...

 Jaime

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Do you find this behavior surprising?

[Benjamin Root]

I fail to see the wtf.

flags = a.flags

So, flags at this point is just an alias to a.flags, just like any
other variable in python

flags.writeable = False would then be equivalent to a.flags.writeable =
False. There is nothing numpy-specific here. a.flags is mutable object.
This is how Python works.

Ben Root



On Wed, Mar 25, 2015 at 4:36 PM, Jaime Fernández del Río 
jaime.f...@gmail.com wrote:


  import numpy as np
  a = np.arange(10)
  flags = a.flags
  flags
   C_CONTIGUOUS : True
   F_CONTIGUOUS : True
   OWNDATA : True
   WRITEABLE : True
   ALIGNED : True
   UPDATEIFCOPY : False
  flags.writeable = False
  a.flags
   C_CONTIGUOUS : True
   F_CONTIGUOUS : True
   OWNDATA : True
   WRITEABLE : False  --- WTF!!??
   ALIGNED : True
   UPDATEIFCOPY : False

 I understand why this is happening, and that there is no other obvious way
 to make

 a.flags.writeable = False

 work than to have the return of a.flags linked to a under the hood.

 But I don't think this is documented anywhere, and wonder if perhaps it
 should.

 Jaime

 --
 (\__/)
 ( O.o)
 (  ) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes
 de dominación mundial.

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy 1.10

[Benjamin Root]

Release minion? Sounds a lot like an academic minion:
https://twitter.com/academicminions

On Fri, Mar 13, 2015 at 2:51 AM, Ralf Gommers ralf.gomm...@gmail.com
wrote:



 On Fri, Mar 13, 2015 at 7:29 AM, Jaime Fernández del Río 
 jaime.f...@gmail.com wrote:

 On Thu, Mar 12, 2015 at 10:16 PM, Charles R Harris 
 charlesr.har...@gmail.com wrote:



 On Sun, Mar 8, 2015 at 3:43 PM, Ralf Gommers ralf.gomm...@gmail.com
 wrote:



 On Sat, Mar 7, 2015 at 12:40 AM, Charles R Harris 
 charlesr.har...@gmail.com wrote:

 Hi All,

 Time to start thinking about numpy 1.10.


 Sounds good. Do we have a volunteer for release manager already?


 I guess it is my turn, unless someone else wants the experience.


 What does a release manager do? I will eventually want to be able to tell
 my grandchildren that I once managed a numpy release, but am not sure if I
 can successfully handle it on my own right now. I will probably need to up
 my git foo, which is nothing to write home about...

 Maybe for this one I can sign up for release minion, so you have someone
 to offload menial tasks?


 I have no doubt that you can do this job well right now - you are vastly
 more experienced than I was when I picked up that role. It's not rocket
 science.

 I have to run now, but here's a start to give you an idea of what it
 entails (some details and version numbers may be slightly outdated):
 https://github.com/numpy/numpy/blob/master/doc/HOWTO_RELEASE.rst.txt

 Cheers,
 Ralf


 Jaime

 --
 (\__/)
 ( O.o)
 (  ) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes
 de dominación mundial.

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion



 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy where

[Benjamin Root]

I think the question is if scalars should be acceptable for the first
argument, not if it should be for the 2nd and 3rd argument.

If scalar can be given for the first argument, the the first three makes
sense. Although, I have no clue why we would allow that.

Ben Root
On Mar 12, 2015 9:25 PM, Nathaniel Smith n...@pobox.com wrote:

 On Mar 12, 2015 5:02 PM, Charles R Harris charlesr.har...@gmail.com
 wrote:
 
  Hi All,
 
  This is apropos gh-5582 dealing with some corner cases of np.where. The
 following are the current behavior
 
   import numpy
   numpy.where(True)  # case 1
  ... (array([0]),)
   numpy.where(True, None, None)  # case 2
  ... array(None, dtype=object)
   numpy.ma.where(True)  # case 3
  ... (array([0]),)
   numpy.ma.where(True, None, None)  # case 4
  ... (array([0]),)
 
  The question is, what exactly should be done in these cases? I'd be
 inclined to raise an error for cases 1 and 3. Case two looks correct to me
 if we agree that scalar inputs are acceptable. Case 4 looks wrong.

 I can't think of any reason scalars wouldn't be acceptable. So everything
 you suggest sounds right to me.

 -n
 Hi All,

 This is apropos gh-5582 https://github.com/numpy/numpy/pull/5582
 dealing with some corner cases of np.where. The following are the current
 behavior

  import numpy
  numpy.where(True)  # case 1
 ... (array([0]),)
  numpy.where(True, None, None)  # case 2
 ... array(None, dtype=object)
  numpy.ma.where(True)  # case 3
 ... (array([0]),)
  numpy.ma.where(True, None, None)  # case 4
 ... (array([0]),)

 The question is, what exactly should be done in these cases? I'd be
 inclined to raise an error for cases 1 and 3. Case two looks correct to me
 if we agree that scalar inputs are acceptable. Case 4 looks wrong.

 Thoughts?

 Chuck

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Adding keyword to asarray and asanyarray.

[Benjamin Root]

On Fri, Mar 6, 2015 at 7:59 AM, Charles R Harris charlesr.har...@gmail.com
wrote:

 Datetime64 seems to use the highest precision

 In [12]: result_type(ones(1, dtype='datetime64[D]'), 'datetime64[us]')
 Out[12]: dtype('M8[us]')

 In [13]: result_type(ones(1, dtype='datetime64[D]'), 'datetime64[Y]')
 Out[13]: dtype('M8[D]')



Ah, yes, that's what I'm looking for. +1 from me to have this in
asarray/asanyarray. Of course, there is always the usual caveats about
converting your datetime data in this manner, but this would be helpful in
many situations in writing functions that expect to deal with temporal data
at the resolution of minutes or somesuch.

Cheers!
Ben Root
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3

[Benjamin Root]

A slightly different way to look at this is one of sharing data. If I am
working on a system with 3.4 and I want to share data with others who may
be using a mix of 2.7 and 3.3 systems, this problem makes npz format much
less attractive.

Ben Root

On Fri, Mar 6, 2015 at 12:51 PM, Charles R Harris charlesr.har...@gmail.com
 wrote:



 On Fri, Mar 6, 2015 at 10:34 AM, Sebastian se...@sebix.at wrote:


 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA256

 Hi all,

 As this also affects .npy files, which uses pickle internally, why can't
 this be done by Numpy itself? This breaks backwards compatibility in a
 very bad way in my opinion.

 The company I worked for uses Numpy and consorts a lot and also has many
 data in .npy and pickle files. They currently work with 2.7, but I also
 tried to develop my programs to be compatible with Py 3. But this was
 not possible when it came to the point of dumping and loading npy files.
 I think this will be major reason why people won't take the step forward
 to Py3 and Numpy is not considered to be compatible to Python 3.


 Are you suggesting adding a flag to the files to mark the python version
 in which they were created? The *.npy format is versioned, so something
 could probably be done with that.

 Chuck

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Adding keyword to asarray and asanyarray.

[Benjamin Root]

dare I say... datetime64/timedelta64 support?

::ducks::

Ben Root

On Thu, Mar 5, 2015 at 11:40 AM, Charles R Harris charlesr.har...@gmail.com
 wrote:

 Hi All,

 This is apropos gh-5634 https://github.com/numpy/numpy/pull/5634, a PR
 adding a precision keyword to asarray and asanyarray. The PR description is

  The precision keyword differs from the current dtype keyword in the
 following way.

- It specifies a minimum precision. If the precision of the input is
greater than the specified precision, the input precision is preserved.
- Complex types are preserved. A specifies floating precision applies
to the dtype of the real and complex parts separately.

 For example, both complex128 and float64 dtypes have the
 same precision and an array of dtype float64 will be unchanged if the
 specified precision is float32.

 Ideally the precision keyword would be pushed down into the array
 constructor so that the resulting dtype could be determined before the
 array is constructed, but that would require adding new functions as the
 current constructors are part of the API and cannot have their
 signatures changed.

 The name of the keyword is open to discussion, as well as its acceptable
 values. And of course, anything else that might come to mind ;)

 Thoughts?

 Chuck

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Adding keyword to asarray and asanyarray.

[Benjamin Root]

On Thu, Mar 5, 2015 at 12:04 PM, Chris Barker chris.bar...@noaa.gov wrote:

 well, the precision of those is 64 bits, yes? so if you asked for less
 than that, you'd still get a dt64. If you asked for 64 bits, you'd get it,
 if you asked for datetime128  -- what would you get???

 a 128 bit integer? or an Exception, because there is no 128bit datetime
 dtype.



I was more thinking of datetime64/timedelta64's ability to specify the time
units.

Ben Root
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion