Re: [Numpy-discussion] Behavior of .reduceat()

2016-05-22 Thread Feng Yu
Hi Marten, As a user of reduceat I seriously like your new proposal! I notice that in your current proposal, each element in the 'at' list shall be interpreted asif they are parameters to `slice`. I wonder if it is meaningful to define reduceat on other `fancy` indexing types? Cheers, - Yu

Re: [Numpy-discussion] Numpy arrays shareable among related processes (PR #7533)

2016-05-13 Thread Feng Yu
> > Personally I prefer a parallel programming style with queues – either to > scatter arrays to workers and collecting arrays from workers, or to chain > workers together in a pipeline (without using coroutines). But exactly how > you program is a matter of taste. I want to make it as inexpensive

Re: [Numpy-discussion] Numpy arrays shareable among related processes (PR #7533)

2016-05-12 Thread Feng Yu
> Again, not everyone uses Unix. > > And on Unix it is not trival to pass data back from the child process. I > solved that problem with Sys V IPC (pickling the name of the segment). > I wonder if it is neccessary insist being able to pass large amount of data back from child to the parent

Re: [Numpy-discussion] Numpy arrays shareable among related processes (PR #7533)

2016-05-11 Thread Feng Yu
Hi, I've been thinking and exploring this for some time. If we are to start some effort I'd like to help. Here are my comments, mostly regarding to Sturla's comments. 1. If we are talking about shared memory and copy-on-write inheritance, then we are using 'fork'. If we are free to use fork,

Re: [Numpy-discussion] Changes to generalized ufunc core dimension checking

2016-03-19 Thread Feng Yu
Hi, Here is another example. To write pix2ang (and similar functions) to a ufunc, one may want to have implicit scalar broadcast on `nested` and `nsides` arguments. The function is described here:

Re: [Numpy-discussion] Changes to generalized ufunc core dimension checking

2016-03-19 Thread Feng Yu
Fox-Rabinovitz <jfoxrabinov...@gmail.com> wrote: > On Thu, Mar 17, 2016 at 10:03 AM, Nathaniel Smith <n...@pobox.com> wrote: >> On Mar 17, 2016 1:22 AM, "Feng Yu" <rainwood...@gmail.com> wrote: >>> >>> Hi, >>> >>> Here is ano

Re: [Numpy-discussion] Changes to generalized ufunc core dimension checking

2016-03-18 Thread Feng Yu
Thanks for the explanation. I see the point now. On Thu, Mar 17, 2016 at 3:21 PM, Nathaniel Smith <n...@pobox.com> wrote: > On Thu, Mar 17, 2016 at 2:04 PM, Feng Yu <rainwood...@gmail.com> wrote: >> Hi, >> >> ang2pix is used in astronomy to pixelize coordinate in

Re: [Numpy-discussion] resizeable arrays using shared memory?

2016-02-09 Thread Feng Yu
Hi, If the base address and size of the anonymous memory map are 'shared', then one can protect them with a lock, grow the memmap with remap (or unmap and map, or other tricks), and release the lock. During the 'resize' call, any reference to the array from Python in other processes could just

Re: [Numpy-discussion] Fast Access to Container of Numpy Arrays on Disk?

2016-01-14 Thread Feng Yu
Hi Ryan, Did you consider packing the arrays into one(two) giant array stored with mmap? That way you only need to store the start & end offsets, and there is no need to use a dictionary. It may allow you to simplify some numerical operations as well. To be more specific, start : numpy.intp

Re: [Numpy-discussion] [SciPy-Dev] Setting up a dev environment with conda

2015-10-18 Thread Feng Yu
Hi Luke, Could you check if you have "/Users/lzkelley/Programs/public/numpy/ in your PYTHONPATH? I would also suggest you add a print(np) line before the crash in nosetester.py. I got something like this (which didn't crash): If you see something not starting with 'numpy/build', then it is

Re: [Numpy-discussion] Notes from the numpy dev meeting at scipy 2015

2015-08-25 Thread Feng Yu
Hi Nathaniel, Thanks for the notes. In some sense, the new dtype class(es) will provided a way of formalizing these `weird` metadata, and probably exposing them to Python. May I add that please consider adding a way to declare the sorting order (priority and direction) of fields in a structured

[Numpy-discussion] Fwd: Reverse(DESC)-ordered sorting

2015-08-19 Thread Feng Yu
Dear list, This is forwarded from issue 6217 https://github.com/numpy/numpy/issues/6217 What is the way to implement DESC ordering in the sorting routines of numpy? (I am borrowing DESC/ASC from the SQL notation) For a stable DESC ordering sort, one can not revert the sorted array via