[Numpy-discussion] Fwd: Reverse(DESC)-ordered sorting

2015-08-19 Thread Feng Yu
Dear list, This is forwarded from issue 6217 https://github.com/numpy/numpy/issues/6217 "What is the way to implement DESC ordering in the sorting routines of numpy?" (I am borrowing DESC/ASC from the SQL notation) For a stable DESC ordering sort, one can not revert the sorted array via argsor

Re: [Numpy-discussion] Notes from the numpy dev meeting at scipy 2015

2015-08-25 Thread Feng Yu
Hi Nathaniel, Thanks for the notes. In some sense, the new dtype class(es) will provided a way of formalizing these `weird` metadata, and probably exposing them to Python. May I add that please consider adding a way to declare the sorting order (priority and direction) of fields in a structured

Re: [Numpy-discussion] [SciPy-Dev] Setting up a dev environment with conda

2015-10-18 Thread Feng Yu
Hi Luke, Could you check if you have "/Users/lzkelley/Programs/public/numpy/ in your PYTHONPATH? I would also suggest you add a print(np) line before the crash in nosetester.py. I got something like this (which didn't crash): If you see something not starting with 'numpy/build', then it is aga

Re: [Numpy-discussion] Fast Access to Container of Numpy Arrays on Disk?

2016-01-14 Thread Feng Yu
Hi Ryan, Did you consider packing the arrays into one(two) giant array stored with mmap? That way you only need to store the start & end offsets, and there is no need to use a dictionary. It may allow you to simplify some numerical operations as well. To be more specific, start : numpy.intp end

Re: [Numpy-discussion] resizeable arrays using shared memory?

2016-02-09 Thread Feng Yu
Hi, If the base address and size of the anonymous memory map are 'shared', then one can protect them with a lock, grow the memmap with remap (or unmap and map, or other tricks), and release the lock. During the 'resize' call, any reference to the array from Python in other processes could just spi

Re: [Numpy-discussion] Changes to generalized ufunc core dimension checking

2016-03-18 Thread Feng Yu
Thanks for the explanation. I see the point now. On Thu, Mar 17, 2016 at 3:21 PM, Nathaniel Smith wrote: > On Thu, Mar 17, 2016 at 2:04 PM, Feng Yu wrote: >> Hi, >> >> ang2pix is used in astronomy to pixelize coordinate in forms of >> (theta, phi). healpy is a

Re: [Numpy-discussion] Changes to generalized ufunc core dimension checking

2016-03-19 Thread Feng Yu
, 2016 at 8:09 AM, Joseph Fox-Rabinovitz wrote: > On Thu, Mar 17, 2016 at 10:03 AM, Nathaniel Smith wrote: >> On Mar 17, 2016 1:22 AM, "Feng Yu" wrote: >>> >>> Hi, >>> >>> Here is another example. >>> >>> To write pix2ang

Re: [Numpy-discussion] Changes to generalized ufunc core dimension checking

2016-03-19 Thread Feng Yu
Hi, Here is another example. To write pix2ang (and similar functions) to a ufunc, one may want to have implicit scalar broadcast on `nested` and `nsides` arguments. The function is described here: http://healpy.readthedocs.org/en/latest/generated/healpy.pixelfunc.pix2ang.html#healpy.pixelfunc.p

Re: [Numpy-discussion] Numpy arrays shareable among related processes (PR #7533)

2016-05-11 Thread Feng Yu
Hi, I've been thinking and exploring this for some time. If we are to start some effort I'd like to help. Here are my comments, mostly regarding to Sturla's comments. 1. If we are talking about shared memory and copy-on-write inheritance, then we are using 'fork'. If we are free to use fork, then

Re: [Numpy-discussion] Numpy arrays shareable among related processes (PR #7533)

2016-05-12 Thread Feng Yu
> Again, not everyone uses Unix. > > And on Unix it is not trival to pass data back from the child process. I > solved that problem with Sys V IPC (pickling the name of the segment). > I wonder if it is neccessary insist being able to pass large amount of data back from child to the parent process

Re: [Numpy-discussion] Numpy arrays shareable among related processes (PR #7533)

2016-05-13 Thread Feng Yu
> > Personally I prefer a parallel programming style with queues – either to > scatter arrays to workers and collecting arrays from workers, or to chain > workers together in a pipeline (without using coroutines). But exactly how > you program is a matter of taste. I want to make it as inexpensive

Re: [Numpy-discussion] Behavior of .reduceat()

2016-05-22 Thread Feng Yu
Hi Marten, As a user of reduceat I seriously like your new proposal! I notice that in your current proposal, each element in the 'at' list shall be interpreted asif they are parameters to `slice`. I wonder if it is meaningful to define reduceat on other `fancy` indexing types? Cheers, - Yu On

Re: [Numpy-discussion] Preserving NumPy views when pickling

2016-10-25 Thread Feng Yu
Hi, Just another perspective. base' and 'data' in PyArrayObject are two separate variables. base can point to any PyObject, but it is `data` that defines where data is accessed in memory. 1. There is no clear way to pickle a pointer (`data`) in a meaningful way. In order for `data` member to mak