Re: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver

[Juha Jeronen]

On 01.10.2015 03:32, Sturla Molden wrote:

On 01/10/15 02:20, Juha Jeronen wrote:


Then again, the matter is further complicated by considering codes that
run on a single machine, versus codes that run on a cluster.Threads
being local to each node in a cluster,


You can run MPI programs on a single machine and you get OpenMP 
implementations for clusters. Just pick an API and stick with it.


Mm. I've quite often run MPI locally (it's nice for multicore scientific 
computing on Python), but I had no idea that OpenMP had cluster 
implementations. Thanks for the tip.


I've got the impression that the way these APIs market themselves is 
that MPI is for processes, while OpenMP is for threads, even if this is 
not completely true across all implementations.


(If I wanted maximal control over what each process/thread is doing, I'd 
go for ZeroMQ :) )



 -J

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver

[Daπid]

On 2 October 2015 at 11:58, Juha Jeronen  wrote:

>
>>
> First version done and uploaded:
>
>
> https://yousource.it.jyu.fi/jjrandom2/miniprojects/trees/master/misc/polysolve_for_numpy
>

Small comment: now you are checking if the input is a scalar or a ndarray,
but it should also accept any array-like. If I pass a list, I expect it to
work, internally converting it into an array.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver

[Slavin, Jonathan]

​Personally I like atleast_1d, which will convert a scalar into a 1d array
but will leave arrays untouched (i.e. won't change the dimensions.  Not
sure what the advantages/disadvantages are relative to asarray.

Jon​


On Fri, Oct 2, 2015 at 7:05 AM,  wrote:

> From: Juha Jeronen 
> To: Discussion of Numerical Python 
> Cc:
> Date: Fri, 2 Oct 2015 13:31:47 +0300
> Subject: Re: [Numpy-discussion] Cython-based OpenMP-accelerated quartic
> polynomial solver
> On 02.10.2015 13:07, Daπid wrote:
>
>
> On 2 October 2015 at 11:58, Juha Jeronen  wrote:
>
>>
>>>
>> First version done and uploaded:
>>
>>
>> https://yousource.it.jyu.fi/jjrandom2/miniprojects/trees/master/misc/polysolve_for_numpy
>>
>
> Small comment: now you are checking if the input is a scalar or a ndarray,
> but it should also accept any array-like. If I pass a list, I expect it to
> work, internally converting it into an array.
>
>
> Good catch.
>
> Is there an official way to test for array-likes? Or should I always
> convert with asarray()? Or something else?
>
>
>  -J
>




-- 

Jonathan D. Slavin Harvard-Smithsonian CfA
jsla...@cfa.harvard.edu   60 Garden Street, MS 83
phone: (617) 496-7981   Cambridge, MA 02138-1516
cell: (781) 363-0035 USA

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver

[Ryan May]

numpy.asanyarray() would be my preferred goto, as it will leave subclasses
of ndarray untouched; asarray() and atleast_1d() force ndarray. It's nice
to do the whenever possible.

Ryan

On Fri, Oct 2, 2015 at 6:52 AM, Slavin, Jonathan 
wrote:

> ​Personally I like atleast_1d, which will convert a scalar into a 1d array
> but will leave arrays untouched (i.e. won't change the dimensions.  Not
> sure what the advantages/disadvantages are relative to asarray.
>
> Jon​
>
>
> On Fri, Oct 2, 2015 at 7:05 AM, 
> wrote:
>
>> From: Juha Jeronen 
>> To: Discussion of Numerical Python 
>> Cc:
>> Date: Fri, 2 Oct 2015 13:31:47 +0300
>> Subject: Re: [Numpy-discussion] Cython-based OpenMP-accelerated quartic
>> polynomial solver
>>
>> On 02.10.2015 13:07, Daπid wrote:
>>
>>
>> On 2 October 2015 at 11:58, Juha Jeronen  wrote:
>>
>>>

>>> First version done and uploaded:
>>>
>>>
>>> https://yousource.it.jyu.fi/jjrandom2/miniprojects/trees/master/misc/polysolve_for_numpy
>>>
>>
>> Small comment: now you are checking if the input is a scalar or a
>> ndarray, but it should also accept any array-like. If I pass a list, I
>> expect it to work, internally converting it into an array.
>>
>>
>> Good catch.
>>
>> Is there an official way to test for array-likes? Or should I always
>> convert with asarray()? Or something else?
>>
>>
>>  -J
>>
>
>
>
>
> --
> 
> Jonathan D. Slavin Harvard-Smithsonian CfA
> jsla...@cfa.harvard.edu   60 Garden Street, MS 83
> phone: (617) 496-7981   Cambridge, MA 02138-1516
> cell: (781) 363-0035 USA
> 
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 
Ryan May
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver

[Juha Jeronen]

On 02.10.2015 13:07, Daπid wrote:


On 2 October 2015 at 11:58, Juha Jeronen > wrote:




First version done and uploaded:


https://yousource.it.jyu.fi/jjrandom2/miniprojects/trees/master/misc/polysolve_for_numpy


Small comment: now you are checking if the input is a scalar or a 
ndarray, but it should also accept any array-like. If I pass a list, I 
expect it to work, internally converting it into an array.


Good catch.

Is there an official way to test for array-likes? Or should I always 
convert with asarray()? Or something else?



 -J

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver

[Juha Jeronen]

On 01.10.2015 03:52, Sturla Molden wrote:

On 01/10/15 02:32, Juha Jeronen wrote:


Sounds good. Out of curiosity, are there any standard fork-safe
threadpools, or would this imply rolling our own?


You have to roll your own.

Basically use pthreads_atfork to register a callback that shuts down 
the threadpool before a fork and another callback that restarts it. 
Python's threading module does not expose the pthreads_atfork 
function, so you must call it from Cython.


I believe there is also a tiny atfork module in PyPI.


Ok. Thanks. This approach fixes the issue of the threads not being there 
for the child process.


I think it still leaves open the issue of creating the correct number of 
threads in the pools for each of the processes when the pool is 
restarted (so that in total there will be as many threads as cores 
(physical or virtual, whichever the user desires)). But this is again 
something that requires context...




So maybe it would be better, at least at first, to make a pure-Cython
version with no attempt at multithreading?


I would start by making a pure Cython version that works correctly. 
The next step would be to ensure that it releases the GIL. After that 
you can worry about parallel processing, or just tell the user to use 
threads or joblib.


First version done and uploaded:

https://yousource.it.jyu.fi/jjrandom2/miniprojects/trees/master/misc/polysolve_for_numpy

OpenMP support removed; this version uses only Cython.

The example program has been renamed to main.py, and setup.py has been 
cleaned, removing the irrelevant module.


This folder contains only the files for the polynomial solver.


As I suspected, removing OpenMP support only required changing a few 
lines, and dropping the import for Cython.parallel. The "prange"s have 
been replaced with "with nogil" and "range".


Note that both the original version and this version release the GIL 
when running the processing loops.



It may be better to leave this single-threaded for now. Using Python 
threads isn't that difficult and joblib sounds nice, too.


What's the next step?

 -J

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver

[Daπid]

On 1 October 2015 at 09:05, Nathaniel Smith  wrote:

>
> >> - gcc + OpenMP on linux still breaks multiprocessing. There's a patch to
> >> fix this but they still haven't applied it; alternatively there's a
> >> workaround you can use in multiprocessing (not using fork mode), but
> this
> >> requires every user update their code and the workaround has other
> >> limitations. We're unlikely to use OpenMP while this is the case.
> >
> > Any idea when is this going to be released?
>
> Which? The gcc patch? I spent 2 full release cycles nagging them and
> they still can't be bothered to make a decision either way, so :-(. If
> anyone has some ideas for how to get traction in gcc-land then I'm
> happy to pass on details...
>

:(

Have you tried asking Python-dev for help with this? Hopefully they would
have some weight there.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver

[Sturla Molden]

Juha Jeronen  wrote:

> Mm. I've quite often run MPI locally (it's nice for multicore scientific 
> computing on Python), but I had no idea that OpenMP had cluster 
> implementations. Thanks for the tip.

Intel has been selling one, I think there are others too. 

OpenMP has a flush pragma for synchronizing shared variables. This means
that OpenMP is not restricted to shared memory hardware. A "pragma omp
flush" can just as well invoke some IPC mechanism, even network
communication. 

Sturla

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver

[Sturla Molden]

Sturla Molden  wrote:

> OpenMP has a flush pragma for synchronizing shared variables. This means
> that OpenMP is not restricted to shared memory hardware. A "pragma omp
> flush" can just as well invoke some IPC mechanism, even network
> communication. 

By the way, while this is the case for C and Fortran, it is certainly not
the case for Cython. In a Cython prange block, a shared variable is
accessed by dereferencing its address. This requires shared memory. Pure
OpenMP in C does not, because shared variables are not accessed through
pointers, but are rather normal variables that are synchronized with a
pragma.

Cython actually requires that there is a shared address space, and it
invokes something that strictly speaking has undefined behavior under the
OpenMP standard. So thus, a prange block in Cython is expected to work
correctly on a laptop with a multicore processor, but it is not expected to
work correctly on a cluster.

IIRC, Intel's cluster OpenMP is based on MPI, which means the compiler will
internally translate code with OpenMP pragmas into equivalent code that
calls MPI functions. A program written for OpenMP can then run on any
cluster that provides an MPI implementation.

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver

[Sturla Molden]

Sturla Molden  wrote:

> Cython actually requires that there is a shared address space, and it
> invokes something that strictly speaking has undefined behavior under the
> OpenMP standard. So thus, a prange block in Cython is expected to work
> correctly on a laptop with a multicore processor, but it is not expected to
> work correctly on a cluster.

OpenMP does not guarrantee that dereferencing a pointer in a parallel block
will dereference the same object across all processors. It only guarrantees
that the value of a shared object can be synchronized. There are many who
use OpenMP and think only in terms of threads that do this incorrectly.
Cython is actually among those.

S.M.

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Fwd: Numpy for data manipulation

[Alex Rogozhnikov]

I would suggest

%matplotlib notebook

It will still have to a nice png, but you get an interactive figure 
when it is live.


Amazing, thanks. I was using mpld3 for this.
(for some strange reason I need to put %matplotlib notebook before each 
plot)


The recommendation of inverting a permutation by argsort'ing it, while 
it works, is suboptimal, as it takes O(n log(n)) time, and you can do 
it in linear time:
Actually, there is (later in post) a linear solution using bincount, but 
your code is definitely better. Thanks!

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Fwd: Numpy for data manipulation

[Kiko]

2015-10-02 9:38 GMT+02:00 Alex Rogozhnikov :

> I would suggest
>>
>> %matplotlib notebook
>>
>> It will still have to a nice png, but you get an interactive figure when
>> it is live.
>>
>
> Amazing, thanks. I was using mpld3 for this.
> (for some strange reason I need to put %matplotlib notebook before each
> plot)
>

You should create a figure before each plot instead of putthon %matplotlib
notebook
plt.figure()



>
> The recommendation of inverting a permutation by argsort'ing it, while it
>> works, is suboptimal, as it takes O(n log(n)) time, and you can do it in
>> linear time:
>>
> Actually, there is (later in post) a linear solution using bincount, but
> your code is definitely better. Thanks!
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Fwd: Numpy for data manipulation

[Kiko]

2015-10-02 9:48 GMT+02:00 Kiko :

>
>
> 2015-10-02 9:38 GMT+02:00 Alex Rogozhnikov :
>
>> I would suggest
>>>
>>> %matplotlib notebook
>>>
>>> It will still have to a nice png, but you get an interactive figure when
>>> it is live.
>>>
>>
>> Amazing, thanks. I was using mpld3 for this.
>> (for some strange reason I need to put %matplotlib notebook before each
>> plot)
>>
>
> You should create a figure before each plot instead of putthon %matplotlib
> notebook
> plt.figure()
> 
>

putthon == putting


>
>
>>
>> The recommendation of inverting a permutation by argsort'ing it, while it
>>> works, is suboptimal, as it takes O(n log(n)) time, and you can do it in
>>> linear time:
>>>
>> Actually, there is (later in post) a linear solution using bincount, but
>> your code is definitely better. Thanks!
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Let's move forward with the current governance document.

[Travis Oliphant]

Hi everyone,

After some further thought and spending quite a bit of time re-reading the
discussion on a few threads, I now believe that my request to be on the
steering council might be creating more trouble than it's worth.
Nothing matters to me more than seeing NumPy continue to grow and improve.

So, I'm switching my position to supporting the adoption of the governance
model outlined and just contributing as I can outside the steering council.
   The people on the steering council are committed to the success of NumPy
and will do a great job --- they already have in contributing to the
community over the past year(s).We can always revisit the question in a
year if difficulties arise with the model.

If my voice and other strong voices remain outside the council, perhaps we
can all encourage that the intended community governance of NumPy does in
fact happen, and most decisions continue to be made in the open.

I had the pleasure last night of meeting one of the new NumPy core
contributors, Allan Haldane.   This only underscored my confidence in
everyone who is contributing to NumPy today.   This confidence has already
been established by watching the great contributions of many talented
developers who have given their time and talents to the project over the
past several years.

I hope that we can move on from the governance discussion and continue to
promote the success of the project together.

Best,

-Travis
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion