thanks, do I need to resubmit questions or just to wait for Carlos to
answer?
On Thursday, May 30, 2013 10:32:05 AM UTC-5, David wrote:
Hi,
Sorry, won't have time to go into more detail for now. If you want
a more relevant answer, you could try to formulate a very concrete
question. But please bear in mind that this is the spyder mailing
list, and as such we do not deal with complex parallel stuff such
as MPI, multithreading Python, IPython, the GIL, etc.
As a side note: Carlos is a more reliable source since he is a
Spyder dev, I am just jumping in from time to time on the mailing
list.
Never worked with QThread (just believe it is being used by
Spyder), more info to be found on the web:
https://duckduckgo.com/?q=QThread <https://duckduckgo.com/?q=QThread>
Regards,
David
On 30 May 2013 17:13, <[email protected] <javascript:>> wrote:
Thanks I like your answers but I have some more questions:)
On Wednesday, May 29, 2013 10:41:25 AM UTC-5, David wrote:
Hi,
I'll try to explain how I understand your question, and
how I would approach this issue. Please correct me if I am
wrong. There are many possible ways of using parallel
computational power, and the best solution heavily depends
upon your specific problem. I am far from an expert, but
from the little that I've learned over the past years I
know that there is no easy answer.
1) Spyder uses multi-threading on the application level
(using QThread if I understand correctly) so things like
code completion, monitor, documentation lookup, ...
doesn't freeze the whole program while doing. However,
this is not related to running your python scripts.
Spyders multiprocessing implementation is, as far as I
understand it, not related at all to the scripts you want
to run. Python/IPython consoles are run in process that is
separate from the main spyder process.
What is QThread?
So it is useful if you want to use python as a wrapper to run
other jobs?
When you want to use the parallel power of your computer
cluster, you need to launch a python or ipython console in
spyder that is aware of all those CPUs/GPUs. From there
you can use the built in modules threading and
multiprocessing to start explicitly using all that
computational power. Note that depending on what you are
trying to achieve, programming parallel applications can
be challenging. The summer school "Advanced Scientific
Programming in Python" [1] has some very nice lectures on
this, and I would recommend you to have a look at those
informative slides.
I am not sure why would that be useful, can't I just
grep processor /proc/cpuinfo | tail -n 1 | awk '{print $3+1}'
?
2) I don't think spyder uses http to connect to other
python consoles, not sure how http is related to the
problem at hand
In previous reply Carlos said it uses sockets, what
protocol/service does it use to use sockets?
3) interactive python console refers to an interactive
work-flow and does not necessarily has anything to do with
qsub (I assume you refer to your clusters mechanism to
submit and queue jobs?)
So is it interactive or not? Why do you call it a workflow?
When I look at how we have our clusters configured, I
could theoretically imagine the following work flow:
* launch an IPython instance on the cluster with qsub and
let it use as many CPUs as you see fit (for instance,
specify #PBS -lnodes=xxx:ppn=yyy in your PBS script), and
give it as much wall time as you think you need.
* connect to that IPython console using the notebook/web
interface (I know people do that, I just don't know how.
There has to be some documentation available somewhere, or
ask on the IPython mailinglist, or see [8])
* Within Spyder, connect to the IPython console running on
the cluster, but I don't think that is already
implemented, not sure if that's planned.
* or, checkout [8]: Using IPython for parallel computing
* rent an IPython instance on cluster configured for you
at [11]
You can release the GIL when using Cython, see for example
the slides on Cython [1]. You can not release the GIL in a
Python script. For explicit concurrency directly in your
Python script, use the build in modules threading and
multiprocessing.
How would you proceed in releasing the GIL and whether you
think it will have consequences?
Numpy/Scipy/Numba and other Python modules already use, in
some cases, the parallel power of your machine. Some
examples on Numba are can be found here [2] and here [3]:
For Numpy/Scipy, this is dependent on your BLAS/LAPACK
implementation (such as MKL, OpenBlas, ACML): the low
level number crunching routines on which they are built.
Building NumPy against a BLAS/LAPACK implementation
optimized for your machine from source can be challenging
depending on your experience/skills [9] [10], but
performance can be increased significantly [4].
Other more exotic libraries that can help unleash parallel
power of CPUs/GPUs (for which I only know they exist):
Magma [5], Plasma [6], CUBLAS [7].
[1]
https://python.g-node.org/python-summerschool-2012/schedule
<https://python.g-node.org/python-summerschool-2012/schedule>
[2]
http://jakevdp.github.io/blog/2012/08/24/numba-vs-cython/
<http://jakevdp.github.io/blog/2012/08/24/numba-vs-cython/>
[3]
http://www.continuum.io/blog/simple-wave-simulation-with-numba-and-pygame
<http://www.continuum.io/blog/simple-wave-simulation-with-numba-and-pygame>
[4]
https://dpinte.wordpress.com/2010/01/15/numpy-performance-improvement-with-the-mkl/
<https://dpinte.wordpress.com/2010/01/15/numpy-performance-improvement-with-the-mkl/>
[5] http://icl.cs.utk.edu/magma/index.html
<http://icl.cs.utk.edu/magma/index.html>
[6] http://icl.cs.utk.edu/plasma/index.html
<http://icl.cs.utk.edu/plasma/index.html>
[7] https://developer.nvidia.com/cublas
<https://developer.nvidia.com/cublas>
[8] http://ipython.org/ipython-doc/dev/parallel/
<http://ipython.org/ipython-doc/dev/parallel/>
[9]
http://osdf.github.io/blog/numpyscipy-with-openblas-for-ubuntu-1204-second-try.html
<http://osdf.github.io/blog/numpyscipy-with-openblas-for-ubuntu-1204-second-try.html>
[10]
http://www.der-schnorz.de/2012/06/optimized-linear-algebra-and-numpyscipy/
<http://www.der-schnorz.de/2012/06/optimized-linear-algebra-and-numpyscipy/>
[11] http://continuum.io/wakari.html
<http://continuum.io/wakari.html>
ok, this reply exploded in length and I can not vouch for
its quality or its usefulness...I'll better stop here :-)
Regards,
David
On 28 May 2013 20:47, <[email protected]> wrote:
Hi Carols,
thank you so much for this kind answers. I have just
few more questions:
1. how different is multiprocessing from spyder in
terms of implementation?
2. Does it mean that spyder uses http?
3. ipython stand for interactive python? How are I'm
going to deal with that and qsub?
Appreciate your help,
Ana
On Wednesday, May 22, 2013 1:28:46 PM UTC-5,
[email protected] wrote:
Hello,
please help me answer those questions, along with:
do numpy and scipy thread or spyder does?
I am planing to install spyder on Cray XE6
machine, with intention do so some python MPI and
threading, and debugging
Would spyder be of use for me?
Thanks
Ana
--
You received this message because you are subscribed
to the Google Groups "spyder" group.
To unsubscribe from this group and stop receiving
emails from it, send an email to
[email protected].
To post to this group, send email to
[email protected].
Visit this group at
http://groups.google.com/group/spyderlib?hl=en
<http://groups.google.com/group/spyderlib?hl=en>.
For more options, visit
https://groups.google.com/groups/opt_out
<https://groups.google.com/groups/opt_out>.
--
You received this message because you are subscribed to the
Google Groups "spyder" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to [email protected] <javascript:>.
To post to this group, send email to [email protected]
<javascript:>.
Visit this group at
http://groups.google.com/group/spyderlib?hl=en
<http://groups.google.com/group/spyderlib?hl=en>.
For more options, visit
https://groups.google.com/groups/opt_out
<https://groups.google.com/groups/opt_out>.
--
You received this message because you are subscribed to the Google
Groups "spyder" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/spyderlib?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
