Hi, Sorry, won't have time to go into more detail for now. If you want a more relevant answer, you could try to formulate a very concrete question. But please bear in mind that this is the spyder mailing list, and as such we do not deal with complex parallel stuff such as MPI, multithreading Python, IPython, the GIL, etc. As a side note: Carlos is a more reliable source since he is a Spyder dev, I am just jumping in from time to time on the mailing list.
Never worked with QThread (just believe it is being used by Spyder), more info to be found on the web: https://duckduckgo.com/?q=QThread Regards, David On 30 May 2013 17:13, <[email protected]> wrote: > Thanks I like your answers but I have some more questions:) > > > > > On Wednesday, May 29, 2013 10:41:25 AM UTC-5, David wrote: >> >> Hi, >> >> I'll try to explain how I understand your question, and how I would >> approach this issue. Please correct me if I am wrong. There are many >> possible ways of using parallel computational power, and the best solution >> heavily depends upon your specific problem. I am far from an expert, but >> from the little that I've learned over the past years I know that there is >> no easy answer. >> >> 1) Spyder uses multi-threading on the application level (using QThread if >> I understand correctly) so things like code completion, monitor, >> documentation lookup, ... doesn't freeze the whole program while doing. >> However, this is not related to running your python scripts. Spyders >> multiprocessing implementation is, as far as I understand it, not related >> at all to the scripts you want to run. Python/IPython consoles are run in >> process that is separate from the main spyder process. >> > > What is QThread? > So it is useful if you want to use python as a wrapper to run other jobs? > >> >> When you want to use the parallel power of your computer cluster, you >> need to launch a python or ipython console in spyder that is aware of all >> those CPUs/GPUs. From there you can use the built in modules threading and >> multiprocessing to start explicitly using all that computational power. >> Note that depending on what you are trying to achieve, programming parallel >> applications can be challenging. The summer school "Advanced Scientific >> Programming in Python" [1] has some very nice lectures on this, and I would >> recommend you to have a look at those informative slides. >> > I am not sure why would that be useful, can't I just > grep processor /proc/cpuinfo | tail -n 1 | awk '{print $3+1}' > ? > > >> >> 2) I don't think spyder uses http to connect to other python consoles, >> not sure how http is related to the problem at hand >> > In previous reply Carlos said it uses sockets, what protocol/service does > it use to use sockets? > >> >> 3) interactive python console refers to an interactive work-flow and does >> not necessarily has anything to do with qsub (I assume you refer to your >> clusters mechanism to submit and queue jobs?) >> > So is it interactive or not? Why do you call it a workflow? > >> >> When I look at how we have our clusters configured, I could theoretically >> imagine the following work flow: >> * launch an IPython instance on the cluster with qsub and let it use as >> many CPUs as you see fit (for instance, specify #PBS -lnodes=xxx:ppn=yyy in >> your PBS script), and give it as much wall time as you think you need. >> * connect to that IPython console using the notebook/web interface (I >> know people do that, I just don't know how. There has to be some >> documentation available somewhere, or ask on the IPython mailinglist, or >> see [8]) >> * Within Spyder, connect to the IPython console running on the cluster, >> but I don't think that is already implemented, not sure if that's planned. >> * or, checkout [8]: Using IPython for parallel computing >> * rent an IPython instance on cluster configured for you at [11] >> >> You can release the GIL when using Cython, see for example the slides on >> Cython [1]. You can not release the GIL in a Python script. For explicit >> concurrency directly in your Python script, use the build in modules >> threading and multiprocessing. >> How would you proceed in releasing the GIL and whether you think it will >> have consequences? >> > > >> Numpy/Scipy/Numba and other Python modules already use, in some cases, >> the parallel power of your machine. Some examples on Numba are can be found >> here [2] and here [3]: For Numpy/Scipy, this is dependent on your >> BLAS/LAPACK implementation (such as MKL, OpenBlas, ACML): the low level >> number crunching routines on which they are built. Building NumPy against a >> BLAS/LAPACK implementation optimized for your machine from source can be >> challenging depending on your experience/skills [9] [10], but performance >> can be increased significantly [4]. >> >> Other more exotic libraries that can help unleash parallel power of >> CPUs/GPUs (for which I only know they exist): Magma [5], Plasma [6], CUBLAS >> [7]. >> >> [1] >> https://python.g-node.org/**python-summerschool-2012/**schedule<https://python.g-node.org/python-summerschool-2012/schedule> >> [2] >> http://jakevdp.github.io/blog/**2012/08/24/numba-vs-cython/<http://jakevdp.github.io/blog/2012/08/24/numba-vs-cython/> >> [3] http://www.continuum.io/blog/**simple-wave-simulation-with-** >> numba-and-pygame<http://www.continuum.io/blog/simple-wave-simulation-with-numba-and-pygame> >> [4] https://dpinte.wordpress.com/**2010/01/15/numpy-performance-** >> improvement-with-the-mkl/<https://dpinte.wordpress.com/2010/01/15/numpy-performance-improvement-with-the-mkl/> >> [5] >> http://icl.cs.utk.edu/magma/**index.html<http://icl.cs.utk.edu/magma/index.html> >> [6] >> http://icl.cs.utk.edu/plasma/**index.html<http://icl.cs.utk.edu/plasma/index.html> >> [7] >> https://developer.nvidia.com/**cublas<https://developer.nvidia.com/cublas> >> [8] >> http://ipython.org/ipython-**doc/dev/parallel/<http://ipython.org/ipython-doc/dev/parallel/> >> [9] http://osdf.github.io/blog/**numpyscipy-with-openblas-for-** >> ubuntu-1204-second-try.html<http://osdf.github.io/blog/numpyscipy-with-openblas-for-ubuntu-1204-second-try.html> >> [10] http://www.der-schnorz.de/**2012/06/optimized-linear-** >> algebra-and-numpyscipy/<http://www.der-schnorz.de/2012/06/optimized-linear-algebra-and-numpyscipy/> >> [11] http://continuum.io/wakari.**html <http://continuum.io/wakari.html> >> >> ok, this reply exploded in length and I can not vouch for its quality or >> its usefulness...I'll better stop here :-) >> >> Regards, >> David >> >> >> On 28 May 2013 20:47, <[email protected]> wrote: >> >>> Hi Carols, >>> >>> thank you so much for this kind answers. I have just few more questions: >>> 1. how different is multiprocessing from spyder in terms of >>> implementation? >>> 2. Does it mean that spyder uses http? >>> 3. ipython stand for interactive python? How are I'm going to deal with >>> that and qsub? >>> >>> Appreciate your help, >>> Ana >>> >>> >>> On Wednesday, May 22, 2013 1:28:46 PM UTC-5, [email protected]: >>>> >>>> Hello, >>>> >>>> please help me answer those questions, along with: do numpy and scipy >>>> thread or spyder does? >>>> I am planing to install spyder on Cray XE6 machine, with intention do >>>> so some python MPI and threading, and debugging >>>> Would spyder be of use for me? >>>> >>>> Thanks >>>> Ana >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "spyder" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to spyderlib+...@**googlegroups.com. >>> To post to this group, send email to [email protected]. >>> >>> Visit this group at >>> http://groups.google.com/**group/spyderlib?hl=en<http://groups.google.com/group/spyderlib?hl=en> >>> . >>> For more options, visit >>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >>> . >>> >>> >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "spyder" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/spyderlib?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- You received this message because you are subscribed to the Google Groups "spyder" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/spyderlib?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
