thanks, do I need to resubmit questions or just to wait for Carlos to answer?
On Thursday, May 30, 2013 10:32:05 AM UTC-5, David wrote: > > Hi, > > Sorry, won't have time to go into more detail for now. If you want a more > relevant answer, you could try to formulate a very concrete question. But > please bear in mind that this is the spyder mailing list, and as such we do > not deal with complex parallel stuff such as MPI, multithreading Python, > IPython, the GIL, etc. > As a side note: Carlos is a more reliable source since he is a Spyder dev, > I am just jumping in from time to time on the mailing list. > > Never worked with QThread (just believe it is being used by Spyder), more > info to be found on the web: https://duckduckgo.com/?q=QThread > > Regards, > David > > > On 30 May 2013 17:13, <[email protected] <javascript:>> wrote: > >> Thanks I like your answers but I have some more questions:) >> >> >> >> >> On Wednesday, May 29, 2013 10:41:25 AM UTC-5, David wrote: >>> >>> Hi, >>> >>> I'll try to explain how I understand your question, and how I would >>> approach this issue. Please correct me if I am wrong. There are many >>> possible ways of using parallel computational power, and the best solution >>> heavily depends upon your specific problem. I am far from an expert, but >>> from the little that I've learned over the past years I know that there is >>> no easy answer. >>> >>> 1) Spyder uses multi-threading on the application level (using QThread >>> if I understand correctly) so things like code completion, monitor, >>> documentation lookup, ... doesn't freeze the whole program while doing. >>> However, this is not related to running your python scripts. Spyders >>> multiprocessing implementation is, as far as I understand it, not related >>> at all to the scripts you want to run. Python/IPython consoles are run in >>> process that is separate from the main spyder process. >>> >> >> What is QThread? >> So it is useful if you want to use python as a wrapper to run other jobs? >> >>> >>> When you want to use the parallel power of your computer cluster, you >>> need to launch a python or ipython console in spyder that is aware of all >>> those CPUs/GPUs. From there you can use the built in modules threading and >>> multiprocessing to start explicitly using all that computational power. >>> Note that depending on what you are trying to achieve, programming parallel >>> applications can be challenging. The summer school "Advanced Scientific >>> Programming in Python" [1] has some very nice lectures on this, and I would >>> recommend you to have a look at those informative slides. >>> >> I am not sure why would that be useful, can't I just >> grep processor /proc/cpuinfo | tail -n 1 | awk '{print $3+1}' >> ? >> >> >>> >>> 2) I don't think spyder uses http to connect to other python consoles, >>> not sure how http is related to the problem at hand >>> >> In previous reply Carlos said it uses sockets, what protocol/service does >> it use to use sockets? >> >>> >>> 3) interactive python console refers to an interactive work-flow and >>> does not necessarily has anything to do with qsub (I assume you refer to >>> your clusters mechanism to submit and queue jobs?) >>> >> So is it interactive or not? Why do you call it a workflow? >> >>> >>> When I look at how we have our clusters configured, I could >>> theoretically imagine the following work flow: >>> * launch an IPython instance on the cluster with qsub and let it use as >>> many CPUs as you see fit (for instance, specify #PBS -lnodes=xxx:ppn=yyy in >>> your PBS script), and give it as much wall time as you think you need. >>> * connect to that IPython console using the notebook/web interface (I >>> know people do that, I just don't know how. There has to be some >>> documentation available somewhere, or ask on the IPython mailinglist, or >>> see [8]) >>> * Within Spyder, connect to the IPython console running on the cluster, >>> but I don't think that is already implemented, not sure if that's planned. >>> * or, checkout [8]: Using IPython for parallel computing >>> * rent an IPython instance on cluster configured for you at [11] >>> >>> You can release the GIL when using Cython, see for example the slides on >>> Cython [1]. You can not release the GIL in a Python script. For explicit >>> concurrency directly in your Python script, use the build in modules >>> threading and multiprocessing. >>> How would you proceed in releasing the GIL and whether you think it >>> will have consequences? >>> >> >> >>> Numpy/Scipy/Numba and other Python modules already use, in some cases, >>> the parallel power of your machine. Some examples on Numba are can be found >>> here [2] and here [3]: For Numpy/Scipy, this is dependent on your >>> BLAS/LAPACK implementation (such as MKL, OpenBlas, ACML): the low level >>> number crunching routines on which they are built. Building NumPy against a >>> BLAS/LAPACK implementation optimized for your machine from source can be >>> challenging depending on your experience/skills [9] [10], but performance >>> can be increased significantly [4]. >>> >>> Other more exotic libraries that can help unleash parallel power of >>> CPUs/GPUs (for which I only know they exist): Magma [5], Plasma [6], CUBLAS >>> [7]. >>> >>> [1] >>> https://python.g-node.org/**python-summerschool-2012/**schedule<https://python.g-node.org/python-summerschool-2012/schedule> >>> [2] >>> http://jakevdp.github.io/blog/**2012/08/24/numba-vs-cython/<http://jakevdp.github.io/blog/2012/08/24/numba-vs-cython/> >>> [3] http://www.continuum.io/blog/**simple-wave-simulation-with-** >>> numba-and-pygame<http://www.continuum.io/blog/simple-wave-simulation-with-numba-and-pygame> >>> [4] https://dpinte.wordpress.com/**2010/01/15/numpy-performance-** >>> improvement-with-the-mkl/<https://dpinte.wordpress.com/2010/01/15/numpy-performance-improvement-with-the-mkl/> >>> [5] >>> http://icl.cs.utk.edu/magma/**index.html<http://icl.cs.utk.edu/magma/index.html> >>> [6] >>> http://icl.cs.utk.edu/plasma/**index.html<http://icl.cs.utk.edu/plasma/index.html> >>> [7] >>> https://developer.nvidia.com/**cublas<https://developer.nvidia.com/cublas> >>> [8] >>> http://ipython.org/ipython-**doc/dev/parallel/<http://ipython.org/ipython-doc/dev/parallel/> >>> [9] http://osdf.github.io/blog/**numpyscipy-with-openblas-for-** >>> ubuntu-1204-second-try.html<http://osdf.github.io/blog/numpyscipy-with-openblas-for-ubuntu-1204-second-try.html> >>> >>> [10] http://www.der-schnorz.de/**2012/06/optimized-linear-** >>> algebra-and-numpyscipy/<http://www.der-schnorz.de/2012/06/optimized-linear-algebra-and-numpyscipy/> >>> >>> [11] http://continuum.io/wakari.**html <http://continuum.io/wakari.html> >>> >>> ok, this reply exploded in length and I can not vouch for its quality or >>> its usefulness...I'll better stop here :-) >>> >>> Regards, >>> David >>> >>> >>> On 28 May 2013 20:47, <[email protected]> wrote: >>> >>>> Hi Carols, >>>> >>>> thank you so much for this kind answers. I have just few more questions: >>>> 1. how different is multiprocessing from spyder in terms of >>>> implementation? >>>> 2. Does it mean that spyder uses http? >>>> 3. ipython stand for interactive python? How are I'm going to deal with >>>> that and qsub? >>>> >>>> Appreciate your help, >>>> Ana >>>> >>>> >>>> On Wednesday, May 22, 2013 1:28:46 PM UTC-5, [email protected]: >>>>> >>>>> Hello, >>>>> >>>>> please help me answer those questions, along with: do numpy and scipy >>>>> thread or spyder does? >>>>> I am planing to install spyder on Cray XE6 machine, with intention do >>>>> so some python MPI and threading, and debugging >>>>> Would spyder be of use for me? >>>>> >>>>> Thanks >>>>> Ana >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "spyder" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to spyderlib+...@**googlegroups.com. >>>> To post to this group, send email to [email protected]. >>>> >>>> Visit this group at >>>> http://groups.google.com/**group/spyderlib?hl=en<http://groups.google.com/group/spyderlib?hl=en> >>>> . >>>> For more options, visit >>>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >>>> . >>>> >>>> >>>> >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "spyder" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected]<javascript:> >> . >> Visit this group at http://groups.google.com/group/spyderlib?hl=en. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> > > -- You received this message because you are subscribed to the Google Groups "spyder" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/spyderlib?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
