On May 21, 8:14 pm, Magnitus <[email protected]> wrote: > Now that you mention it, I recall reading in the Python/C API that > Python wasn't really thread-safe and that Python objects shouldn't be > accessed from multiple C threads (they recommended using the Python > threading API which was exposed in the Python/C API instead, but that > didn't interest me as its not true multi-threading).
Python is thread safe so long as you abide by the contract of acquiring the Python GIL. If you are going to ignore that required contract, then obviously it will break. This is the norm for any Python system in as much as you need to acquire a lock when accessing a shared resource. In the Python case there so happens to be one single global lock around any C API accesses where as in a custom C/C++ application which is multithreaded, you might have more fine grained locking around specific data structures of parts of the API. > My solution to this was make my C++ code C++ only and then glue parts > that I wanted to expose to Python so that Python calls on the C++ > compiled code, but not the reverse. Which is exactly what many C extension modules do. That is, they move data between Python and C worlds so that they can then release the GIL and process the data. That why it can still make use of multi core/ processor systems. This is why Apache/mod_wsgi isn't constrained by the Python GIL because everything Apache itself does doesn't use the Python GIL and can be properly parallelised. Graham > Hence, I probably bypassed a large part of the problematic discussed > above by not delving deeply into Python's threading API. > > On May 21, 5:00 am, Yarko Tymciurak <[email protected]> > wrote: > > > > > On May 21, 3:33 am, Magnitus <[email protected]> wrote: > > > > But if you create "tasks" without doing it at the OS level, doesn't > > > that means that you won't really be able to take full advantage of > > > multi-processor hardware (since the OS handles the hardware and if the > > > OS doesn't know about it, it won't be able to do the required > > > optimizations with the hardware)? > > > With the GIL, python itself does not utilize multiple processors, so > > web2py is processor-bound (the only > > effect of multi-core is that the o/s itself can "leave" a core to the > > main python task, e.g. > > it can grab an alternate core... other than that, you're running on > > one core regardless - > > unless you fire multiple instances of python interpreters, in which > > case you are really only > > going to communicate thru services anyway.... > > > See some of the discussion > > athttp://bugs.python.org/issue7946,http://stackoverflow.com/questions/9...... > > > ... and so forth... > > > > Maybe I've done C/C++ for too long and am trying to micro-manage too > > > much, but a solution to I like to use for the problem of creating/ > > > tearing down process threads is just to pre-create a limited number of > > > them (optimised for the number of CPUs you have) and recycle them to > > > do various tasks as needed. > > > Well - since you don't have that with python, you run the risk of I/O > > blocking .... which is why really lightweight > > tasklets are so desireable (CCP Games > > runshttp://en.wikipedia.org/wiki/Eve_Online > > with many tens of thousands of simultaneous users, if I recall > > correctly, and maintain stackless for this purpose). > > > > Of course, that works best when you give your threads/processes longer > > > tasks to perform in parallel (else, the extra cost of managing it will > > > probably outweight the benefits of running it in parallel). > > > There is much to cover in this - and I suppose reason to be happy that > > python traditionally hasn't run multi-core. > > See, for example, the discussions > > at:http://stackoverflow.com/questions/203912/does-python-support-multipr... > > > andhttp://docs.python.org/library/multiprocessing.html > > > Lots to read! ;-) > > > - Yarko > > > > On May 20, 2:12 pm, Yarko Tymciurak <[email protected]> > > > wrote: > > > > > On May 19, 6:18 pm, Yarko Tymciurak <[email protected]> > > > > wrote: > > > > > > On May 19, 5:41 pm, amoygard <[email protected]> wrote: > > > > > .... > > > > > > So - in general, you do not start subprocesses - with the exception of > > > > > cron. Seehttp://www.web2py.com/book/default/section/4/17 > > > > > I might better have said you do not _want_ to be starting subprocesses > > > > - besides the cost (compute time, memory, etc.), if you generally did > > > > this. This (the inneficiency of spawning subrocesses) is why > > > > stackless was created - and (among other things) used in a a very > > > > busy online game. A lot of thought went into avoiding the costs of > > > > spawning subprocesses. > > > > > If you haven't heard of it, stackless is an implementation of python > > > > that does not use the traditional "C" stack for local variables, > > > > etc. Among other things, it has added "tasklets" to the language, so > > > > you can create and schedule tasks - without the overhead of doing so > > > > in your operating system. There is a lot of discussion of benefit, > > > > efficiency. Although there might be some discussion questioning the > > > > approach, other alternative approaches, one thing is clear: the > > > > motivation to stay away from creating threads / subprocesses, and the > > > > costs involved. it might be interesting to read up on it. > > > > > - Yarko > > > > > > - Yarko- Hide quoted text - > > > - Show quoted text -
