Re: [BangPypers] real use case of gevent context switch gevent.sleep(0)
Thanks noufal, I got it. After gevent.spawn, we have to do join or gevent.joinall or gevent.sleep or patched library call to start the greenlets, right ? I have see the code which not using any of these after gevent.spawn, How it is working there ? https://github.com/Juniper/contrail-controller/blob/master/src/opserver/uveserver.py#L656 Thanks, On Sun, Jun 21, 2015 at 1:01 PM, Noufal Ibrahim KV nou...@nibrahim.net.in wrote: On Sun, Jun 21 2015, anu sree wrote: [...] This code has two pause (gevent.sleep(0)), in Consumer.start and Worker.run. Here control goes to Worker.run when Consumer.start pauses and Consumer.start gets control back when Worker.run pauses. There may be benefit from this switching, but I am still not understanding it. [...] The idea is that any non blocking call is an opportunity to switch between greenlets. The monkey patching system will enable this. If you don't have any place in your greenlets that are monkey patched (and therefore allow switching), you can be a good citizen and drop in a patched sleep call so that other greenlets will get time to run. Otherwise, you'll simply hog the interpreter and no one else will be able to run. This the principle. I still don't understand your question. -- Cordially, Noufal http://nibrahim.net.in ___ BangPypers mailing list BangPypers@python.org https://mail.python.org/mailman/listinfo/bangpypers ___ BangPypers mailing list BangPypers@python.org https://mail.python.org/mailman/listinfo/bangpypers
Re: [BangPypers] real use case of gevent context switch gevent.sleep(0)
On 2015-06-21 19:48, anu sree wrote: Thanks noufal, I got it. After gevent.spawn, we have to do join or gevent.joinall or gevent.sleep or patched library call to start the greenlets, right ? join (and joinall) will pause the current greenlet till the ones you've joined terminate (similar to the threading.Thread join method). gevent.sleep will suspend the execution of the current greenlet and let other greenlets get some time to execute. I have see the code which not using any of these after gevent.spawn, How it is working there ? https://github.com/Juniper/contrail-controller/blob/master/src/opserver/uveserver.py#L656 Neither of these are *necessary*. Just creating a greenlet and `.start`ing it will make it run similar to a thread as long as other greenlets don't hog the CPU and have some calls that can context switch. ___ BangPypers mailing list BangPypers@python.org https://mail.python.org/mailman/listinfo/bangpypers
Re: [BangPypers] real use case of gevent context switch gevent.sleep(0)
On Sun, Jun 21 2015, anu sree wrote: [...] This code has two pause (gevent.sleep(0)), in Consumer.start and Worker.run. Here control goes to Worker.run when Consumer.start pauses and Consumer.start gets control back when Worker.run pauses. There may be benefit from this switching, but I am still not understanding it. [...] The idea is that any non blocking call is an opportunity to switch between greenlets. The monkey patching system will enable this. If you don't have any place in your greenlets that are monkey patched (and therefore allow switching), you can be a good citizen and drop in a patched sleep call so that other greenlets will get time to run. Otherwise, you'll simply hog the interpreter and no one else will be able to run. This the principle. I still don't understand your question. -- Cordially, Noufal http://nibrahim.net.in ___ BangPypers mailing list BangPypers@python.org https://mail.python.org/mailman/listinfo/bangpypers
Re: [BangPypers] real use case of gevent context switch gevent.sleep(0)
Hi Krace, I am still not understanding from code (practical example) where we are getting the benefit of PAUSE and let other greenlet to execute. I have created a simple worker program from your example https://github.com/dcramer/taskmaster/blob/79a312c5cb3c34d00829fe9cf4204aeb478a0166/src/taskmaster/client.py#L26 . This code has two pause (gevent.sleep(0)), in Consumer.start and Worker.run. Here control goes to Worker.run when Consumer.start pauses and Consumer.start gets control back when Worker.run pauses. There may be benefit from this switching, but I am still not understanding it. Please check the attached code and output. Is my example code valid ? CODE == import gevent class Worker(object): def __init__(self, consumer, target): self.consumer = consumer self.target = target def run(self): print Worker.run, START self.started = True while self.started: print Greenlet thread (from Worker.run) Pauses, START gevent.sleep(0) print Greenlet thread (from Worker.run) Pauses, END job = self.consumer.get_job() self.target(job) print Worker.run, END class Consumer(object): def __init__(self, target): self.target = target def start(self): print Consumer.start, START self.started = True worker = Worker(self, self.target) gevent.spawn(worker.run) while self.started: print Main Interpreter greenlet (from Consumer.start) Pauses, START gevent.sleep(0) print Main Interpreter greenlet (from Consumer.start) Pauses, END \n print Consumer.start, END def get_job(self): return 1 # def my_target_1(n): print my_target_1, START nums = [] while n 0: nums.append(n) n -= 1 print my_target_1, END c = Consumer(my_target_1) c.start() OUTPUT === Consumer.start, START Main Interpreter greenlet (from Consumer.start) Pauses, START Worker.run, START Greenlet thread (from Worker.run) Pauses, START Main Interpreter greenlet (from Consumer.start) Pauses, END Main Interpreter greenlet (from Consumer.start) Pauses, START Greenlet thread (from Worker.run) Pauses, END my_target_1, START my_target_1, END Greenlet thread (from Worker.run) Pauses, START Main Interpreter greenlet (from Consumer.start) Pauses, END Main Interpreter greenlet (from Consumer.start) Pauses, START Greenlet thread (from Worker.run) Pauses, END my_target_1, START my_target_1, END Greenlet thread (from Worker.run) Pauses, START Main Interpreter greenlet (from Consumer.start) Pauses, END ___ BangPypers mailing list BangPypers@python.org https://mail.python.org/mailman/listinfo/bangpypers
[BangPypers] real use case of gevent context switch gevent.sleep(0)
Hi, I have seen an example of gevent context switch http://sdiehl.github.io/gevent-tutorial/#synchronous-asynchronous-execution Could you please give some real use case with example. It would be fine If you can share link of github project which uses gevent context gevent.sleep(0). Thanks, ___ BangPypers mailing list BangPypers@python.org https://mail.python.org/mailman/listinfo/bangpypers
Re: [BangPypers] real use case of gevent context switch gevent.sleep(0)
Citation from Taskmaster [1], [2]. [1]: https://github.com/dcramer/taskmaster/blob/79a312c5cb3c34d00829fe9cf4204aeb478a0166/src/taskmaster/client.py#L26 [2]: https://github.com/dcramer/taskmaster/blob/79a312c5cb3c34d00829fe9cf4204aeb478a0166/src/taskmaster/server.py#L199 The reason to put sleep(0) is to ensure other greenlets gets a chance to execute. On Sat, Jun 20, 2015 at 2:41 PM, anu sree anusree@gmail.com wrote: Hi, I have seen an example of gevent context switch http://sdiehl.github.io/gevent-tutorial/#synchronous-asynchronous-execution Could you please give some real use case with example. It would be fine If you can share link of github project which uses gevent context gevent.sleep(0). Thanks, ___ BangPypers mailing list BangPypers@python.org https://mail.python.org/mailman/listinfo/bangpypers -- *Thanks RegardskracekumarTalk is cheap, show me the code -- Linus Torvaldshttp://kracekumar.com http://kracekumar.com* ___ BangPypers mailing list BangPypers@python.org https://mail.python.org/mailman/listinfo/bangpypers
Re: [BangPypers] real use case of gevent context switch gevent.sleep(0)
On Sat, Jun 20 2015, anu sree wrote: Hi, I have seen an example of gevent context switch http://sdiehl.github.io/gevent-tutorial/#synchronous-asynchronous-execution Could you please give some real use case with example. It would be fine If you can share link of github project which uses gevent context gevent.sleep(0). [...] The difference between a regular sleep and a gevent sleep is that the gevent sleep will not make the whole interpreter process sleep. It will instead, only make the current greenlet sleep which essentially tells the internal scheduler to not execute it till the time is over. You can see this happen like this. Make a greenlet in your interpreter like so from gevent import Greenlet class Printer(Greenlet): ... def _run(self): ... n = 0 ... while True: ... n += 1 ... print n This simply prints numbers starting from 1. now you can create this greenlet and start it p = Printer() p.start() Nothing will happen. This is cooperative multitasking where the current Greenlet (your interpreter shell) has to give up its time slice and let p execute. You can do this by monkey patching sleep from gevent import monkey monkey.patch_time() import time time.sleep(1) Now, control is handed over to the Greenlet. However, it is not cooperative and runs a tight loop without ever giving up control. This means, that you'll never be able to execute your own interpreter again. So you see, the various greenlets have to cooperate to get things done. This is in counter distinction to how preemptive multitasking works where the OS can force the currently executing thread or process to stop and then grant execution to something else. They don't need to be nice. If your greenlet looked like this class Printer(Greenlet): ...def _run(self): ... n = 0 ... while True: ... n += 1 ... time.sleep(0.1) ... print n the slot would be given up inside each iteration and other greenlets can execute. Try it. So people stick in a call to a monkey patched version of sleep inside tight loops to suspend the current interpreter for a short period of time thereby letting other greenlets execute. Krace's example is a good one. -- Cordially, Noufal http://nibrahim.net.in ___ BangPypers mailing list BangPypers@python.org https://mail.python.org/mailman/listinfo/bangpypers
Re: [BangPypers] real use case of gevent context switch gevent.sleep(0)
Thanks naufal and Krace. I have tried following code. I have given gevent.sleep(0.1), that means each greenlet let others to execute 0.1 secoond. But here each greenlet waiting for others to complete, why? Is it because of greenlet.joinall ? Here I have created 3 greenlet threads (A, B, C). I think, A is the parent greenlet, right ? Once C get finished control should goes to B, right ? But here it going to A. Please check the ouput. import gevent def task(num): thread_id = id(gevent.getcurrent()) print Start: , num, thread_id for i in range(10001): if i == 1: print num + str(i) gevent.sleep(0.1) for i in range(10001): if i == 1: print num + str(i) print End: , num, thread_id threads = [gevent.spawn(task, x) for x in ['A', 'B', 'C']] gevent.joinall(threads) Output === Start: A 139984038735696 A1 Start: B 139984008421616 B1 Start: C 139984008421776 C1 A1 End: A 139984038735696 B1 End: B 139984008421616 C1 End: C 139984008421776 ___ BangPypers mailing list BangPypers@python.org https://mail.python.org/mailman/listinfo/bangpypers
Re: [BangPypers] real use case of gevent context switch gevent.sleep(0)
On Sun, Jun 21, 2015 at 5:55 AM, anu sree anusree@gmail.com wrote:
Thanks naufal and Krace.
I have tried following code. I have given gevent.sleep(0.1), that means
each greenlet let others to execute 0.1 secoond. But here each greenlet
waiting for others to complete, why? Is it because of greenlet.joinall ?
Gevent has the hub/scheduler. The whole job is to schedule the tasks. AFAIK,
the hub uses queue to schedule them. If the task has a sleep [1], task is
enqueued for later execution [2].
By saying sleep, gevent puts task out of execution and doesn't guarantee
execution once time is elapsed.
Here I have created 3 greenlet threads (A, B, C). I think, A is the parent
greenlet, right ? Once C get finished control should goes to B, right ?
But here it going to A.
Please check the ouput.
import gevent
def task(num):
thread_id = id(gevent.getcurrent())
print Start: , num, thread_id
for i in range(10001):
if i == 1:
print num + str(i)
gevent.sleep(0.1)
for i in range(10001):
if i == 1:
print num + str(i)
print End: , num, thread_id
threads = [gevent.spawn(task, x) for x in ['A', 'B', 'C']]
gevent.joinall(threads)
Output
===
Start: A 139984038735696
A1
Start: B 139984008421616
B1
Start: C 139984008421776
C1
A1
End: A 139984038735696
B1
End: B 139984008421616
C1
End: C 139984008421776
Parent in gevent context is Hub(run inside an event loop). Like unix
process concept, gevent doesn't have parent process and child process.
Let's say if a task is spawn inside gevent task, they are still individual
tasks.
In [24]: def task(langs):
print inside task, gevent.getcurrent(), gevent.getcurrent().parent
def echo(msg):
print msg, gevent.getcurrent(), gevent.getcurrent().parent
tasks = [gevent.spawn(echo, lang) for lang in langs]
gevent.joinall(tasks)
:
In [25]: gevent.joinall([gevent.spawn(task, langs) for langs in [('Python',
'Go'), ('Lisp', 'Clojure')]])
inside task Greenlet at 0x1025269b0: task(('Python', 'Go')) Hub at
0x1025262d0 select default pending=0 ref=1
inside task Greenlet at 0x102526f50: task(('Lisp', 'Clojure')) Hub at
0x1025262d0 select default pending=0 ref=2
Python Greenlet at 0x102526cd0: echo('Python') Hub at 0x1025262d0 select
default pending=0 ref=3
Go Greenlet at 0x102526d70: echo('Go') Hub at 0x1025262d0 select default
pending=0 ref=3
Lisp Greenlet at 0x1025265f0: echo('Lisp') Hub at 0x1025262d0 select
default pending=0 ref=3
Clojure Greenlet at 0x102526c30: echo('Clojure') Hub at 0x1025262d0
select default pending=0 ref=3
___
BangPypers mailing list
BangPypers@python.org
https://mail.python.org/mailman/listinfo/bangpypers
[1]: https://github.com/gevent/gevent/blob/master/gevent/hub.py#L80
[2]:
https://github.com/gevent/gevent/blob/1599210203836560698a5facac015c31735cac3d/gevent/corecffi.py#L402
--
*Thanks RegardskracekumarTalk is cheap, show me the code -- Linus
Torvaldshttp://kracekumar.com http://kracekumar.com*
___
BangPypers mailing list
BangPypers@python.org
https://mail.python.org/mailman/listinfo/bangpypers
