Re: How to benchmark a HTTP connection with requests module?

[Etienne Robillard]

Hi Dennis,

Here's my code so far:

import os
import requests
import threading
import time

from Queue import Queue
from test_support import unittest

test_queue = Queue()

def worker(ident, url, queue):

    while True:
    start = time.clock()
    r = requests.get(url)
    end = time.clock()
    queue.put((ident, r.status_code, end - start))
    queue.task_done()
    return queue

class HTTPBenchmarkTestCase(unittest.TestCase):

    url = 'http://localhost/benchmark/'
    threads = 5

    def setUp(self):
    self.hold = True


    def test_benchmark_concurrency(self):
    for i in range(self.threads):
    t = threading.Thread(target=worker, args=(i, self.url, 
test_queue))

    t.daemon = True
    t.start()
    obj = test_queue.get()
    print obj
    test_queue.join()


erob@marina:~/src/django-hotsauce-0.9/tests$ pypy ./run.py -C benchmarks
DEBUG:Starting new HTTP connection (1): localhost
DEBUG:http://localhost:80 "GET /benchmark/ HTTP/1.1" 200 76
DEBUG:Starting new HTTP connection (1): localhost
(0, 200, 0.1299950349943)
DEBUG:Starting new HTTP connection (1): localhost
DEBUG:http://localhost:80 "GET /benchmark/ HTTP/1.1" 200 76
DEBUG:Starting new HTTP connection (1): localhost
(1, 200, 0.05324292899606)
DEBUG:Starting new HTTP connection (1): localhost
DEBUG:http://localhost:80 "GET /benchmark/ HTTP/1.1" 200 76
DEBUG:Starting new HTTP connection (1): localhost
(0, 200, 0.1122124810025)
DEBUG:Starting new HTTP connection (1): localhost
DEBUG:http://localhost:80 "GET /benchmark/ HTTP/1.1" 200 76
DEBUG:Starting new HTTP connection (1): localhost
(1, 200, 0.1207582250033)
DEBUG:Starting new HTTP connection (1): localhost
DEBUG:http://localhost:80 "GET /benchmark/ HTTP/1.1" 200 76
DEBUG:Starting new HTTP connection (1): localhost
DEBUG:http://localhost:80 "GET /benchmark/ HTTP/1.1" 200 76
DEBUG:Starting new HTTP connection (1): localhost
(0, 200, 0.1259027660028)
.
--
Ran 1 test in 0.468s

OK


What do you think? :-)

Etienne


Le 2018-02-16 à 10:58, Dennis Lee Bieber a écrit :

On Fri, 16 Feb 2018 06:22:04 -0500, Etienne Robillard 
declaimed the following:


Hi Dennis,

Nice pseudo code! :-)

Is it possible benchmark/measure the latency of a HTTP connection for
each threads by timing the duration of the requests.get method to complete?


Wrap each .get with calls for start time and end time? (Need to check
if time.time or time.clock is more precise on your hardware, docs recommend
time.clock for benchmarking)


def worker(ID):
while hold: pass

st = time.clock()

r = requests.get(...)

en = time.clock()
resultQ.put( (ID, r.status_code, en - st) )
#return duration as third parameter



for _ in range(NUMBEROFREQUESTS):

(ID, code, duration) = resultQ.get()

requestTasks[ID].join()


--
Etienne Robillard
tkad...@yandex.com
https://www.isotopesoftware.ca/

--
https://mail.python.org/mailman/listinfo/python-list

Re: How to benchmark a HTTP connection with requests module?

[Etienne Robillard]

Hi Dennis,

Nice pseudo code! :-)

Is it possible benchmark/measure the latency of a HTTP connection for 
each threads by timing the duration of the requests.get method to complete?


I also plan to test with gevent.monkey extension for enabling 
cooperative multithreading support:


from gevent import monkey

monkey.patch_all()


Etienne

Le 2018-02-15 à 11:56, Dennis Lee Bieber a écrit :


Keyword: threads

Create a number of threads, each handling one request, and use a global
flag to start them. And maybe a queue to retrieve

Pseudo-code (I haven't checked the manual for correct API):

hold = True
resultQ = Queue.Queue()

def worker(ID):
while hold: pass
r = requests.get(...)
resultQ.put( (ID, r.status_code) )


def benchmark():
requestTasks = [ threading.thread( worker, args=(ID) )
for ID in range(NUMBEROFREQUESTS) ]
for rT in requestTasks.start()  #or is it .run()

#workers are now busy waiting for hold to False
#better would be to use threading.Condition and .notifyAll()
#having each thread wait on a global condition variable,
#rather than spinning on a busy wait
hold = False

for _ in range(NUMBEROFREQUESTS):
(ID, code) = resultQ.get()
requestTasks[ID].join()

#put any asserts here


--
Etienne Robillard
tkad...@yandex.com
https://www.isotopesoftware.ca/

--
https://mail.python.org/mailman/listinfo/python-list