Looking for the right library for a simple HTTP client
I am trying to implement a simple client that can do the following: 1)to send the following kinds of HTTP requests and validate responses 1.1 GET 1.2 POST with application/x-www-form-urlencoded encoding 1.3 POST with multipart/form-data encoding 2)to set any number of (even duplicate) headers. For example, I may intentionally add the following Cookie: headers to a request: Cookie: id_1=v1;Domain=sample.com;Path=/ Cookie: id_1=v1;Domain=sample.com;Path=/ <--same as the one above Cookie: id_2=v1;Domain=sample.com;Path=/ 3)to set proxy cfg so requests can go through a proxy server to reach the target server. 4)to send multiple requests simultaneously. I have Python 2.4.1 on Solaris 9 and 10, and I don't have any plan to upgrade to latest version of Python. I looked around and I found httplib and urllib. Are they sufficient for my tasks 1 to 3 above? Any good sample codes will be great. Thanks. -- http://mail.python.org/mailman/listinfo/python-list
logging outgoing HTTP POST message and incoming response message
I am sending a HTTP POST by using the following codes:
opener = urllib2.build_opener(proxyHandler, MultipartPostHandler)
params = { "audio" : "http://sample.com/my.wav";,
"data" : open(file, "rb") }
print opener.open(myURL, params).read()
How do I log or print out the actual POST message (including headers)
that is sent out?
How do I log or print out the response headers as well?
Thanks
--
http://mail.python.org/mailman/listinfo/python-list
Re: logging outgoing HTTP POST message and incoming response message
On Jul 22, 1:54 pm, "Diez B. Roggisch" wrote: > You can use proxy-tools such as tcpmon or sniff traffic using wireshark. > > Diez Thanks, but I am trying to enable some debug mode to log all outgoing and incoming messages for certain period of time, and running another proxy-tool is not very ideal. It would be great to log it in some log file. -- http://mail.python.org/mailman/listinfo/python-list
regex: multiple matching for one string
For example, I have a string "#a=valuea;b=valueb;c=valuec;", and I
will like to take out the values (valuea, valueb, and valuec). How do
I do that in Python? The group method will only return the matched
part. Thanks.
p = re.compile('#a=*;b=*;c=*;')
m = p.match(line)
if m:
print m.group(),
--
http://mail.python.org/mailman/listinfo/python-list
passing data to a liburl2 opener object
I have prepared my headers and data for a HTTP POST message; however,
I am not sure how to pass the data to the opener. Can you guys
provide some suggestions? Thanks.
proxy_handler = urllib2.ProxyHandler({'http': 'http://my.proxy.com:
3128/'})
opener = urllib2.build_opener(proxy_handler)
url = "http://sample.company.com:/service?";
headers['Content-Type'] = 'multipart/form-data; boundary=%s' %
myBoundar
headers['Cookie'] = 'somevalidcookiecontents'
#there is an opener.addheaders method for adding headers
data = multipart_encode(myDict)
#how to pass data to the opener???
With the codes above, I need to send out a HTTP POST like the
following one:
POST http://sample.company.com:/service?
Content-Type: multipart/form-data;
boundary=---1234
---1234
Content-Disposition: form-data; name="picture"
PICTURE contents here
---1234
Content-Disposition: form-data; name="id"
whateverid
---1234--
--
http://mail.python.org/mailman/listinfo/python-list
how can a child thread notify a parent thread its status?
My parent thread keeps a counter for the number of free child workers (say 100) and initializes some child threads and call child.start(). Once the number of free child workers reach 0, the parent thread will wait until some at least one child thread finishes and then it will initialize another child thread. My question is, how can a child thread notify the parent that it's done so that the parent can call join() on it? I am not sure how a child thread can send a signal to its parent while it may not even know anything about it's parent. Can you guys please provide some suggestions? Some code samples will be nice. Thanks. -- http://mail.python.org/mailman/listinfo/python-list
time in milliseconds by calling time.time()
I am trying to measure some system response time by using the time.time () or time.clock() in my script. However, the numbers I get are in 10s of milliseconds. For example, 1248481670.34 #from time.time() 0.08 #from time.clock() That won't work for me, since the response time may be only a few milliseconds. My environment is Solaris 10 with Python 2.4.4 (#7, Feb 9 2007, 22:10:21). SunOS 5.10 Generic_137112-07 i86pc i386 i86pc The tricky thing is, if I run the python interpreter and import the time module, I can get a time floating number in better precision by calling time.time(). Do you guys have any suggestion on debugging this problem? Or, is there any other module I can try? Thanks. $ python Python 2.4.4 (#7, Feb 9 2007, 22:10:21) [GCC 3.4.6] on sunos5 Type "help", "copyright", "credits" or "license" for more information. >>> import time >>> time.time() 1248481930.8023829 <--I like this! >>> time.clock() 0.0 >>> -- http://mail.python.org/mailman/listinfo/python-list
Re: how can a child thread notify a parent thread its status?
First of all, let me say thank you to all of you. I have asked many questions (some of them are dump questions), and you have kindly helped me. I am not going to reply every message to say thank-you since that would be annoying for such group with such high daily traffics. Thank you very much. Let's get back to topic of this message. Here's how I have implemented it so far, and I am taking the queue of work load items approach. In my child thread, I will keep checking for available work load item until a duration is reached. #inside the child# while endTime > time.time(): try: item = self.q.get(True, 3) except Queue.Empty: #what's wrong? AttributeError: class Queue has no attribute 'Empty' print 'cant find any work load item, so lets wait and try again later' time.sleep(1) #wait and then check again continue except: print "Unexpected error:", sys.exc_info()[0] raise #do the real work with load item In my parent thread, I will initialize X (depending on a cfg file) child threads and keep adding load items to a shared q until the duration is reached. #inside the parent# callCounter = 0 workers = [] #a list of child threads totalWorkers = 250 endTime = time.time() + duration for i in range(totalWorkers): w = Worker(q, duration, i) w.start() #worker, do your job now! workers.append(w) while endTime > time.time(): time.sleep(1) q.put(getWorkloadItem()) #add workload itmes callCounter += 1 #actually can we guarantee that the call will be sent?? #should we ask each child to report the number of calls they make? for i in range(totalWorkers): workers[i].join()# Wait for the child threads to finish Overall, it seems to be working now. Though, I still have a couple of problems to resolve. 1. I got the following error for the codes that attempt to catch Empty Queue exception. What's the right way to use it? except Queue.Empty: AttributeError: class Queue has no attribute 'Empty' 2. What's the best way to have each child thread to report the number of requests they send when they are done? To add the numbers to another queue? 3. I will need to do some logging for response time as well as some response contents. I have two choices, one big log file for all threads (both child and parent), and one log file for each thread. Given the fact that I may have to log tons of data, I think opening and maintaining a bunch of smaller logs may be better than dealing with a big one (it may grow very fast). Is there any best prastice for logging in Python? If I change my mind and go with one big log file (pass it to each thread), is there anything I should be aware of for multi-thread access (writting) to the same log file? Again, thank you. -- http://mail.python.org/mailman/listinfo/python-list
Re: how can a child thread notify a parent thread its status?
I decided to go with one big log file, which will be shared by all
threads (child and parent). A log message Queue is used to store all
log entries, and a customized logger thread will get log entries from
the Queue.
#from the logger thread#
def run(self):
while self.flag == 1: #if the flag is set to 0, the logger
thread should exit
try:
entry = self.q.get()
except Empty:
self.logger.debug('cant find any log entry')
continue
except:
self.logger.error("Unexpected error:", sys.exc_info()
[0])
raise
#do whatever that should be done
self.logger.info("logger thread done") #should see this
message in log file as well
def off(self):
self.logger.info('turning off flag')
self.flag = 0
#in parent thread#
logItemQ.put('We are done, lets stop the logger now.')
time.sleep(1) #it seems that the logger thread cannot exit if
I put a sleep here
myLog.off() #off is called successfully
myLog.join()
I put an off method to turn off a flag so the logger thread knows it
should exit. However, the last log message (the one 'We are done,
lets stop the logger now.') won't be logged if I call myLog.off() and
myLog.join() immediately. So, I put a time.sleep(1) to make sure the
logger thread has enough time to finish it's job. Unfortunately, now
the logger thread simply won't exit, and I don't see the message
'logger thread done'. I can't figure out at which point it hangs,
since I don't any new log entry but the thread simply won't exit.
Am I taking a right approach by using a flag? Should I lock the flag?
--
http://mail.python.org/mailman/listinfo/python-list
