Re: Batching HTTP requests with httplib (Python 2.7)
On 2012-09-14, Xavier Combelle xavier.combe...@free.fr wrote: Le 14/09/2012 12:56, Dwight Hutto a ?crit : service_num_list = [num for num in range(0,5)] for service_num in service_num_list: eval(web_service_call%i(%i) % (service_num,service_num)) service_num_list = [num for num in range(0,5)] service_num_list = list(range(5)) -- Neil Cerutti -- http://mail.python.org/mailman/listinfo/python-list
Re: Batching HTTP requests with httplib (Python 2.7)
instead of Le 14/09/2012 12:56, Dwight Hutto a écrit : service_num_list = [num for num in range(0,5)] for service_num in service_num_list: eval(web_service_call%i(%i) % (service_num,service_num)) service_num_list = [num for num in range(0,5)] for service_num in service_num_list: locals()[web_service_call%i % (service_num,)](service_num) would be safer eval is really bad -- http://mail.python.org/mailman/listinfo/python-list
Re: Batching HTTP requests with httplib (Python 2.7)
On 14Sep2012 10:53, Chicken McNuggets chic...@mcnuggets.com wrote:
| On 14/09/2012 03:31, Cameron Simpson wrote:
| On 13Sep2012 19:34, Chicken McNuggets chic...@mcnuggets.com wrote:
| | I'm writing a simple library that communicates with a web service and am
| | wondering if there are any generally well regarded methods for batching
| | HTTP requests?
| |
| | The problem with most web services is that they require a list of
| | sequential commands to be executed in a certain order to complete a
| | given task (or at least the one I am using does) so having to manually
| | call each command is a bit of a pain. How would you go about the design
| | of a library to interact with these services?
|
| Maybe I'm missing something. What's hard about:
|
| - wrapping the web services calls in a simple wrapper which
| composes the call, runs it, and returns the result parts
| This lets you hide all the waffle about the base URL,
| credentials etc in the wrapper and only supply the essentials
| at call time.
|
| - writing your workflow thing then as a simple function:
|
| def doit(...):
| web_service_call1(...)
| web_service_call2(...)
| web_service_call3(...)
|
| with whatever internal control is required?
|
| This has worked for me for simple things.
| What am I missing about the larger context?
|
| That is what I have at the moment but it is ugly as hell. I was
| wondering if there was a somewhat more elegant solution that I was missing.
Leading disclaimer: I'm no web service expert.
Well, I presume you're using a web service library like ZSI or suds?
I'm using suds; started with ZSI but it had apparent maintenance
stagnancy and also some other issues. I moved to suds and it's been
fine.
I make a class wrapping a suds Client object and some convenience
functions. A suds Client object is made from the WSDL file or URL
and fills out all the services with callable methods accepting the web
service parameters. So the class has a setup method like this:
def set_client(self, wsdlfile, baseurl):
''' Common code to set the .client attribute to a suds SOAP Client.
'''
from suds.client import Client
self.client = Client('file://'+wsdlfile)
self.client.set_options(location = baseurl)
self.client.set_options(proxy = {'https': 'proxy:3128'})
wsdlfile points at a local WSDL file, removing dependence on some server
to provide the WSDL. Makes testing easier too.
Then there's a general call wrapper. The wscall() method below is a
slightly scoured version of something I made for work. You hand it the
service name and any parameters needed for that service call and it looks
it up in the Client, calls it, sanity checks the result in a generic
sense. If things are good it returns the suds reply object. But if
anything goes wrong it logs it and returns None. Goes wrong includes
a successful call whose internal result contains an error response -
obviously that's application dependent.
The upshot is that the wrapper for a specific web service call then becomes:
def thing1(self, blah, blah, ...):
reply = self.wscall('thing1', blah, blah, ...)
if reply is None:
# badness happened; it has been logged, just return in whatever
# fashion you desire - return None, raise exception, etc
...
# otherwise we pick apart the reply object to get the bits that
# matter and return them
return stuff from the reply object
and your larger logic looks like:
def doit(self, ...):
self.thing1(1,2,3)
self.thing2(...)
and so forth.
The general web service wrapper wscall() below has some stuff ripped out,
but I've left in some application specific stuff for illustration:
- optional _client and _cred keyword parameters to override the
default Client and credentials from the main class instance
- the Pfx context manager stuff arranges to prefix log messages and
exception strings with context, which I find very handy in debugging and
logging
- likewise the LogTime context manager logs things that exceed a time
threshold
- the wsQueue.submit(...) stuff punts the actual web service call
through a capacity queue (a bit like the futures module) because
this is a capacity limited multithreaded app.
For your purposes you could write:
websvc = getattr(_client.service, servicename)
with LogTime(servicename, threshold=3):
try:
reply = websvc(userId=_cred.username, password=_cred.password,
*args, **kwargs)
except:
exc_info = sys.exc_info
reply = None
else:
exc_info = None
which is much easier to follow.
- the 'callStatus' thing is application specific, left as an example
of a post-call sanity check you would add for calls that complete
but return with an error indication
Anyway, all that said, the general wrapper looks like this:
def
Re: Batching HTTP requests with httplib (Python 2.7)
On 14/09/2012 03:31, Cameron Simpson wrote: On 13Sep2012 19:34, Chicken McNuggets chic...@mcnuggets.com wrote: | I'm writing a simple library that communicates with a web service and am | wondering if there are any generally well regarded methods for batching | HTTP requests? | | The problem with most web services is that they require a list of | sequential commands to be executed in a certain order to complete a | given task (or at least the one I am using does) so having to manually | call each command is a bit of a pain. How would you go about the design | of a library to interact with these services? | | Any help is greatly appreciated :). Maybe I'm missing something. What's hard about: - wrapping the web services calls in a simple wrapper which composes the call, runs it, and returns the result parts This lets you hide all the waffle about the base URL, credentials etc in the wrapper and only supply the essentials at call time. - writing your workflow thing then as a simple function: def doit(...): web_service_call1(...) web_service_call2(...) web_service_call3(...) with whatever internal control is required? This has worked for me for simple things. What am I missing about the larger context? That is what I have at the moment but it is ugly as hell. I was wondering if there was a somewhat more elegant solution that I was missing. -- http://mail.python.org/mailman/listinfo/python-list
Re: Batching HTTP requests with httplib (Python 2.7)
| The problem with most web services is that they require a list of | sequential commands to be executed in a certain order to complete a | given task (or at least the one I am using does) so having to manually | call each command is a bit of a pain. How would you go about the design | of a library to interact with these services? | | Any help is greatly appreciated :). Maybe I'm missing something. What's hard about: - wrapping the web services calls in a simple wrapper which composes the call, runs it, and returns the result parts This lets you hide all the waffle about the base URL, credentials etc in the wrapper and only supply the essentials at call time. - writing your workflow thing then as a simple function: def doit(...): web_service_call1(...) web_service_call2(...) web_service_call3(...) with whatever internal control is required? This has worked for me for simple things. What am I missing about the larger context? That is what I have at the moment but it is ugly as hell. I was wondering if there was a somewhat more elegant solution that I was missing. -- Well if the code works it works. There probably is a better way than what I came up with: def web_service_call0(y): print Service call = %i) % y def web_service_call1(y): print Service call = %i) % y def web_service_call2(y): print Service call = %i) % y def web_service_call3(y): print Service call = %i) % y def web_service_call4(y): print Service call = %i) % y service_num_list = [num for num in range(0,5)] for service_num in service_num_list: eval(web_service_call%i(%i) % (service_num,service_num)) I just define the service calls, and iterate through them with an eval that places in the service call num, but you would have to have a matching list of the params that would go with each one. I'm almost positive there will be some more well written ones on the way. -- Best Regards, David Hutto CEO: http://www.hitwebdevelopment.com -- http://mail.python.org/mailman/listinfo/python-list
Re: Batching HTTP requests with httplib (Python 2.7)
On 13Sep2012 19:34, Chicken McNuggets chic...@mcnuggets.com wrote: | I'm writing a simple library that communicates with a web service and am | wondering if there are any generally well regarded methods for batching | HTTP requests? | | The problem with most web services is that they require a list of | sequential commands to be executed in a certain order to complete a | given task (or at least the one I am using does) so having to manually | call each command is a bit of a pain. How would you go about the design | of a library to interact with these services? | | Any help is greatly appreciated :). Maybe I'm missing something. What's hard about: - wrapping the web services calls in a simple wrapper which composes the call, runs it, and returns the result parts This lets you hide all the waffle about the base URL, credentials etc in the wrapper and only supply the essentials at call time. - writing your workflow thing then as a simple function: def doit(...): web_service_call1(...) web_service_call2(...) web_service_call3(...) with whatever internal control is required? This has worked for me for simple things. What am I missing about the larger context? -- Cameron Simpson c...@zip.com.au Clymer's photographs of this procedure show a very clean head. This is a lie. There is oil in here, and lots of it. - Mike Mitten, rec.moto, 29sep1993 -- http://mail.python.org/mailman/listinfo/python-list
