Hi!,
On Thursday 27 August 2009 15:43:16 erikcw wrote:
> Hi,
>
> I can't seem to figure out how to make the HTTPClient work with a
> proxy server. My existing code uses urllib2 which works well with the
> Proxy -- I'd keep using it in Kamaeila, but if I'm not mistaken it
> will block and break concurrency (right?)
The HTTPClient component doesn't really handle proxies, which is why you're
not able to get that working.
In an Axon.Component.component, grabbing something using urllib2 would indeed
block the system, however, if you use a threaded component it won't.
As a quick example that will play nicely.
class UrlLib2Client(Axon.ThreadedComponent.threadedcomponent):
def main(self):
while not self.dataReady("control"):
for url in self.Inbox("inbox"):
X = urllib2.open(url)
page = X.read()
X.close()
self.send(page, "outbox")
self.pause()
self.send(self.recv("control"), "signal")
(Admitedly I've not tested this, but you get the idea :-)
The component above sits there waiting for URLs on its main inbox "inbox", and
sends complete pages out its outbox "outbox". Similarly it also honours
shutdown messages in that any message sent to its control inbox will cause it
to shutdown and also pass on the shutdown message to the next component.
Ideally HTTPClient would be extended to support proxies, but this is a
pragmatic solution, which will work.
Generally speaking, if you have any operations which you need to handle which
are blocking, and you can't transform them to non-blocking, that's really the
purpose behind the threaded components. It's possible at some point the two
component sorts may be unified and threading changed to a decorator which
delcares the component to be blocking, but kamaelia's dev tends to be driven
by pragmatic needs :)
>From a development perspective regarding threaded components ....
* Their main() function is just a normal function, not a generator
* self.pause() in threaded components actually contains a micro-sleep
inside it, and can also be used like this:
* self.pause(some_sleep_time)
Which is an interruptable sleep, unlike time.sleep(some_sleep_time)
which is not.
* Global services shouldn't (at present) be accessed directly from inside
one (I can expand on this, but this would be rare inside such a
component anyway - you're more likely to use a backplane)
* You continue to use the normal inbox/outbox API
* You use them in pipelines, graphlines, linkages etc in exactly the same
way as normal components.
* The inbox/outboxes are implemented behind the API differently however
using threadsafe queues.
... ie more or less what you'd want :-)
If you have any more q's please do say. Also, it'd be really interesting to
hear what you're building or playing with :-)
Best Regards,
Michael.
--
http://yeoldeclue.com/blog
http://twitter.com/kamaelian
http://www.kamaelia.org/Home
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"kamaelia" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/kamaelia?hl=en
-~----------~----~----~----~------~----~------~--~---
