Re: [Twisted-Python] twistd/tac : when does logging start?
On Wed, 2011-11-02 at 04:38 -0700, Tobias Oberstein wrote: I've got a Twisted application started via a tac file. Within the tac, I also initialize a database and create a db pool for use with adbapi. Now it seems that log messages from the network services are logged, but not from the initial DB setup stuff. When exactly does Twistd start the logging? How can I make it start logging from the very beginning? I've looked at the docs: Before startLogging is called, log messages will be discarded and errors will be written to stderr. If you are using twistd to run your daemon, it will take care of calling startLogging for you, ... but that doesn't help me. Any hints are welcome Hi Tobias, You need to do your intialisation (database connections, etc) in a startService method of your application, rather than (as you're probably doing) in the __init__ method. Hope this helps. -- Best Regards, Luke Marsden CTO, Hybrid Logic Ltd. Web: http://www.hybrid-cluster.com/ Hybrid Web Cluster - cloud web hosting Mobile: +447791750420 (UK) / +1-415-449-1165 (US) ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Announcement: txHybridCluster client released
Hi all, From a blustery grey Monday morning in Bristol, England, comes the announcement of the immediate availability of a Twisted client for interacting with the Hybrid Cluster API. https://github.com/hybridlogic/txHybridCluster http://www.hybrid-cluster.com/ http://www.hybrid-cluster.com/api/ The Hybrid Cluster API allows you to set up websites, databases, mailboxes, add SSL certificates, configure custom DNS, and more (over 100 API functions are documented). If anyone's interested in building cool stuff with this, contact me off-list and we can get you set up with a cluster to play with ;-) -- Best Regards, Luke Marsden CTO, Hybrid Logic Ltd. Web: http://www.hybrid-cluster.com/ Hybrid Web Cluster - cloud web hosting Mobile: +447791750420 (UK) / +1-415-449-1165 (US) ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Moving Twisted off Trac and SVN to somewhere nicer
Hi all, On Fri, 2011-07-01 at 12:00 +0100, Reza Lotun wrote: On Fri, Jul 1, 2011 at 11:29 AM, Laurens Van Houtven _...@lvh.cc wrote: Although I've hated git for a long while (and I still don't like it very much), I firmly believe Github is the right thing for Twisted. My incredibly unscientific poll amongst people who like Twisted but aren't devs is that they all love or at least like Github, and a surprising number has a distaste for Launchpad (unfamiliarity with Bazaar, perceived developer-unfriendly UI, slowness). I vote for GitHub too. Git's a pain but powerful (no need for combinator), and GitHub has a pretty good API into everything - low level repo innards, to issues/tickets. Also, GitHub's webhook system can easily integrate with buildbot (we used to do it at TweetDeck). Code review is doable by pull requests too. You can even map twistedmatrix.com to a github hosted website, which itself would be a repo. Oh, and the wikis are git repos too. Given all these tools, I see mappings for all of Twisted's bits and pieces (unless I'm missing something). I agree, possibly the biggest win with GitHub is the way it encourages fellow users to fork a project and contribute patches via pull requests. Popularity doesn't always equate to quality but in this case (amongst developers) I think it *is* indicative that they've got something right. Pull requests can be used to implement the Twisted review model as they form a good centralised place to review a set of changes. GitHub's issue tracker used to be pretty shoddy but has had a big upgrade recently and is almost good now. Git is powerful, and while it can also be confusing at times StackOverflow almost always has the answer ;-) Also, managing branches in git really is real pleasure compared to the mish-mash of merging branches with SVN and various external scripts. For example, to merge 'somebranch' into master (i.e. trunk) and push it to GitHub (the 'origin' remote), after committing to somebranch, it's just: git checkout master; git merge somebranch; git push origin master Addressing the concern of getting your data out of GitHub, since git is a DVCS, every repository is a complete copy of the entire revision history. Therefore, GitHub cannot lock you in. I suppose the issue tracker might be a different story, but it has an API. -- Best Regards, Luke Marsden CTO, Hybrid Logic Ltd. Mobile: +447791750420 www.hybrid-cluster.com - Cloud web hosting platform ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Question about starving the reactor
On Mon, 2011-06-20 at 11:09 -0400, Glyph Lefkowitz wrote: On Jun 20, 2011, at 7:39 AM, Phil Mayers wrote: On 20/06/11 11:29, Orestis Markou wrote: It might be there's no way to do guarantee something like that, so it might be just that everyone should be careful about this. I don't think you can stop callers of a function doing silly things with the return value. +1. The best way to deal with this is to make your APIs nice and simple, and their implementation straightforward. If you try to do weird tricks to make callbacks on your Deferreds cooperative, then you might break an otherwise reasonable strategy on the part of the client code to be well-behaved on their part. For the code that is itself trying to be well-behaved, there are things like twisted.internet.task.cooperate. There might not be any way to stop consumers of your Deferreds executing blocking operations, but when trying to track down which pieces of code are the culprits, I found exarkun's BigTimesliceTimer very useful: http://twistedmatrix.com/trac/browser/sandbox/exarkun/btt.py It uses a reactor-independent timing mechanism (the setitimer sigcall on Linux, contact me if you want the FreeBSD version) to set an alarm which the reactor then has to race to unset, else the itimer handler gets pre-emptively executed. When the alarm runs, it prints a traceback of the current execution point in the client code. By setting the alarm frequency sufficiently low and by watching for sufficiently long you can sample the code which is running while the reactor is blocked. Code paths which show up frequently are therefore *more likely* to be the culprits. One day it would be nice to turn this into some kind of statistical tool for highlighting which code paths are the hot-spots in your code, so that you can optimise the blockiest bits first. Premature optimisation, etc. Hope this helps; it helped me :-) -- Best Regards, Luke Marsden CTO, Hybrid Logic Ltd. Web: http://www.hybrid-cluster.com/ Hybrid Web Cluster - cloud web hosting Phone: +447791750420 ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] 100% CPU on high opened descriptors
Hi Juan, You could try: print reactor.__class__ This should allow you to check that the twistd command-line argument is working. Example: luke@pow:~$ twistd -noy foo.py |head -n 1 class 'twisted.internet.selectreactor.SelectReactor' luke@pow:~$ twistd -r poll -noy foo.py |head -n 1 class 'twisted.internet.pollreactor.PollReactor' luke@pow:~$ cat foo.py from twisted.internet import reactor from twisted.application import service print reactor.__class__ application = service.Application(nothing) -- Best Regards, Luke Marsden CTO, Hybrid Logic Ltd. Web: http://www.hybrid-cluster.com/ Hybrid Web Cluster - cloud web hosting Phone: +447791750420 On Thu, 2011-05-19 at 21:57 +0200, Juan Antonio Ibañez Santorum wrote: Is there any way to check if it is using the the poll/epoll reactor? Regards 2011/4/26 Luke Marsden luke-li...@hybrid-logic.co.uk On Tue, 2011-04-26 at 17:42 +0200, Juan Antonio Ibañez Santorum wrote: Hello! I have a Linux box serving my twisted app. All works ok except after some days running (where I can see my CPU going to 100%). I saw that CPU goes 100% usage when the app reaches to 1024 opened descriptors (sockets). You can also switch to using the poll reactor, which will allow you to sustain more than 1024 connections. You should still figure out what is keeping your connections open and find a way to reap them however, as this will just delay the problem. As Glyph suggested an application level ping may help here. In the mean-time you can switch to the poll reactor by adding -r poll to your twistd command. -- Best Regards, Luke Marsden CTO, Hybrid Logic Ltd. Web: http://www.hybrid-cluster.com/ Hybrid Web Cluster - cloud web hosting Phone: +447791750420 ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] 100% CPU on high opened descriptors
2011/4/26 Luke Marsden luke-li...@hybrid-logic.co.uk You can also switch to using the poll reactor On Wed, 2011-04-27 at 08:01 +0200, Juan Antonio Ibañez Santorum wrote: Is that the unique way to get mor than 1024 simultaneous connections? Regads On Linux and BSD, yes. See http://twistedmatrix.com/documents/current/core/howto/choosing-reactor.html for all the options for different platforms. -- Best Regards, Luke Marsden CTO, Hybrid Logic Ltd. Web: http://www.hybrid-cluster.com/ Hybrid Web Cluster - cloud web hosting Phone: +447791750420 ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Announcing txMySQL - native async Twisted MySQL protocol
Hi everyone, I'm pleased to announce the release of txMySQL, a native Twisted MySQL protocol implementation at https://github.com/hybridlogic/txMySQL The bulk of this code is courtesy of _habnabit (thank you!), I just added authentication support and fixed a couple of bugs which were stopping the MySQL protocol parser working. As we will be using this in production at Hybrid Logic I am happy to take over maintenance of it. As you can tell from the TODO list in the README, this is still a fairly rough draft, but it works well enough to .fetchall() basic results sets and .query() any other MySQL statements you care to run. See example.py. The incentive to produce this was due to a critical bug in MySQLdb with threading under load (on FreeBSD) which was causing Python to lock up. I decided that a native client protocol implementation would be less work than tracking down that bug, and besides it's something I've wanted to do for a while ;-) It might be interesting to see how adbapi could be modified to use a DeferredSemaphore to implement a completely threadless pool of MySQL connections with this...? Feel free to fork, tweak, fix, use, report issues, etc. -- Best Regards, Luke Marsden CTO, Hybrid Logic Ltd. Web: http://www.hybrid-cluster.com/ Hybrid Web Cluster - cloud web hosting Phone: +441172232002 / +16179496062 ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Boston Area Mini-Sprint
Damn, I'm going to be in Cambridge for 10 weeks from Wednesday 19th onwards (usually based here in the UK). So I miss out by 3 days! I'd really like to meet up with any Twisted folk when I arrive though. Is anyone Boston-based up for a drink some time after the 19th? FYI I'm the guy behind http://twistedmatrix.com/trac/wiki/SuccessStories#HybridLogic :-) -- Best Regards, Luke Marsden CTO, Hybrid Logic Ltd. Web: http://www.hybrid-cluster.com/ Hybrid Web Cluster - cloud web hosting Mobile: +447791750420 On Thu, 2011-01-13 at 12:32 -0500, Tenth wrote: Apologies in advance for the short notice, but I'm planning to host a Boston area Twisted mini-sprint this weekend on Sunday, January 16th. Space is fairly limited (hence the mini- prefix), but if you're on twisted-python and would be interested in the details of this and/or future Boston area sprints, let me know, and I'll add you to the invitation. Thanks, - Dave ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] how to implement sleep?
Hi Ruslan, Try this to drop in to your code, as in: sleep(3) def sleep(secs): d = defer.Deferred() reactor.callLater(secs, d.callback, None) return d Cheers, Luke On Sun, 2010-09-19 at 16:04 +0400, ruslan usifov wrote: Hello Twisted is great, but how cant i emulate sleep behaviour in syested. I know that exists callLater, but how to use it in yield scenario, like here: @inlineCallbacks def work(self): l_attempts = 0; while l_attemps 3: try: resp = yield some twisted call break; except: yield sleep(3); l_attempts +=1 else: log.err(cant communicate); Here i make 3 attempts to connect busy service, with pause 3 seconds between attempts. How cant i implement this in twisted ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python -- Best Regards, Luke Marsden Hybrid Logic Ltd. Web: http://www.hybrid-cluster.com/ Hybrid Web Cluster - cloud web hosting based on FreeBSD and ZFS Mobile: +447791750420 ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted HTTP client supporting failover for multiple A records?
Hi David, That's not to say using multiple A records isn't a helpful practice for many sorts of outages (especially to permit controlled maintenance). Just don't expect it to necessarily be sufficient in all failure modes depending on the behavior you want clients to experience. Indeed, in our application it's considered an optimisation over DNS failover. This is why we also use a low TTL (30 seconds) to purge the bad A records out of the pool as soon as possible. If this is strictly limited to a client you control, it's much less of an issue, since you can drop the TCP connect timeout much lower than what it defaults to, though you still probably can't match how fast it can happen for rejected connections, since you'll want to leave enough room for occasional latency or response time issues without immediately failing over. But you can do a lot better than the system defaults. Unfortunately we have no control over the clients' configuration (this is a LAMP web hosting environment). But 30 seconds is considered much more acceptable than the days it can often take a manual repair job if a server goes down. -- Best Regards, Luke Marsden Hybrid Logic Ltd. Web: http://www.hybrid-cluster.com/ Hybrid Web Cluster - cloud web hosting based on FreeBSD and ZFS Mobile: +447791750420 ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Twisted HTTP client supporting failover for multiple A records?
Hi all, Is there any existing support for any Twisted HTTP client to simulate the behaviour of all modern browsers in that -- if an address returns multiple A records -- and if one IP fails (connection refused, etc) then the client attempts a number of the other IPs before giving up? If not, where should I start? I understand that client.Agent is more modern than client.getPage. Thanks for an awesome framework! -- Best Regards, Luke Marsden Hybrid Logic Ltd. Web: http://www.hybrid-cluster.com/ Hybrid Web Cluster - cloud web hosting based on FreeBSD and ZFS Mobile: +447791750420 ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted HTTP client supporting failover for multiple A records?
On Thu, 2010-07-15 at 08:06 -0400, Itamar Turner-Trauring wrote: On Thu, 2010-07-15 at 10:46 +0100, Reza Lotun wrote: As for connecting to hosts that resolve to multiple A records - I presume as a means of load balancing via DNS round robin We're actually using it to provide redundancy in this instance. In our application any request for any site can be made to any (live) server, so having dead servers in the pool of A records doesn't matter so long as real web browsers failover to some other A record within a second, which they do! http://crypto.stanford.edu/dns/dns-rebinding.pdf The problem is that my test application uses client.getPage which, because it uses the reactor's standard DNS lookup mechanism, picks just one A record and sticks to it. So, it reports connection errors (some fraction of the time, as A records are randomised) even when the user of a real web browser would not experience them. These errors go away when the dead server(s) drop out of the DNS pool and reactor's lookups stops returning the dead IP, but this takes some time. Gar. I should read better. Twisted uses a threadpool of gethostname by default, but you can plug in your own resolver (e.g. you can use twisted.names): http://twistedmatrix.com/documents/10.1.0/api/twisted.internet.interfaces.IReactorPluggableResolver.html#installResolver The question is whether the client code re-resolves on each re-connect, and whether the current lookup interface is sufficient for this use case. Alas, I'm pretty sure the answer is no. You could however always just do the DNS lookup yourself, passing resulting correct IP to connectTCP, just make sure you don't block (e.g. by using deferToThread to call gethostbyname_ex). Thanks Itamar, this is massively useful. I'll try subclassing twisted.web.client.Agent to do its own DNS lookups with twisted.names so as to be aware of the full list of A records returned. It would then attempt all the IP addresses in turn until it finds one which works, giving up only if all the IPs yield connection errors. This should mirror the behaviour of the majority of web browsers in the wild. Would you be interested in having this code contributed back to Twisted if I can get it working? It might be a useful addition to the Agent. -- Best Regards, Luke Marsden Hybrid Logic Ltd. Web: http://www.hybrid-cluster.com/ Hybrid Web Cluster - cloud web hosting based on FreeBSD and ZFS Mobile: +447791750420 ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted HTTP client supporting failover for multiple A records?
On Thu, 2010-07-15 at 14:28 +0100, Reza Lotun wrote: I suspect you can do this without subclassing... pass in IP address, and just make sure you pass correct Host header. I forget the exact API though. This makes sense. Conceptually I had considered it to be the responsibility of the web client itself to handle the reconnection, not the calling code, hence my plan for a subclass. But a separate class which uses the Agent's API makes a lot more sense, and it can equally provide the same interface as Agent so that any code which uses Agent can use it without modifications. Yeah, I was about to say, why not just call socket.gethostbyname_ex in deferToThread and in the callback do a regular Agent.request? Sounds like a plan! Thanks guys. -- Best Regards, Luke Marsden Hybrid Logic Ltd. Web: http://www.hybrid-cluster.com/ Hybrid Web Cluster - cloud web hosting based on FreeBSD and ZFS Mobile: +447791750420 ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
