Re: [Launchpad-dev] The with statement
On Wed, Jan 27, 2010 at 12:31:04PM -0500, Barry Warsaw wrote:
Now that we're on Python 2.5, we have a few new language features at our
disposal. In today's AMEU reviewer meeting, I brought up the 'with' statement
and was asked to write something up about it. The with statement will be okay
to use in Launchpad code, so you'll see more of it crop up, and it's good to
know how and when to use it.
First off, in Python 2.5 you need a future import to enable the with
statement. 'with' is a keyword in Python 2.6, so when we move to that version
we can get rid of all the future imports:
Thanks for the write-up, Barry. I think the with statement is good, but
I'd like to re-iterate one of Jeroen's concern (at least part of it, I
think).
For example, if you wanted to print every line in /etc/password, and you were
anal like me, you'd do something like:
fp = open('/etc/passwd')
try:
... for line in fp:
... sys.stdout.write(line)
... finally:
... # Don't wait for gc to free the file descriptor.
... fp.close()
Look at how much nicer this is using the with statement. This works because
built-in file objects already support the context manager protocol:
with open('/etc/passwd') as fp:
... for line in fp:
... sys.stdout.write(line)
I think this is an excellent use of with. It's well-known that files
work like this, so it's easy to understand, and the code gets cleaner.
It's also very easy to write your own context managers to work with the
with-statement. The documentation has all the gory details, but essentially,
you just need to implement an __enter__() method and an __exit__() method.
The former gets called when the with-statement starts, and the latter gets
called when it's done. The signature of __exit__() is a little tricky to
allow it to handle exceptions in the body of the with-statement, but usually
you don't care and will just let those percolate up.
think we should be a bit careful using it for other things than files.
I'm sure there are good use cases for using with with custom
implementations, but we should be careful not to over-use it. Just
because it's easy to write, doesn't mean that we should do it. It's very
easy to get carried away, wanting to use new features just because. It's
important to think about the end result.
So, when writing context manager implementations, you (and the reviewer)
should ask youself the question: how easy is it to see what's going on
when the with statement is used? The with statement does add a level of
magic. If you need to chase down the implementation to figure out what's
going on, it's not a good design. As I see it, the with statement is
mainly useful wehre you have a well-known behaviour, where it's easy to
find out what's going on (for example from the name), or when the
clean-up step isn't important. This doesn't mean that we should avoid
writing our own context managers at all cost, just that we should
evaluate the benefit. The more an object is used in the code, the more
likely it is that the behaviour will be well-known.
--
Björn Tillenius | https://launchpad.net/~bjornt
___
Mailing list: https://launchpad.net/~launchpad-dev
Post to : launchpad-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~launchpad-dev
More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] The with statement
On Feb 02, 2010, at 06:21 PM, Jeroen Vermeulen wrote:
(Irrelevant gripes: it's a thin layer of sugar;
True, but it's a tasty bit of sugar. I find that carefully used,
with-statements are more readable than try/finallys.
it doesn't scale well for more than resource;
'one'?--^
It's awkward to do in Python 2.6, but possible:
from contextlib import nested
with nested(open('/etc/passwd'), open('/etc/group')) as (a, b):
... pass
But easier in Python 2.7 and 3:
with open('/etc/passwd') as a, open('/etc/group') as b:
... pass
The justification for this design was transactions: it lets you write
transaction classes don't need explicit commits or aborts, because they
can see for themselves if the code block exits normally or with an
exception.
I'm not sure the with-statement was designed for transactions, but one good
reason why it shouldn't be used for transactions is because (IME) proper
handling doesn't fit into pure try/finally:
txn = transaction.begin()
try:
modify_the_database()
except:
# Yes, bare except.
txn.abort()
else:
# Ah, everything succeeded.
txn.commit()
That's not a good fit for 'with'. I suppose you can wedge it into the context
manager protocol, but it wouldn't be the first thing I'd reach for.
I guess that means we (mostly) agree! :)
-Barry
signature.asc
Description: PGP signature
___
Mailing list: https://launchpad.net/~launchpad-dev
Post to : launchpad-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~launchpad-dev
More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] The with statement
On Feb 10, 2010, at 05:28 PM, Bjorn Tillenius wrote:
think we should be a bit careful using it for other things than files.
I'm sure there are good use cases for using with with custom
implementations, but we should be careful not to over-use it. Just
because it's easy to write, doesn't mean that we should do it. It's very
easy to get carried away, wanting to use new features just because. It's
important to think about the end result.
I completely agree that we should be careful, and tasteful, but of course
that's true of any wacky Python feature, not just with-statements. :)
I sort of envision that we'll find some common patterns in our code that fit
well as context managers. I'm hoping that we can write a few generic classes,
and that we'll get used to seeing them in our code. Things like temporarily
changing the umask or cwd come to mind. Like a lot of these, it'll take some
time to propagate across our collective consciousness, so when the dev and
reviewer agree that a useful context manager has been written, documenting it
in the wiki and/or mailing list would be helpful.
So, when writing context manager implementations, you (and the reviewer)
should ask youself the question: how easy is it to see what's going on
when the with statement is used? The with statement does add a level of
magic. If you need to chase down the implementation to figure out what's
going on, it's not a good design. As I see it, the with statement is
mainly useful wehre you have a well-known behaviour, where it's easy to
find out what's going on (for example from the name), or when the
clean-up step isn't important. This doesn't mean that we should avoid
writing our own context managers at all cost, just that we should
evaluate the benefit. The more an object is used in the code, the more
likely it is that the behaviour will be well-known.
Right. Above all, if you do decide to write a context manager, making it read
well in English when used as the target of the 'with' is half the battle.
with cwd('/path/to/stuff'):
# ...
is pretty obvious.
-Barry
signature.asc
Description: PGP signature
___
Mailing list: https://launchpad.net/~launchpad-dev
Post to : launchpad-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~launchpad-dev
More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] The with statement
On Wed, 2010-02-10 at 14:42 -0500, Barry Warsaw wrote:
with cwd('/path/to/stuff'):
# ...
is pretty obvious.
And extremely dangerous, as well as trivially broken. Not safe to use
from any server code.
(And if its not obvious why its dangerous or breakable, that should
serve as pause!)
-Rob
signature.asc
Description: This is a digitally signed message part
___
Mailing list: https://launchpad.net/~launchpad-dev
Post to : launchpad-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~launchpad-dev
More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] Using HTTP basic auth on windmill tests
Nobody replied yet. +1 on using basic auth for me. I like having the test openid provider on a separate domain, and this limitation doesn't seem too bad, at least for now.. Gary On Feb 3, 2010, at 9:38 AM, Guilherme Salgado wrote: Hi there, I have a branch on which I changed the +login page to use OpenID, and when developing or testing it uses testopenid.lp.dev as the provider. That's a problem for windmill tests because windmill can't cleanly switch between domains[1], so I thought of using HTTP basic auth (+basiclogin) instead, which works ok in this case as our tests don't switch between domains anyway. Does anybody see that as a problem in any way? An alternative would be to not use a separate vhost for the testopenid provider, but that would make it slightly trickier to disable the test provider in production -- when using a separate vhost it's very easy to do that. [1] I tried really hard to make it possible to login using the test OpenID provider in windmill tests and almost succeeded (with a few hacks here and there), but in the end I wasn't sure it'd be possible at all, so I gave up. http://trac.getwindmill.com/ticket/279 -- Guilherme Salgado salg...@canonical.com ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
[Launchpad-dev] doctest size, refactoring, moving corner cases to unittests, etc
Long doctests can accumulate large amounts of state. Object, test data and variable reuse causes headaches for developers and reviewers. This bothers me, so I brought it up at a reviewer's meeting a long long time ago: Because of the accumulation of state in doctests I think it would be good to limit new doctests to 200 lines or less. gmb wrote a new doctest last week of just under 200 lines and it was still perfectly comprehensible. That's where the number comes from. The consensus was that a 200 line limit wasn't the answer, but there were lots of other ideas, so let's discuss it. (I tried to make notes from the logs, but it was a content packed and concise discussion in itself. I've attached a formatted log for reference.) Gavin. bac [topic] Limit doctests to 200 lines [allenap] bac gavin, take it away allenap Right, allenap Because of the accumulation of state in doctests allenap I think it would be good to limit new doctests to 200 lines or less. allenap gmb wrote a new doctest last week of just under 200 lines and it was still perfectly comprehensible. allenap So that's where the number comes from. * gmb notes that it got changed into a unittest anyway during the review process... danilos I don't think that'd be right for large classes and corresponding doctests, 200 sounds as artificially low limit allenap Although manuel (?) adds a reset-globs directive whihc might make this moot. flacoste and there is an alternative to the state problem danilos perhaps large classes are the problem, but we do have them flacoste exactly, reset-globs in manuel gary_poster agree with danilos. Also, agree with allenap's observation of manuel sinzui allenap: I have thought about breaking many registry doctests into smaller themes flacoste manuel also adds the option of running specific section of a doctest sinzui allenap: but my motivation it to control the layer the test is run on. intellectronica i think limiting the length is not the best way to handle this. instead i suggest dividing tests thematically mars I agree, I don't see the problem as much with larger narratives as it is with naughty test code state. danilos sinzui, I think the problem with existing doctests is that we don't have a good structure of tests in general (oh, I'd like to clean most of translations ones as well) intellectronica that is, stop and start a new file when you're really testing something new (with new state and so on) bac intellectronica: i agree allenap The main problem I have is trying to understand the object and database state by the end of the test. That can't really be fixed by reset-globs or manuel as I understand it. danilos intellectronica, +1, basic doctests (those which are usually large) should be basic class documentation; specific tests should be unit test allenap intellectronica: The limit is an incentive to split :) allenap intellectronica: But I agree. danilos allenap, I see the point, but artificial limit would have to be broken in so many valid cases intellectronica allenap: yes, i understand that, but somehow it feels a bit too artificial to me. are you concerned that without a hard limit developers/reviewers won't bother? bac allenap: it is a good point but i'd rather see us have smaller, more themed tests as a guide rather than adopting a limit gary_poster We do have a 800 line limit for reviews that is a guideline but often breached intellectronica also, i'll play sinzui and ask: who's going to split all the existing test? ;) danilos how about instead spreading a rule enforce corner-case testing through unit tests among reviewers instead? those tend to make doctests harder to read mars instead of a hard limit, can we call it a refactor point? As in, over X lines, the reviewer should look for a possible refactoring allenap Okay, sounds good. Is there a way to measure or have more concrete guidelines, or is it at developer/reviewer discretion? mars it's ok if they do not find such a point mars kind of like review lint intellectronica allenap: there is a measure. variables shouldn't be recycled danilos mars, I still believe that those complex doctests should really be turned into unittests, and that's exactly where you can't reuse variables :) allenap intellectronica: I don't think we should split the existing tests... until it is no longer possible to not split them. allenap intellectronica: I like that :) allenap intellectronica: There's still a problem of db state. mars danilos, well, if they are over 800 lines, then you would recommend the refactoring, wouldn't you? :) intellectronica allenap: if we adopt this as a policy we will eventually have to split the tests, if only so that they don't serve as a negative example for new contributors allenap danilos: +1 mars danilos, or 400, or whatever sinzui intellectronica: One line that creates a file or email
Re: [Launchpad-dev] [RFC] Launchpad Enhancement Proposals
On 10 February 2010 13:22, Jonathan Lange j...@canonical.com wrote: Hello, I've been thinking a bit about how we can achieve three[1] things: * avoid reworking features * avoid working on silly features * make our features so good that we have annoying fanboys queueing outside our retail outlets on release day Ultimately, it's not going to happen unless we all refuse to make anything short of excellent. However, maybe some structure can help. In particular, I think a little structure around the area of defining features will go a long way toward achieving those three goals. By a coincidence that I can only explain by appealing to astrology, I've written three documents to help with this. 1. https://dev.launchpad.net/ReadyToCode It's a very short list. I'm proposing that before we begin work on _anything at all_, we run through this checklist: bugs, refactoring, features, whatever. It might be a useful thing to use for pre-implementation calls. 2. https://dev.launchpad.net/LaunchpadEnhancementProposalProcess If getting ready to code is actually non-trivial, then you might want to use this to help get there. I've tried hard to keep it as simple as possible. It's worth noting that the LEP is *absolutely not* the place where implementation design happens. It's where we collect requirements. I only grudgingly included the UI mockup section. In the process, I suggest that we use blueprints for keeping track of these proposals. I think it's time we started eating our own dogfood again.[2] 3. https://dev.launchpad.net/LaunchpadEnhancementProposalTemplate This is a template for 2. Should be quite straightforward. Finally, I said this earlier in an email with smaller circulation, but it bears repeating: I really, really believe that big design up front is a wasteful practice that produces crappy software and unhappy developers. I would only be happy with these things becoming a part of our process if they were iterative and if the documents do not become ends in themselves. With that in mind, I want to know your thoughts on the whole thing. I think this is a bloody good experiment, thanks for writing (and thinking) this up. There's one emotional reaction I noticed erupting as I read the process proposal: oh no, more wikis to edit?! Perhaps some of the work we can do on Blueprint which will actually give us value is work that will rid us of the need to edit wikis to keep structured documents. The template you are proposing, for example, could be very easily encoded in a form. Also there are other document management solutions that don't make you sweat so much. Tom ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] [RFC] Launchpad Enhancement Proposals
Jonathan Lange wrote: Hello, I've been thinking a bit about how we can achieve three[1] things: * avoid reworking features * avoid working on silly features * make our features so good that we have annoying fanboys queueing outside our retail outlets on release day In my experience, great products with annoying fanboys are the end result of 1000s of great decisions: every label, every widget, every workflow, every error message, etc. The primary challenge in a large software development team is how to distribute (wise) decision making. Ultimately, it's ideal if every software engineer gets the vision and makes smart choices accordingly. Failing that, team leaders need to be capable of getting it right. I've been reading Mary Tom's latest book on Lean SD this week. They put a huge emphasis on managing flow: finding what's slowing you down from delivering value faster. They also caution against introducing new processes that solve occasional problems: it can be better to wear the cost of rework (say) now and then than to slow down every change. Sometimes the fastest way to improve quality is to simply fix lots of bugs. The primary argument against doing that is opportunity cost: could the time be better spent on delivering new features instead? That's never an easy call. I think your templates are great for new features. If fixing small bugs takes too long though, then that process needs to be improved IMO. I'd be hesitant to impose additional overhead on small changes. I really, really believe that big design up front is a wasteful practice that produces crappy software and unhappy developers. I would only be happy with these things becoming a part of our process if they were iterative and if the documents do not become ends in themselves. With that in mind, I want to know your thoughts on the whole thing. The essential thing is to not make yourself a bottleneck. That's *very* easy to do in a role like Product Strategist or Architect. Also, one trick that's appropriate now and then is to write documents that have ongoing value (e.g. end user documentation) instead of documents that become obsolete. HTH, Ian C. ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] [RFC] Launchpad Enhancement Proposals
I think the guts of what Aaron is getting at here is the following: * Do you expect _all_ questions to be answered for _every_ piece of work? This doesn't really work for trivial or simple bug fixes * Is the Launchpad Enhancement Proposal Process necessary for _every_ enhancement? Even minor ones? Tim ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] [RFC] Launchpad Enhancement Proposals
On Thu, 11 Feb 2010 12:31:27 +1000, Ian Clatworthy ian.clatwor...@canonical.com wrote: Also, one trick that's appropriate now and then is to write documents that have ongoing value (e.g. end user documentation) instead of documents that become obsolete. Indeed, I like combining specs and documentation, but I don't do enough of it. You do the UI design by writing the documentation for the feature despite it not existing, and where you would put a screenshot you put a mockup. You can then add another page that explains more about what is going on under the hood, which can act as your architectural specification for the feature. This means that you get the benefits of a specification, but it keeps its value after the implementation is done, and doesn't grow out of date like a spec will as you keep the documentation up to date as things change over their lifetime. It also includes a user experience focused approach as it forces you to think about how people will use it first, and the goal of a simple to describe interface nicely meshes with your desire to write less documentation. There are clearly some problems though, it adds a fairly large amount to the cost of changing direction in the middle of implementation, good documentation doesn't always match up with what you need to think about when designing something, we don't all make great documenters, and rolling out a half-finished feature becomes trickier as you have to edit the documentation so people don't get confused. Thanks, James ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] RFC on build from branch UI
On Tue, 02 Feb 2010 23:01:49 Michael Nelson wrote: https://dev.launchpad.net/BuildBranchToArchiveUI Hi Michael, Firstly I think you have done an awesome job with the UI mock-ups. I talked with Michael Hudson today, and here is the summary of what came up. I'm not expecting you to answer all of these, but to give us notes to work from. We agreed that it is important to focus on the initial workflow of creating a new daily build of a branch into a PPA. We realised that we are missing the schedule information for the recipe itself in order to support daily builds properly. We are missing any form of UI mock-ups for the branch page itself. What changes on the branch page when a new build has been requested? How are we going to show builds in progress? What if the user wants to remove a daily build? Or just disable it? Is it really feasible to share recipes? When wanting to build into a PPA, what are we really accomplishing by choosing someone else's recipe? We are likely to have multiple recipes for each branch that is being built daily as we are going to target multiple distroseries. Am I right on this assumption? We will want some general recipe oriented views later, but what we have is a good start. Tim ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] RFC on build from branch UI
Tim Penhey wrote: On Tue, 02 Feb 2010 23:01:49 Michael Nelson wrote: https://dev.launchpad.net/BuildBranchToArchiveUI Hi Michael, Firstly I think you have done an awesome job with the UI mock-ups. I talked with Michael Hudson today, and here is the summary of what came up. I'm not expecting you to answer all of these, but to give us notes to work from. We agreed that it is important to focus on the initial workflow of creating a new daily build of a branch into a PPA. One part of the mock up I was a bit uncomfortable with was the view details section of selecting a recipe. It seems as if this allowed editing the recipe -- if this is the case, I think it's a bit wrong. The dialog that lets you edit a recipe should have a button labelled Save at the bottom of it, not build, to emphasise that one is editing a shared resource, not something specific to this build. So perhaps the workflow should be select recipe, edit recipe, this pops up another dialog, save, goes back to selection screen with edited recipe selected, then click build. I think James already said something like this. Cheers, mwh ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] Using HTTP basic auth on windmill tests
On Wed, Feb 10, 2010 at 04:09:52PM -0500, Gary Poster wrote: Nobody replied yet. +1 on using basic auth for me. I like having the test openid provider on a separate domain, and this limitation doesn't seem too bad, at least for now.. I'm also +1 on using basic auth. I don't think any of our Windmill tests include the login step as part of the workflow that is tested. The login is mostly test setup, so how we do it isn't important. As long as we have non-Windmill tests that make sure logging in without Javascript works, we should be fine. -- Björn Tillenius | https://launchpad.net/~bjornt ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] doctest size, refactoring, moving corner cases to unittests, etc
On Wed, Feb 10, 2010 at 09:17:36PM +, Gavin Panella wrote: Long doctests can accumulate large amounts of state. Object, test data and variable reuse causes headaches for developers and reviewers. This bothers me, so I brought it up at a reviewer's meeting a long long time ago: Because of the accumulation of state in doctests I think it would be good to limit new doctests to 200 lines or less. gmb wrote a new doctest last week of just under 200 lines and it was still perfectly comprehensible. That's where the number comes from. The consensus was that a 200 line limit wasn't the answer, but there were lots of other ideas, so let's discuss it. This is an excellent thing to discuss. In fact, this is one of the things that is on my plate to deal with during this cycle, so hearing what people think is interesting. I have some ideas on how to improve the situate, and reading the IRC log, it looks like at least some people. I don't know what the right thing to do is. I think this requires some experimentation, testing a few different strategies and see which one looks the best, and is most useful. My basic ideas so far is to make the doctests into actual documentation. For each major part of Launchpad, there should be a doctest describing the part. That doctest can link to other doctests which describes the part in more detail, or describes sub-parts. The doctest should be high-level, and not describe too much detail, preventing them from breaking due to trivial changes. More detailed tests could go into Python tests. This is the basic idea that I'm going to experiment with first. It might turn out it's not feasible. For example, good documentation is hard to write, and it's only feasible to write good documentation if someone actually reads it. So another challenge will be to actually make use of the documentation, providing in a way that people will find it useful, and actually read it. The less useful documentation proves to be, the less we should spend time writing it. That said, documentation can actually be useful for the one writing it. This should not be under-estimated. Explaining how things work, and most importantly explaining why things work the way they do, forces you to think about how easy things are to understand. If you have trouble explaining it, maybe your design is not so good. I know that I have realised this a few times, and I'm sure others would as well, if they made the effort. -- Björn Tillenius | https://launchpad.net/~bjornt ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
