Re: [Launchpad-dev] brainstorm: cheaper API collection iteration
On Fri, 25 Mar 2011 15:11:19 +1300, Robert Collins robert.coll...@canonical.com wrote: But how do we teach launchpadlib to do this? And how do we encode this in the wadl? My understanding, may be way off. Pages in collections are just links, so you can generate what you like. You can generate the bare link to the collection to get the first page, and then when that page is returned it would say next_page_link: https://api.launchpad.net/devel/some_collection?start_key=endkey_of_this_collection then when the next page is needed launchpadlib simply hits that URL. Therefore I think this would be fairly straightforward to change in the current webservice. It also looks as though just changing lazr.batchnavigator will get most of the way there, so it may be that fixing the web UI fixes the API too. Thanks, James ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] brainstorm: cheaper API collection iteration
On March 24, 2011, Martin Pool wrote: Hi, thanks for raising this. Nearly any practical operation on bugs is going to take O(n) round trips, because nearly anything will want both the bug and the bugtask(s) and you cannot get them both in batches. It would be good to work out a way to send them all at once to avoid potato programming. I don't know about the limitations of the mapping layer but conceptually and in terms of network representation it should not be that hard to say that tasks are value objects sent inline with the enclosing bug. Fixing this kind of thing would speed up many clients, and I expect also reduce the server load. Similarly for merge proposals. That's what is covered by the expand/filter proposal. I find it kind of weird that Launchpad in some ways is quite pedantic about HTTP and being restful, and then in others has very rpc-like non-restful methods to get collections and do searches. I would like if the URLs looked more like collections, ie /~mbp/bugs/status=In+Progress/ or something. It might make them more cacheable, and more understandable for client developers. I have a sense that people do not look enough at what actually goes across the wire, and things would be better if they did. That was the original proposed design and still a direction that we wanted to take the web service into. But that purely REST model was put on hold because it was thought that a) creating one object model is hard already, let's not have two (internal + external); so have minimal skew between internal and external API b) getting this REST stuff is hard, much simpler to go with what people know: methods Please don't try to argue why these two assumptions were bad. I think we have seen the limitations of those in practice. -- Francis J. Lacoste francis.laco...@canonical.com signature.asc Description: This is a digitally signed message part. ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
[Launchpad-dev] JavaScript step debugging
Hi, all.
I've had a couple occasions in the last 2-3 days where people didn't
know you can do step-wise debugging of JavaScript. So just in case
others are suffering in Y.log pain, I'll point out the debugger
statement, which works in all the developer/js consoles like Firebug
and Chromium's inspector.
Add debugger; anywhere to trigger the debug console:
function foo() {
debugger;
doingSomething(bar);
}
You can also go to the script tab in said tools and click a line
number and reload the page to trigger step debugging on the fly.
Hope that helps!
Cheers,
deryck
--
Deryck Hodge
https://launchpad.net/~deryck
http://www.devurandom.org/
___
Mailing list: https://launchpad.net/~launchpad-dev
Post to : launchpad-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~launchpad-dev
More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] brainstorm: cheaper API collection iteration
On March 25, 2011, James Westby wrote: On Fri, 25 Mar 2011 15:11:19 +1300, Robert Collins robert.coll...@canonical.com wrote: But how do we teach launchpadlib to do this? And how do we encode this in the wadl? My understanding, may be way off. Pages in collections are just links, so you can generate what you like. You can generate the bare link to the collection to get the first page, and then when that page is returned it would say next_page_link: https://api.launchpad.net/devel/some_collection?start_key=endkey_of_this_ collection then when the next page is needed launchpadlib simply hits that URL. Therefore I think this would be fairly straightforward to change in the current webservice. It also looks as though just changing lazr.batchnavigator will get most of the way there, so it may be that fixing the web UI fixes the API too. Generally, lazr.restfulclient uses next_colletion_link to navigate the collection. There is one special case which is if the starting index requested within the collection isn't already loaded, it sets ws.start to control the starting point. In a away, launchpadlib and lazr.restful offers the slice protocol on all colletions and assume that these understands ws.start to start at any arbitrary point. The change suggested by Robert means that we should only support the iteration protocol, and maybe a slice one, where the slices aren't numeric index. -- Francis J. Lacoste francis.laco...@canonical.com signature.asc Description: This is a digitally signed message part. ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] bug notifications database utilization
On March 25, 2011, Ian Booth wrote: 2. Why do some calls to sendmail() take so long to complete. And given they do, what can be done about it. Within data-centre sendmail is still doing some form of spam-prevention checking (not spamassassin) but DNS checks and such. We have found similar issues with email sending from app servers, and there are no plans to change this. (Ideally, we'd like to move to asynchronous email sending but there are bugs to fix before we can re-enable the default ZopeQueuedEmailRunner.) And that wouldn't really help with email sending from scripts. -- Francis J. Lacoste francis.laco...@canonical.com signature.asc Description: This is a digitally signed message part. ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] brainstorm: cheaper API collection iteration
My understanding, may be way off. Pages in collections are just links, so you can generate what you like. You can generate the bare link to the collection to get the first page, and then when that page is returned it would say next_page_link: https://api.launchpad.net/devel/some_collection?start_key=endkey_of_this_collection then when the next page is needed launchpadlib simply hits that URL. Therefore I think this would be fairly straightforward to change in the current webservice. It also looks as though just changing lazr.batchnavigator will get most of the way there, so it may be that fixing the web UI fixes the API too. In general, this is right. lazr.restfulclient follows the URL it finds in next_collection_link without caring what that URL is. launchpadlib doesn't mess with the collection links at all, and lazr.restfulclient never uses previous_collection_link. So if you come up with a better way to paginate records, you can just change next_collection_link and the client will adapt. I know of two exceptions, both optimizations in the code that handles slices (_get_slice() in lazr.restfulclient). 1. If you ask for a slice like launchpad.bugs[:76], lazr.restful gets the first page (which happens to have 75 entries), and then this code runs: if more_needed 0 and more_needed first_page_size: # An optimization: it's likely that we need less than # a full page of entries, because the number we need # is less than the size of the first page we got. # Instead of requesting a full-sized page, we'll # request only the number of entries we think we'll # need. If we're wrong, there's no problem; we'll just # keep looping. page_url = self._with_url_query_variable_set( page_url, 'ws.size', more_needed) So, we take next_collection_link, and we hack ws.size to only give us (in this case) one additional entry. We don't need another 75 entries, we only need one. I think this will continue to work no matter what next_collection_link looks like, so long as ws.size continues to work. Worst case, we can simply remove the optimization. 2. When you ask for a slice like 'launchpad.bugs[70:200]', this code runs: # No part of this collection has been loaded yet, or the # slice starts beyond the part that has been loaded. We'll # use our secret knowledge of lazr.restful to set a value for # the ws.start variable. That way we start reading entries # from the first one we want. first_page_size = None entry_dicts = [] page_url = self._with_url_query_variable_set( self._wadl_resource.url, 'ws.start', start) This secret knowledge of lazr.restful would be invalidated by the change. We could stop supporting this syntax in later versions of launchpadlib, or we could just remove the optimization: make lazr.restfulclient load subsequent pages from the beginning until it has 200, and then perform the slice. I did a quick search of Launchpad for references to ws.start and ws.size. It shows up a lot in tests, but that's about it. In particular, it doesn't seem to be used in the Javascript code (though client.js has support for it). I did see some batch navigation code in picker.js. It looks like Launchpad code, not web service code, but it may have to be changed for the same reasons as the web service needs to be changed. Leonard ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
[Launchpad-dev] Quick'n'dirty solution sought: Locking down certain bugs even further
Hello Launchpadders, I recently had a chat with the Kernel developers about how they use Launchpad. They posed me an interesting problem. They have bugs that they must keep public, for reasons of both principle and practicality. However, they don't really want anyone outside of an authorized set of people actually *doing* anything to those bugs. Outsiders change tags, statuses and add unhelpful comments and the Ubuntu developers would like to stop that from happening. What options do they have with Launchpad as it stands? What could we cheaply change in Launchpad to meet that need better? I long to hear your thoughts, jml ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] brainstorm: cheaper API collection iteration
On Sat, Mar 26, 2011 at 5:43 AM, Leonard Richardson leonard.richard...@canonical.com wrote: ... I did a quick search of Launchpad for references to ws.start and ws.size. It shows up a lot in tests, but that's about it. In particular, it doesn't seem to be used in the Javascript code (though client.js has support for it). I did see some batch navigation code in picker.js. It looks like Launchpad code, not web service code, but it may have to be changed for the same reasons as the web service needs to be changed. awesome, thanks! I think its ok to leave that optimisation in - most folk will iterate, and deciding to disable indexed jumping is a larger discussion - we've enough immediate work that will befit by fixing the common case that i feel no urge to make the problem bigger than it is already :) -Rob ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] brainstorm: cheaper API collection iteration
On Thu, Mar 24, 2011 at 10:11 PM, Robert Collins robert.coll...@canonical.com wrote: This is about optimising a particular part of the current webservice; its still useful in an expand/filter world I think, because some collections are massive. If I'm understanding the various moving parts correctly, the filter/expand mechanism would remove the need for this optimization. Not that we shouldn't do it for the short term gains (about which I have no opinion one way or the other). How would filter/expand help in this scenario? It will first return the membership of whatever set you specify and the client would then batch using that information. There isn't any need to do batch calculations on the server side, avoiding the expensive surrogate key calculations. -- Benji York ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] brainstorm: cheaper API collection iteration
On Sat, Mar 26, 2011 at 9:04 AM, Benji York benji.y...@canonical.com wrote: On Thu, Mar 24, 2011 at 10:11 PM, Robert Collins robert.coll...@canonical.com wrote: This is about optimising a particular part of the current webservice; its still useful in an expand/filter world I think, because some collections are massive. If I'm understanding the various moving parts correctly, the filter/expand mechanism would remove the need for this optimization. Not that we shouldn't do it for the short term gains (about which I have no opinion one way or the other). How would filter/expand help in this scenario? It will first return the membership of whatever set you specify and the client would then batch using that information. There isn't any need to do batch calculations on the server side, avoiding the expensive surrogate key calculations. the filter/expand system still runs into trouble in very large batches; we don't know how large 'very large' is because we haven't written it yet. A trivial example of where very large is too large: ubuntu/allbugsever So, making batching /possible/ with very high offsets is necessary at some point [or we have to stop offering stupendously large collections in the way we do today] -Rob ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] brainstorm: cheaper API collection iteration
On Fri, Mar 25, 2011 at 4:17 PM, Robert Collins robe...@robertcollins.net wrote: On Sat, Mar 26, 2011 at 9:04 AM, Benji York benji.y...@canonical.com wrote: On Thu, Mar 24, 2011 at 10:11 PM, Robert Collins robert.coll...@canonical.com wrote: This is about optimising a particular part of the current webservice; its still useful in an expand/filter world I think, because some collections are massive. If I'm understanding the various moving parts correctly, the filter/expand mechanism would remove the need for this optimization. Not that we shouldn't do it for the short term gains (about which I have no opinion one way or the other). How would filter/expand help in this scenario? It will first return the membership of whatever set you specify and the client would then batch using that information. There isn't any need to do batch calculations on the server side, avoiding the expensive surrogate key calculations. the filter/expand system still runs into trouble in very large batches; we don't know how large 'very large' is because we haven't written it yet. A trivial example of where very large is too large: ubuntu/allbugsever Indeed, we can't be sure because we haven't built it yet, but I'm hopeful that we'll find a way. For example, 1 million bug URIs is 47 megabytes. Gzipped they take up 2.5 meg. If that's any indication, we might be able to avoid batching set membership altogether. -- Benji York ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] brainstorm: cheaper API collection iteration
On Sat, Mar 26, 2011 at 9:58 AM, Benji York benji.y...@canonical.com wrote: Indeed, we can't be sure because we haven't built it yet, but I'm hopeful that we'll find a way. For example, 1 million bug URIs is 47 megabytes. Gzipped they take up 2.5 meg. If that's any indication, we might be able to avoid batching set membership altogether. oh, its duration thats a problem: doing straight sql to bring all distro bugtasks back : 155194.069 ms ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] Quick'n'dirty solution sought: Locking down certain bugs even further
On 26 March 2011 05:56, Jonathan Lange j...@canonical.com wrote: Hello Launchpadders, I recently had a chat with the Kernel developers about how they use Launchpad. They posed me an interesting problem. They have bugs that they must keep public, for reasons of both principle and practicality. However, they don't really want anyone outside of an authorized set of people actually *doing* anything to those bugs. Outsiders change tags, statuses and add unhelpful comments and the Ubuntu developers would like to stop that from happening. What options do they have with Launchpad as it stands? What could we cheaply change in Launchpad to meet that need better? I think it would be interesting and perhaps reasonably cheap to add two checkboxes per bug: lock status and lock comments. They can be changed by bug supervisors of any task. If the former is set, task fields (status, importance, etc) can't be changed and new tasks can't be created, except by bug supervisors. If the latter is set, comments and attachments can't be added, again except by bug supervisors. (Perhaps s/bug supervisors/some other role/; the intention being that only developers can change it.) I don't anticipate this would be set on many bugs but there are some very hot bugs where it could be useful. I like this idea because it may be a path to removing some special case ACLs that exist at the moment (eg limits on moving into triage state; wishes for limits on moving out of triage state; opinion state; wishes for locking to wontfix). Unlike those ACLs this can be put in place only when something is actually a problem rather than slowing down community triage in general. (There is some similarity to Wikipedia page protection.) If combined with a final comment by a developer explaining where the bug ought to be discussed, how people can express their desire it be fixed, or what else people can do, it might help. Martin ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] brainstorm: cheaper API collection iteration
On Fri, Mar 25, 2011 at 5:24 PM, Robert Collins robe...@robertcollins.net wrote: On Sat, Mar 26, 2011 at 9:58 AM, Benji York benji.y...@canonical.com wrote: Indeed, we can't be sure because we haven't built it yet, but I'm hopeful that we'll find a way. For example, 1 million bug URIs is 47 megabytes. Gzipped they take up 2.5 meg. If that's any indication, we might be able to avoid batching set membership altogether. oh, its duration thats a problem: doing straight sql to bring all distro bugtasks back : 155194.069 ms Ah! Gotcha. I wonder if there's anything creative we can do on that front. After all if the user really wants all 15 seconds worth of data they'll get it eventually (spending more overhead than one request would entail too). -- Benji York ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] bug notifications database utilization
On Mar 25, 2011, at 12:21 AM, Ian Booth wrote: Hi folks An update on the status of this. The system appears ok *for now*. Thank you, Ian. Also thanks to Danilo for his help today in looking at the symptoms and talking over possible solutions. I'm regarding this as critical and no longer an incident, and I think that's where you left this. Practically, that means I think I work on it until it is fixed, but we don't have to work on it over the weekend. A cowboy was applied to prod (the optimisation gary refers to below) but in reality it didn't make much difference to the sql being run. By far the latest number of sql statements were single selects to get subscribers' email details and subscription details. There were 100's of them. These should each be consolidated into a bulk select. Or something else done to address the issue. See bug https://bugs.launchpad.net/launchpad/+bug/742230 I analyzed the new bug and the logs more. I used a trick to add some timestamps to our SQL logging: http://pastebin.ubuntu.com/585447/ . It's a bit fussy, but gave a lot more visibility than nothing at all. Running this on qastaging led to a red herring: sendmail was consuming 2/3 of the runtime on qastaging. That's doubly odd because we don't even send anything out from qastaging to my knowledge. However, returning to the production logs, this symptom does not seem to be the case: I don't believe sendmail is the culprit for the particular problem we are experiencing now, because correlating the logs with the code (and the timestamped version we saw on qastaging), we were most often sending emails quite fast--86 a second in one case Danilo looked at. I got one run from the LOSAs of that SQL logging trick on production, but the bug notification queue was empty; the LOSAs did not get around to fulfilling my request for a second run. Limited LOSA availability has made looking at this problem even harder than it might be otherwise. I did some more work on optimizing the query problems we saw. This branch reduces the pertinent queries in my analysis locally, and should make a particular difference for reducing queries for notifications based on team subscriptions. That is, on a small scale of notifications, this code is 1/3 to 1/2 better; but I believe the characteristics of these changes will make a much bigger difference in production, hopefully in an order of magnitude. lp:~gary/launchpad/bug741684 I'll try to get that cowboyed Monday to see if that addresses the query situation better. We also should run representative unique SQL generated by this code with EXPLAIN ANALYZE, ideally in the production database, since qastaging and staging do not generate notifications in the quantity that production does. Since I did not get a response for the next step of my previously mentioned request to the LOSAs, I never got to making this request of them. The latest dbr report shows the bugnotification user using approx 10% cpu over the past few hours. Bug email is going out but the cpu is still far too high. Why is the system running ok now? I'm not sure and that's a worry. As I saw some other people hypothesize, I'm guessing we're running just barely OK most of the time right now, but we can be flipped over into cascading errors whenever we tip over into extra capacity. Another issue I noticed from the logs is as follows. The cronjob runs every 5 minutes. The logs over the past few days show that there are times when notifications are still being sent when the next script invocation occurs eg 2011-03-23 15:15:08 INFONotifying xxx about bug 736049. 2011-03-23 15:15:08 INFONotifying xxx about bug 736049. 2011-03-23 15:15:08 INFONotifying xxx about bug 736049. 2011-03-23 15:15:08 INFOCreating lockfile: /var/lock/launchpad-send-bug-notifications.lock 2011-03-23 15:15:08 INFONotifying xxx about bug 736049. 2011-03-23 15:15:08 INFONotifying xxx about bug 736049. ... 2011-03-23 15:15:16 INFONotifying xxx about bug 733732. Often also the notification lines in the log are several seconds or more apart, indicating the call to sendmail() blocks for a time. So I have 2 questions: 1. How is the new script invocation happening if the old one appears to still be running? My theory is that the new script starts and blocks until the old one finishes. And if the next one is slow too, then it all compounds That doesn't quite jibe with what I think we see here, but I could be wrong. The core issue does appear to be that it seems to be possible for a script to run simultaneously, though we haven't caught that smoking gun yet. I don't intend to address this at the moment, though I'll be interested in tackling it when we go to bug rotation if someone else has not already looked at it. 2. Why do some calls to sendmail() take so long to complete. And given they do, what can be done about it. This is a good question, but
Re: [Launchpad-dev] brainstorm: cheaper API collection iteration
On Sat, Mar 26, 2011 at 2:54 PM, Benji York benji.y...@canonical.com wrote: On Fri, Mar 25, 2011 at 5:24 PM, Robert Collins robe...@robertcollins.net wrote: On Sat, Mar 26, 2011 at 9:58 AM, Benji York benji.y...@canonical.com wrote: Indeed, we can't be sure because we haven't built it yet, but I'm hopeful that we'll find a way. For example, 1 million bug URIs is 47 megabytes. Gzipped they take up 2.5 meg. If that's any indication, we might be able to avoid batching set membership altogether. oh, its duration thats a problem: doing straight sql to bring all distro bugtasks back : 155194.069 ms Ah! Gotcha. I wonder if there's anything creative we can do on that front. After all if the user really wants all 15 seconds worth of data they'll get it eventually (spending more overhead than one request would entail too). 155 seconds - you dropped a 0 :) yes, we need to figure something out, but its going to require some heavy lifting, I think. -Rob ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
