Hi Toby, First off, you're correct about the swarm issue. Let me explain why, then apologize, and then propose some alternatives. I'd like to your opinion in order to update the module in a way that works for everyone.
WHAT THE MODULE DOES The module displays a list of your tags and lets you click on a tag to see an inline list of the posts. You can click on different tags and see the posts tagged with that tag. Its an easy way to navigate all your tags without page reloads. THE MECHANICS OF FETCHING DATA Originally the module just got a list of of a users posts in a single request and then rendered the tag cloud. and managed all the posts in memory. But there's a catch, Delicious limits the number of posts that get returned to 100. So if you have 101 posts the first thing you posted doesn't show up in the module anymore. So I updated the module to do the following: 1) get the list of tags (still limited to 100, but so far not an issue that people I know are hitting) 2) when someone clicks on a tag, make a request for the posts with that tag. (again with the 100 limit, but so far not an issue for any one tag) THE SWARM 3) There is an optional setting in the module to automatically fetch all of the posts for each tag as soon as you launch the module. The purpose is to improve UI response time. At the time I implemented it I never through del.icio.us would even notice the traffic. Clearly that isn't the case. MY APOLOGY I'm really sorry if the module I created is causing an inappropriate load on del.icio.us. Even more so I'm sorry that it seems to have caused a DOS response from del.icio.us to the web service connection between google's iGoogle proxy service and the del.icio.us JSON feed. I hope you'll accept my apology. GOING FORWARD Some Alternatives How would you like me to proceed? I see the following options: 1) I can easily remove the code from the module that causes it to prefect all of the posts for all of a users tags. Doing so should update to all of the module users over a short period of time since the module is hosted on my web site and fetched in real time when people use it. This will eliminate the SWARM you are seeing. You will still see requests for posts but each one will be the result of a user clicking on a tag. My guess is you will see a 99% drop in traffic from my module overall. It will add a slight delay to the module UI responsiveness to tag clicks, but that seems more than a fair price to pay to be able to play nice with del.icio.us which I view as on of the top free services available on the internet. OR 2) The del.icio.us team could look at what the module is doing and decide that the traffic is worth it. I'm a heavy user f the module and have to admit that option 1 sounds like it makes more sense, but assumptions have not worked out well for me in this situation thus far, so I'll throw this out there. Thanks for your quick response on my first post an for making del.icio.us available! Paul --- In ydn-delicious@yahoogroups.com, Toby Elliott <[EMAIL PROTECTED]> wrote: > > > On Jun 5, 2007, at 7:10 AM, parussel wrote: > > > Hi, > > > > It appears that the Delicious JSON API is no longer responding to > > requests from google. > > > > The URL http://del.icio.us/feeds/json/russelldad/books?raw works fine > > when navigated to in a web browser. > > > > However, when I try to get the same URL via google's home page proxy I > > get the following result: > > http://russelldad.googlepages.com/yahoo_error.html > > > > > > Thanks for the message. > > I've escalated the problem-submission error to the appropriate group so > that they can look into that bug and hopefully they'll have it fixed > shortly. > > As to the error, it's because whenever a user adds the module to their > homepage (and, presumably on some regular basis after that), it > triggers a swarm of queries to our servers for all the user's json tag > pages from a proxy server. Unsurprisingly, our servers throttle that > flood of queries as more users add this module and the error you are > seeing there is the fallout from that. Without understanding in > finer-grained detail the inner workings of your module I can't really > analyze the situation beyond that, but I would hope that there'd be at > least some caching and algorithmic backing-off on the proxy server > side. > > Regards, > Toby Elliott > del.icio.us >
