On Tue, Oct 4, 2011 at 6:13 AM, Graham Binns <gra...@canonical.com> wrote: > Calling all folks, hello folks... >> It may be easiest to fix this by moving uploads out of the zope >> appservers, which our infrastructure is not geared around supporting >> big uploads too. E.g. upload directly to the librarian, then register >> the content with LP (/handwave) > > And I'm interested in investigating this further. As far as I can tell, > we have the following issues to solve:
\o/ > 1. How do we upload directly to the librarian from the browser? (AIUI > the librarian uses an almost-but-not-quite-HTTP-like protocol, which > makes things awkward, but if I've misunderstood then please feel > free to tell me and so make me very happy). Theres no reason its not an HTTP POST (PUT implies users choosing the URL so it perhaps not appropriate) other than hysterical raisins. Changing it to be one is a small task for a twisted fluent person, or a moderate one for a non fluent person, IMNSHO. (Switch from custom protocol to a standard one pulling our metadata out of the the form variables) > 2. How do we register the content with LP once the upload has > completed? (The answer that both Gary and I have arrived at involves > the words "hey, look at this RabbitMQ thing over here...") Less handwavy time then :P - Several options appear to me here: A: Have the user do it: on completion of the upload return HTTP 30x and a link to the linking page in the appserver. There are some security ramifications here (e.g. can I trigger someone to link hostile content to their account somehow) but that only means pay attention to detail, AFAICT B: Have the librarian do it: include the object to link to as a form header, have the upload service run in the same domain as the appserver (so auth cookies are available) and make an XMLRPC call to the appserver before returning the HTTP status code to the user. In fact the XMLRPC call to the appservers could return the next_url to redirect to, giving a seamless experience. C: Offload it to a backend asynchronously - e.g. Rabbit. I think this will prevent us giving the user a clear signal about their upload and so is undesirable. Remember that there is nothing to wait /for/ if the upload went straight to the librarian - we wouldn't be double handling uploads anymore, so I don't think there is any need to queue things up. > Really, I'm looking for any information that anyone has on how the > Librarian works, since I don't have a great understanding of it. If I'm > wrong about the protocol then this is a relatively simple bug to fix. If > I'm right then I suspect I'll need your help to work it out. Its a twisted daemon, launched via a tac file. That means that the tac file declares a Service with associated protocol factories that listen on various ports. Each of those then fires up a protocol instance when something connects to it. You don't need to touch the existing factories to fix this bug - you need to add a new HTTP POST upload factory (and I suggest this set all LFA's it creates to private, and let the linking step in the appserver decide if it should instead be public). If you need further details to get the ball rolling, I, or one of the other twisted heads here, can certainly do that :) my recommendation is B. -Rob _______________________________________________ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
