Thanks for the feedback On Thu, Nov 18, 2010 at 3:53 AM, Robert Collins < [email protected]> wrote:
> It's late for me right now, but I'm not going to be able to comment > more until Monday, so I thought I'd give you some brief, quick, > feedback. > > I know that jml is interested in this *sort* of thing eventually in > Launchpad, so I'm going to limit my response to technical performance > and scaling considerations. > > On Thu, Nov 18, 2010 at 3:37 PM, Seif Lotfy <[email protected]> wrote: > > mashed together. So again this is to provide a timeline per > > project/team/individual. > > This suggests doing at least one extra write per write we do, and > either scatter-gather aggregation on reads or expansion onto related > objects asynchronously. We have users where this will require > correlation onto thousands of project+team objects. > > > First thing we need for that is a Zeitgeist running on a server with an > SQL > > DB different SQLite. > > I can't quite parse this, but I think you mean 'different to SQLite'. > Well, yes, we couldn't run on SQLite. The launchpad production > database is ~260GB of data. Running on a different database would mean > either accepting inaccurate data or using some mechanism such as a > post-commit work queue to load data, or accepting that some updates > would either (depending on sequencing) not reach zeitgeist or reach > zeitgeist but not actually happen. Why would we want a different > database? No its not different to SQL. I mean PostgreSQL or MySQL. This makes the changes minimal to Zeitgeist. I think only 2 files need to be changed for that. > > This should be no problem. > > Pushing activity from launchpad into zeitgeist means we need to hook into > > launchpad and push > > what the user does into Server-Zeitgeist. > > This assumes we know what the user does :) - its reasonably high up my > 'we should do this' list to have a web API event triggered out of > Launchpad which things [like Zeitgeist] can hook into. > I think its possible to know what the user does or what is happening on launchpad else I wouldn't be able to see who filed a bug > > > Now this is just an example an far from perfect and I would like to > discuss > > the idea with you guys here. > > One of the issues we will face is user privacy. Again that should not be > an > > issue since a logic can be > > implemented between launchpad and Zeitgeist for querying for information. > > Allowing timelines to be either > > public or private to the respected teams/individuals. > > Privacy (disclosure and access) is a very nuanced topic - its not at > all amenable to a handwave: I suggest you read through the series of > topics Curtis has posted about the overall efforts we're entering into > to make privacy and disclosure in Launchpad rock. > Can you please hook me up? > > I expect we'd do about 5-10 insertions a second, and if we were to > make timelines omnipresent then maybe 50 queries a second : how many > milliseconds does zeitgeist take to answer a query, how much context > does it need, and how does it scale as you add millions of actions? > How much disk space would we need to support it per month, and does it > grow linearly or sublinearly (or worse, does it grow supralinearly?) > I will give you a more detailed answer on that tomorrow since we underwent some major performance improvments. But I think on a good Server we are pretty quick and scaling should not be an issue at all. > Please feel free to criticize the idea or help evolve it to something that > > we can all together work on. > > Its not clear to me that we'd want to use zeitgeist as is rather than: > - have an zeitgeist API server we could call - loose coupling, > separated concerns > - or implement the algorithms directly against the Launchpad data model > Algorithms wont work without a good Log. And Zeitgeist will provide you the log, the API to access that log and options on working algorithms on top of this log. > > As far as all working together, I'm fine in principle with patches to > add something like this - as long as jml concurs and we get some > serious design cycles to user test the impact of adding this to the > Launchpad web UI, but I very much doubt there are any cycles in the > work-day for paid Launchpad developers to work on this: we have a > significant queue of items which are all considered critical by the > stakeholders who provide planning input to what paid development time > is spent on. > > -Rob > I will get into more details tomorrow, Cheers Seif
_______________________________________________ Mailing list: https://launchpad.net/~launchpad-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
