Re: [Repoze-dev] GAE or other 'cloud' choices for bfg apps?
Hi Tim, On Sat, Apr 17, 2010 at 7:27 PM, Tim Hoffman wrote: > HI Darryl > > Here's a couple of quick tips and these where a big win for us. > > 1. Memcache page output heavily, and if possible serve the pages > without even starting up or initialising bfg. > We found we could do a cold startup and serve a page from cache in > 200ms cached page from memcache in 30ms. In this scenario you don't > want to import any thirdparty libraries, just use what comes out of > the box from appengine. ( ie importing any module not part of the > core gae env always seems to take longer.) Whereas just starting bfg > before doing anything could take between 3-4 sec and on a bad day a > lot longer. Then if its not in the cache start bfg and process the > page request. > > 2. Defer loading/importing stuff where possible. For instance we use > formish/schemaish for forms. But if no one is logged in > then we don't need to do the formish/schema imports. If someone is > logged in then we do import formish. That way we only incur import > costs when we actually use things. This is hard to control with zcml > and a lot easier to do if you are directly registering things in > python. Also it means you don't have the cost of parsing zcml as > well. Thanks a million for sharing! Very best, Darryl > > Rgds > > T > > > > On Sat, Apr 17, 2010 at 3:52 AM, Darryl Cousins > wrote: >> Hi Tim, >> >> On Thu, Apr 15, 2010 at 1:55 PM, Tim Hoffman wrote: >>> Hi Iain >>> >>> I have a number of projects on app engine. Some using repoze.bfg >>> (www.polytechnic.wa.edu.au (paid work), www.fishandlily.com.au (my >>> small business)) and others just using zope.component and bobo (not >>> public yet). >>> >>> We are using app engines persistence model which is simple and >>> straight forward. (http://code.google.com/p/bfg-pages/ has some >>> examples of implementing a very simple cms on bfg and appengine, it >>> implements traversal over entities in the datastore as folders and >>> content). >>> >>> I think bfg is a good fit with appengine. A couple of pointers, you >>> are basically using the view, traversal, component registry >>> mechanisms and not zodb/persistence (which isn't really core to bfg). >>> We are not currently using chameleon but straight zpt with a custom >>> bindings see the link above. You need to watch startup times so I am >>> moving away from using zcml and using python to register things. >> >> Yes, start up time is something I've noticed too. Aside from not using >> zcml for component registration do you have any other pointers to help >> with that problem? >> >> Thanks in advance, >> Darryl Cousins >> >>> >>> The really big advantage of app engine as I see it, is not having to >>> deal with the system platform (ie cpu's disk, memory, networks, etc..) >>> app engine is purely a runtime and services. So if you have little in >>> the way of IT admin support, it is a big win. >>> You do have to make some concessions/design compromises to take into >>> account the restrictions enforced by the platform. >>> >>> As for economics www.polytechnic.wa.edu.au gets about 7000 unique >>> visitors a day, around 20,000-50,000 page views a day and >>> we have billing enabled but rarely get over 50% of the free quota so >>> it's currently not costing anything at the moment to run. >>> >>> Rgds >>> >>> T >>> >>> >>> On Thu, Apr 15, 2010 at 7:46 AM, Iain Duncan >>> wrote: Hey all, I have an app that eventually is intended to be used as a subscriber service. I'm totally new to the cloud thing, wondering if experienced bfg'ers would like to weigh in with their experiences - what is your preferred cloud deployment for bfg apps? - what persistence mechanism gets used? Is there some kind of facade over the bigtable? How do you handle it? - what is your experience with using cloud deployment, do you think it's even worth it? thanks! Iain ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev >>> ___ >>> Repoze-dev mailing list >>> Repoze-dev@lists.repoze.org >>> http://lists.repoze.org/listinfo/repoze-dev >>> >> > ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
Re: [Repoze-dev] GAE or other 'cloud' choices for bfg apps?
Oops pressed send with out proof reading. 2nd line of item 1. Should read "We found we could do a cold startup and serve a page from cache in 200ms. and a cached page from a running instance from memcache in around 30ms. T On Sat, Apr 17, 2010 at 3:27 PM, Tim Hoffman wrote: > HI Darryl > > Here's a couple of quick tips and these where a big win for us. > > 1. Memcache page output heavily, and if possible serve the pages > without even starting up or initialising bfg. > We found we could do a cold startup and serve a page from cache in > 200ms cached page from memcache in 30ms. In this scenario you don't > want to import any thirdparty libraries, just use what comes out of > the box from appengine. ( ie importing any module not part of the > core gae env always seems to take longer.) Whereas just starting bfg > before doing anything could take between 3-4 sec and on a bad day a > lot longer. Then if its not in the cache start bfg and process the > page request. > > 2. Defer loading/importing stuff where possible. For instance we use > formish/schemaish for forms. But if no one is logged in > then we don't need to do the formish/schema imports. If someone is > logged in then we do import formish. That way we only incur import > costs when we actually use things. This is hard to control with zcml > and a lot easier to do if you are directly registering things in > python. Also it means you don't have the cost of parsing zcml as > well. > > Rgds > > T > > > > On Sat, Apr 17, 2010 at 3:52 AM, Darryl Cousins > wrote: >> Hi Tim, >> >> On Thu, Apr 15, 2010 at 1:55 PM, Tim Hoffman wrote: >>> Hi Iain >>> >>> I have a number of projects on app engine. Some using repoze.bfg >>> (www.polytechnic.wa.edu.au (paid work), www.fishandlily.com.au (my >>> small business)) and others just using zope.component and bobo (not >>> public yet). >>> >>> We are using app engines persistence model which is simple and >>> straight forward. (http://code.google.com/p/bfg-pages/ has some >>> examples of implementing a very simple cms on bfg and appengine, it >>> implements traversal over entities in the datastore as folders and >>> content). >>> >>> I think bfg is a good fit with appengine. A couple of pointers, you >>> are basically using the view, traversal, component registry >>> mechanisms and not zodb/persistence (which isn't really core to bfg). >>> We are not currently using chameleon but straight zpt with a custom >>> bindings see the link above. You need to watch startup times so I am >>> moving away from using zcml and using python to register things. >> >> Yes, start up time is something I've noticed too. Aside from not using >> zcml for component registration do you have any other pointers to help >> with that problem? >> >> Thanks in advance, >> Darryl Cousins >> >>> >>> The really big advantage of app engine as I see it, is not having to >>> deal with the system platform (ie cpu's disk, memory, networks, etc..) >>> app engine is purely a runtime and services. So if you have little in >>> the way of IT admin support, it is a big win. >>> You do have to make some concessions/design compromises to take into >>> account the restrictions enforced by the platform. >>> >>> As for economics www.polytechnic.wa.edu.au gets about 7000 unique >>> visitors a day, around 20,000-50,000 page views a day and >>> we have billing enabled but rarely get over 50% of the free quota so >>> it's currently not costing anything at the moment to run. >>> >>> Rgds >>> >>> T >>> >>> >>> On Thu, Apr 15, 2010 at 7:46 AM, Iain Duncan >>> wrote: Hey all, I have an app that eventually is intended to be used as a subscriber service. I'm totally new to the cloud thing, wondering if experienced bfg'ers would like to weigh in with their experiences - what is your preferred cloud deployment for bfg apps? - what persistence mechanism gets used? Is there some kind of facade over the bigtable? How do you handle it? - what is your experience with using cloud deployment, do you think it's even worth it? thanks! Iain ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev >>> ___ >>> Repoze-dev mailing list >>> Repoze-dev@lists.repoze.org >>> http://lists.repoze.org/listinfo/repoze-dev >>> >> > ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
Re: [Repoze-dev] GAE or other 'cloud' choices for bfg apps?
HI Darryl Here's a couple of quick tips and these where a big win for us. 1. Memcache page output heavily, and if possible serve the pages without even starting up or initialising bfg. We found we could do a cold startup and serve a page from cache in 200ms cached page from memcache in 30ms. In this scenario you don't want to import any thirdparty libraries, just use what comes out of the box from appengine. ( ie importing any module not part of the core gae env always seems to take longer.) Whereas just starting bfg before doing anything could take between 3-4 sec and on a bad day a lot longer. Then if its not in the cache start bfg and process the page request. 2. Defer loading/importing stuff where possible. For instance we use formish/schemaish for forms. But if no one is logged in then we don't need to do the formish/schema imports. If someone is logged in then we do import formish. That way we only incur import costs when we actually use things. This is hard to control with zcml and a lot easier to do if you are directly registering things in python. Also it means you don't have the cost of parsing zcml as well. Rgds T On Sat, Apr 17, 2010 at 3:52 AM, Darryl Cousins wrote: > Hi Tim, > > On Thu, Apr 15, 2010 at 1:55 PM, Tim Hoffman wrote: >> Hi Iain >> >> I have a number of projects on app engine. Some using repoze.bfg >> (www.polytechnic.wa.edu.au (paid work), www.fishandlily.com.au (my >> small business)) and others just using zope.component and bobo (not >> public yet). >> >> We are using app engines persistence model which is simple and >> straight forward. (http://code.google.com/p/bfg-pages/ has some >> examples of implementing a very simple cms on bfg and appengine, it >> implements traversal over entities in the datastore as folders and >> content). >> >> I think bfg is a good fit with appengine. A couple of pointers, you >> are basically using the view, traversal, component registry >> mechanisms and not zodb/persistence (which isn't really core to bfg). >> We are not currently using chameleon but straight zpt with a custom >> bindings see the link above. You need to watch startup times so I am >> moving away from using zcml and using python to register things. > > Yes, start up time is something I've noticed too. Aside from not using > zcml for component registration do you have any other pointers to help > with that problem? > > Thanks in advance, > Darryl Cousins > >> >> The really big advantage of app engine as I see it, is not having to >> deal with the system platform (ie cpu's disk, memory, networks, etc..) >> app engine is purely a runtime and services. So if you have little in >> the way of IT admin support, it is a big win. >> You do have to make some concessions/design compromises to take into >> account the restrictions enforced by the platform. >> >> As for economics www.polytechnic.wa.edu.au gets about 7000 unique >> visitors a day, around 20,000-50,000 page views a day and >> we have billing enabled but rarely get over 50% of the free quota so >> it's currently not costing anything at the moment to run. >> >> Rgds >> >> T >> >> >> On Thu, Apr 15, 2010 at 7:46 AM, Iain Duncan >> wrote: >>> Hey all, I have an app that eventually is intended to be used as a >>> subscriber service. I'm totally new to the cloud thing, wondering if >>> experienced bfg'ers would like to weigh in with their experiences >>> - what is your preferred cloud deployment for bfg apps? >>> - what persistence mechanism gets used? Is there some kind of facade over >>> the bigtable? How do you handle it? >>> - what is your experience with using cloud deployment, do you think it's >>> even worth it? >>> thanks! >>> Iain >>> ___ >>> Repoze-dev mailing list >>> Repoze-dev@lists.repoze.org >>> http://lists.repoze.org/listinfo/repoze-dev >>> >>> >> ___ >> Repoze-dev mailing list >> Repoze-dev@lists.repoze.org >> http://lists.repoze.org/listinfo/repoze-dev >> > ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
Re: [Repoze-dev] GAE or other 'cloud' choices for bfg apps?
Hey Attila, On Sat, Apr 17, 2010 at 9:32 AM, Attila Oláh wrote: > Hi, > > On Fri, Apr 16, 2010 at 21:52, Darryl Cousins > wrote: >> Hi Tim, >> >> On Thu, Apr 15, 2010 at 1:55 PM, Tim Hoffman wrote: >>> Hi Iain >>> >>> I have a number of projects on app engine. Some using repoze.bfg >>> (www.polytechnic.wa.edu.au (paid work), www.fishandlily.com.au (my >>> small business)) and others just using zope.component and bobo (not >>> public yet). >>> >>> We are using app engines persistence model which is simple and >>> straight forward. (http://code.google.com/p/bfg-pages/ has some >>> examples of implementing a very simple cms on bfg and appengine, it >>> implements traversal over entities in the datastore as folders and >>> content). >>> >>> I think bfg is a good fit with appengine. A couple of pointers, you >>> are basically using the view, traversal, component registry >>> mechanisms and not zodb/persistence (which isn't really core to bfg). >>> We are not currently using chameleon but straight zpt with a custom >>> bindings see the link above. You need to watch startup times so I am >>> moving away from using zcml and using python to register things. >> >> Yes, start up time is something I've noticed too. Aside from not using >> zcml for component registration do you have any other pointers to help >> with that problem? > > GAE caches your app for one minute, IIRC. Some Django folks use a cron > job to ping the app every minute to keep it in cache, since Django is > a monster when it comes to startup times. Something like: > http://www.morkeleb.com/2009/12/16/appengine-grails-cron/ Thanks very much for the tip and the link. Best, Darryl > > Attila > ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
Re: [Repoze-dev] GAE or other 'cloud' choices for bfg apps?
Hi, On Fri, Apr 16, 2010 at 21:52, Darryl Cousins wrote: > Hi Tim, > > On Thu, Apr 15, 2010 at 1:55 PM, Tim Hoffman wrote: >> Hi Iain >> >> I have a number of projects on app engine. Some using repoze.bfg >> (www.polytechnic.wa.edu.au (paid work), www.fishandlily.com.au (my >> small business)) and others just using zope.component and bobo (not >> public yet). >> >> We are using app engines persistence model which is simple and >> straight forward. (http://code.google.com/p/bfg-pages/ has some >> examples of implementing a very simple cms on bfg and appengine, it >> implements traversal over entities in the datastore as folders and >> content). >> >> I think bfg is a good fit with appengine. A couple of pointers, you >> are basically using the view, traversal, component registry >> mechanisms and not zodb/persistence (which isn't really core to bfg). >> We are not currently using chameleon but straight zpt with a custom >> bindings see the link above. You need to watch startup times so I am >> moving away from using zcml and using python to register things. > > Yes, start up time is something I've noticed too. Aside from not using > zcml for component registration do you have any other pointers to help > with that problem? GAE caches your app for one minute, IIRC. Some Django folks use a cron job to ping the app every minute to keep it in cache, since Django is a monster when it comes to startup times. Something like: http://www.morkeleb.com/2009/12/16/appengine-grails-cron/ Attila ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
Re: [Repoze-dev] GAE or other 'cloud' choices for bfg apps?
Hi Tim, On Thu, Apr 15, 2010 at 1:55 PM, Tim Hoffman wrote: > Hi Iain > > I have a number of projects on app engine. Some using repoze.bfg > (www.polytechnic.wa.edu.au (paid work), www.fishandlily.com.au (my > small business)) and others just using zope.component and bobo (not > public yet). > > We are using app engines persistence model which is simple and > straight forward. (http://code.google.com/p/bfg-pages/ has some > examples of implementing a very simple cms on bfg and appengine, it > implements traversal over entities in the datastore as folders and > content). > > I think bfg is a good fit with appengine. A couple of pointers, you > are basically using the view, traversal, component registry > mechanisms and not zodb/persistence (which isn't really core to bfg). > We are not currently using chameleon but straight zpt with a custom > bindings see the link above. You need to watch startup times so I am > moving away from using zcml and using python to register things. Yes, start up time is something I've noticed too. Aside from not using zcml for component registration do you have any other pointers to help with that problem? Thanks in advance, Darryl Cousins > > The really big advantage of app engine as I see it, is not having to > deal with the system platform (ie cpu's disk, memory, networks, etc..) > app engine is purely a runtime and services. So if you have little in > the way of IT admin support, it is a big win. > You do have to make some concessions/design compromises to take into > account the restrictions enforced by the platform. > > As for economics www.polytechnic.wa.edu.au gets about 7000 unique > visitors a day, around 20,000-50,000 page views a day and > we have billing enabled but rarely get over 50% of the free quota so > it's currently not costing anything at the moment to run. > > Rgds > > T > > > On Thu, Apr 15, 2010 at 7:46 AM, Iain Duncan > wrote: >> Hey all, I have an app that eventually is intended to be used as a >> subscriber service. I'm totally new to the cloud thing, wondering if >> experienced bfg'ers would like to weigh in with their experiences >> - what is your preferred cloud deployment for bfg apps? >> - what persistence mechanism gets used? Is there some kind of facade over >> the bigtable? How do you handle it? >> - what is your experience with using cloud deployment, do you think it's >> even worth it? >> thanks! >> Iain >> ___ >> Repoze-dev mailing list >> Repoze-dev@lists.repoze.org >> http://lists.repoze.org/listinfo/repoze-dev >> >> > ___ > Repoze-dev mailing list > Repoze-dev@lists.repoze.org > http://lists.repoze.org/listinfo/repoze-dev > ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
Re: [Repoze-dev] GAE or other 'cloud' choices for bfg apps?
Am 15.04.2010, 15:54 Uhr, schrieb Wichert Akkerman : > Isn 't it incredibly hard to predict costs for cloud services? Not a known and predictable requirement basis. There are problems with some of the cheapest options such as Amazon's new auction-based service. But for anything where large resources are required but time is not a given, it's a good model. Indeed The Economist even recently covered the nascent market for resources. http://www.economist.com/business-finance/displaystory.cfm?story_id=15663898 We will at some point come to forget that Amazon ever sold things apart from CPU cycles! There will be issues about concentration - I remember someone at Europython last year euphorically welcoming the fact that 25% (or relatively high percentage) of all sites are now on one of four providers. A thought that fills me with the collywobbles, I'm not afraid to admit. But hopefully utility computing can find its way into smaller data centres both in-house and down the road. Charlie -- Charlie Clark Managing Director Clark Consulting & Research German Office Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-600-3657 Mobile: +49-178-782-6226 ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
Re: [Repoze-dev] GAE or other 'cloud' choices for bfg apps?
On 4/15/10 14:57 , Charlie Clark wrote: > Am 15.04.2010, 01:48 Uhr, schrieb Iain Duncan: > >> Also curious about the economics of cloud deployment vs renting servers, >> input welcome. > > That's very much apples and spaceships. Utility services (cloud is such a > deliberately "nebulous" (sic) and misleading team) can be unbeatable on > price for calculation which is why genome analysis is often done with > them. Isn 't it incredibly hard to predict costs for cloud services? Wichert. ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
Re: [Repoze-dev] GAE or other 'cloud' choices for bfg apps?
Am 15.04.2010, 01:48 Uhr, schrieb Iain Duncan : > Also curious about the economics of cloud deployment vs renting servers, > input welcome. That's very much apples and spaceships. Utility services (cloud is such a deliberately "nebulous" (sic) and misleading team) can be unbeatable on price for calculation which is why genome analysis is often done with them. And, as Tim says, save you sys admin resources. But SLA's and, particularly, data protection and confidentiality can be major issues - a lot of EC2-based stuff dumps data on S3 servers which simple tokens for protection. Shared hosting or virtual servers are a reasonable alternative if you do need greater control of the environment and, at around € 15 / a month are very reasonably priced. Charlie -- Charlie Clark Managing Director Clark Consulting & Research German Office Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-600-3657 Mobile: +49-178-782-6226 ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
Re: [Repoze-dev] GAE or other 'cloud' choices for bfg apps?
Hi Iain http://code.google.com/appengine/docs/python/datastore/entitiesandmodels.html Will get you started on app engines high level datastore peristence. T ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
Re: [Repoze-dev] GAE or other 'cloud' choices for bfg apps?
Hi Iain I have a number of projects on app engine. Some using repoze.bfg (www.polytechnic.wa.edu.au (paid work), www.fishandlily.com.au (my small business)) and others just using zope.component and bobo (not public yet). We are using app engines persistence model which is simple and straight forward. (http://code.google.com/p/bfg-pages/ has some examples of implementing a very simple cms on bfg and appengine, it implements traversal over entities in the datastore as folders and content). I think bfg is a good fit with appengine. A couple of pointers, you are basically using the view, traversal, component registry mechanisms and not zodb/persistence (which isn't really core to bfg). We are not currently using chameleon but straight zpt with a custom bindings see the link above. You need to watch startup times so I am moving away from using zcml and using python to register things. The really big advantage of app engine as I see it, is not having to deal with the system platform (ie cpu's disk, memory, networks, etc..) app engine is purely a runtime and services. So if you have little in the way of IT admin support, it is a big win. You do have to make some concessions/design compromises to take into account the restrictions enforced by the platform. As for economics www.polytechnic.wa.edu.au gets about 7000 unique visitors a day, around 20,000-50,000 page views a day and we have billing enabled but rarely get over 50% of the free quota so it's currently not costing anything at the moment to run. Rgds T On Thu, Apr 15, 2010 at 7:46 AM, Iain Duncan wrote: > Hey all, I have an app that eventually is intended to be used as a > subscriber service. I'm totally new to the cloud thing, wondering if > experienced bfg'ers would like to weigh in with their experiences > - what is your preferred cloud deployment for bfg apps? > - what persistence mechanism gets used? Is there some kind of facade over > the bigtable? How do you handle it? > - what is your experience with using cloud deployment, do you think it's > even worth it? > thanks! > Iain > ___ > Repoze-dev mailing list > Repoze-dev@lists.repoze.org > http://lists.repoze.org/listinfo/repoze-dev > > ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
Re: [Repoze-dev] GAE or other 'cloud' choices for bfg apps?
Also curious about the economics of cloud deployment vs renting servers, input welcome. Iain On Wed, Apr 14, 2010 at 4:46 PM, Iain Duncan wrote: > Hey all, I have an app that eventually is intended to be used as a > subscriber service. I'm totally new to the cloud thing, wondering if > experienced bfg'ers would like to weigh in with their experiences > > - what is your preferred cloud deployment for bfg apps? > - what persistence mechanism gets used? Is there some kind of facade over > the bigtable? How do you handle it? > - what is your experience with using cloud deployment, do you think it's > even worth it? > > thanks! > Iain > ___ Repoze-dev mailing list Repoze-dev@lists.repoze.org http://lists.repoze.org/listinfo/repoze-dev
