I wish i could +1 on postings like this ! Gonna try it out tomorrow ;] Thnx.
Bart Thate programming schizofrenic - "till freedom come!" [!] http://protocol.by/botfather [!] http://botfather.blogspot.com [!] http://jsonbot.org On Sat, Sep 24, 2011 at 8:23 PM, Adam Sah <[email protected]> wrote: > hey guys-- > > If you're using Tasks and haven't tried public, dynamic Backends yet, > they're wonderful: > - for tiny apps, you can offload another $0.72/day for free (they're > dynamic!) > - for medium apps, it's a super easy way to get more RAM > - easy migration from Tasks running on frontends to running them on > public, dynamic backends (incl development, debugging, etc.) > - nearly identical debugging/production methods as frontend Tasks > - since they're public, you can launch Tasks directly on the backend, > removing the frontend from the equation, e.g. you can test out how a > Task will perform on a backend, before committing to the architecture > change. > - cool trick: keep the fast Tasks running on frontends, and use the > (expensive) backends for RAM-intensive jobs-- this is a tiny code > change, and also makes it easier to debug large jobs because the > console isn't flooded with little jobs. For my app, I use Tasks for > various indexing jobs in an online marketplace-- now, I send the > "large" sellers with 1000s of SKUs, to a B4 backend. > > GAE team: > - I've noticed that my backends can 500-error but not produce a stack > trace in the logs? are other people seeing this? > > my choice of migration: (python) > > 1. create backends.xml, define *one* backend, e.g. > backends: > - name: backend2647324 # since it's public, give it a hard-to- > guess name > class: B4 # big+fast without blowing the free budget > options: dynamic, public > > 2. add backend-enabled queued, by copying queues from frontends: > > queue: > - name: image-processor > ...other options here... > > - name: image-processor-backend # note how I append "-backend" > to the name > target: backend2647324 > ...other options here... > > 3. in your code, start firing Tasks at the backend > > # see bottom for taskname() trick > taskqueue.add(url=..., name=taskname("some-identifier-i-will- > recognize"), > queue_name="image-thumbnailer"+("-backend" if > is_gigantic_image(...) else "")) > > 4. modify your deploy script to update both the front & backends. > Mine (legacy) is in perl: > > # deploy bar ==> deploys to foo-bar and to foo-bar-backend > # deploy bar-backend ==> deploys to foo-bar-backend only > # deploy bar-frontend ==> deploys to foo-bar only > $DEST = pop(@ARGV); > my $push_frontend = 1; > if ($DEST =~ s/-backend//) { $push_frontend = 0; } > my $push_backend = 1; > if ($DEST =~ s/-frontend//) { $push_backend = 0; } > if ($push_frontend) { > open(FH, "echo ".$ENV{"GAEPASSWD"}."|python2.5 ".$ENV{"GAEDIR"}."/ > appcfg.py --email=".$ENV{"GAEUSER"}." --passin --application=foo-$DEST > update .|"); > while (<FH>) { print; } > close FH; > } > if ($push_backend) { > open(FH, "echo ".$ENV{"GAEPASSWD"}."|python2.5 ".$ENV{"GAEDIR"}."/ > appcfg.py --email=".$ENV{"GAEUSER"}." --passin --application=foo-$DEST > backends . update|"); > while (<FH>) { print; } > close FH; > } > > > that's it! now, some jobs will go to the frontends and others to the > backends. > > hope this helps, > adam > > > def taskname(string): > """safely create Google App Engine Task names.""" > # add timestamp and random for ultimate uniqueness, strip disallowed > chars > rnd_salt = str(random.randint(100000, 999999)) > return re.sub(r'[^a-zA-Z0-9-]', '-', string + > str(datetime.datetime.now())) + "-" + rnd_salt > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
