http://code.google.com/appengine/docs/python/taskqueue/overview.html
First, pop a task to start your crawl. From that task fire another task to continue it. Keep doing that until you get what you need or run out of resources. You'll probably want to use either named or transactional tasks to prevent a fork bomb. Use cron, or a user request to initiate your 'spider.' Or to ensure the process keeps running. Robert On Sun, Oct 31, 2010 at 22:34, sombra XX <[email protected]> wrote: > task-queue? how? > > 2010/10/31 Robert Kluin <[email protected]> >> >> Then use the other part of djidjadji's suggestion: task-queue. >> >> >> Robert >> >> >> >> >> On Sun, Oct 31, 2010 at 12:17, sombra XX <[email protected]> wrote: >> > djidjadji: Yes, I thought about the cron, but the intervals are every 1 >> > minute, and I need to be every 5 seconds >> > >> > 2010/10/31 djidjadji <[email protected]> >> >> >> >> Every request should finish within 30 seconds. >> >> You can setup a cron job and/or task queue chain to simulate a >> >> continuous processing. >> >> You have to keep state information in the datastore or part of the >> >> task arguments. >> >> >> >> 2010/10/31 SombraX <[email protected]>: >> >> > Hello everyone. I try to build a program that works non-stop, is an >> >> > infinite while that collects information from different Web sites and >> >> > record the information in the database. The problem is that at >> >> > startup >> >> > is interpreted as a request and get the timeout error. >> >> > Is it possible to build a service daemon with python and boot from >> >> > the >> >> > console app enginne? >> >> > >> >> > (my english is terrible , sorry....cago en die). >> >> > >> >> > -- >> >> > You received this message because you are subscribed to the Google >> >> > Groups "Google App Engine" group. >> >> > To post to this group, send email to >> >> > [email protected]. >> >> > To unsubscribe from this group, send email to >> >> > [email protected]. >> >> > For more options, visit this group at >> >> > http://groups.google.com/group/google-appengine?hl=en. >> >> > >> >> > >> >> >> >> -- >> >> You received this message because you are subscribed to the Google >> >> Groups >> >> "Google App Engine" group. >> >> To post to this group, send email to [email protected]. >> >> To unsubscribe from this group, send email to >> >> [email protected]. >> >> For more options, visit this group at >> >> http://groups.google.com/group/google-appengine?hl=en. >> >> >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups >> > "Google App Engine" group. >> > To post to this group, send email to [email protected]. >> > To unsubscribe from this group, send email to >> > [email protected]. >> > For more options, visit this group at >> > http://groups.google.com/group/google-appengine?hl=en. >> > >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
