There is no way around the 30 second limit yet. You must break up
your work into tasks - probably start one task for each for each URL.
The Deferred library linked to below makes this easy to manage.
On 2 Aug 2010, at 23:13, Sandeep Ghosh wrote:
Hi,
Twig is good but the problem i am facing is that from the
browser i am passing a load command as parameter like this
http://localhost:8888/sbphishlock?load
which tells my code that i need to load the data.
now in the logic i have written few logic to process in a loop
1)Reads a csv file containg a list of URL's(may be 3000)
2)downloads each URL HTML pages and tokenize it
3)Creating a Object and storing in the database
Problem is first 2 points itself takes a lot of time
I think i need to use Task/Queues so that i can start a task for
each url
But i dont know what i need to do for task/Queues
moreover is there any other way of doing this
Can u help me regarding this
Thanx
Sandeep
On Sun, Aug 1, 2010 at 1:25 AM, John Patterson
<[email protected]> wrote:
I mean "bulk put" - that is using DatastoreService.put(Iterable) or
JDO's makePersistentAll(...)
If you do one call with many instances instead of many calls with
one instance it will be a lot faster.
On 31 Jul 2010, at 14:49, Sandeep Ghosh wrote:
Hi,
can i do bulk load in Java.
Can you Please give me a sample code.
Thanx
Sandeep
On Fri, Jul 30, 2010 at 6:22 PM, John Patterson <[email protected]
> wrote:
On 30 Jul 2010, at 18:10, Sandeep Ghosh wrote:
Hi,
I am facing promlem of entering many data into the Google App
engine database as its taking more then 30 sec time limit.
PersistenceManager pm = PMF.get().getPersistenceManager();
for(;;)
{
EnterData data = new EnterData(key,value,new Text(totaData));//
EnterData is my class Entity
pm.makePersistent(data);
}
How can i use the Tasks/Queues in this scenario
You could try using bulk put rather than making a datastore call
for every iteration.
You can also do many store operations in parallel if you use Twig
like this:
for (...)
{
datastore.store().instance(data).later();
}
this is an non-blocking async call that returns immediately and
lets you run all your operations in parallel.
Otherwise, to use task queues I would recommend using the Deferred
library to chain tasks:
http://groups.google.com/group/google-appengine-java/browse_thread/thread/6f736624cc820d5b
--
You received this message because you are subscribed to the Google
Groups "Google App Engine for Java" group.
To post to this group, send email to [email protected]
.
To unsubscribe from this group, send email to [email protected]
.
For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en
.
--
"Kind words can be short and easy to speak,
but their echoes are truly endless."
Mother Teresa
--
You received this message because you are subscribed to the Google
Groups "Google App Engine for Java" group.
To post to this group, send email to [email protected]
.
To unsubscribe from this group, send email to [email protected]
.
For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en
.
--
You received this message because you are subscribed to the Google
Groups "Google App Engine for Java" group.
To post to this group, send email to [email protected]
.
To unsubscribe from this group, send email to [email protected]
.
For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en
.
--
"Kind words can be short and easy to speak,
but their echoes are truly endless."
Mother Teresa
--
You received this message because you are subscribed to the Google
Groups "Google App Engine for Java" group.
To post to this group, send email to [email protected]
.
To unsubscribe from this group, send email to [email protected]
.
For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en
.
--
You received this message because you are subscribed to the Google Groups "Google
App Engine for Java" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
