Hi,
Twig is good but the problem i am facing is that from the browser i am
passing a load command as parameter like this
*http://localhost:8888/sbphishlock?load*
which tells my code that i need to load the data.
now in the logic i have written few logic to process in a loop
1)Reads a csv file containg a list of URL's(may be 3000)
2)downloads each URL HTML pages and tokenize it
3)Creating a Object and storing in the database
Problem is first 2 points itself takes a lot of time
I think i need to use Task/Queues so that i can start a task for each url
But i dont know what i need to do for task/Queues
moreover is there any other way of doing this
Can u help me regarding this
Thanx
Sandeep
On Sun, Aug 1, 2010 at 1:25 AM, John Patterson <[email protected]>wrote:
> I mean "bulk put" - that is using DatastoreService.put(Iterable) or JDO's
> makePersistentAll(...)
>
> If you do one call with many instances instead of many calls with one
> instance it will be a lot faster.
>
> On 31 Jul 2010, at 14:49, Sandeep Ghosh wrote:
>
> Hi,
> can i do bulk load in Java.
> Can you Please give me a sample code.
>
> Thanx
> Sandeep
>
> On Fri, Jul 30, 2010 at 6:22 PM, John Patterson <[email protected]>wrote:
>
>>
>> On 30 Jul 2010, at 18:10, Sandeep Ghosh wrote:
>>
>> Hi,
>>> I am facing promlem of entering many data into the Google App
>>> engine database as its taking more then 30 sec time limit.
>>>
>>> PersistenceManager pm = PMF.get().getPersistenceManager();
>>> for(;;)
>>> {
>>> EnterData data = new EnterData(key,value,new Text(totaData));//
>>> EnterData is my class Entity
>>> pm.makePersistent(data);
>>> }
>>>
>>> How can i use the Tasks/Queues in this scenario
>>>
>>
>> You could try using bulk put rather than making a datastore call for every
>> iteration.
>>
>> You can also do many store operations in parallel if you use Twig like
>> this:
>>
>> for (...)
>> {
>> datastore.store().instance(data).later();
>> }
>>
>> this is an non-blocking async call that returns immediately and lets you
>> run all your operations in parallel.
>>
>> Otherwise, to use task queues I would recommend using the Deferred library
>> to chain tasks:
>>
>>
>> http://groups.google.com/group/google-appengine-java/browse_thread/thread/6f736624cc820d5b
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine for Java" group.
>> To post to this group, send email to
>> [email protected].
>> To unsubscribe from this group, send email to
>> [email protected]<google-appengine-java%[email protected]>
>> .
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine-java?hl=en.
>>
>>
>
>
> --
> "Kind words can be short and easy to speak,
> but their echoes are truly endless."
> Mother Teresa
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine for Java" group.
> To post to this group, send email to
> [email protected].
> To unsubscribe from this group, send email to
> [email protected].
> For more options, visit this group at
> http://groups.google.com/group/google-appengine-java?hl=en.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine for Java" group.
> To post to this group, send email to
> [email protected].
> To unsubscribe from this group, send email to
> [email protected]<google-appengine-java%[email protected]>
> .
> For more options, visit this group at
> http://groups.google.com/group/google-appengine-java?hl=en.
>
--
"Kind words can be short and easy to speak,
but their echoes are truly endless."
Mother Teresa
--
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
