bulkloader.py
1. multi-threads --> so fast.
2. unicode --> ?? haha !
Regards
Tom Wu
2009/4/14 Alkis Evlogimenos ('Αλκης Ευλογημένος) <[email protected]>
> If you are using bulkloader you might want to tune some flags from:
>
>
> http://code.google.com/appengine/docs/python/tools/uploadingdata.html#Command_Line_Arguments
>
> The particular ones that might interest you are:
>
> --num_threads
> --batch_size
> --bandwidth_limit
> --rps_limit
> --http_limit
>
> Running with the defaults will probably lead to really slow uploads
> especially with very small entities that can be batched 1000 at a time. That
> would give you a 100x speedup right off the bat.
>
> 2009/4/13 Josh Steiner <[email protected]>
>
>>
>> So lets see, I need to import 12 million records...
>>
>> 83.33 days to import.
>>
>> On the latest GAE blog posting it says:
>>
>> "Database import: move GBs of data easily into your App Engine app.
>> Matching export capabilities are coming soon, hopefully within a
>> month."
>>
>> Clearly this is going to have to be something other than bulkloader.py
>> , anyone have any more info on this new capability?
>>
>> -josh
>>
>> On Sun, Apr 12, 2009 at 5:26 AM, 风笑雪 <[email protected]> wrote:
>> > You should do it this way:
>> > http://code.google.com/appengine/docs/python/tools/uploadingdata.html
>> > It will take about 1 second to insert 100 records, so I think 4000+
>> record
>> > will be uploaded in 1 minute.
>> >
>> > 2009/4/12 sevenstar <[email protected]>
>> >>
>> >> hi all,
>> >> I have some 4000+ records to be uploaded on the appengine.
>> >> Does GAE allow to upload so much data, if yes then i have some
>> >> questions?
>> >> 1) How to upload that data?
>> >> 2) How much time will it take to complete this thing?
>> >> 3) What is the fastest method to upload it.
>> >>
>> >> Please reply ASAP. I have been reading about it but couldn't find a
>> >> solution.
>> >>
>> >
>> >
>> > >
>> >
>>
>>
>>
>
>
> --
>
> Alkis
>
> >
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~----------~----~----~----~------~----~------~--~---
