The batch get,put,delete will be added for bulk performance. We have discussed adding db table namespaces based on application names on the list, and it will also be added.
http://groups.google.com/group/web2py/browse_thread/thread/57fd68115975382f/6205b3d0f8e0b913?hl=en&lnk=gst&q=dal+namespace#6205b3d0f8e0b913 Robin On Jan 30, 9:00 am, Iceberg <[email protected]> wrote: > Thank Robin for the insight. I guess you are saying: Although writing > only one record is somewhat slow in GAE, writing multiple (dozens/ > hundreds/thousands/or even more) records can be finished in similar > time due to the scalability of GAE, if using the batch get/put/delete > properly. That is cool. And hopefully web2py's GQLDB will have built- > in batch operation soon. Only then, web2py fully deserves the > reputation which "write same code once, then the application can run > both with or without GAE". > > On the other hand, I think using bulk uploader is helpful but not an > elegant way from web2py's point of view, because it forces web2py > developers to define a duplicated data model in both web2py and in > bulk load handler, via different syntax. So I perfer to solve this > problem in web2py's way, perhaps a functional admin app with its csv > import function will do. > > Besides, batch delete is equally important. Currently (without batch > delete yet) web2py on GAE can do 80~100 deletion per second, that is > not fast enough if you are going to do a db reset on GAE which invokes > thousands of records. > > By the way, if I upload only one web2py as my "GAE application" (such > as web2py.appspot.com), then I upload/deploy many "web2py > applications" (say welcome, demo, t3), are they sharing the same > Google DB name space therefore we must be careful not to have same > table name? > > On Jan 28, 12:32 am, Robin B <[email protected]> wrote: > > > Hey, > > > > Did I miss something? Do you guys encounter same problem when playing > > > with GAE? Does your application successfully insert hundreds of > > > records during one request? Any feedback will be highly appreciated! > > > On GAE, writes are very expensive because each record is actually > > written to disk, indexes are updated and written to disk and all this > > happens synchronously, so it takes a long time to serially write > > records. > > > For GAE, take a look at batch get,put,delete which let you do > > operations in parallel as opposed to serially. These batch operations > > will be added to GQLDB eventually. > > > Either way, to prepopulate a db on GAE, you have to do it over a > > series of requests, because each request can only be 10 seconds max. > > See bulk uploader:http://code.google.com/appengine/articles/bulkload.html > > > Robin > > > On Jan 27, 10:13 am, Iceberg <[email protected]> wrote: > > > > Hi pals, > > > > Don't know whether this is a web2py problem or a GAE problem, anyway I > > > still post it here and hope to get some kindly hints. > > > > I started my GAE experience just today. I managed to upload web2py > > > 1.55.2 and its welcome application and later one of my homebrew "no-db- > > > required" application, all of them work fine. Then I tried to port my > > > another "serious" application to GAE. The application requires some > > > initial data (20000+ records, about 500KB total) before it can do some > > > real job. > > > > In my local developing environment, I can easily setup an SQLite db, > > > then visit something like "/myapp/default/putDataIntoSQLDB" to trigger > > > the data initialization without any problem. > > > > But when talking about GAE, I guess there is no way to supply these > > > data in a predefined storage.db file, probably because GAE doesn't > > > support file operation, am I correct? In fact, I tried defining > > > "db=SQLDB('sqlite://storage.db')" in my db.py but it caused an error > > > in GAE: > > > NameError: global name 'sqlite3' is not defined > > > > So I used "db=GQLDB()" instead. Then the applications can start, BUT, > > > when I visit "/myapp/default/putDataIntoGQLDB", my application ALWAYS > > > results in "DeadlineExceededError" after about 8 seconds of execution, > > > during this time only several dozens of records are inserted into > > > GQLDB. If this is because I called db.mytable.insert(...) for too many > > > times, may I have some other way to do batch insert()? > > > > By the way, later when I tried to reset the faulty db data, I found > > > out that there are more than 50% chances that GAE complains: > > > Timeout: datastore timeout: operation took too long. > > > even when I simply invoke: > > > db(db.mytable.id>0).delete() > > > > I search the web and found some rumors, such as: > > > http://highscalability.com/google-appengine-second-look > > > But I still can't believe the performance is as bad as what I > > > mentioned above. > > > > Did I miss something? Do you guys encounter same problem when playing > > > with GAE? Does your application successfully insert hundreds of > > > records during one request? Any feedback will be highly appreciated! > > > > Sincerely, > > > Iceberg --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "web2py Web Framework" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~----------~----~----~----~------~----~------~--~---
