perhaps i have led you astray...i have not done that sort of download
from non-GAE to GAE. references will break for sure, but if you are
lucky enough to not have them, maybe the CSV export/import all tables
at once (see section 6.25.2 of version 3 of the book) will work for
you. when i was first experimenting with the app engine i recall
having the reference problem. included is my method for fixing up
references. this is specific to my model, so it won't just work for
you...but hopefully you can make it work for you. in my model i had
several multi-value fields (strings of id's separated with "|") so
that is what you see going on there.
let me know if you have more questions, and i'll do my best to help!
def replace_db():
"""
Truncate all tables, and replace with data from the uploaded CSV
file.
Note that this is intended to load data from the web2py formatted
csv file
as downloaded from L{export}
"""
id_map = None
form = FORM(INPUT(_type='file', _name='data'),
INPUT(_type='submit'))
if form.accepts(request.vars):
for table in db.tables:
db[table].truncate()
id_map = {}
db.import_from_csv_file(form.vars.data.file,id_map=id_map,unique=False)
#...@todo: fix up song media references
songs = db(db.song.id>0).select()
for song in songs:
if not song.media:
continue
new_media = ''
medias = song.media.split('|')[1:-1]
for m in medias:
new_media += "|" + str(id_map['media'][m])
new_media += "|"
song.update_record(media = new_media)
#...@todo: fix up recording.media references
recordings = db(db.recording.id>0).select()
for r in recordings:
if not r.media:
continue
new_media = ''
medias = r.media.split('|')[1:-1]
for m in medias:
if id_map['media'].has_key(m):
new_media += "|" + str(id_map['media'][m])
new_media += "|"
r.update_record(media = new_media)
#...@todo: fix up product.elements references
products = db(db.product.id>0).select()
for p in products:
if not p.elements:
continue
new_song = ''
songs = p.elements.split('|')[1:-1]
for s in songs:
if id_map['song'].has_key(s):
new_song += "|" + str(id_map['song'][s])
new_song += "|"
p.update_record(elements = new_song)
return dict(form=form, id_map=id_map)
On Aug 17, 8:11 am, Jurgis Pralgauskis <jurgis.pralgaus...@gmail.com>
wrote:
> I was experimenting on bulk download/upload.
> I have simple table (Languages), which I first fill with a couple of
> records.
> Then I download them -- seems nice.
>
> The problem I get, is that when Uploaded (actually the same stuff),
> id values are stored in some strange column "Key Name", but not in
> "id".
> and I get TypeError: int() argument must be a string or a number, not
> 'NoneType'
> And I can't see the uploaded records in appadmin.
>
> screenshots, csv and
> bulkloader.yaml:http://ftp.akl.lt/users/jurgis/etc/db_migrate_web2py-gae/
>
> By the way GAE has "ID" name (not "id")
>
> if I change "__key__" to "id" in bulkuploader.yaml,
> I get extra column "id" -- which is clearly different from GAE "ID",
> as GAE "ID" then are inserted independently as if I would not have
> them.
>
> ===== Another Idea =========
> maybe Web2Py could have "Backup"/"Restore" feature per DB,
> alongside with "Export"/"Import" per table.
>
> As I need to keep my id's unchanged when migrating (as I have
> hierarchical/nested/recursive relation in one table, and also some
> foreign keys)
>
> On Aug 16, 6:43 am, howesc <how...@umich.edu> wrote:
>
> > take a look at the bulk loader from
> > google:http://code.google.com/appengine/docs/python/tools/uploadingdata.html
>
> > i think it takes a CSV format, so if you can get your data from sqlite
> > in proper CSV format you should be able to upload via bulk loader.
>
> > good luck!
>
> > christian
>
> > On Aug 15, 8:33 am, Jurgis Pralgauskis <jurgis.pralgaus...@gmail.com>
> > wrote:
>
> > > Hello,
> > > what's the best way to import my currently sqlite db to GAE?
>
> > > I think for each table:
> > > 1) export CSV on nonGAE,
> > > 2) then paste it into some form in GAE environment and adapt
> > > import_from_csv_file... ?
>
> > > but maybe there are other/better/ready recipies?
