I just wanted to let you know that with taps, as soon as you get above about 500 megs, it gets REALLLLLLLY slow. I don't see how you could ever import a gigabyte of data with taps. It would be really helpful if taps could do one table at a time.
I tried to hack up my own one-table-at-a-time version with YamlDb, but then I ran into severe utf8 issues that I could never work through. I ended up removing enough data from the initial database to get it small enough to import with taps, then I added various csv files with the removed data to the apps source and wrote a script to import them bit by bit. On Fri, Mar 12, 2010 at 4:46 PM, Mike <[email protected]> wrote: > Those are really good ideas. Would there be any way you can think of > to push data back up to the server? > > On Mar 12, 9:43 am, Daniele <[email protected]> wrote: > > For the import part if it has not to be "atomic" you could create > > another app and use it to feed via a web service the data on the > > production database while the main app is live. > > > > To download only the main dataset create a boundle, animate it on a > > new application, delete the huge data with a migration or with a sql > > query in the console and the pull the database locally. > > > > By the way I'm investigating about how to work with big db too. Coming > > from a standard server world I love Heroku but some steps could be a > > bit less flexible. Nothing is perfect :) > > > > On 12 Mar, 07:29, Mike <[email protected]> wrote: > > > > > I'm going to be adding a number of discrete, but enormous (maybe many > > > gigs each), datasets to my Heroku app's database. In many ways, I'm > > > in a similar situation faced by Tobin in another current post, but > > > with a different question: > http://groups.google.com/group/heroku/browse_thread/thread/141c3ef84b... > > > > > Right now I still haven't merged the datasets into my database yet. > > > What's the best way for me to approach this? > > > > > The lack of ability to push individual tables with taps suggests to me > > > I'm going to want to do this probably as a one shot deal, rather than > > > doing each dataset sequentially and testing that one before proceeding > > > to the next. I'm thinking about doing a db:pull to get the current > > > state of my database, and then shutting down my application in > > > maintenance mode, running a local merge of the datasets (maybe taking > > > days I'm guessing just to process the enormous things), doing some > > > exhaustive local testing on the result, and then doing a push back to > > > Heroku (maybe taking days again), before reactivating my app. Because > > > of their massive size, it seems like after I've done one, doing any > > > further db:pulls is going to be basically impossible. Just the idea > > > of possibly having made a mistake in merging the datasets that I don't > > > catch until after it's been pushed to the site gives me the shivers. > > > Overall, I wonder if there could be a better way that I'm overlooking. > > > > > One possible alternative I thought of is would it be possible to do > > > something involving creating a local bundle from my database using > > > YamlDB? But then I'm not sure how to get the bundle back onto the > > > server and then to restore from it? The documentation on Heroku > > > doesn't seem to really talk about that possibility. > > > > > Also, in my case this data is integral to the application, so I'm not > > > going to be able to split it up into a separate Heroku application > > > like in Tobin's case. Is there going to be any practical way for me > > > to be pulling just the non-dataset data from the server in order to > > > use on a development machine? > > > > > Does anyone have any ideas on how they would approach this problem? > > > If so, I'd be filled with gratitude. > > > > > Mike > > > > > > -- > You received this message because you are subscribed to the Google Groups > "Heroku" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]<heroku%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
