Yes, there is a faster alternative. Most databases have a way to import CSV directly. Postgresql, mysql, sqlite all can import CSV with different degrees of complication, but I can say it is pretty easy and much faster. The nice thing of web2py is that is independent of the database, and this comes with a price in terms of speed.
2013/8/26 Joe Barnhart <[email protected]> > In general processing almost a gig of CSV data is going to take a > measurable amount of time. It's simply unavoidable. > > I process similar size CSV files and I have found the Scheduler to be the > ideal solution. I don't actually care that much how long it takes -- I > just want my website to remain responsive while it loads. This is exactly > why the scheduler exists, to fork something long-running to another process > so your site is unaffected. And it truly is another process -- not a > thread. If you have a quad-core you can support a whale of a lot of > processing using the scheduler. > > I built in a little "messaging" table so when my scheduler is done it > sends a message to me and I can see the results. It's a marvelous piece of > work and one of the best features of web2py. > > -- Joe > > > On Sunday, August 25, 2013 3:15:28 PM UTC-7, Sebastian Demian wrote: >> >> Hi guys, I am working on an application using web2py framework, and I >> need to import a csv file that has a size of 700MB. Is there a faster way >> to import it into the database ? I have tried the database administration >> tool, but it takes forever. >> >> PS: I am a newbie. >> >> Thank you. >> > -- > > --- > You received this message because you are subscribed to the Google Groups > "web2py-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > -- --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
