Rewrite or provide an alternative to ? -Thadeus
On Sat, Jan 30, 2010 at 2:51 PM, mdipierro <[email protected]> wrote: > I think we should consider rewriting the export function using a > tmpfile instead of StringIO and doing pagination. > > On Jan 30, 2:49 pm, Thadeus Burgess <[email protected]> wrote: >> Perhaps you need to write your own csv export. Save a csv file to /tmp >> instead and then when your finished creating your csv file then stream >> it for download. >> >> This might reduce the memory requirements? Though not sure what you >> can do about the fact web2py cache's your select into memory, maybe do >> it in chunks as you generate the csv file? >> >> -Thadeus >> >> On Sat, Jan 30, 2010 at 2:37 PM, mdipierro <[email protected]> wrote: >> > The problem is that all records are copied into memory before being >> > serialized + the entire csv is copied into memory. We need to find a >> > better way when exporting a large of records. >> >> > On Jan 30, 12:19 pm, Johann Spies <[email protected]> wrote: >> >> It happened more than once that a csv-export using appadmin resulted >> >> in incomplete data both with Postgresql-backend and sqlite. >> >> >> This one was on sqlite: >> >> >> Selected records: 139413 >> >> csv-exported (first effort): 287 records. >> >> csv-exported (second effort) 216 records >> >> >> This after a lot of time and slowing down the computer to a near >> >> standstill. >> >> >> I suppose it might be easier and more reliable directly form the database. >> >> >> Regards >> >> Johann >> >> > -- >> > You received this message because you are subscribed to the Google Groups >> > "web2py-users" group. >> > To post to this group, send email to [email protected]. >> > To unsubscribe from this group, send email to >> > [email protected]. >> > For more options, visit this group >> > athttp://groups.google.com/group/web2py?hl=en. >> >> > > -- > You received this message because you are subscribed to the Google Groups > "web2py-users" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/web2py?hl=en. > > -- You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/web2py?hl=en.
