Ben Tilly wrote:
I am having a problem with a SELECT query against a large dataset. (Several million rows.) Because DBD::Pg pre-fetches data before allowing me to process it, I'm using up a very large amount of RAM. Enough that I'm concerned about running out.
I remember hearing about this problem a couple years ago and asking about it on #postgresql on freenode. The crux of the issue was that the underlying libpq functions were doing the prefetching. Whether that is (still) accurate I do not know, I haven't followed development closely enough, I always managed to work with a data slice that was within my memory limits.
Is there any way to tell it that I wish to process data incrementally and do not want all of the data in RAM at once?
You could use a cursor to fetch the data incrementally: http://groups.google.com/group/pgsql.general/browse_thread/thread/f38a21bd4fc765b/c9357adab6bbacac%23c9357adab6bbacac
