Thanks, pg_getcopydata sounds very promising.
I'll try to implement an alternative solution based on pg_getcopydata and
compare it with my current approach.
--
kmx
On 12.11.2014 16:48, Vikas N Kumar wrote:
On 11/12/2014 07:43 AM, kmx wrote:
my $dbh = DBI->connect($dsn);
my $pdl = pdl($dbh->selectall_arrayref($sql_query));
But it does not scale well for very large data (millions of rows).
Hi KMX
If you're using Postgresql you should use the DBD::Pg->pg_getcopydata
using the "COPY mytable to STDOUT" functionality for accessing millions
of rows. You can do this in async or sync mode. This will get you there
faster than using selectall_arrayref(). This allows you to get the rows
without having to redesign your DB.
SQLite has a stream API but I am not familiar with it.
--Vikas
_______________________________________________
Perldl mailing list
[email protected]
http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
