On 1/5/07 6:22 PM, Neal Clark wrote:
> The last row in the table has an id of 23503496. Am I wrong to think
> I can say:
>
> my $objects = MalwareLog::Manager->get_malwarelogs(id => { gt =>
> 23503495 });
>
> and get an array of two objects back?
You're not wrong. That should work fine.
The error looks to me like something in DBD::mysql itself, and it's failing
on $sth->execute(), before any rows are fetched:
> Out of memory (Needed 1461356 bytes)
> DBD::mysql::st execute failed: MySQL client ran out of memory at /usr/
> local/lib/perl5/site_perl/5.8.8/Rose/DB/Object/Manager.pm line 1680.
> get_objects() - DBD::mysql::st execute failed: MySQL client ran out
> of memory at /usr/local/lib/perl5/site_perl/5.8.8/Rose/DB/Object/
> Manager.pm line 1680.
Have you tried replacing the Manager call with a series of plain DBI calls
using the same SQL? (Set $Rose::DB::Object::Manager::Debug = 1 before the
Manager call to see the SQL it's using.)
> Ideally, I will be making iterators from SQL calls. The SQL will
> limit the number of rows as best as I can, but there will be times I
> need to iterate over every row. I read in one of the CPAN docs that
> Rose::DB::Object::Iterator returns 'true iterators', i.e. the rows
> are not loaded until you call next, so it seems like I should also be
> able to say:
>
> my $sql = "SELECT * from malware_logs";
> my $iterator = MalwareLog::Manager->get_objects_iterator_from_sql($sql);
> while (my $object = $iterator->next) {
> ## row gets loaded
> ## i do stuff
> ## memory is released at the end of this block
> }
>
> is that correct?
Yep, that's the idea.
> That also does not work, same memory error.
If the error is on $sth->execute(), as it seems to be, then it happens well
before any iterator/non-iterator stuff.
> when I make a new table (called 'abc') from the same schema
> definition and load it with 1,000,000 rows from the malware_logs
> table, everything works fine. is it just a limitation of Rose::DB
> that it won't work with tables of this size? or is it a DBI issue?
It looks to me like a DBI issue, but you should try using plain DBI in
exactly the same context as the failing Manager calls to confirm this. I've
personally had no difficulty dealing with multi-million row tables using
RDBO with Informix.
> any suggestions would be very appreciated. like i said, i spent tons
> of time writing software using this ORM scheme that worked great
> until the table got this big, so I'm quite desperate for a way to
> make this work.
What version of DBD::mysql are you running? Does it make any difference if
you dump the table and recreate the table in question? Have you tried using
the InnoDB engine?
-John
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Rose-db-object mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/rose-db-object
