On Sat, 5 Jan 2008 15:27:34 -0500 "John Siracusa" <[EMAIL PROTECTED]> wrote:
JS> On Jan 5, 2008 1:25 PM, Ted Zlatanov <[EMAIL PROTECTED]> wrote: >> I realize much of this question belongs to database-specific domains, >> but here goes: I want RDBO insert operations to be faster. I need to >> load thousands of records per second into a database. JS> The first thing to investigate is using plain DBI. If that's not fast JS> enough, then RDBO will never be any faster so you'll have to look JS> elsewhere. JS> The second thing to consider is bulk loading rather than INSERT JS> statements. You can perhaps use RDBO to produce the (usually JS> db-specific) bulk-load files themselves, then point the database at JS> the files using DBI and whatever the "load from file" syntax is for JS> your db. I'll try this. The RDBO code will be read-only to let me know, essentially, if a row already exists (using RDBO::Cached). Each run will generate a tab-separated file per table affected. At the end of a "run" of new objects, I'll just exit the agent process and run COPY FROM rather than trying to flush all the caches; this is also necessary because of Perl's habit of releasing memory whenever it feels like doing it. So I'll have to repopulate the cache occasionally, but that's not as bad as the churn I was seeing before with a 100% RDBO-driven insert process. JS> You can always make your product class inherit from JS> Rose::DB::Object::Cached, so each product will only be SELECTed once. JS> (This assumes you're actually load()ing the products, as opposed to JS> getting them with a JOIN via with => ... etc.) RDBO::Cached made a difference for sure. I still have to use COPY FROM, though, to keep up with the data flow. Damn referential integrity. Ted ------------------------------------------------------------------------- Check out the new SourceForge.net Marketplace. It's the best place to buy or sell services for just about anything Open Source. http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace _______________________________________________ Rose-db-object mailing list Rose-db-object@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/rose-db-object
