>>>>> "Creigh" == Creigh Shank <[EMAIL PROTECTED]> writes:

Creigh> Using an Apache/PHP/MySQL/Linux (Redhat 8.0) solution,
Creigh> PHPList, to create an e-mailing list for our 5.6 million book
Creigh> club members.  Unfortunately, the import speed for importing
Creigh> records (at record number 150,000 the rate is about 2,000
Creigh> records per hour).  We're running on the following:

Creigh> P4 (1.5 Ghz), 1.2 Gbytes RAM (650 Mbytes RAM Disk using
Creigh> ramfs), IDE drive (72,00 rpm)

Creigh> So far we've moved the MySQL data files (var/lib/mysql), PHP
Creigh> /tmp and upload directories and PHPlist web site files to RAM
Creigh> Disk (still just testing - not yet dealing with data safety
Creigh> issues).  With all of this tuning we're still at only 2,000
Creigh> records per hour for uploading.

Creigh> We need to be at 100,000 records per hour (uploading and
Creigh> sending seem to run at about the same rate - we need to be
Creigh> able to send to all book club members in the same week).  Any
Creigh> suggestions?

What is your data source for the records?  Are they in a bunch of flat
files?  If so, cut out the overhead of Apache/PHP and use Perl with
the DBI modules to insert the records directly into your database.

Also, make sure you have the proper indexes for your database that can
be a killer.  Also, for that size of DB, I'd make sure you're using
InnoDB tables.  

What are your mysql settings like as well?  You can tune them up quite
a bit since your import will be the big problem at first, but then it
will mostly be just reads on the tables when you send out emails.

   John Stoffel - Senior Unix Systems Administrator - Lucent Technologies
         [EMAIL PROTECTED] - http://www.lucent.com - 978-952-7830

PHP Database Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Reply via email to