I have a program which read rows from one table (new_data) and attempts to find a matching record in another table, called master_table. Upon determining a that we have seen a record from this client (in new_data) before, based upon some exact matching and probability matching logic, we insert the new record into the master_table.
The typical size of the input table (new_data) is 40-65 million rows). So the master_table will grow substantially over time, and the speed of processing an inbound table against the master_table will fall off too. We expect that, anyway. We've have partitioned the perl program with record keys that it should process, such that we can run 2 or more copies of the process and get through the new_data table(s) quickly without stepping on each others toes. The following is the configuration: 2way 2.8Ghz Intel, RHAS 3.0 (Hyper-threaded) 6Gb Memory 14x72 Gb 10k RPM drives MySQL 4.1.1-alpha-standard-log (using all MyISAM tables) Perl v5.8.0 built for i386-linux-thread-multi DBI 1.42 my.cnf: [mysqld] datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock key_buffer_size=3G table_cache=1024 # Save our bodies from carpel tunnel <sp> lower_case_table_names = 1 # Log slow queries for inspection later. long_query_time=2 log-long-format log-slow-queries = /home/jgd/SlowQuery.log # End slow query log params [mysql.server] user=mysql basedir=/var/lib [safe_mysqld] err-log=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid We were processing a test set of 7.5 million rows in a new_data table and hit between 339 and 505 rows per second. However, in the logs, I noticed some (potentially) concerning messages. Even though all rows were processed and accounted for, I am at a loss to explain the errors, this one in particular: -------------------------------------------------------------- [Fri May 28 03:30:37 2004] mp_thread4.pl: DBD::mysql::st execute failed: Got error 127 from storage engine at cmp.pl line 4555. -------------------------------------------------------------- Any pointers? Or did I leave anything out in this overly written message? TIA. -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe: http://lists.mysql.com/[EMAIL PROTECTED]
