I got back from vacation today and I didn't see that anyone else has
responded yet so I figure that later is better than never.... ;-)
Your INSERT speed will be directly related to how fast all of your JOINS
can be accomplished. Your JOIN speed is related to two things: the size of
your tables and your index structures.
Could you post the SHOW INDEX FROM results for: url_servers, url_paths,
url_queries, user_agents, and url_visit? Could you also post the results
of:
EXPLAIN SELECT bt.user, bt.time, bt.ip, uv.urlid, bt.timetaken,
bt.cs_size, bt.sc_size, bt.method_ID, bt.action_ID, bt.virus, ua.ID
FROM bulk_table bt
INNER JOIN url_servers us ON us.server=bt.server
INNER JOIN url_paths up ON up.path=bt.path
INNER JOIN url_queries uq ON uq.query=bt.query
INNER JOIN user_agents ua ON ua.useragent=bt.useragent
INNER JOIN url_visit uv ON uv.url_server_ID=us.ID
AND uv.url_path_ID=up.ID
AND uv.url_scheme_ID=bt.scheme_ID
AND uv.url_query_ID=uq.ID };
Yours,
Shawn Green
Database Administrator
Unimin Corporation - Spruce Pine
|---------+---------------------------->
| | "J S" |
| | <[EMAIL PROTECTED]|
| | com> |
| | |
| | 07/01/2004 11:22 |
| | AM |
| | |
|---------+---------------------------->
>--------------------------------------------------------------------------------------------------------------------------------|
|
|
| To: [EMAIL PROTECTED]
|
| cc:
|
| Fax to:
|
| Subject: odd problem
|
>--------------------------------------------------------------------------------------------------------------------------------|
Hi,
The code in my function below (insert_internet_usage) has suddenly
dramatically slowed down and I don't understand why. The logs I'm parsing
are the same size as before, but what used to take 15 minutes has shot up
to
about 4 hours. It's almost as if the indexes had stopped working.
I wonder if someone here can help me out please? I've tried to include all
the information which I thought might be relevant below
but if you need more let me know. There haven't been any errors reported in
the log since I restarted mysql. Here's my script log:
Thu Jul 1 08:56:18 2004: PARSING
/sawmill/rawlog/SG_CSGL02_main_470612210000.log.gz ...
Thu Jul 1 09:00:37 2004: BULK_TABLE_INSERT ...
Thu Jul 1 09:01:43 2004: INSERT_URL_SERVERS ...
Thu Jul 1 09:02:26 2004: INSERT_URL_PATHS ...
Thu Jul 1 09:03:54 2004: INSERT_URL_QUERIES ...
Thu Jul 1 09:04:32 2004: INSERT_USER_AGENTS ...
Thu Jul 1 09:05:09 2004: INSERT_URL_VISITS ...
Thu Jul 1 09:10:06 2004: INSERT_INTERNET_USAGE ...
Thu Jul 1 09:19:31 2004: DELETE_BULK_TABLE ...
Thu Jul 1 09:19:32 2004: PARSING
/sawmill/rawlog/SG_CSGL02_main_470611210000.log.gz ...
Thu Jul 1 09:29:02 2004: BULK_TABLE_INSERT ...
Thu Jul 1 09:31:36 2004: INSERT_URL_SERVERS ...
Thu Jul 1 09:33:21 2004: INSERT_URL_PATHS ...
Thu Jul 1 09:36:26 2004: INSERT_URL_QUERIES ...
Thu Jul 1 09:38:07 2004: INSERT_USER_AGENTS ...
Thu Jul 1 09:39:37 2004: INSERT_URL_VISITS ...
<=== This is where the
time difference has
Thu Jul 1 09:54:19 2004: INSERT_INTERNET_USAGE ...
<=== suddenly
increased
Thu Jul 1 13:28:37 2004: DELETE_BULK_TABLE ...
Thu Jul 1 13:28:38 2004: PARSING
/sawmill/rawlog/SG_CSGL02_main_470610210000.log.gz ...
Thu Jul 1 13:39:09 2004: BULK_TABLE_INSERT ...
Thu Jul 1 13:41:59 2004: INSERT_URL_SERVERS ...
Thu Jul 1 13:43:52 2004: INSERT_URL_PATHS ...
Thu Jul 1 13:47:11 2004: INSERT_URL_QUERIES ...
Thu Jul 1 13:48:55 2004: INSERT_USER_AGENTS ...
Thu Jul 1 13:50:35 2004: INSERT_URL_VISITS ...
Thu Jul 1 14:06:55 2004: INSERT_INTERNET_USAGE ...
time now is 16:13 and it's still going.
This is the function:
sub insert_internet_usage(){ my $sql = qq {INSERT internet_usage
(uid,time,ip,urlid,timetaken,
cs_size,sc_size,method_ID,action_ID,virus_ID,userag
ent_ID)
SELECT bt.user, bt.time, bt.ip, uv.urlid, bt.timetaken,
bt.cs_size, bt.sc_size, bt.method_ID, bt.action_ID, b
t.virus, ua.ID
FROM bulk_table bt
INNER JOIN url_servers us ON us.server=bt.server
INNER JOIN url_paths up ON up.path=bt.path
INNER JOIN url_queries uq ON uq.query=bt.query
INNER JOIN user_agents ua ON
ua.useragent=bt.useragent
INNER JOIN url_visit uv ON uv.url_server_ID=us.ID
AND uv.url_path_ID=up.ID
AND uv.url_scheme_ID=bt.scheme_ID
AND uv.url_query_ID=uq.ID };
my $timenow=localtime();
print PL "\n$timenow: INSERT_INTERNET_USAGE ...";
$dbh2->do($sql) or die $dbh2->errstr;
$dbh2->commit() or die $dbh2->errstr;
}
Here's a bit of extra info:
mysql> desc url_visit;
+-----------------+---------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------------+---------+------+-----+---------+----------------+
| urlid | int(11) | | PRI | NULL | auto_increment |
| url_scheme_ID | int(11) | | | 0 | |
| url_server_ID | int(11) | | MUL | 0 | |
| url_path_ID | int(11) | | | 0 | |
| url_query_ID | int(11) | | | 0 | |
| url_category_ID | int(11) | | | 0 | |
+-----------------+---------+------+-----+---------+----------------+
6 rows in set (0.01 sec)
mysql> desc internet_usage;
+--------------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+--------------+-------------+------+-----+---------+-------+
| uid | varchar(10) | YES | MUL | NULL | |
| time | datetime | YES | | NULL | |
| ip | int(10) | YES | | 0 | |
| urlid | int(11) | | | 0 | |
| timetaken | int(11) | YES | | 0 | |
| cs_size | int(11) | YES | | 0 | |
| sc_size | int(11) | YES | | 0 | |
| method_ID | int(11) | YES | | 0 | |
| action_ID | int(11) | YES | | 0 | |
| virus_ID | int(11) | YES | | 0 | |
| useragent_ID | int(11) | YES | | 0 | |
+--------------+-------------+------+-----+---------+-------+
11 rows in set (0.00 sec)
mysql> show status;
+--------------------------+------------+
| Variable_name | Value |
+--------------------------+------------+
| Aborted_clients | 0 |
| Aborted_connects | 0 |
| Bytes_received | 16714 |
| Bytes_sent | 4250908493 |
| Com_admin_commands | 0 |
| Com_alter_table | 0 |
| Com_analyze | 0 |
| Com_backup_table | 0 |
| Com_begin | 0 |
| Com_change_db | 1 |
| Com_change_master | 0 |
| Com_check | 0 |
| Com_commit | 2 |
| Com_create_db | 0 |
| Com_create_function | 0 |
| Com_create_index | 0 |
| Com_create_table | 0 |
| Com_delete | 2 |
| Com_delete_multi | 0 |
| Com_drop_db | 0 |
| Com_drop_function | 0 |
| Com_drop_index | 0 |
| Com_drop_table | 0 |
| Com_flush | 0 |
| Com_grant | 0 |
| Com_ha_close | 0 |
| Com_ha_open | 0 |
| Com_ha_read | 0 |
| Com_insert | 98 |
| Com_insert_select | 18 |
| Com_kill | 0 |
| Com_load | 3 |
| Com_load_master_data | 0 |
| Com_load_master_table | 0 |
| Com_lock_tables | 1 |
| Com_optimize | 0 |
| Com_purge | 0 |
| Com_rename_table | 0 |
| Com_repair | 0 |
| Com_replace | 0 |
| Com_replace_select | 0 |
| Com_reset | 0 |
| Com_restore_table | 0 |
| Com_revoke | 0 |
| Com_rollback | 0 |
| Com_savepoint | 0 |
| Com_select | 18 |
| Com_set_option | 13 |
| Com_show_binlog_events | 0 |
| Com_show_binlogs | 0 |
| Com_show_create | 12 |
| Com_show_databases | 0 |
| Com_show_fields | 14 |
| Com_show_grants | 0 |
| Com_show_keys | 2 |
| Com_show_logs | 0 |
| Com_show_master_status | 0 |
| Com_show_new_master | 0 |
| Com_show_open_tables | 0 |
| Com_show_processlist | 0 |
| Com_show_slave_hosts | 0 |
| Com_show_slave_status | 0 |
| Com_show_status | 2 |
| Com_show_innodb_status | 0 |
| Com_show_tables | 14 |
| Com_show_variables | 0 |
| Com_slave_start | 0 |
| Com_slave_stop | 0 |
| Com_truncate | 0 |
| Com_unlock_tables | 1 |
| Com_update | 0 |
| Connections | 9 |
| Created_tmp_disk_tables | 15 |
| Created_tmp_tables | 16 |
| Created_tmp_files | 0 |
| Delayed_insert_threads | 0 |
| Delayed_writes | 0 |
| Delayed_errors | 0 |
| Flush_commands | 1 |
| Handler_commit | 0 |
| Handler_delete | 0 |
| Handler_read_first | 3 |
| Handler_read_key | 28924102 |
| Handler_read_next | 23389 |
| Handler_read_prev | 0 |
| Handler_read_rnd | 4649 |
| Handler_read_rnd_next | 118372628 |
| Handler_rollback | 0 |
| Handler_update | 0 |
| Handler_write | 57423139 |
| Key_blocks_used | 157632 |
| Key_read_requests | 2897986745 |
| Key_reads | 480921 |
| Key_write_requests | 13481590 |
| Key_writes | 389143 |
| Max_used_connections | 2 |
| Not_flushed_key_blocks | 17697 |
| Not_flushed_delayed_rows | 0 |
| Open_tables | 19 |
| Open_files | 31 |
| Open_streams | 0 |
| Opened_tables | 25 |
| Questions | 207 |
| Qcache_queries_in_cache | 2 |
| Qcache_inserts | 5 |
| Qcache_hits | 0 |
| Qcache_lowmem_prunes | 0 |
| Qcache_not_cached | 12 |
| Qcache_free_memory | 33503944 |
| Qcache_free_blocks | 1 |
| Qcache_total_blocks | 7 |
| Rpl_status | NULL |
| Select_full_join | 8 |
| Select_full_range_join | 0 |
| Select_range | 0 |
| Select_range_check | 0 |
| Select_scan | 29 |
| Slave_open_temp_tables | 0 |
| Slave_running | OFF |
| Slow_launch_threads | 0 |
| Slow_queries | 23 |
| Sort_merge_passes | 0 |
| Sort_range | 0 |
| Sort_rows | 4649 |
| Sort_scan | 1 |
| Table_locks_immediate | 199 |
| Table_locks_waited | 0 |
| Threads_cached | 0 |
| Threads_created | 3 |
| Threads_connected | 3 |
| Threads_running | 2 |
| Uptime | 28728 |
+--------------------------+------------+
132 rows in set (0.00 sec)
mysql> show index from internet_usage;
+----------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name |
Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+----------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| internet_usage | 1 | uid | 1 | uid | A
| 57073 | NULL | NULL | YES | BTREE | |
+----------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
1 row in set (0.01 sec)
mysql> show index from url_visit;
+-----------+------------+---------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name |
Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-----------+------------+---------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+
| url_visit | 0 | PRIMARY | 1 | urlid | A
| 6864184 | NULL | NULL | | BTREE | |
| url_visit | 0 | url_server_ID | 1 | url_server_ID | A
| NULL | NULL | NULL | | BTREE | |
| url_visit | 0 | url_server_ID | 2 | url_path_ID | A
| NULL | NULL | NULL | | BTREE | |
| url_visit | 0 | url_server_ID | 3 | url_query_ID | A
| NULL | NULL | NULL | | BTREE | |
| url_visit | 0 | url_server_ID | 4 | url_scheme_ID | A
| 6864184 | NULL | NULL | | BTREE | |
+-----------+------------+---------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+
5 rows in set (0.00 sec)
Thanks for any help,
js.
_________________________________________________________________
It's fast, it's easy and it's free. Get MSN Messenger today!
http://www.msn.co.uk/messenger
--
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql
To unsubscribe: http://lists.mysql.com/[EMAIL PROTECTED]
--
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql
To unsubscribe: http://lists.mysql.com/[EMAIL PROTECTED]
