(Sent on behalf of Giovanni <[email protected]> who encountered problems with the list.)
Hi, I am trying to re-upload my entire database (~10M of MarcXML records) in Invenio. The last time I did it, disabling the population of most of the bibxxx tables the insertion rate was around 100 records per second, while today, with the same configuration on the server, I am uploading at a rate of 7-15 records per second. The bottleneck seems to be MySQL, since in "top" I see: 29532 mysql 15 0 29.3g 5.1g 4040 S 97.8 3.6 11:49.56 mysqld (98% of CPU usage) while bibupload 29597 apache 15 0 514m 274m 5516 S 3.0 0.2 0:34.36 bibupload (3% of CPU usage) This is weird because my server is quite powerful (148 GB RAM, 24 total cores, hard drives SCSI in RAID 5). According to this server configuration I have the following parameters in my.cnf: [client] port = 3306 socket = /var/lib/mysql/mysql.sock [mysqld] port = 3306 socket = /var/lib/mysql/mysql.sock skip-locking key_buffer = 29472M max_allowed_packet = 1536M table_cache = 1024 sort_buffer_size = 148M read_buffer_size = 148M read_rnd_buffer_size = 296M myisam_sort_buffer_size = 4736M thread_cache_size = 8 query_cache_size = 256M query_cache_limit = 3M thread_concurrency = 48 log-slow-queries log=/tmp/mysql-query.log server-id = 2007097 [mysqldump] quick max_allowed_packet = 2147483647 [mysql] no-auto-rehash [isamchk] key_buffer = 256M sort_buffer_size = 256M read_buffer = 2M write_buffer = 2M [myisamchk] key_buffer = 256M sort_buffer_size = 256M read_buffer = 2M write_buffer = 2M [mysqlhotcopy] interactive-timeout I tried to read some documentation to tune some of these parameters, but I was not successful. And In general my MySQL instance is running with the following parameters: http://pastebin.com/8Kge9wzU The version of Invenio I am using is the latest (updated this morning). The linux version on the server is CentOS 5.8 Any help is welcome! Thanks, Giovanni
