It seems that SMW_refreshData gets slower with growing size of the dataset.
I didn't do much of troubleshooting of the issue, but first 50000 pages from
my dataset were processed faster then second 50000 pages.
I'm going to start upgrade over for RC2 and will try to look at it in terms
of speed of the process, but I think there might be a reason for it in some
indexes getting bigger with more data (which can be avoided by dropping
indexes prior to refresh and rebuilding them right after) or MySQL not
liking that many temporary tables created so rapidly.
Also, I'm wondering if parts of the dataset can be processed in parallel? it
seems that single run of the script doesn't load CPU that much and
alternates between PHP and MySQL processes which is not optimal for
multi-processor boxes where these loads can be spread across all the CPUs.
Sergey
--
Sergey Chernyshev
http://www.sergeychernyshev.com/
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Semediawiki-devel mailing list
Semediawiki-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
