As we have introduced object oriented programming to the Ngram Statistics Package, we and a number of users have noticed some slowdown as we moved from version 0.73 (the last non-OO version) to versions 0.91, 0.93, and 0.95. For example, the time for running a large number of ll tests went from 4.9 seconds in 0.73 to 11.7 seconds in version 0.95.
We have largely resolved these performance issues in version 0.97, which is now available. This was done by avoiding the creation of objects and reducing some of the passing of function arguments, however, we have still retained the hierarchy of measures that was introduced in version 0.91. So, version 0.97 is something of a hybrid, it has a hiearchical organization of measures, but without creating objects. You can download this version from http://www.d.umn.edu/~tpederse/nsp.html or http://ngram.sourceforge.net The speed of 0.97 is comparable we have found to 0.73, which was a much messier implementation we feel (with lots of code duplication, and no real organization among the measures). So we think 0.97 represents the best of both worlds, a reasonable organization of code that should be easy to maintain and extend, while still running quickly on even relatively large amounts of data. Here are the times for our test will ll, with more detailed reports below. These are created using the profile option in Perl. Total Elapsed Time = 4.897306 Seconds (version 0.73) Total Elapsed Time = 11.72926 Seconds (version 0.95) Total Elapsed Time = 4.759184 Seconds (version 0.97) ===================================================================== Performance of ll on NSP version 0.73 ===================================================================== Exporter::export has -1 unstacked calls in outer Exporter::Heavy::heavy_export has 1 unstacked calls in outer Total Elapsed Time = 4.897306 Seconds User+System Time = 2.097306 Seconds Exclusive Times %Time ExclSec CumulS #Calls sec/call Csec/c Name 18.8 0.396 0.960 45672 0.0000 0.0000 ll::calculateStatistic 13.4 0.281 0.281 45672 0.0000 0.0000 measure2d::getObservedValues 13.4 0.281 0.281 45672 0.0000 0.0000 measure2d::getExpectedValues 11.9 0.250 0.250 1 0.2500 0.2500 main::unformattedPrinting 4.91 0.103 0.144 45673 0.0000 0.0000 ll::errorCode 1.95 0.041 0.041 45673 0.0000 0.0000 measure2d::errorCode 0.95 0.020 0.030 3 0.0067 0.0099 main::BEGIN 0.48 0.010 0.010 3 0.0033 0.0033 vars::BEGIN 0.05 0.001 0.001 45672 0.0000 0.0000 measure2d::getTotalBigrams 0.00 0.000 0.000 1 0.0000 0.0000 Exporter::Heavy::heavy_export 0.00 - -0.000 1 - - ll::getStatisticName 0.00 - -0.000 1 - - measure2d::initializeStatistic 0.00 - -0.000 1 - - Getopt::Long::ConfigDefaults 0.00 - -0.000 1 - - Getopt::Long::Configure 0.00 - -0.000 1 - - Config::TIEHASH ===================================================================== Performance of ll on NSP version 0.95 ===================================================================== Exporter::export has -1 unstacked calls in outer Exporter::Heavy::heavy_export has 1 unstacked calls in outer Total Elapsed Time = 11.72926 Seconds User+System Time = 5.319261 Seconds Exclusive Times %Time ExclSec CumulS #Calls sec/call Csec/c Name 21.0 1.119 4.035 45672 0.0000 0.0001 Text::NSP::Measures::2D::MI::ll::c alculateStatistic 16.8 0.894 2.310 45672 0.0000 0.0001 Text::NSP::Measures::2D::MI::calcu lateStatistic 11.3 0.606 0.606 182688 0.0000 0.0000 Text::NSP::Measures::2D::MI::compu tePMI 10.1 0.541 0.541 45672 0.0000 0.0000 Text::NSP::Measures::2D::computeMa rginalTotals 9.27 0.493 1.034 45672 0.0000 0.0000 Text::NSP::Measures::2D::computeOb servedValues 7.16 0.381 0.381 45672 0.0000 0.0000 Text::NSP::Measures::2D::computeEx pectedValues 4.14 0.220 0.220 1 0.2200 0.2200 main::unformattedPrinting 1.71 0.091 0.091 45672 0.0000 0.0000 Text::NSP::Measures::getErrorCode 0.38 0.020 0.030 4 0.0050 0.0074 main::BEGIN 0.19 0.010 0.010 3 0.0033 0.0033 vars::BEGIN 0.00 0.000 0.000 1 0.0000 0.0000 Exporter::Heavy::heavy_export 0.00 - -0.000 1 - - Text::NSP::Measures::initializeSta 0.0000 tistic 0.00 - -0.000 1 - - Getopt::Long::ConfigDefaults 0.00 - -0.000 1 - - Getopt::Long::Configure 0.00 - -0.000 1 - - Config::TIEHASH ===================================================================== Performance of ll on NSP version 0.97 ===================================================================== Exporter::export has -4 unstacked calls in outer Exporter::Heavy::heavy_export has 4 unstacked calls in outer Total Elapsed Time = 4.759184 Seconds User+System Time = 2.769184 Seconds Exclusive Times %Time ExclSec CumulS #Calls sec/call Csec/c Name 25.2 0.699 1.915 45672 0.0000 0.0000 Text::NSP::Measures::2D::MI::ll::c alculateStatistic 14.3 0.396 0.850 45672 0.0000 0.0000 Text::NSP::Measures::2D::MI::getVa lues 13.2 0.366 0.366 182688 0.0000 0.0000 Text::NSP::Measures::2D::MI::compu tePMI 9.43 0.261 0.261 45672 0.0000 0.0000 Text::NSP::Measures::2D::computeMa rginalTotals 8.31 0.230 0.230 1 0.2300 0.2300 main::unformattedPrinting 5.45 0.151 0.151 45672 0.0000 0.0000 Text::NSP::Measures::2D::computeOb servedValues 2.20 0.061 0.061 45672 0.0000 0.0000 Text::NSP::Measures::getErrorCode 1.48 0.041 0.041 45672 0.0000 0.0000 Text::NSP::Measures::2D::computeEx pectedValues 0.36 0.010 0.010 4 0.0025 0.0025 Exporter::as_heavy 0.36 0.010 0.010 4 0.0025 0.0025 Text::NSP::Measures::2D::MI::BEGIN 0.36 0.010 0.030 4 0.0025 0.0074 main::BEGIN 0.36 0.010 0.010 19 0.0005 0.0005 Getopt::Long::BEGIN 0.00 0.000 0.000 4 0.0000 0.0000 Exporter::Heavy::heavy_export 0.00 - -0.000 1 - - Getopt::Long::ConfigDefaults 0.00 - -0.000 1 - - Getopt::Long::Configure -- Ted Pedersen http://www.d.umn.edu/~tpederse ------------------------ Yahoo! Groups Sponsor --------------------~--> Check out the new improvements in Yahoo! Groups email. http://us.click.yahoo.com/6pRQfA/fOaOAA/yQLSAA/x3XolB/TM --------------------------------------------------------------------~-> Yahoo! Groups Links <*> To visit your group on the web, go to: http://groups.yahoo.com/group/nlpatumd/ <*> To unsubscribe from this group, send an email to: [EMAIL PROTECTED] <*> Your use of Yahoo! Groups is subject to: http://docs.yahoo.com/info/terms/
