Sergey, I'm writing a github readme and a blog post that will explain everything in more detail (Matt, if you're reading this, sorry for the delay!).
In the beginning I tuned everything by hand. I used the classic hold-out strategy that separates the available data intro training and validation (I did not use the test data for this), so I kept changing some parameters by hand and estimated how well the model would predict the energy load for December 1998. Since it was my first time doing this, it was a very random process. The results improved but I wanted to try different strategies (see the "Many-steps prediction" thread for more details) so I tried to get swarming running to speed up the process. In the end I was having trouble to get swarming functional so I thought it'd be wiser to try new strategies without much tuning, otherwise I'd run out of personal time. My final model wasn't tuned at all. It just used the basic parameters from the hotgym example, perhaps with a different pamLength. Using a smarter strategy by discarding useless data and by using the prediction expectation (see "Scalar prediction tips" thread for more details) made a huge difference, while parameter tuning only made a small difference when I tried it. Pedro. On Fri, Oct 25, 2013 at 7:21 AM, Sergey Zuev <[email protected]> wrote: > Pedro, > > You have done great job! Thank you for explanations and source code. > > Did you use swarming for models creation or you tuned everything manually? > > > Best regards, > Sergey Zuev. > > > On Mon, Oct 14, 2013 at 9:24 AM, Pedro Tabacof <[email protected]> wrote: > >> Hello, >> >> As promised elsewhere, here is the presentation of the results of NuPIC >> application to the electricity load forecast competition. This dataset >> comes from a 2001 competition on the prediction of max energy loads for the >> whole month of January 1999 of the electrical grid of Slovakia. The >> training data consists of two years (1997 and 1998) of half-hourly energy >> loads and daily temperatures, with holiday information. This seems to have >> been a serious competition as the winners got real prizes and they made >> sure it wouldn't be possible to access the test data before the submission. >> There were 56 registered competitor but only 26 of them submitted their >> predictions. >> >> The winners of the competition used SVM regression (SVR) for one-step >> predictions, but there were many different models used. Just as a >> curiosity, the winners were part of the group that wrote the popular >> "libsvm" library, so they are actually machine learning experts. >> >> Since this was my first time using NuPIC, I tried many different >> strategies. My first one was to just select the daily max energy load and >> do 31-days predictions, and this got me a good result (3.3% MAPE - among >> the top 10). This good start motivated me to try to improve the results, so >> I tried to use the whole half-hourly data, and while this gave me a great >> improvement for the first half of the test month, the last half was not >> good enough for an overall improvement. >> >> In the end a variation of the first idea was the simplest and quickest >> method and achieved the best results: 2.5% MAPE, which would place me among >> the top 3 best results. To get it down from 3.3% I had to remove summer >> data and also use a different way of doing prediction (I will start another >> thread about it). I also tried to use a multi-model approach, doing >> 1,2,...,30,31 days predictions with 31 different models but it didn't >> improve anything and was slower. I didn't use swarming and tried to tune >> everything by hand, but in the end the parameters stayed very close to the >> hotgym example, which is very motivating. >> >> I've attached the code, data and some graphs with comparisons of NuPIC >> against the best results of the competition. It's very important to stress >> that since I was in possession of the test data, the results are not really >> "scientific". But I think that the fact I had never used the new NuPIC >> before cancels that out! >> >> Pedro. >> >> -- >> Pedro Tabacof, >> Unicamp - Eng. de Computação 08. >> >> _______________________________________________ >> nupic mailing list >> [email protected] >> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >> >> > > _______________________________________________ > nupic mailing list > [email protected] > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org > > -- Pedro Tabacof, Unicamp - Eng. de Computação 08.
_______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
