Hello Pedro & Ralph, I've been following this thread closely and would like to weigh-in since it seems to have died off and I see a very important opportunity here as it relates to system ID and control. Unfortunately, I'm less familiar with NuPIC than I would like to be in order to speak to some of the possibilities of using NuPIC for model learning in the context of implementing an MPC controller. However, the main point brought up by Pedro about the key drawback of MPC is very important, and is also where potentially NuPIC could be useful. In particular, a model of sufficient accuracy & fidelity is required to support a reasonable level of confidence in the control sequence that comes from the optimization of the cost function and constraints associated with MPC. Having said that, there are multiple issues in this thread that I would like to address, which I've outlined as follows:
1. I think it is important to develop a very clear and rigorous characterization of the model in order for it to be used for MPC. For example, linear vs. nonlinear, parameterized vs. non-parametric, etc. The answers to these questions may be obvious to those who are more familiar with the nuances of CLA/NuPIC, and that's what I'm hoping to gain some insight on from the experts that frequent this list. If we could establish the model framework it would help to determine the level of abstraction and thus complexity required to use MPC. 2. Incidentally, there was a paper recently published at ECC2013 in Zurich that may be of interest [1]. This approach considers system-ID as a constraint in the context of a linear model. To clarify, the distinction between [1] and what Pedro has proposed below is that the system-ID part in [1] is considered as a constraint and runs as on-line identification where Pedro's algorithm "turns off" NuPIC learning during the control update and is not considered as a constraint. However, NuPIC provides a more general (presumably nonlinear CLA-based) modeling framework with which to make the future predictions, not a parameterized linear system. 3. Another question is then: is it possible to formulate NuPIC learning as a constraint, as in [1], such that there is no need to "turn off" the learning ? This may require quantification of parameter estimation error in a formal manner. 4. Furthermore, if the learning is turned off as suggested by Pedro below, can CLA fit into a "linear model" paradigm as found in [1] ? Restricting the model in this way limits the accuracy and fidelity of the model, however it opens up the MPC optimization problem to a richer and more well established set of solutions to certain kinds of constraints (i.e. chance constraints, etc.). 5. I don't see the distinction between the proposed variant of the MPC algorithm that Pedro advocates below and standard MPC. Am I missing something ? The only difference is that the model is learned adaptively based upon new data rather than using a fixed model. 6. The problem of closed-loop vs. open loop system identification is well known, and I do not think the perils of the former are limited to using NuPIC to model the dynamics. In fact, I disagree and think that performing open-loop system ID would in fact be useful as a first cut. If you make any progress on this I'd be very interested in hearing the results. [1] Christian A. Larsson, Mariette Annergren, Håkan Hjalmarsson, Cristian R. Rojas, Xavier Bombois, Ali Mesbah and Per Erik Modén. "Model predictive control with integrated experiment design for output error systems." 2013 European Control Conference (ECC) July 17-19, 2013, Zürich, Switzerland, pp. 3790 - 3795. Regards, Rodney A. Martin Pedro Tabacof<http://www.mail-archive.com/[email protected]&q=from:%22Pedro+Tabacof%22> Fri, 30 Aug 2013 07:32:48 -0700<http://www.mail-archive.com/[email protected]&q=date:20130830> Ralph, The problem is that the controller dynamics interfere with the system dynamics, thus changing its overall structure. So when you apply any kind of black-box identification procedure it can mistake the real system with the controller, because the outputs will always seem to be chasing the reference point. A very extensive reference can be found in [1] (I'm still reading it). In the case of online learning, I'm not sure what is going to happen, but I haven't had good experience with offline learning using other techniques on closed loop data. For example, once I trained an Echo State Network (powerful and easy to use recurrent neural nets) on some real closed loop data and the result seemed very good first, but when I simulated a step response, I found out the network had created an equilibrium point around the controller reference, which clearly wasn't the case for the true system. Since the controller stabilized the system around the reference, the neural net thought it was an equilibrium point, but the real reference was a very unstable place! (The true system was a pH neutralizer plant and the reference was pH = 7) Here is the MPC algorithm I propose: While control is active: Turn NuPIC learning off While cost function is still changing (cost function usually is squared difference between outputs and reference plus regularization term): 1) Guess next N inputs via some optimization procedure (N <= K) 2) Predict next K steps with the N guessed inputs (K usually around 20-30) Apply only the first of all optimized inputs to the real system Retrieve output from the real system Turn learning on and use real inputs and outputs to update NuPIC Does this seem feasible? I guess the basic scalar encoder can be used as interface. The optimization part is probably very time-consuming, but we can use warm starting to greatly increase convergence (just initialize new input guess with last input guess). One problem I see is that the number of prediction steps is actually greater than the number of applied steps to the system, so the learning would have to be done with less than the number of prediction steps. Is this possible? Pedro. [1] Forssell, Urban, and Lennart Ljung. "Closed-loop identification revisited."*Automatica* 35.7 (1999): 1215-1241. http://photon.isy.liu.se:81/research/pub-db-supplemental/phd/566/main.pdf On Thu, Aug 29, 2013 at 1:34 PM, Ralph Dratman <[email protected]>wrote: > Pedro, > > This is well expressed and I find it very interesting. Can you suggest a > reference for your point about noise behavior in a closed loop? > > You present a situation in which a cortical algorithm ought to adapt in > some way. But the algorithm you describe is not an elementary loop, and I > would like to give the implications more thought. > > Ralph > > > > On Tuesday, August 27, 2013, Pedro Tabacof wrote: > >> Hello, >> >> I've used the old NuPic for spatial classification (achieving almost the >> same error rate as a SVM classifier) and read the whitepaper on the CLA, >> but I haven't used the current implementation or been up to date on recent >> developments. >> >> I was reading the thread about the Ski game and started to think if it >> would be possible to apply NuPIC for MPC (Model Predictive Control), a >> mature and popular technology for advanced process control. The main >> difficult associated with MPC is having a reliable model of the system to >> be controlled, so if a smart online learner such as NuPIC could be used to >> model the system dynamics, it would be an interesting opportunity for >> academic or even commercial research. >> >> MPC is an optimization problem where the controlled inputs are chosen to >> minimize some cost function subject to the system dynamics and restrictions >> for a given time horizon. After the control inputs are chosen for the whole >> time window, only the next input is actually applied to the real system, >> the model is updated with the system response to the chosen input, and the >> optimization is done all over again. This picture sums it up: >> [image: Inline image 1] >> >> NuPIC could be used to predict the system outputs given the input/output >> history. The search for the best inputs on the control horizon could be >> done with a nonlinear least squares using a numerical derivative or some >> global heuristic (such as an evolutionary strategy). This of course leads >> to a local minimum, but sometimes this is more than enough. Learning would >> be off during the search, and turned on only after the next input is chosen >> and applied to the system. >> >> Is the CLA robust to white noise? Has anyone used NuPIC within a closed >> stable loop? This actually changes the noise dynamics of the system, so I'm >> not sure if the current learning algorithm can be applied indiscriminately. >> >> Do you think the current evaluation speed is fast enough for a local >> search with numerical derivatives or a global heuristic search? On more >> concrete terms, how long does it take to predict 30 time slices for 5 >> outputs on a regular PC on a moderately sized network? >> >> The inputs and outputs are all real numbers, any idea on how to convert >> this to a proper SDR? >> >> Thanks, >> Pedro. >> >> -- >> Pedro Tabacof, >> State University of Campinas, Brazil. >> > > _______________________________________________ > nupic mailing list > [email protected] > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org > > -- Pedro Tabacof, Unicamp - Eng. de Computação 08. _______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
_______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
