Pedro, I'm going to be stupid in public here so please correct me:
1. A system you want to put under control has a desired set point 2. The distance to that set point can be measured at each time step 3. That system is being modeled by [something] 4. You have one or more inputs that can be applied to the system 5. The system model can predict the system state for K steps (usually 20-30) into the future 6. You are searching for the control inputs that move those predicted steps toward the set point, such that within the predicted window you minimize the sum of errors. 7. The above minimization preferences smooth curves rather than sudden jumps [somehow] 8. You would like to use CLA as the system model 9. You're worried CLA will start believing that the reference point is the most likely next state. 10. You want to know how fast CLA is for the optimization step. For 10, a temporal inference with learning takes 20-30ms. If you ask CLA to predict the next 20-30 steps with each new input the memory requirements will be large, but the inference time shouldn't be 20x. To properly explore the predicted space with various control inputs, you'll also need to feed in a chunk of previous data to get CLA to 'lock on' to the patterns in the data. (Imagine someone composing a song, they may replay the previous section several times as they explore what should come next). For 9, this will probably happen, but if you do prediction first with 'no control' as part of the input CLA should make a prediction closer to the actual system. A lot of training data on the system before control, or with control turned on then off could help. I think this is a really interesting problem. I'm also very interested to know if there is a way to represent the set point and measure error using SDRs. Having this all external is cool, but getting this type of loop fully integrated would be a huge step. Ian On Sun, Sep 1, 2013 at 11:37 PM, Ralph Dratman <[email protected]>wrote: > Pedro, > > I wish I could advise you on the feasibility of your plan, but I don't > yet know enough about CLA to be of any help. I am not part of Numenta. I am > only an onlooker. > > On Friday, August 30, 2013, Pedro Tabacof wrote: > >> Ralph, >> >> The problem is that the controller dynamics interfere with the system >> dynamics, thus changing its overall structure. So when you apply any kind >> of black-box identification procedure it can mistake the real system with >> the controller, because the outputs will always seem to be chasing the >> reference point. A very extensive reference can be found in [1] (I'm still >> reading it). In the case of online learning, I'm not sure what is going to >> happen, but I haven't had good experience with offline learning using other >> techniques on closed loop data. >> >> For example, once I trained an Echo State Network (powerful and easy to >> use recurrent neural nets) on some real closed loop data and the result >> seemed very good first, but when I simulated a step response, I found out >> the network had created an equilibrium point around the controller >> reference, which clearly wasn't the case for the true system. Since the >> controller stabilized the system around the reference, the neural net >> thought it was an equilibrium point, but the real reference was a very >> unstable place! (The true system was a pH neutralizer plant and the >> reference was pH = 7) >> >> Here is the MPC algorithm I propose: >> >> While control is active: >> >> Turn NuPIC learning off >> >> While cost function is still changing (cost function usually is squared >> difference between outputs and reference plus regularization term): >> >> 1) Guess next N inputs via some optimization procedure (N <= K) >> >> 2) Predict next K steps with the N guessed inputs (K usually around 20-30) >> >> Apply only the first of all optimized inputs to the real system >> Retrieve output from the real system >> Turn learning on and use real inputs and outputs to update NuPIC >> >> Does this seem feasible? I guess the basic scalar encoder can be used as >> interface. The optimization part is probably very time-consuming, but we >> can use warm starting to greatly increase convergence (just initialize new >> input guess with last input guess). >> >> One problem I see is that the number of prediction steps is actually >> greater than the number of applied steps to the system, so the learning >> would have to be done with less than the number of prediction steps. Is >> this possible? >> >> Pedro. >> >> [1] Forssell, Urban, and Lennart Ljung. "Closed-loop identification >> revisited."*Automatica* 35.7 (1999): 1215-1241. >> http://photon.isy.liu.se:81/research/pub-db-supplemental/phd/566/main.pdf >> >> >> On Thu, Aug 29, 2013 at 1:34 PM, Ralph Dratman >> <[email protected]>wrote: >> >>> Pedro, >>> >>> This is well expressed and I find it very interesting. Can you suggest a >>> reference for your point about noise behavior in a closed loop? >>> >>> You present a situation in which a cortical algorithm ought to adapt in >>> some way. But the algorithm you describe is not an elementary loop, and >>> I would like to give the implications more thought. >>> >>> Ralph >>> >>> >>> >>> On Tuesday, August 27, 2013, Pedro Tabacof wrote: >>> >>>> Hello, >>>> >>>> I've used the old NuPic for spatial classification (achieving almost >>>> the same error rate as a SVM classifier) and read the whitepaper on the >>>> CLA, but I haven't used the current implementation or been up to date on >>>> recent developments. >>>> >>>> I was reading the thread about the Ski game and started to think if it >>>> would be possible to apply NuPIC for MPC (Model Predictive Control), a >>>> mature and popular technology for advanced process control. The main >>>> difficult associated with MPC is having a reliable model of the system to >>>> be controlled, so if a smart online learner such as NuPIC could be used to >>>> model the system dynamics, it would be an interesting opportunity for >>>> academic or even commercial research. >>>> >>>> MPC is an optimization problem where the controlled inputs are chosen >>>> to minimize some cost function subject to the system dynamics and >>>> restrictions for a given time horizon. After the control inputs are chosen >>>> for the whole time window, only the next input is actually applied to the >>>> real system, the model is updated with the system response to the chosen >>>> input, and the optimization is done all over again. This picture sums it >>>> up: >>>> [image: Inline image 1] >>>> >>>> NuPIC could be used to predict the system outputs given the >>>> input/output history. The search for the best inputs on the control horizon >>>> could be done with a nonlinear least squares using a numerical derivative >>>> or some global heuristic (such as an evolutionary strategy). This of course >>>> leads to a local minimum, but sometimes this is more than enough. Learning >>>> would be off during the search, and turned on only after the next input is >>>> chosen and applied to the system. >>>> >>>> Is the CLA robust to white noise? Has anyone used NuPIC within a closed >>>> stable loop? This actually changes the noise dynamics of the system, so I'm >>>> not sure if the current learning algorithm can be applied indiscriminately. >>>> >>>> Do you think the current evaluation speed is fast enough for a local >>>> search with numerical derivatives or a global heuristic search? On more >>>> concrete terms, how long does it take to predict 30 time slices for 5 >>>> outputs on a regular PC on a moderately sized network? >>>> >>>> The inputs and outputs are all real numbers, any idea on how to convert >>>> this to a proper SDR? >>>> >>>> Thanks, >>>> Pedro. >>>> >>>> -- >>>> Pedro Tabacof, >>>> State University of Campinas, Brazil. >>>> >>> >>> _______________________________________________ >>> nupic mailing list >>> [email protected] >>> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >>> >>> >> >> >> -- >> Pedro Tabacof, >> Unicamp - Eng. de Computação 08. >> > > _______________________________________________ > nupic mailing list > [email protected] > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org > >
_______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
